├── .gitignore ├── MANIFEST.in ├── README.rst ├── example-filter.py ├── mqtt-watchdir.py ├── setup.py └── version.py /.gitignore: -------------------------------------------------------------------------------- 1 | RELEASE-VERSION 2 | *.pyc 3 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include RELEASE-VERSION 2 | include version.py 3 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | mqtt-watchdir 2 | ============= 3 | 4 | This simple Python program portably watches a directory recursively and 5 | publishes the content of newly created and modified files as payload to 6 | an `MQTT `_ broker. Files which are deleted are 7 | published with a NULL payload. 8 | 9 | The path to the directory to watch recursively (default ``.``), as well 10 | as a list of files to ignore (``*.swp``, ``*.o``, ``*.pyc``), the broker 11 | host (``localhost``) and port number (``1883``) must be specified in via 12 | environment variables , together with the topic prefix to which to 13 | publish to (``watch/``). 14 | 15 | Installation 16 | ------------ 17 | 18 | :: 19 | 20 | git clone https://github.com/jpmens/mqtt-watchdir.git 21 | cd mqtt-watchdir 22 | virtualenv watchdir 23 | source watchdir/bin/activate 24 | pip install -e . 25 | 26 | Configuration 27 | ------------- 28 | 29 | Set the following optional environment variables before invoking 30 | *mqtt-watchdir.py*: 31 | 32 | - ``MQTTHOST`` (default ``"localhost"``) is the name/address of the MQTT broker. 33 | - ``MQTTPORT`` (default ``1883``) is the TCP port number of the broker. 34 | - ``MQTTUSERNAME`` is the username to connect to the broker. 35 | - ``MQTTPASSWORD`` is the password to connect to the broker. 36 | - ``MQTTWATCHDIR`` (default: ``"."``) is the path to the directory to watch. 37 | - ``MQTTQOS`` (default: ``0``) is the MQTT Quality of Service (QoS) to 38 | use on publish. Allowed values are ``0``, ``1``, or ``2``. 39 | - ``MQTTRETAIN`` (default: ``0``) specifies whether the "retain" flag 40 | should be set on publish. Set to ``1`` if you want messages to be retained. 41 | - ``MQTTPREFIX`` (default: ``"watch"``) is the prefix to be prepended 42 | (with a slash) to the MQTT topic name. The topic name is formed from 43 | this prefix plus the path name of the file that is being modified 44 | (i.e. watched). You can set this to an empty string (``""``) to avoid 45 | prefixing the topic name. 46 | - ``MQTTFILTER`` (default None) allows modifying payload (see below). 47 | Takes path to a Python file (e.g. ``"/path/to/example-filter.py"``. 48 | - ``MQTTFIXEDTOPIC`` (default None) sets a MQTT topic to which 49 | all messages are published. If set, the ``MQTTPREFIX`` setting is 50 | overruled and ignored. 51 | 52 | - Set ``WATCHDEBUG`` (default: ``0``) to ``1`` to show debugging 53 | information. 54 | 55 | Testing 56 | ------- 57 | 58 | Launch ``mosquitto_sub``: 59 | 60 | :: 61 | 62 | mosquitto_sub -v -t 'watch/#' 63 | 64 | Launch this program and, in another terminal, try something like this: 65 | 66 | :: 67 | 68 | echo Hello World > message 69 | echo JP > myname 70 | rm myname 71 | 72 | whereupon, on the first window, you should see: 73 | 74 | :: 75 | 76 | watch/message Hello World 77 | watch/myname JP 78 | watch/myname (null) 79 | 80 | Filters 81 | ------- 82 | 83 | Without a filter (the default), *mqtt-watchdir* reads the content of a 84 | newly created or modified file and uses that as the MQTT payload. By 85 | creating and enabling a so-called *filter*, *mqtt-watchdir* can pass the 86 | payload into said filter (a Python function) to have a payload 87 | translated. 88 | 89 | Consider the included ``example-filter.py``: 90 | 91 | :: 92 | 93 | def mfilter(filename, topic, payload): 94 | '''Return a tuple [pub, newpayload] indicating whether this event 95 | should be published (True or False) and a new payload string 96 | or None''' 97 | 98 | print "Filter for topic %s" % topic 99 | 100 | if filename.endswith('.jpg'): 101 | return False, None 102 | 103 | if payload is not None: 104 | return True, payload.replace("\n", "-").replace(" ", "+") 105 | return True, None 106 | 107 | The *mfilter* function is passed the fully qualified path to the file, 108 | the (possibly prefixed) MQTT topic name and the payload. In this simple 109 | example, spaces and newlines in the payload are replaced by dashes and 110 | plusses. 111 | 112 | The function must return a tuple with two elements: 113 | 114 | 1. The first specifies whether the payload will be published (True) or 115 | not (False) 116 | 2. The second is a string with a possibly modified payload or None. If 117 | the returned payload is *None*, the original payload is not modified. 118 | 119 | Possible uses of filters include 120 | 121 | - Limiting payload lengths 122 | - Conversion to JSON 123 | - Ignore certain file types (e.g. binary data) 124 | - Process content of files, say, YAML or JSON, and extract elements 125 | returning as string 126 | 127 | Requirements 128 | ------------ 129 | 130 | - `watchdog `_, a Python 131 | library to monitor file-system events. 132 | - `Paho-MQTT `_'s Python module 133 | 134 | Related utilities & Credits 135 | --------------------------- 136 | 137 | - Roger Light (of `Mosquitto `_ fame) created 138 | `mqttfs `_, a FUSE driver (in C) 139 | which works similarly. 140 | - Roger Light (yes, the same busy gentleman) also made 141 | `treewatch `_, a program to 142 | watch a set of directories and execute a program when there is a 143 | change in the files within the directories. 144 | - Thanks to Karl Palsson for the ``setup.py`` and ``version.py`` magic. 145 | 146 | -------------------------------------------------------------------------------- /example-filter.py: -------------------------------------------------------------------------------- 1 | def mfilter(filename, topic, payload): 2 | '''Return a tuple [pub, newpayload] indicating whether this event 3 | should be published (True or False) and a new payload string 4 | or None''' 5 | 6 | print "Filter for topic %s" % topic 7 | 8 | if filename.endswith('.jpg'): 9 | return False, None 10 | 11 | if payload is not None: 12 | return True, payload.replace("\n", "-").replace(" ", "+") 13 | return True, None 14 | -------------------------------------------------------------------------------- /mqtt-watchdir.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2013 Jan-Piet Mens 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are met: 8 | # 9 | # 1. Redistributions of source code must retain the above copyright notice, 10 | # this list of conditions and the following disclaimer. 11 | # 2. Redistributions in binary form must reproduce the above copyright 12 | # notice, this list of conditions and the following disclaimer in the 13 | # documentation and/or other materials provided with the distribution. 14 | # 3. Neither the name of mosquitto nor the names of its 15 | # contributors may be used to endorse or promote products derived from 16 | # this software without specific prior written permission. 17 | # 18 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 21 | # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 22 | # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 23 | # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 24 | # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 25 | # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 26 | # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 27 | # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 28 | # POSSIBILITY OF SUCH DAMAGE. 29 | 30 | __author__ = "Jan-Piet Mens" 31 | __copyright__ = "Copyright (C) 2013-2015 by Jan-Piet Mens" 32 | 33 | import os, sys 34 | import signal 35 | import time 36 | import paho.mqtt.client as paho 37 | # https://github.com/gorakhargosh/watchdog 38 | from watchdog.events import PatternMatchingEventHandler 39 | from watchdog.observers import Observer 40 | import platform 41 | import imp 42 | 43 | MQTTHOST = os.getenv('MQTTHOST', 'localhost') 44 | MQTTPORT = int(os.getenv('MQTTPORT', 1883)) 45 | MQTTUSERNAME = os.getenv('MQTTUSERNAME', None) 46 | MQTTPASSWORD = os.getenv('MQTTPASSWORD', None) 47 | MQTTWATCHDIR = os.getenv('MQTTWATCHDIR', '.') 48 | MQTTQOS = int(os.getenv('MQTTQOS', 0)) 49 | MQTTRETAIN = int(os.getenv('MQTTRETAIN', 0)) 50 | 51 | # May be None in which case neither prefix no separating slash are prepended 52 | MQTTPREFIX = os.getenv('MQTTPREFIX', 'watch') 53 | MQTTFILTER = os.getenv('MQTTFILTER', None) 54 | 55 | # Publish all messages to a fixed topic. E.g. if the file contents already/also 56 | # contains the name of the file or in certain situations with retained messages. 57 | # Overrules and ignores the MQTTPREFIX setting. 58 | MQTTFIXEDTOPIC = os.getenv('MQTTFIXEDTOPIC', None) 59 | 60 | WATCHDEBUG = os.getenv('WATCHDEBUG', 0) 61 | 62 | if MQTTPREFIX == '': 63 | MQTTPREFIX = None 64 | 65 | if MQTTFIXEDTOPIC == '': 66 | MQTTFIXEDTOPIC = None 67 | 68 | if MQTTFIXEDTOPIC: 69 | print 'Publishing ALL messages to the topic: %s' % MQTTFIXEDTOPIC 70 | 71 | ignore_patterns = [ '*.swp', '*.o', '*.pyc' ] 72 | 73 | # Publish with retain (True or False) 74 | if MQTTRETAIN == 1: 75 | MQTTRETAIN=True 76 | else: 77 | MQTTRETAIN=False 78 | 79 | # Ensure absolute path (incl. symlink expansion) 80 | DIR = os.path.abspath(os.path.expanduser(MQTTWATCHDIR)) 81 | 82 | OS = platform.system() 83 | 84 | mf = None 85 | if MQTTFILTER is not None: 86 | try: 87 | mf = imp.load_source('mfilter', MQTTFILTER) 88 | except Exception, e: 89 | sys.exit("Can't import filter from file %s: %s" % (MQTTFILTER, e)) 90 | 91 | clientid = 'mqtt-watchdir-%s' % os.getpid() 92 | mqtt = paho.Client(clientid, clean_session=True) 93 | if MQTTUSERNAME is not None or MQTTPASSWORD is not None: 94 | mqtt.username_pw_set(MQTTUSERNAME, MQTTPASSWORD) 95 | 96 | def on_publish(mosq, userdata, mid): 97 | pass 98 | # print("mid: "+str(mid)) 99 | 100 | def on_disconnect(mosq, userdata, rc): 101 | print "disconnected" 102 | time.sleep(5) 103 | 104 | def signal_handler(signal, frame): 105 | """ Bail out at the top level """ 106 | 107 | mqtt.loop_stop() 108 | mqtt.disconnect() 109 | 110 | sys.exit(0) 111 | 112 | class MyHandler(PatternMatchingEventHandler): 113 | """ 114 | React to changes in files, handling create, update, unlink 115 | explicitly. Ignore directories. Warning: does not handle move 116 | operations (i.e. `mv f1 f2' isn't handled). 117 | """ 118 | 119 | def catch_all(self, event, op): 120 | 121 | if event.is_directory: 122 | return 123 | 124 | path = event.src_path 125 | 126 | if OS == 'Linux' and op != 'DEL': 127 | 128 | try: 129 | # On Linux, a new file is NEW and MOD. Ensure we publish once only 130 | ctime = os.path.getctime(path) 131 | mtime = os.path.getmtime(path) 132 | 133 | if op == 'NEW' and mtime == ctime: 134 | return 135 | except: 136 | pass 137 | 138 | # Create relative path name and append to topic prefix 139 | filename = path.replace(DIR + '/', '') 140 | 141 | if MQTTFIXEDTOPIC is not None: 142 | topic = MQTTFIXEDTOPIC 143 | else: 144 | if MQTTPREFIX is not None: 145 | topic = '%s/%s' % (MQTTPREFIX, filename) 146 | else: 147 | topic = filename 148 | 149 | if WATCHDEBUG: 150 | print "%s %s. Topic: %s" % (op, filename, topic) 151 | 152 | if op == 'DEL': 153 | payload = None 154 | else: 155 | try: 156 | f = open(path) 157 | payload = f.read() 158 | f.close() 159 | payload = payload.rstrip() 160 | except Exception, e: 161 | print "Can't open file %s: %s" % (path, e) 162 | return 163 | 164 | # If we've loaded a filter, run data through the filter to obtain 165 | # a (possibly) modified payload 166 | 167 | if mf is not None: 168 | try: 169 | publish, new_payload = mf.mfilter(path, topic, payload) 170 | if publish is False: 171 | if WATCHDEBUG: 172 | print "NOT publishing %s" % path 173 | return 174 | if new_payload is not None: 175 | payload = new_payload 176 | except Exception, e: 177 | print "mfilter: %s" % (e) 178 | 179 | mqtt.publish(topic, payload, qos=MQTTQOS, retain=MQTTRETAIN) 180 | 181 | def on_created(self, event): 182 | self.catch_all(event, 'NEW') 183 | 184 | def on_modified(self, event): 185 | self.catch_all(event, 'MOD') 186 | 187 | def on_deleted(self, event): 188 | self.catch_all(event, 'DEL') 189 | 190 | def main(): 191 | 192 | mqtt.on_disconnect = on_disconnect 193 | mqtt.on_publish = on_publish 194 | 195 | mqtt.connect(MQTTHOST, MQTTPORT) 196 | 197 | mqtt.loop_start() 198 | 199 | signal.signal(signal.SIGINT, signal_handler) 200 | while 1: 201 | 202 | observer = Observer() 203 | event_handler = MyHandler( ignore_patterns=ignore_patterns ) 204 | observer.schedule(event_handler, DIR, recursive=True) 205 | observer.start() 206 | try: 207 | while True: 208 | time.sleep(1) 209 | except KeyboardInterrupt: 210 | observer.stop() 211 | observer.join() 212 | 213 | if __name__ == '__main__': 214 | main() 215 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | # from https://gist.github.com/dcreager/300803 with "-dirty" support added 4 | from version import get_git_version 5 | 6 | # From http://bugs.python.org/issue15881 7 | try: 8 | import multiprocessing 9 | except ImportError: 10 | pass 11 | 12 | long_description = '' 13 | 14 | with open('README.rst') as file: 15 | long_description = file.read() 16 | 17 | setup( 18 | name='mqtt-watchdir', 19 | version=get_git_version(), 20 | author='Jan-Piet Mens', 21 | author_email='jpmens@gmail.com', 22 | url="https://github.com/jpmens/mqtt-watchdir", 23 | description='Recursively watch a directory for modifications and publish file content to an MQTT broker', 24 | license='BSD License', 25 | long_description=long_description, 26 | keywords = [ 27 | "MQTT", 28 | "files", 29 | "notify" 30 | ], 31 | scripts=[ 32 | "mqtt-watchdir.py" 33 | ], 34 | data_files=[ 35 | "README.rst" 36 | ], 37 | install_requires=[ 38 | 'watchdog', 39 | 'paho-mqtt', 40 | ], 41 | classifiers=[ 42 | 'Development Status :: 4 - Beta', 43 | 'Intended Audience :: Developers', 44 | 'License :: OSI Approved :: BSD License', 45 | 'Operating System :: MacOS :: MacOS X', 46 | 'Operating System :: POSIX', 47 | 'Programming Language :: Python', 48 | 'Programming Language :: Python :: 2.6', 49 | 'Programming Language :: Python :: 2.7', 50 | 'Topic :: Communications', 51 | 'Topic :: Internet', 52 | ] 53 | 54 | ) 55 | -------------------------------------------------------------------------------- /version.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Author: Douglas Creager 3 | # This file is placed into the public domain. 4 | 5 | # Calculates the current version number. If possible, this is the 6 | # output of “git describe”, modified to conform to the versioning 7 | # scheme that setuptools uses. If “git describe” returns an error 8 | # (most likely because we're in an unpacked copy of a release tarball, 9 | # rather than in a git working copy), then we fall back on reading the 10 | # contents of the RELEASE-VERSION file. 11 | # 12 | # To use this script, simply import it your setup.py file, and use the 13 | # results of get_git_version() as your package version: 14 | # 15 | # from version import * 16 | # 17 | # setup( 18 | # version=get_git_version(), 19 | # . 20 | # . 21 | # . 22 | # ) 23 | # 24 | # 25 | # This will automatically update the RELEASE-VERSION file, if 26 | # necessary. Note that the RELEASE-VERSION file should *not* be 27 | # checked into git; please add it to your top-level .gitignore file. 28 | # 29 | # You'll probably want to distribute the RELEASE-VERSION file in your 30 | # sdist tarballs; to do this, just create a MANIFEST.in file that 31 | # contains the following line: 32 | # 33 | # include RELEASE-VERSION 34 | 35 | __all__ = ("get_git_version") 36 | 37 | from subprocess import Popen, PIPE 38 | 39 | 40 | def call_git_describe(abbrev): 41 | try: 42 | p = Popen(['git', 'describe', '--abbrev=%d' % abbrev], 43 | stdout=PIPE, stderr=PIPE) 44 | p.stderr.close() 45 | line = p.stdout.readlines()[0] 46 | return line.strip() 47 | 48 | except: 49 | return None 50 | 51 | 52 | def is_dirty(): 53 | try: 54 | p = Popen(["git", "diff-index", "--name-only", "HEAD"], 55 | stdout=PIPE, stderr=PIPE) 56 | p.stderr.close() 57 | lines = p.stdout.readlines() 58 | return len(lines) > 0 59 | except: 60 | return False 61 | 62 | 63 | def read_release_version(): 64 | try: 65 | f = open("RELEASE-VERSION", "r") 66 | 67 | try: 68 | version = f.readlines()[0] 69 | return version.strip() 70 | 71 | finally: 72 | f.close() 73 | 74 | except: 75 | return None 76 | 77 | 78 | def write_release_version(version): 79 | f = open("RELEASE-VERSION", "w") 80 | f.write("%s\n" % version) 81 | f.close() 82 | 83 | def get_git_version(abbrev=7): 84 | # Read in the version that's currently in RELEASE-VERSION. 85 | 86 | release_version = read_release_version() 87 | 88 | # First try to get the current version using “git describe”. 89 | 90 | version = call_git_describe(abbrev) 91 | if version is not None and is_dirty(): # JPM 92 | version += "-dirty" 93 | 94 | # If that doesn't work, fall back on the value that's in 95 | # RELEASE-VERSION. 96 | 97 | if version is None: 98 | version = release_version 99 | 100 | # If we still don't have anything, that's an error. 101 | 102 | if version is None: 103 | raise ValueError("Cannot find the version number!") 104 | 105 | # If the current version is different from what's in the 106 | # RELEASE-VERSION file, update the file to be current. 107 | 108 | if version != release_version: 109 | write_release_version(version) 110 | 111 | # Finally, return the current version. 112 | 113 | return version 114 | 115 | 116 | if __name__ == "__main__": 117 | print get_git_version() 118 | --------------------------------------------------------------------------------