├── .travis.yml ├── images ├── figures.pptx ├── mqtt_message_conversion.png ├── libelium_sensor_data_collection.png └── sensor_data_uploads_from_tiny_computers.png ├── Gemfile ├── .gitignore ├── test ├── helper.rb └── plugin │ ├── test_in_mqtt.rb │ └── test_out_mqtt.rb ├── Rakefile ├── CHANGELOG.md ├── fluent-plugin-mqtt-io.gemspec ├── lib └── fluent │ └── plugin │ ├── in_mqtt.rb │ ├── out_mqtt.rb │ └── mqtt_proxy.rb ├── README.md ├── LICENSE └── README-old.md /.travis.yml: -------------------------------------------------------------------------------- 1 | language: ruby 2 | rvm: 3 | - 2.2.2 4 | before_install: gem install bundler -v 1.10.2 5 | -------------------------------------------------------------------------------- /images/figures.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/toyokazu/fluent-plugin-mqtt-io/HEAD/images/figures.pptx -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Specify your gem's dependencies in fluent-plugin-mqtt-io.gemspec 4 | gemspec 5 | -------------------------------------------------------------------------------- /images/mqtt_message_conversion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/toyokazu/fluent-plugin-mqtt-io/HEAD/images/mqtt_message_conversion.png -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /.bundle/ 2 | /.yardoc 3 | /Gemfile.lock 4 | /_yardoc/ 5 | /coverage/ 6 | /doc/ 7 | /pkg/ 8 | /spec/reports/ 9 | /tmp/ 10 | -------------------------------------------------------------------------------- /images/libelium_sensor_data_collection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/toyokazu/fluent-plugin-mqtt-io/HEAD/images/libelium_sensor_data_collection.png -------------------------------------------------------------------------------- /images/sensor_data_uploads_from_tiny_computers.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/toyokazu/fluent-plugin-mqtt-io/HEAD/images/sensor_data_uploads_from_tiny_computers.png -------------------------------------------------------------------------------- /test/helper.rb: -------------------------------------------------------------------------------- 1 | require 'bundler/setup' 2 | require 'test/unit' 3 | 4 | $LOAD_PATH.unshift(File.join(__dir__, '..', 'lib')) 5 | $LOAD_PATH.unshift(__dir__) 6 | require 'fluent/test' 7 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | require "bundler/gem_tasks" 2 | require "rake/testtask" 3 | 4 | Rake::TestTask.new(:test) do |t| 5 | t.libs << "test" 6 | t.libs << "lib" 7 | t.test_files = FileList['test/**/test_*.rb'] 8 | end 9 | 10 | task :default => :test 11 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## 0.4.0 (2018-01-05) 2 | Features: 3 | - change fluentd dependency from ~> 0.14 to >= 0.14 4 | 5 | 6 | ## 0.3.0 (2017-03-20) 7 | 8 | Features: 9 | - compatible with fluentd v0.14 10 | - change license from MIT License to Apache License Version 2.0 11 | 12 | Deprecations: 13 | 14 | - deprecated support of fluentd v0.12 15 | -------------------------------------------------------------------------------- /fluent-plugin-mqtt-io.gemspec: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | lib = File.expand_path('../lib', __FILE__) 3 | $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 4 | 5 | Gem::Specification.new do |spec| 6 | spec.name = "fluent-plugin-mqtt-io" 7 | spec.version = "0.5.0" 8 | spec.authors = ["Toyokazu Akiyama"] 9 | spec.email = ["toyokazu@gmail.com"] 10 | 11 | spec.summary = %q{fluentd input/output plugin for mqtt broker} 12 | spec.description = %q{fluentd input/output plugin for mqtt broker} 13 | spec.homepage = "https://github.com/toyokazu/fluent-plugin-mqtt-io" 14 | spec.license = "Apache License Version 2.0" 15 | 16 | spec.files = `git ls-files`.gsub(/images\/[\w\.]+\n/, "").split($/) 17 | spec.bindir = spec.files.grep(%r{^bin/}) { |f| File.basename(f) } 18 | spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) } 19 | spec.test_files = spec.files.grep(%r{^(test|spec|features)/}) 20 | spec.require_paths = ["lib"] 21 | 22 | spec.required_ruby_version = '>= 2.2.0' 23 | 24 | spec.add_dependency 'fluentd', [">= 0.14.0", "< 2"] 25 | spec.add_dependency "mqtt", "~> 0.5" 26 | 27 | spec.add_development_dependency "bundler", [">= 1.14", "< 2.3"] 28 | spec.add_development_dependency "rake", "~> 13.0" 29 | spec.add_development_dependency "test-unit" 30 | end 31 | -------------------------------------------------------------------------------- /test/plugin/test_in_mqtt.rb: -------------------------------------------------------------------------------- 1 | require_relative '../helper' 2 | require 'fluent/test/driver/input' 3 | require 'fluent/plugin/in_mqtt' 4 | 5 | class MqttInputTest < Test::Unit::TestCase 6 | def setup 7 | Fluent::Test.setup 8 | end 9 | 10 | 11 | CONFIG = %[ 12 | ] 13 | 14 | def create_driver(conf = CONFIG, opts = {}) 15 | Fluent::Test::Driver::Input.new(Fluent::Plugin::MqttInput, opts: opts).configure(conf) 16 | end 17 | 18 | sub_test_case "configure" do 19 | test "host and port" do 20 | d = create_driver %[ 21 | host 127.0.0.1 22 | port 1300 23 | client_id aa-bb-cc-dd 24 | 25 | recv_time true 26 | 27 | 28 | use_tls true 29 | 30 | ca_file /cert/cacert.pem 31 | key_file /cert/private.key 32 | cert_file /cert/cert.pem 33 | 34 | 35 | 36 | ] 37 | assert_equal '127.0.0.1', d.instance.host 38 | assert_equal 1300, d.instance.port 39 | assert_equal 'aa-bb-cc-dd', d.instance.client_id 40 | 41 | assert_equal 'none', d.instance.parser_configs.first[:@type] 42 | 43 | assert_equal true, d.instance.monitor.recv_time 44 | 45 | assert_equal true, d.instance.security.use_tls 46 | assert_equal '/cert/cacert.pem', d.instance.security.tls.ca_file 47 | assert_equal '/cert/private.key', d.instance.security.tls.key_file 48 | assert_equal '/cert/cert.pem', d.instance.security.tls.cert_file 49 | 50 | end 51 | end 52 | end 53 | -------------------------------------------------------------------------------- /test/plugin/test_out_mqtt.rb: -------------------------------------------------------------------------------- 1 | require_relative '../helper' 2 | require 'fluent/test/driver/output' 3 | require 'fluent/plugin/out_mqtt' 4 | 5 | class MqttOutputTest < Test::Unit::TestCase 6 | def setup 7 | Fluent::Test.setup 8 | end 9 | 10 | 11 | CONFIG = %[ 12 | ] 13 | 14 | def create_driver(conf = CONFIG, opts = {}) 15 | Fluent::Test::Driver::Output.new(Fluent::Plugin::MqttOutput, opts: opts).configure(conf) 16 | end 17 | 18 | sub_test_case 'non-buffered' do 19 | test 'configure' do 20 | d = create_driver %[ 21 | host 127.0.0.1 22 | port 1300 23 | client_id aa-bb-cc-dd 24 | retain true 25 | qos 2 26 | 27 | async true 28 | 29 | 30 | send_time true 31 | 32 | 33 | use_tls true 34 | 35 | ca_file /cert/cacert.pem 36 | key_file /cert/private.key 37 | cert_file /cert/cert.pem 38 | 39 | 40 | ] 41 | assert_equal '127.0.0.1', d.instance.host 42 | assert_equal 1300, d.instance.port 43 | assert_equal 'aa-bb-cc-dd', d.instance.client_id 44 | assert_equal true, d.instance.retain 45 | assert_equal 2, d.instance.qos 46 | 47 | assert_equal true, d.instance.buffer_config.async 48 | 49 | assert_equal true, d.instance.monitor.send_time 50 | 51 | # cannot test formatter configuration (default: single_value) 52 | 53 | assert_equal true, d.instance.security.use_tls 54 | assert_equal '/cert/cacert.pem', d.instance.security.tls.ca_file 55 | assert_equal '/cert/private.key', d.instance.security.tls.key_file 56 | assert_equal '/cert/cert.pem', d.instance.security.tls.cert_file 57 | end 58 | end 59 | end 60 | -------------------------------------------------------------------------------- /lib/fluent/plugin/in_mqtt.rb: -------------------------------------------------------------------------------- 1 | require 'fluent/plugin/input' 2 | require 'fluent/event' 3 | require 'fluent/time' 4 | require 'fluent/plugin/mqtt_proxy' 5 | 6 | module Fluent::Plugin 7 | class MqttInput < Input 8 | include MqttProxy 9 | include Fluent::TimeMixin::Formatter 10 | 11 | Fluent::Plugin.register_input('mqtt', self) 12 | 13 | helpers :compat_parameters, :parser 14 | 15 | desc 'The topic to subscribe.' 16 | config_param :topic, :string, default: '#' 17 | 18 | config_section :parse do 19 | desc 'The format to receive.' 20 | config_param :@type, :string, default: 'none' 21 | end 22 | 23 | config_section :monitor, required: false, multi: false do 24 | desc 'Record received time into message or not.' 25 | config_param :recv_time, :bool, default: false 26 | desc 'Specify the attribute name of received time.' 27 | config_param :recv_time_key, :string, default: 'recv_time' 28 | desc 'Specify time type of recv_time (string, unixtime, float).' 29 | config_param :time_type, :string, default: 'string' 30 | desc 'Specify time format of recv_time (e.g. %FT%T.%N%:z).' 31 | config_param :time_format, :string, default: nil 32 | end 33 | 34 | def configure(conf) 35 | super 36 | configure_parser(conf) 37 | if !@monitor.nil? 38 | @recv_time_formatter = time_formatter_create( 39 | type: @monitor.time_type.to_sym, format: @monitor.time_format 40 | ) 41 | end 42 | end 43 | 44 | def configure_parser(conf) 45 | compat_parameters_convert(conf, :parser) 46 | parser_config = conf.elements('parse').first 47 | @parser = parser_create(conf: parser_config) 48 | end 49 | 50 | def start 51 | super 52 | start_proxy 53 | end 54 | 55 | def shutdown 56 | shutdown_proxy 57 | super 58 | end 59 | 60 | def current_plugin_name 61 | :in_mqtt 62 | end 63 | 64 | def exit_thread 65 | @get_thread.exit if !@get_thread.nil? 66 | end 67 | 68 | def disconnect 69 | begin 70 | @client.disconnect if @client.connected? 71 | rescue => e 72 | log.error "Error in in_mqtt#disconnect,#{e.class},#{e.message}" 73 | end 74 | exit_thread 75 | end 76 | 77 | def terminate 78 | exit_thread 79 | super 80 | end 81 | 82 | def after_connection 83 | if @client.connected? 84 | @client.subscribe(@topic) 85 | @get_thread = thread_create(:in_mqtt_get) do 86 | @client.get do |topic, message| 87 | emit(topic, message) 88 | end 89 | end 90 | end 91 | @get_thread 92 | end 93 | 94 | def add_recv_time(record) 95 | if !@monitor.nil? && @monitor.recv_time 96 | # recv_time is recorded in ms 97 | record.merge({"#{@monitor.recv_time_key}": @recv_time_formatter.format(Fluent::EventTime.now)}) 98 | else 99 | record 100 | end 101 | end 102 | 103 | def parse(message) 104 | @parser.parse(message) do |time, record| 105 | if time.nil? 106 | log.debug "Since time_key field is nil, Fluent::EventTime.now is used." 107 | time = Fluent::EventTime.now 108 | end 109 | return [time, record] 110 | end 111 | end 112 | 113 | def emit(topic, message) 114 | begin 115 | tag = topic.gsub("/","\.") 116 | time, record = parse(message) 117 | if record.is_a?(Array) 118 | mes = Fluent::MultiEventStream.new 119 | record.each do |single_record| 120 | log.debug "MqttInput#emit: #{tag}, #{time}, #{add_recv_time(single_record)}" 121 | mes.add(@parser.parse_time(single_record), add_recv_time(single_record)) 122 | end 123 | router.emit_stream(tag, mes) 124 | else 125 | log.debug "MqttInput#emit: #{tag}, #{time}, #{add_recv_time(record)}" 126 | router.emit(tag, time, add_recv_time(record)) 127 | end 128 | rescue Exception => e 129 | log.error error: e.to_s 130 | log.debug_backtrace(e.backtrace) 131 | end 132 | end 133 | end 134 | end 135 | -------------------------------------------------------------------------------- /lib/fluent/plugin/out_mqtt.rb: -------------------------------------------------------------------------------- 1 | require 'fluent/plugin/output' 2 | require 'fluent/event' 3 | require 'fluent/time' 4 | require 'fluent/plugin/mqtt_proxy' 5 | 6 | module Fluent::Plugin 7 | class MqttOutput < Output 8 | include MqttProxy 9 | include Fluent::TimeMixin::Formatter 10 | 11 | Fluent::Plugin.register_output('mqtt', self) 12 | 13 | helpers :compat_parameters, :formatter, :inject 14 | 15 | 16 | desc 'Topic rewrite matching pattern.' 17 | config_param :topic_rewrite_pattern, :string, default: nil 18 | desc 'Topic rewrite replacement string.' 19 | config_param :topic_rewrite_replacement, :string, default: nil 20 | 21 | desc 'Retain option which publishing' 22 | config_param :retain, :bool, default: false 23 | desc 'QoS option which publishing' 24 | config_param :qos, :integer, default: 1 25 | 26 | config_section :format do 27 | desc 'The format to publish' 28 | config_param :@type, :string, default: 'single_value' 29 | desc 'Add newline' 30 | config_param :add_newline, :bool, default: false 31 | end 32 | 33 | config_section :monitor, required: false, multi: false do 34 | desc 'Recording send time for monitoring.' 35 | config_param :send_time, :bool, default: false 36 | desc 'Recording key name of send time for monitoring.' 37 | config_param :send_time_key, :string, default: "send_time" 38 | desc 'Specify time type of send_time (string, unixtime, float).' 39 | config_param :time_type, :string, default: 'string' 40 | desc 'Specify time format of send_time (e.g. %FT%T.%N%:z).' 41 | config_param :time_format, :string, default: nil 42 | end 43 | 44 | config_section :buffer, required: false, multi: false do 45 | desc 'Prefer asynchronous buffering' 46 | config_param :async, :bool, default: false 47 | end 48 | 49 | # This method is called before starting. 50 | # 'conf' is a Hash that includes configuration parameters. 51 | # If the configuration is invalid, raise Fluent::ConfigError. 52 | def configure(conf) 53 | super 54 | compat_parameters_convert(conf, :formatter, :inject, :buffer, default_chunk_key: "time") 55 | formatter_config = conf.elements(name: 'format').first 56 | @formatter = formatter_create(conf: formatter_config) 57 | @has_buffer_section = conf.elements(name: 'buffer').size > 0 58 | if !@monitor.nil? 59 | @send_time_formatter = time_formatter_create( 60 | type: @monitor.time_type.to_sym, format: @monitor.time_format 61 | ) 62 | end 63 | end 64 | 65 | def rewrite_tag(tag) 66 | if @topic_rewrite_pattern.nil? 67 | tag.gsub("\.", "/") 68 | else 69 | tag.gsub("\.", "/").gsub(Regexp.new(@topic_rewrite_pattern), @topic_rewrite_replacement) 70 | end 71 | end 72 | 73 | def prefer_buffered_processing 74 | @has_buffer_section 75 | end 76 | 77 | def prefer_delayed_commit 78 | @has_buffer_section && @buffer_config.async 79 | end 80 | 81 | # This method is called when starting. 82 | # Open sockets or files here. 83 | def start 84 | super 85 | start_proxy 86 | end 87 | 88 | # This method is called when shutting down. 89 | # Shutdown the thread and close sockets or files here. 90 | def shutdown 91 | shutdown_proxy 92 | exit_thread 93 | super 94 | end 95 | 96 | def exit_thread 97 | @dummy_thread.exit if !@dummy_thread.nil? 98 | end 99 | 100 | def disconnect 101 | begin 102 | @client.disconnect if @client.connected? 103 | rescue => e 104 | log.error "Error in out_mqtt#disconnect,#{e.class},#{e.message}" 105 | end 106 | exit_thread 107 | end 108 | 109 | def terminate 110 | exit_thread 111 | super 112 | end 113 | 114 | def after_connection 115 | @dummy_thread = thread_create(:out_mqtt_dummy) do 116 | Thread.stop 117 | end 118 | @dummy_thread 119 | end 120 | 121 | def current_plugin_name 122 | :out_mqtt 123 | end 124 | 125 | def add_send_time(record) 126 | if !@monitor.nil? && @monitor.send_time 127 | # send_time is recorded in ms 128 | record.merge({"#{@monitor.send_time_key}": @send_time_formatter.format(Fluent::EventTime.now)}) 129 | else 130 | record 131 | end 132 | end 133 | 134 | def publish_event_stream(tag, es) 135 | log.debug "publish_event_stream: #{es.class}" 136 | es = inject_values_to_event_stream(tag, es) 137 | es.each do |time, record| 138 | rescue_disconnection do 139 | publish(tag, time, record) 140 | end 141 | end 142 | log.flush 143 | end 144 | 145 | def process(tag, es) 146 | publish_event_stream(tag, es) 147 | end 148 | 149 | def format(tag, time, record) 150 | record = inject_values_to_record(tag, time, record) 151 | [tag, time, record].to_msgpack 152 | end 153 | 154 | def formatted_to_msgpack_binary 155 | true 156 | end 157 | 158 | def publish(tag, time, record) 159 | log.debug "MqttOutput::#{caller_locations(1,1)[0].label}: #{rewrite_tag(tag)}, #{time}, #{add_send_time(record)}" 160 | @client.publish( 161 | rewrite_tag(tag), 162 | @formatter.format(tag, time, add_send_time(record)), 163 | @retain, 164 | @qos 165 | ) 166 | end 167 | 168 | def write(chunk) 169 | return if chunk.empty? 170 | chunk.each do |tag, time, record| 171 | rescue_disconnection do 172 | publish(tag, time, record) 173 | end 174 | end 175 | end 176 | 177 | def try_write(chunk) 178 | return if chunk.empty? 179 | rescue_disconnection do 180 | chunk.each do |tag, time, record| 181 | publish(tag, time, record) 182 | end 183 | commit_write(chunk.unique_id) 184 | end 185 | end 186 | end 187 | end 188 | -------------------------------------------------------------------------------- /lib/fluent/plugin/mqtt_proxy.rb: -------------------------------------------------------------------------------- 1 | require 'mqtt' 2 | module Fluent::Plugin 3 | module MqttProxy 4 | MQTT_PORT = 1883 5 | 6 | def self.included(base) 7 | base.helpers :timer, :thread 8 | 9 | base.desc 'The address to connect to.' 10 | base.config_param :host, :string, default: '127.0.0.1' 11 | base.desc 'The port to connect to.' 12 | base.config_param :port, :integer, default: MQTT_PORT 13 | base.desc 'Client ID of MQTT Connection' 14 | base.config_param :client_id, :string, default: nil 15 | base.desc 'Specify clean session value.' 16 | base.config_param :clean_session, :bool, default: true 17 | base.desc 'Specify keep alive interval.' 18 | base.config_param :keep_alive, :integer, default: 15 19 | base.desc 'Specify initial connection retry interval.' 20 | base.config_param :initial_interval, :integer, default: 1 21 | base.desc 'Specify increasing ratio of connection retry interval.' 22 | base.config_param :retry_inc_ratio, :integer, default: 2 23 | base.desc 'Specify the maximum connection retry interval.' 24 | base.config_param :max_retry_interval, :integer, default: 300 25 | base.desc 'Specify threshold of retry frequency as number of retries per minutes. Frequency is monitored per retry.' 26 | base.config_param :max_retry_freq, :integer, default: 10 27 | 28 | base.config_section :security, required: false, multi: false do 29 | ### User based authentication 30 | desc 'The username for authentication' 31 | config_param :username, :string, default: nil 32 | desc 'The password for authentication' 33 | config_param :password, :string, default: nil 34 | desc 'Use TLS or not.' 35 | config_param :use_tls, :bool, default: nil 36 | config_section :tls, required: false, multi: false do 37 | desc 'Specify TLS ca file.' 38 | config_param :ca_file, :string, default: nil 39 | desc 'Specify TLS key file.' 40 | config_param :key_file, :string, default: nil 41 | desc 'Specify TLS cert file.' 42 | config_param :cert_file, :string, default: nil 43 | end 44 | end 45 | end 46 | 47 | class MqttError < StandardError; end 48 | 49 | class ExceedRetryFrequencyThresholdException < StandardError; end 50 | 51 | def current_plugin_name 52 | # should be implemented 53 | end 54 | 55 | def start_proxy 56 | # Start a thread from main thread for handling a thread generated 57 | # by MQTT::Client#get (in_mqtt). Dummy thread is used for out_mqtt 58 | # to keep the same implementation style. 59 | @proxy_thread = thread_create("#{current_plugin_name}_proxy".to_sym, &method(:proxy)) 60 | end 61 | 62 | def proxy 63 | log.debug "start mqtt proxy for #{current_plugin_name}" 64 | log.debug "start to connect mqtt broker #{@host}:#{@port}" 65 | opts = { 66 | host: @host, 67 | port: @port, 68 | client_id: @client_id, 69 | clean_session: @clean_session, 70 | keep_alive: @keep_alive 71 | } 72 | opts[:username] = @security.username if @security.to_h.has_key?(:username) 73 | opts[:password] = @security.password if @security.to_h.has_key?(:password) 74 | if @security.to_h.has_key?(:use_tls) && @security.use_tls 75 | opts[:ssl] = @security.use_tls 76 | opts[:ca_file] = @security.tls.ca_file 77 | opts[:cert_file] = @security.tls.cert_file 78 | opts[:key_file] = @security.tls.key_file 79 | end 80 | 81 | init_retry_interval 82 | @retry_sequence = [] 83 | @client = MQTT::Client.new(opts) 84 | connect 85 | end 86 | 87 | def shutdown_proxy 88 | disconnect 89 | end 90 | 91 | def init_retry_interval 92 | @retry_interval = @initial_interval 93 | end 94 | 95 | def increment_retry_interval 96 | return @max_retry_interval if @retry_interval >= @max_retry_interval 97 | @retry_interval = @retry_interval * @retry_inc_ratio 98 | end 99 | 100 | def update_retry_sequence(e) 101 | @retry_sequence << {time: Time.now, error: "#{e.class}: #{e.message}"} 102 | # delete old retry records 103 | while @retry_sequence[0][:time] < Time.now - 60 104 | @retry_sequence.shift 105 | end 106 | end 107 | 108 | def check_retry_frequency 109 | return if @retry_sequence.size <= 1 110 | if @retry_sequence.size > @max_retry_freq 111 | log.error "Retry frequency threshold is exceeded: #{@retry_sequence}" 112 | raise ExceedRetryFrequencyThresholdException 113 | end 114 | end 115 | 116 | def retry_connect(e, message) 117 | log.error "#{message},#{e.class},#{e.message}" 118 | log.error "Retry in #{@retry_interval} sec" 119 | update_retry_sequence(e) 120 | check_retry_frequency 121 | disconnect 122 | sleep @retry_interval 123 | increment_retry_interval 124 | connect 125 | # never reach here 126 | end 127 | 128 | def disconnect 129 | # should be implemented 130 | end 131 | 132 | def terminate 133 | end 134 | 135 | def rescue_disconnection 136 | # Errors cannot be caught by fluentd core must be caught here. 137 | # Since fluentd core retries write method for buffered output 138 | # when it caught Errors during the previous write, 139 | # caughtable Error, e.g. MqttError, should be raised here. 140 | begin 141 | yield 142 | rescue MQTT::ProtocolException => e 143 | retry_connect(e, "Protocol error occurs in #{current_plugin_name}.") 144 | rescue Timeout::Error => e 145 | retry_connect(e, "Timeout error occurs in #{current_plugin_name}.") 146 | rescue SystemCallError => e 147 | retry_connect(e, "System call error occurs in #{current_plugin_name}.") 148 | rescue StandardError=> e 149 | retry_connect(e, "The other error occurs in #{current_plugin_name}.") 150 | rescue MQTT::NotConnectedException=> e 151 | # Since MQTT::NotConnectedException is raised only on publish, 152 | # connection error should be catched before this error. 153 | # So, reconnection process is omitted for this Exception 154 | # to prevent waistful increment of retry interval. 155 | #log.error "MQTT not connected exception occurs.,#{e.class},#{e.message}" 156 | #retry_connect(e, "MQTT not connected exception occurs.") 157 | #raise MqttError, "MQTT not connected exception occurs in #{current_plugin_name}." 158 | end 159 | end 160 | 161 | def after_connection 162 | # should be implemented 163 | # returns thread instance for monitor thread to wait 164 | # for Exception raised by MQTT I/O 165 | end 166 | 167 | def connect 168 | rescue_disconnection do 169 | @client.connect 170 | log.debug "connected to mqtt broker #{@host}:#{@port} for #{current_plugin_name}" 171 | init_retry_interval 172 | thread = after_connection 173 | thread.join 174 | end 175 | end 176 | end 177 | end 178 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Fluent::Plugin::Mqtt::IO 2 | 3 | Fluent plugin for MQTT Input/Output. 4 | Mqtt::IO Plugin is deeply inspired by Fluent::Plugin::Mqtt. 5 | 6 | https://github.com/yuuna/fluent-plugin-mqtt 7 | 8 | Mqtt::IO plugin focus on federating components, e.g. sensors, messaging platform and databases. Connection encryption/decryption (TLS) is supported by ruby-mqtt but end-to-end encryption/decryption is not supported in this plugin. [fluent-plugin-jwt-filter](https://github.com/toyokazu/fluent-plugin-jwt-filter) can be used to encrypt/decrypt messages using JSON Web Token technology. 9 | 10 | ## Installation 11 | 12 | Add this line to your application's Gemfile: 13 | 14 | gem 'fluent-plugin-mqtt-io' 15 | 16 | And then execute: 17 | 18 | $ bundle 19 | 20 | Or install it yourself as: 21 | 22 | $ gem install fluent-plugin-mqtt-io 23 | 24 | 25 | ## Usage 26 | 27 | fluent-plugin-mqtt-io provides Input and Output Plugins for MQTT. 28 | 29 | ### Input Plugin (Fluet::MqttInput) 30 | 31 | Input Plugin can be used via source directive in the configuration. 32 | 33 | ``` 34 | 35 | @type mqtt 36 | host 127.0.0.1 37 | port 1883 38 | 39 | @type json 40 | 41 | 42 | 43 | ``` 44 | 45 | When using security options, specify them in security section; for example: 46 | 47 | ``` 48 | 49 | @type mqtt 50 | host 'your_host' 51 | port 'your_port' 52 | 53 | username 'your_username' 54 | password 'your_password' 55 | 56 | 57 | @type json 58 | 59 | 60 | ``` 61 | 62 | The default MQTT topic is "#". Configurable options are the following: 63 | 64 | - **host**: IP address of MQTT broker 65 | - **port**: Port number of MQTT broker 66 | - **client_id**: Client ID that to connect to MQTT broker 67 | - **parser**: Parser plugin can be specified [Parser Plugin](https://docs.fluentd.org/v1.0/articles/parser-plugin-overview) 68 | - **topic**: Topic name to be subscribed 69 | - **security** 70 | - **username**: User name for authentication 71 | - **password**: Password for authentication 72 | - **use_tls**: set true if you want to use SSL/TLS. If set to true, the following parameter must be provided 73 | - **ca_file**: CA certificate file path 74 | - **key_file**: private key file path 75 | - **cert_file**: certificate file path 76 | - **clean_session**: Setting for clean session client option (false persists session offline) 77 | - **keep_alive**: An interval of sending keep alive packet (default 15 sec) 78 | - **monitor**: monitor section. only for fluent-plugin-mqtt-io 79 | - **recv_time**: Add receive time to message in millisecond (ms) as integer for debug and performance/delay analysis (default: false). only for fluent-plugin-mqtt-io 80 | - **recv_time_key**: An attribute of recv_time (default: "recv_time"). only for fluent-plugin-mqtt-io 81 | - **time_type**: Type of time format (string (default), unixtime, float) only for fluent-plugin-mqtt-io 82 | - **time_format**: Time format e.g. %FT%T.%N%:z (refer strftime) only for fluent-plugin-mqtt-io 83 | - **initial_interval**: An initial value of retry interval (s) (default 1) 84 | - **retry_inc_ratio**: An increase ratio of retry interval per connection failure (default 2 (double)). It may be better to set the value to 1 in a mobile environment for eager reconnection. 85 | - **max_retry_interval**: Maximum value of retry interval (default 300) 86 | - **max_retry_freq**: Threshold of retry frequency described by a number of retries per minutes. This option is provided for detecting failure via proxy services, e.g. ssh port forwarding. When the thresold is exceeded, MqttProxy::ExceedRetryFrequencyThresholdException is raised and the fluentd will be restarted. So, it is enough to be specified once for a MQTT server at a source/match directive in your configuration (default 10) 87 | 88 | Input Plugin supports @label directive. 89 | 90 | ### Output Plugin (Fluent::MqttOutput, Fluent::MqttBufferedOutput) 91 | 92 | Output Plugin can be used via match directive. 93 | 94 | ``` 95 | 96 | @type mqtt 97 | host 127.0.0.1 98 | port 1883 99 | 100 | @type json 101 | add_newline false 102 | 103 | 104 | ``` 105 | 106 | The options are basically the same as Input Plugin except for "parser (for Input)/format (for Output)". Additional options for Output Plugin are the following. 107 | 108 | - **qos**: Quality of Service (QoS) for message publishing, 0 or 1 is valid. 2 is not supported by mqtt client. Default is 1. 109 | - **retain**: If set true the broker will keep the message even after sending it to all current subscribers. Default is false 110 | - **time_key**: An attribute name used for timestamp field genarated from fluentd time field. Default is nil (omitted). 111 | If this option is omitted, timestamp field is not appended to the output record. 112 | - **topic_rewrite_pattern**: Regexp pattern to extract replacement words from received topic or tag name 113 | - **topic_rewrite_replacement**: Topic name used for the publish using extracted pattern 114 | - **format**: Formatter plugin can be specified. [Formatter Plugin](https://docs.fluentd.org/v1.0/articles/formatter-plugin-overview) 115 | - **monitor**: monitor section. only for fluent-plugin-mqtt-io 116 | - **send_time**: Add send time to message in millisecond (ms) as integer for debug and performance/delay analysis. only for fluent-plugin-mqtt-io 117 | - **send_time_key**: An attribute of send_time. only for fluent-plugin-mqtt-io 118 | - **time_type**: Type of time format (string (default), unixtime, float) only for fluent-plugin-mqtt-io 119 | - **time_format**: Time format e.g. %FT%T.%N%:z (refer strftime) only for fluent-plugin-mqtt-io 120 | - **buffer**: synchronous/asynchronous buffer is supported. Refer [Buffer section configurations](https://docs.fluentd.org/v1.0/articles/buffer-section) for detailed configuration. 121 | - **async**: Enable asynchronous buffer transfer. Default is false. 122 | 123 | If you use different source, e.g. the other MQTT broker, log file and so on, there is no need to specifie topic rewriting. Skip the following descriptions. 124 | 125 | The topic name or tag name, e.g. "topic", received from an event can not be published without modification because if MQTT input plugin connecting to the identical MQTT broker is used as a source, the same message will become an input repeatedly. In order to support data conversion in single MQTT domain, simple topic rewriting should be supported. Since topic is rewritten using #gsub method, 'pattern' and 'replacement' are the same as #gsub arguments. 126 | 127 | 128 | ``` 129 | 130 | @type mqtt 131 | host 127.0.0.1 132 | port 1883 133 | 134 | @type json 135 | add_newline false 136 | 137 | topic_rewrite_pattern '^([\w\/]+)$' 138 | topic_rewrite_replacement '\1/rewritten' 139 | 140 | ``` 141 | 142 | 143 | ``` 144 | 145 | @type mqtt 146 | host 127.0.0.1 147 | port 1883 148 | 149 | @type json 150 | 151 | topic_rewrite_pattern '^([\w\/]+)$' 152 | topic_rewrite_replacement '\1/rewritten' 153 | # You can specify Buffer Plugin options 154 | 155 | buffer_type memory 156 | flush_interval 1s 157 | 158 | 159 | ``` 160 | 161 | 162 | ## Contributing 163 | 164 | 1. Fork it ( http://github.com/toyokazu/fluent-plugin-mqtt-io/fork ) 165 | 2. Create your feature branch (`git checkout -b my-new-feature`) 166 | 3. Commit your changes (`git commit -am 'Add some feature'`) 167 | 4. Push to the branch (`git push origin my-new-feature`) 168 | 5. Create new Pull Request 169 | 170 | 171 | ## License 172 | 173 | The gem is available as open source under the terms of the [Apache License Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). 174 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | https://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | Copyright 2015-2017 Toyokazu Akiyama 180 | 181 | Licensed under the Apache License, Version 2.0 (the "License"); 182 | you may not use this file except in compliance with the License. 183 | You may obtain a copy of the License at 184 | 185 | https://www.apache.org/licenses/LICENSE-2.0 186 | 187 | Unless required by applicable law or agreed to in writing, software 188 | distributed under the License is distributed on an "AS IS" BASIS, 189 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 190 | See the License for the specific language governing permissions and 191 | limitations under the License. 192 | -------------------------------------------------------------------------------- /README-old.md: -------------------------------------------------------------------------------- 1 | # Fluent::Plugin::Mqtt::IO 2 | 3 | Fluent plugin for MQTT Input/Output. 4 | Mqtt::IO Plugin is deeply inspired by Fluent::Plugin::Mqtt. 5 | 6 | https://github.com/yuuna/fluent-plugin-mqtt 7 | 8 | Mqtt::IO plugin focus on federating components, e.g. sensors, messaging platform and databases. Encryption/Decryption is not supported in this plugin but [fluent-plugin-jwt-filter](https://github.com/toyokazu/fluent-plugin-jwt-filter) can be used to encrypt/decrypt messages using JSON Web Token technology. 9 | 10 | ## Installation 11 | 12 | Add this line to your application's Gemfile: 13 | 14 | gem 'fluent-plugin-mqtt-io' 15 | 16 | And then execute: 17 | 18 | $ bundle 19 | 20 | Or install it yourself as: 21 | 22 | $ gem install fluent-plugin-mqtt-io 23 | 24 | 25 | ## Usage 26 | 27 | fluent-plugin-mqtt-io provides Input and Output Plugins for MQTT. 28 | 29 | ### Input Plugin (Fluet::MqttInput) 30 | 31 | Input Plugin can be used via source directive in the configuration. 32 | 33 | For fluent-plugin-mqtt-io <= 0.2.3 34 | 35 | ``` 36 | 37 | 38 | type mqtt 39 | host 127.0.0.1 40 | port 1883 41 | format json 42 | 43 | 44 | ``` 45 | 46 | For fluent-plugin-mqtt-io >= v0.3.0 47 | 48 | ``` 49 | 50 | 51 | @type mqtt 52 | host 127.0.0.1 53 | port 1883 54 | 55 | @type json 56 | 57 | 58 | 59 | ``` 60 | 61 | 62 | The default MQTT topic is "#". Configurable options are the following: 63 | 64 | - **host**: IP address of MQTT broker 65 | - **port**: Port number of MQTT broker 66 | - **client_id**: Client ID that to connect to MQTT broker 67 | - **format** (mandatory): Input parser can be chosen, e.g. json, xml 68 | - In order to use xml format, you need to install [fluent-plugin-xml-parser](https://github.com/toyokazu/fluent-plugin-xml-parser). 69 | - Default time_key field for json format is 'time' 70 | - **topic**: Topic name to be subscribed 71 | - **bulk_trans**: Enable bulk transfer to support buffered output (mqtt_buf, Fluent::MqttBufferedOutput, defaut: true) only for fluent-plugin-mqtt-io <= 0.2.3 72 | - **bulk_trans_sep**: A message separator for bulk transfer. The default separator is "\t". only for fluent-plugin-mqtt-io <= 0.2.3 73 | - **username**: User name for authentication 74 | - **password**: Password for authentication 75 | - **clean_session**: Setting for clean session client option (false persists session offline) 76 | - **keep_alive**: An interval of sending keep alive packet (default 15 sec) 77 | - **ssl**: set true if you want to use SSL/TLS. If set to true, the following parameter must be provided 78 | - **ca_file**: CA certificate file path 79 | - **key_file**: private key file path 80 | - **cert_file**: certificate file path 81 | - **recv_time**: Add receive time to message in millisecond (ms) as integer for debug and performance/delay analysis. only for fluent-plugin-mqtt-io <= 0.2.3 82 | - **recv_time_key**: An attribute of recv_time. only for fluent-plugin-mqtt-io <= 0.2.3 83 | - **monitor**: monitor section. only for fluent-plugin-mqtt-io >= 0.3.0 84 | - **recv_time**: Add receive time to message in millisecond (ms) as integer for debug and performance/delay analysis (default: false). only for fluent-plugin-mqtt-io >= 0.3.0 85 | - **recv_time_key**: An attribute of recv_time (default: "recv_time"). only for fluent-plugin-mqtt-io >= 0.3.0 86 | - **time_type**: Type of time format (string (default), unixtime, float) only for fluent-plugin-mqtt-io >= 0.3.0 87 | - **time_format**: Time format e.g. %FT%T.%N%:z (refer strftime) only for fluent-plugin-mqtt-io >= 0.3.0 88 | - **initial_interval**: An initial value of retry interval (s) (default 1) 89 | - **retry_inc_ratio**: An increase ratio of retry interval per connection failure (default 2 (double)). It may be better to set the value to 1 in a mobile environment for eager reconnection. 90 | - **max_retry_interval**: Maximum value of retry interval (default 300) only for fluent-plugin-mqtt-io >= 0.3.0 91 | 92 | Input Plugin supports @label directive. 93 | 94 | ### Output Plugin (Fluent::MqttOutput, Fluent::MqttBufferedOutput) 95 | 96 | Output Plugin can be used via match directive. 97 | 98 | For fluent-plugin-mqtt-io <= 0.2.3 99 | 100 | ``` 101 | 102 | 103 | type mqtt 104 | host 127.0.0.1 105 | port 1883 106 | 107 | 108 | ``` 109 | 110 | For fluent-plugin-mqtt-io >= v0.3.0 111 | 112 | ``` 113 | 114 | 115 | @type mqtt 116 | host 127.0.0.1 117 | port 1883 118 | 119 | @type json 120 | add_newline false 121 | 122 | 123 | 124 | ``` 125 | 126 | The options are basically the same as Input Plugin except for "format" and "bulk_trans" (only for Input). Additional options for Output Plugin are the following. 127 | 128 | - **time_key**: An attribute name used for timestamp field genarated from fluentd time field. Default is nil (omitted). 129 | If this option is omitted, timestamp field is not appended to the output record. 130 | - **time_format**: Output format of timestamp field. Default is ISO8601. You can specify your own format by using TimeParser. 131 | - **topic_rewrite_pattern**: Regexp pattern to extract replacement words from received topic or tag name 132 | - **topic_rewrite_replacement**: Topic name used for the publish using extracted pattern 133 | - **send_time**: Add send time to message in millisecond (ms) as integer for debug and performance/delay analysis. only for fluent-plugin-mqtt-io <= 0.2.3 134 | - **send_time_key**: An attribute of send_time. only for fluent-plugin-mqtt-io <= 0.2.3 135 | - **monitor**: monitor section. only for fluent-plugin-mqtt-io >= 0.3.0 136 | - **send_time**: Add send time to message in millisecond (ms) as integer for debug and performance/delay analysis. only for fluent-plugin-mqtt-io >= 0.3.0 137 | - **send_time_key**: An attribute of send_time. only for fluent-plugin-mqtt-io >= 0.3.0 138 | - **time_type**: Type of time format (string (default), unixtime, float) only for fluent-plugin-mqtt-io >= 0.3.0 139 | - **time_format**: Time format e.g. %FT%T.%N%:z (refer strftime) only for fluent-plugin-mqtt-io >= 0.3.0 140 | 141 | If you use different source, e.g. the other MQTT broker, log file and so on, there is no need to specifie topic rewriting. Skip the following descriptions. 142 | 143 | The topic name or tag name, e.g. "topic", received from an event can not be published without modification because if MQTT input plugin connecting to the identical MQTT broker is used as a source, the same message will become an input repeatedly. In order to support data conversion in single MQTT domain, simple topic rewriting should be supported. Since topic is rewritten using #gsub method, 'pattern' and 'replacement' are the same as #gsub arguments. 144 | 145 | For fluent-plugin-mqtt-io <= v0.2.3 146 | 147 | ``` 148 | 149 | 150 | type mqtt 151 | host 127.0.0.1 152 | port 1883 153 | topic_rewrite_pattern '^([\w\/]+)$' 154 | topic_rewrite_replacement '\1/rewritten' 155 | 156 | 157 | ``` 158 | 159 | For fluent-plugin-mqtt-io >= v0.3.0 160 | 161 | ``` 162 | 163 | 164 | @type mqtt 165 | host 127.0.0.1 166 | port 1883 167 | 168 | @type json 169 | add_newline false 170 | 171 | topic_rewrite_pattern '^([\w\/]+)$' 172 | topic_rewrite_replacement '\1/rewritten' 173 | 174 | 175 | ``` 176 | 177 | You can also use mqtt_buf type which is implemented as Fluent::MqttBufferedOutput. 178 | 179 | ``` 180 | 181 | 182 | type mqtt_buf 183 | host 127.0.0.1 184 | port 1883 185 | topic_rewrite_pattern '^([\w\/]+)$' 186 | topic_rewrite_replacement '\1/rewritten' 187 | # You can specify Buffer Plugin options 188 | buffer_type memory 189 | flush_interval 1s 190 | 191 | 192 | ``` 193 | 194 | For fluent-plugin-mqtt-io >= v0.3.0 195 | 196 | ``` 197 | 198 | 199 | @type mqtt 200 | host 127.0.0.1 201 | port 1883 202 | 203 | @type json 204 | 205 | topic_rewrite_pattern '^([\w\/]+)$' 206 | topic_rewrite_replacement '\1/rewritten' 207 | # You can specify Buffer Plugin options 208 | 209 | buffer_type memory 210 | flush_interval 1s 211 | 212 | 213 | 214 | ``` 215 | 216 | When a plugin implemented as Fluent::BufferedOutput, fluentd stores the received messages into buffers (Fluent::MemoryBuffer is used as default) separated by tag names. Writer Threads emit those stored messages periodically in the specified interval. In MqttBufferedOutput, the stored messages are concatenated with 'bulk_trans_sep' (default: "\t"). This function reduces the number of messages sent via MQTT if data producing interval of sensors are smaller than publish interval (flush_interval). The concatinated messages can be properly handled by Fluent::MqttInput plugin by specifying 'bulk_trans' option. 217 | 218 | ## Use case examples 219 | 220 | ### Sensor data collection (not for Libelium now) 221 | 222 | *additional description (2015-12-25)*: Thanks to the updates by Libelium, Meshlium farmware now supports flexible output format configuration including JSON. As a result, data conversion by XmlParser is not required for this use case. The following description is kept just for sharing know-how of data conversion in fluentd. 223 | 224 | There are many kinds of commercial sensor products on the market, e.g. [Libelium](http://www.libelium.com/). Major sensor products support MQTT to upload sensor data. The following example shows how to store uploaded Libelium sensor data into ElasticSearch. 225 | 226 | ![Libelium sensor data collection](https://github.com/toyokazu/fluent-plugin-mqtt-io/blob/master/images/libelium_sensor_data_collection.png "Libelium sensor data") 227 | 228 | As described in the figure, fluent-plugin-mqtt-io, fluent-plugin-xml-parser and fluent-plugin-elasticsearch are used. The following is an example configuration. 229 | 230 | ``` 231 | 232 | type mqtt 233 | host 192.168.1.100 234 | port 1883 235 | topic 'Libelium/+/+' 236 | format xml 237 | time_xpath '["cap:alert/cap:info/cap:onset", "text"]' 238 | time_key '@timestamp' 239 | attr_xpaths '[["cap:alert/cap:info/cap:parameter/cap:valueName", "text"]]' 240 | value_xpaths '[["cap:alert/cap:info/cap:parameter/cap:value", "text"]]' 241 | @label @MQTT_OUT 242 | 243 | 244 | 258 | 259 | ``` 260 | 261 | The following mapping is assumed to be created at ElasticSearch. 262 | 263 | ``` 264 | 265 | curl -XPUT 'http://localhost:9200/libelium/_mapping/smartcity' -d ' 266 | {"smartcity": 267 | {"properties": 268 | { 269 | "sensor_id":{"type": "string"}, 270 | "DUST":{"type": "float"}, 271 | "MCP":{"type": "float"}, 272 | "HUMA":{"type": "float"}, 273 | "TCA":{"type": "float"}, 274 | "BAT":{"type": "float"}, 275 | "@timestamp":{"type":"date","format":"dateOptionalTime"} 276 | } 277 | } 278 | }' 279 | 280 | ``` 281 | 282 | ### MQTT message conversion 283 | 284 | Sometimes, MQTT message conversion must be done in the network because the processing entities does not have the conversion function. In that case, the configuration similar to the above example can be used. The difference resides output configuration. In this example, since the same MQTT broker is used to upload converted data, topic rewriting function is used for separating messages before and after conversion. 285 | 286 | ![MQTT message conversion](https://github.com/toyokazu/fluent-plugin-mqtt-io/blob/master/images/mqtt_message_conversion.png "MQTT message conversion") 287 | 288 | ``` 289 | 290 | type mqtt 291 | host 192.168.1.100 292 | port 1883 293 | topic 'Libelium/+/+' 294 | format xml 295 | time_xpath '["cap:alert/cap:info/cap:onset", "text"]' 296 | time_key '@timestamp' 297 | attr_xpaths '[["cap:alert/cap:info/cap:parameter/cap:valueName", "text"]]' 298 | value_xpaths '[["cap:alert/cap:info/cap:parameter/cap:value", "text"]]' 299 | @label @MQTT_OUT 300 | 301 | 302 | 311 | 312 | ``` 313 | 314 | 315 | ### Sensor data uploads from tiny computers, e.g. Raspberry Pi, Edison, etc 316 | 317 | MQTT output plugin can be used as the following. If you have tiny computers like Raspberry Pi equipped with sensors and their data are outputted as files, you can use fluent-plugin-mqtt-io for uploading those data. 318 | 319 | ![Sensor data uploads from tiny computers](https://github.com/toyokazu/fluent-plugin-mqtt-io/blob/master/images/sensor_data_uploads_from_tiny_computers.png "Sensor data uploads from tiny computers") 320 | 321 | 322 | ## Contributing 323 | 324 | 1. Fork it ( http://github.com/toyokazu/fluent-plugin-mqtt-io/fork ) 325 | 2. Create your feature branch (`git checkout -b my-new-feature`) 326 | 3. Commit your changes (`git commit -am 'Add some feature'`) 327 | 4. Push to the branch (`git push origin my-new-feature`) 328 | 5. Create new Pull Request 329 | 330 | 331 | ## License 332 | 333 | The gem is available as open source under the terms of the [Apache License Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). 334 | --------------------------------------------------------------------------------