├── .gitignore ├── Rakefile ├── Gemfile ├── spec ├── spec_helper.rb └── codecs │ └── fluent_spec.rb ├── NOTICE.TXT ├── .github ├── PULL_REQUEST_TEMPLATE.md ├── ISSUE_TEMPLATE.md └── CONTRIBUTING.md ├── .travis.yml ├── CONTRIBUTORS ├── LICENSE ├── CHANGELOG.md ├── logstash-codec-fluent.gemspec ├── README.md └── lib └── logstash └── codecs └── fluent.rb /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | Gemfile.lock 3 | .bundle 4 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | @files=[] 2 | 3 | task :default do 4 | system("rake -T") 5 | end 6 | 7 | require "logstash/devutils/rake" 8 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Specify your gem's dependencies in logstash-mass_effect.gemspec 4 | gemspec 5 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | require "logstash/devutils/rspec/spec_helper" 3 | require "logstash/codecs/fluent" 4 | -------------------------------------------------------------------------------- /NOTICE.TXT: -------------------------------------------------------------------------------- 1 | Elasticsearch 2 | Copyright 2012-2015 Elasticsearch 3 | 4 | This product includes software developed by The Apache Software 5 | Foundation (http://www.apache.org/). -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/ 2 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | --- 2 | sudo: false 3 | language: ruby 4 | cache: bundler 5 | rvm: 6 | - jruby-1.7.25 7 | script: 8 | - bundle exec rspec spec 9 | jdk: oraclejdk8 10 | before_install: [] 11 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Please post all product and debugging questions on our [forum](https://discuss.elastic.co/c/logstash). Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here. 2 | 3 | For all general issues, please provide the following details for fast resolution: 4 | 5 | - Version: 6 | - Operating System: 7 | - Config File (if you have sensitive info, please remove it): 8 | - Sample Data: 9 | - Steps to Reproduce: 10 | -------------------------------------------------------------------------------- /CONTRIBUTORS: -------------------------------------------------------------------------------- 1 | The following is a list of people who have contributed ideas, code, bug 2 | reports, or in general have helped logstash along its way. 3 | 4 | Contributors: 5 | * Colin Surprenant (colinsurprenant) 6 | * Jordan Sissel (jordansissel) 7 | * João Duarte (jsvd) 8 | * Pier-Hugues Pellerin (ph) 9 | * Richard Pijnenburg (electrical) 10 | * Suyog Rao (suyograo) 11 | 12 | Note: If you've sent us patches, bug reports, or otherwise contributed to 13 | Logstash, and you aren't on the list above and want to be, please let us know 14 | and we'll make sure you're here. Contributions from folks like you are what make 15 | open source awesome. 16 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012–2016 Elasticsearch 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. 14 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## 3.0.2 2 | - Relax constraint on logstash-core-plugin-api to >= 1.60 <= 2.99 3 | 4 | ## 3.0.1 5 | - Republish all the gems under jruby. 6 | ## 3.0.0 7 | - Update the plugin to the version 2.0 of the plugin api, this change is required for Logstash 5.0 compatibility. See https://github.com/elastic/logstash/issues/5141 8 | # 2.0.4 9 | - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash 10 | # 2.0.3 11 | - New dependency requirements for logstash-core for the 5.0 release 12 | ## 2.0.0 13 | - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 14 | instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895 15 | - Dependency on logstash-core update to 2.0 16 | 17 | -------------------------------------------------------------------------------- /logstash-codec-fluent.gemspec: -------------------------------------------------------------------------------- 1 | Gem::Specification.new do |s| 2 | 3 | s.name = 'logstash-codec-fluent' 4 | s.version = '3.1.1' 5 | s.licenses = ['Apache License (2.0)'] 6 | s.summary = "This codec handles fluentd's msgpack schema." 7 | s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program" 8 | s.authors = ["Elastic"] 9 | s.email = 'info@elastic.co' 10 | s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html" 11 | s.require_paths = ["lib"] 12 | 13 | # Files 14 | s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT'] 15 | 16 | # Tests 17 | s.test_files = s.files.grep(%r{^(test|spec|features)/}) 18 | 19 | # Special flag to let us know this is actually a logstash plugin 20 | s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" } 21 | 22 | # Gem dependencies 23 | s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99" 24 | 25 | if RUBY_PLATFORM == 'java' 26 | s.platform = RUBY_PLATFORM 27 | s.add_runtime_dependency 'msgpack-jruby' 28 | else 29 | s.add_runtime_dependency 'msgpack' 30 | end 31 | 32 | s.add_development_dependency 'logstash-devutils', ">= 1.0.0" 33 | end 34 | 35 | -------------------------------------------------------------------------------- /spec/codecs/fluent_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | require_relative "../spec_helper" 3 | require "logstash/plugin" 4 | require "logstash/event" 5 | 6 | describe LogStash::Codecs::Fluent do 7 | 8 | let(:properties) { {:name => "foo" } } 9 | let(:event) { LogStash::Event.new(properties) } 10 | 11 | it "should register without errors" do 12 | plugin = LogStash::Plugin.lookup("codec", "fluent").new 13 | expect { plugin.register }.to_not raise_error 14 | end 15 | 16 | describe "event encoding" do 17 | 18 | it "should encode as message pack format" do 19 | subject.on_event do |event, data| 20 | fields = MessagePack.unpack(data) 21 | expect(fields[0]).to eq("log") 22 | expect(fields[2]["name"]).to eq("foo") 23 | end 24 | subject.encode(event) 25 | end 26 | 27 | end 28 | 29 | describe "event decoding" do 30 | 31 | let(:tag) { "mytag" } 32 | let(:epochtime) { event.timestamp.to_i } 33 | let(:data) { LogStash::Util.normalize(event.to_hash) } 34 | let(:message) do 35 | MessagePack.pack([tag, epochtime, data.merge(LogStash::Event::TIMESTAMP => event.timestamp.to_iso8601)]) 36 | end 37 | 38 | it "should decode without errors" do 39 | subject.decode(message) do |event| 40 | expect(event.get("name")).to eq("foo") 41 | end 42 | end 43 | 44 | end 45 | 46 | describe "event decoding (buckets of events)" do 47 | 48 | let(:tag) { "mytag" } 49 | let(:epochtime) { event.timestamp.to_i } 50 | let(:data) { LogStash::Util.normalize(event.to_hash) } 51 | let(:message) do 52 | MessagePack.pack([tag, 53 | [ 54 | [epochtime, data.merge(LogStash::Event::TIMESTAMP => event.timestamp.to_iso8601)], 55 | [epochtime, data.merge(LogStash::Event::TIMESTAMP => event.timestamp.to_iso8601)], 56 | [epochtime, data.merge(LogStash::Event::TIMESTAMP => event.timestamp.to_iso8601)] 57 | ] 58 | ]) 59 | end 60 | 61 | it "should decode without errors" do 62 | count = 0 63 | 64 | subject.decode(message) do |event| 65 | expect(event.get("name")).to eq("foo") 66 | count += 1 67 | end 68 | 69 | expect(count).to eq(3) 70 | end 71 | 72 | end 73 | 74 | describe "event decoding (broken package)" do 75 | 76 | let(:tag) { "mytag" } 77 | let(:epochtime) { event.timestamp.to_s } 78 | let(:data) { LogStash::Util.normalize(event.to_hash) } 79 | let(:message) do 80 | MessagePack.pack([tag, 81 | epochtime, data.merge(LogStash::Event::TIMESTAMP => event.timestamp.to_iso8601) 82 | ]) 83 | end 84 | 85 | it "should decode with errors" do 86 | subject.decode(message) do |event| 87 | expect(event.get("name")).not_to eq("foo") 88 | end 89 | end 90 | 91 | it "should inject a failure event" do 92 | subject.decode(message) do |event| 93 | expect(event.get("tags")).to include("_fluentparsefailure") 94 | end 95 | end 96 | 97 | end 98 | 99 | end 100 | -------------------------------------------------------------------------------- /.github/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to Logstash 2 | 3 | All contributions are welcome: ideas, patches, documentation, bug reports, 4 | complaints, etc! 5 | 6 | Programming is not a required skill, and there are many ways to help out! 7 | It is more important to us that you are able to contribute. 8 | 9 | That said, some basic guidelines, which you are free to ignore :) 10 | 11 | ## Want to learn? 12 | 13 | Want to lurk about and see what others are doing with Logstash? 14 | 15 | * The irc channel (#logstash on irc.freenode.org) is a good place for this 16 | * The [forum](https://discuss.elastic.co/c/logstash) is also 17 | great for learning from others. 18 | 19 | ## Got Questions? 20 | 21 | Have a problem you want Logstash to solve for you? 22 | 23 | * You can ask a question in the [forum](https://discuss.elastic.co/c/logstash) 24 | * Alternately, you are welcome to join the IRC channel #logstash on 25 | irc.freenode.org and ask for help there! 26 | 27 | ## Have an Idea or Feature Request? 28 | 29 | * File a ticket on [GitHub](https://github.com/elastic/logstash/issues). Please remember that GitHub is used only for issues and feature requests. If you have a general question, the [forum](https://discuss.elastic.co/c/logstash) or IRC would be the best place to ask. 30 | 31 | ## Something Not Working? Found a Bug? 32 | 33 | If you think you found a bug, it probably is a bug. 34 | 35 | * If it is a general Logstash or a pipeline issue, file it in [Logstash GitHub](https://github.com/elasticsearch/logstash/issues) 36 | * If it is specific to a plugin, please file it in the respective repository under [logstash-plugins](https://github.com/logstash-plugins) 37 | * or ask the [forum](https://discuss.elastic.co/c/logstash). 38 | 39 | # Contributing Documentation and Code Changes 40 | 41 | If you have a bugfix or new feature that you would like to contribute to 42 | logstash, and you think it will take more than a few minutes to produce the fix 43 | (ie; write code), it is worth discussing the change with the Logstash users and developers first! You can reach us via [GitHub](https://github.com/elastic/logstash/issues), the [forum](https://discuss.elastic.co/c/logstash), or via IRC (#logstash on freenode irc) 44 | Please note that Pull Requests without tests will not be merged. If you would like to contribute but do not have experience with writing tests, please ping us on IRC/forum or create a PR and ask our help. 45 | 46 | ## Contributing to plugins 47 | 48 | Check our [documentation](https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html) on how to contribute to plugins or write your own! It is super easy! 49 | 50 | ## Contribution Steps 51 | 52 | 1. Test your changes! [Run](https://github.com/elastic/logstash#testing) the test suite 53 | 2. Please make sure you have signed our [Contributor License 54 | Agreement](https://www.elastic.co/contributor-agreement/). We are not 55 | asking you to assign copyright to us, but to give us the right to distribute 56 | your code without restriction. We ask this of all contributors in order to 57 | assure our users of the origin and continuing existence of the code. You 58 | only need to sign the CLA once. 59 | 3. Send a pull request! Push your changes to your fork of the repository and 60 | [submit a pull 61 | request](https://help.github.com/articles/using-pull-requests). In the pull 62 | request, describe what your changes do and mention any bugs/issues related 63 | to the pull request. 64 | 65 | 66 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Logstash Plugin 2 | 3 | [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-codec-fluent.svg)](https://travis-ci.org/logstash-plugins/logstash-codec-fluent) 4 | 5 | This is a plugin for [Logstash](https://github.com/elastic/logstash). 6 | 7 | It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way. 8 | 9 | ## Documentation 10 | 11 | Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/). 12 | 13 | - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive 14 | - For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide 15 | 16 | ## Need Help? 17 | 18 | Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum. 19 | 20 | ## Developing 21 | 22 | ### 1. Plugin Developement and Testing 23 | 24 | #### Code 25 | - To get started, you'll need JRuby with the Bundler gem installed. 26 | 27 | - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example). 28 | 29 | - Install dependencies 30 | ```sh 31 | bundle install 32 | ``` 33 | 34 | #### Test 35 | 36 | - Update your dependencies 37 | 38 | ```sh 39 | bundle install 40 | ``` 41 | 42 | - Run tests 43 | 44 | ```sh 45 | bundle exec rspec 46 | ``` 47 | 48 | ### 2. Running your unpublished Plugin in Logstash 49 | 50 | #### 2.1 Run in a local Logstash clone 51 | 52 | - Edit Logstash `Gemfile` and add the local plugin path, for example: 53 | ```ruby 54 | gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome" 55 | ``` 56 | - Install plugin 57 | ```sh 58 | # Logstash 2.3 and higher 59 | bin/logstash-plugin install --no-verify 60 | 61 | # Prior to Logstash 2.3 62 | bin/plugin install --no-verify 63 | 64 | ``` 65 | - Run Logstash with your plugin 66 | ```sh 67 | bin/logstash -e 'filter {awesome {}}' 68 | ``` 69 | At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash. 70 | 71 | #### 2.2 Run in an installed Logstash 72 | 73 | You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using: 74 | 75 | - Build your plugin gem 76 | ```sh 77 | gem build logstash-filter-awesome.gemspec 78 | ``` 79 | - Install the plugin from the Logstash home 80 | ```sh 81 | # Logstash 2.3 and higher 82 | bin/logstash-plugin install --no-verify 83 | 84 | # Prior to Logstash 2.3 85 | bin/plugin install --no-verify 86 | 87 | ``` 88 | - Start Logstash and proceed to test the plugin 89 | 90 | ## Contributing 91 | 92 | All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin. 93 | 94 | Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here. 95 | 96 | It is more important to the community that you are able to contribute. 97 | 98 | For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file. -------------------------------------------------------------------------------- /lib/logstash/codecs/fluent.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | require "logstash/codecs/base" 3 | require "logstash/util/charset" 4 | require "logstash/timestamp" 5 | require "logstash/util" 6 | 7 | # This codec handles fluentd's msgpack schema. 8 | # 9 | # For example, you can receive logs from `fluent-logger-ruby` with: 10 | # [source,ruby] 11 | # input { 12 | # tcp { 13 | # codec => fluent 14 | # port => 4000 15 | # } 16 | # } 17 | # 18 | # And from your ruby code in your own application: 19 | # [source,ruby] 20 | # logger = Fluent::Logger::FluentLogger.new(nil, :host => "example.log", :port => 4000) 21 | # logger.post("some_tag", { "your" => "data", "here" => "yay!" }) 22 | # 23 | # Notes: 24 | # 25 | # * the fluent uses a second-precision time for events, so you will never see 26 | # subsecond precision on events processed by this codec. 27 | # 28 | class LogStash::Codecs::Fluent < LogStash::Codecs::Base 29 | config_name "fluent" 30 | 31 | def register 32 | require "msgpack" 33 | @decoder = MessagePack::Unpacker.new 34 | end 35 | 36 | def decode(data, &block) 37 | @decoder.feed_each(data) do |item| 38 | decode_event(item, &block) 39 | end 40 | end # def decode 41 | 42 | def encode(event) 43 | tag = event.get("tags") || "log" 44 | epochtime = event.timestamp.to_i 45 | 46 | # use normalize to make sure returned Hash is pure Ruby for 47 | # MessagePack#pack which relies on pure Ruby object recognition 48 | data = LogStash::Util.normalize(event.to_hash) 49 | # timestamp is serialized as a iso8601 string 50 | # merge to avoid modifying data which could have side effects if multiple outputs 51 | @on_event.call(event, MessagePack.pack([tag, epochtime, data.merge(LogStash::Event::TIMESTAMP => event.timestamp.to_iso8601)])) 52 | end # def encode 53 | 54 | private 55 | 56 | def decode_event(data, &block) 57 | tag = data[0] 58 | entries = data[1] 59 | 60 | case entries 61 | when String 62 | # PackedForward 63 | option = data[2] 64 | compressed = (option && option['compressed'] == 'gzip') 65 | if compressed 66 | raise(LogStash::Error, "PackedForward with compression is not supported") 67 | end 68 | 69 | entries_decoder = MessagePack::Unpacker.new 70 | entries_decoder.feed_each(entries) do |entry| 71 | epochtime = entry[0] 72 | map = entry[1] 73 | event = LogStash::Event.new(map.merge( 74 | LogStash::Event::TIMESTAMP => LogStash::Timestamp.at(epochtime), 75 | "tags" => [ tag ] 76 | )) 77 | yield event 78 | end 79 | when Array 80 | # Forward 81 | entries.each do |entry| 82 | epochtime = entry[0] 83 | map = entry[1] 84 | event = LogStash::Event.new(map.merge( 85 | LogStash::Event::TIMESTAMP => LogStash::Timestamp.at(epochtime), 86 | "tags" => [ tag ] 87 | )) 88 | yield event 89 | end 90 | when Fixnum 91 | # Message 92 | epochtime = entries 93 | map = data[2] 94 | event = LogStash::Event.new(map.merge( 95 | LogStash::Event::TIMESTAMP => LogStash::Timestamp.at(epochtime), 96 | "tags" => [ tag ] 97 | )) 98 | yield event 99 | else 100 | raise(LogStash::Error, "Unknown event type") 101 | end 102 | rescue StandardError => e 103 | @logger.error("Fluent parse error, original data now in message field", :error => e, :data => data) 104 | yield LogStash::Event.new("message" => data, "tags" => [ "_fluentparsefailure" ]) 105 | end 106 | 107 | end # class LogStash::Codecs::Fluent 108 | --------------------------------------------------------------------------------