├── .gitignore ├── .travis.yml ├── Appraisals ├── Gemfile ├── LICENSE ├── README.md ├── Rakefile ├── fluent-plugin-rewrite.gemspec ├── gemfiles └── fluentd_0.14.gemfile ├── lib └── fluent │ └── plugin │ ├── filter_rewrite.rb │ ├── out_rewrite.rb │ └── rewrite_rule.rb └── test ├── plugin ├── test_filter_rewrite.rb └── test_out_rewrite.rb └── test_helper.rb /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | *.rbc 3 | .bundle 4 | .config 5 | .yardoc 6 | Gemfile.lock 7 | InstalledFiles 8 | _yardoc 9 | coverage 10 | doc/ 11 | lib/bundler/man 12 | pkg 13 | rdoc 14 | spec/reports 15 | test/tmp 16 | test/version_tmp 17 | tmp 18 | vendor/bundler 19 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: ruby 2 | gemfile: 3 | - gemfiles/fluentd_0.14.gemfile 4 | rvm: 5 | - 2.4.2 6 | - 2.3.5 7 | - 2.2 8 | - 2.1 9 | script: 10 | - bundle exec rake test 11 | -------------------------------------------------------------------------------- /Appraisals: -------------------------------------------------------------------------------- 1 | appraise "fluentd_0.14" do 2 | gem "fluentd", "~> 0.14.0" 3 | end 4 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Specify your gem's dependencies in fluent-plugin-rewrite.gemspec 4 | gemspec 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012 Kentaro Kuribayashi 2 | Apache License, Version 2.0 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # fluent-plugin-rewrite, a plugin for [Fluentd](http://fluentd.org) 2 | 3 | ## Requirements 4 | 5 | | fluent-plugin-rewrite | fluentd | ruby | 6 | |-----------------------|------------|--------| 7 | | >= 0.1.0 | >= v0.14.8 | >= 2.1 | 8 | | < 0.1.0 | >= v0.12.0 | >= 1.9 | 9 | 10 | ## Component 11 | 12 | ### RewriteOutput 13 | 14 | Output plugin to rewrite messages' tags/values along with pattern 15 | matching and re-emit them. 16 | 17 | ## Synopsis 18 | 19 | ``` 20 | 21 | @type rewrite 22 | 23 | remove_prefix apache.log 24 | add_prefix filtered 25 | 26 | 27 | key path 28 | pattern \\?.+$ 29 | replace 30 | 31 | 32 | key path 33 | pattern (/[^/]+)\\?([^=]+)=(\\d) 34 | replace \\1/\\2/\\3 35 | 36 | 37 | key status 38 | pattern ^500$ 39 | ignore true 40 | 41 | 42 | key path 43 | pattern ^\/(users|entries) 44 | append_to_tag true 45 | fallback others 46 | 47 | 48 | key is_loggged_in 49 | pattern 1 50 | append_to_tag true 51 | tag user 52 | 53 | 54 | ``` 55 | 56 | ## Configuration 57 | 58 | ### remove_prefix / add_prefix 59 | 60 | ``` 61 | remove_prefix apache.log 62 | add_prefix filtered 63 | ``` 64 | 65 | - remove_prefix: removes the string from a prefix of tag. 66 | - add_prefix: prepend the string to a tag. 67 | 68 | ### rule: replace 69 | 70 | For example, if you want to filter out query string form URL string: 71 | 72 | ``` 73 | 74 | key path 75 | pattern \\?.+$ 76 | replace 77 | 78 | ``` 79 | 80 | It executes pattern matching against a value related with `key` and replaces it with empty string if it matches. 81 | 82 | ``` 83 | /foo?bar=baz -> /foo 84 | ``` 85 | 86 | This time, if you want to rewrite path string along with some pattern: 87 | 88 | ``` 89 | 90 | key path 91 | pattern (/[^/]+)\\?([^=]+)=(\\d) 92 | replace \\1/\\2/\\3 93 | 94 | ``` 95 | 96 | It executes pattern matching against a value related with `key` and replaces it with `replace` if it matches. 97 | Need `()` in `pattern` if you want to refer values in `replace`. This is Ruby's [capturing feature](http://www.ruby-doc.org/core-2.1.1/doc/regexp_rdoc.html#label-Capturing). 98 | 99 | ``` 100 | /foo?bar=1 -> /foo/bar/1 101 | ``` 102 | 103 | ### rule: ignore 104 | 105 | For example, if you want to skip a message which matches some pattern: 106 | 107 | ``` 108 | 109 | key status 110 | pattern ^500$ 111 | ignore true 112 | 113 | ``` 114 | 115 | It executes pattern matching against a value related with `key` and skip emitting the message if it matches. 116 | 117 | ### rule: append_to_tag 118 | 119 | ``` 120 | 121 | key path 122 | pattern ^\/(users|entries) 123 | append_to_tag true 124 | 125 | ``` 126 | 127 | It executes pattern matching against a value related with `key` and append mathed strings to message tag. 128 | This also need `()` to append values to tag. See [rule: replace](#rule-replace) section. For example: 129 | 130 | ``` 131 | apache.log { "path" : "/users/antipop" } 132 | ``` 133 | 134 | the messabe above will be re-emmited as the message below: 135 | 136 | ``` 137 | apache.log.users { "path" : "/users/antipop" } 138 | ``` 139 | 140 | If you set `fallback` option like below: 141 | 142 | ``` 143 | 144 | key path 145 | pattern ^\/(users|entries) 146 | append_to_tag true 147 | fallback others 148 | 149 | ``` 150 | 151 | the value of `fallback` option will be appended to message tag. 152 | 153 | ``` 154 | apache.log { "path" : "/foo/bar" } 155 | ``` 156 | 157 | This time, the messabe above will be re-emmited as the message below: 158 | 159 | ``` 160 | apache.log.others { "path" : "/foo/bar" } 161 | ``` 162 | 163 | If `tag` option set, 164 | 165 | ``` 166 | 167 | key is_loggged_in 168 | pattern 1 169 | append_to_tag true 170 | tag user 171 | 172 | ``` 173 | 174 | the value designated by `tag` will be appended to the original tag, that is: 175 | 176 | 177 | ``` 178 | test { "is_logged_in" => "1" } 179 | ``` 180 | 181 | will be 182 | 183 | ``` 184 | test.user { "is_logged_in" => "1" } 185 | ``` 186 | 187 | ### rule: last 188 | 189 | If you set `last` option to true, rewriting chain stops applying rule where the pattern matches first. 190 | 191 | ``` 192 | 193 | key path 194 | pattern ^/foo$ 195 | replace /bar 196 | last true 197 | 198 | 199 | key path 200 | pattern ^/bar$ 201 | replace /baz 202 | 203 | ``` 204 | 205 | This rules will be applied like below: 206 | 207 | ``` 208 | { "path" => "/foo" } 209 | ``` 210 | 211 | will be replaced with 212 | 213 | ``` 214 | { "path" => "/bar" } 215 | ``` 216 | 217 | and the chain stops here. Therefore, the second rule is never 218 | applied. 219 | 220 | ``` 221 | { "path" => "/bar" } 222 | ``` 223 | 224 | will be replaced by the second rule as usual. 225 | 226 | ``` 227 | { "path" => "/baz" } 228 | ``` 229 | 230 | ### RewriteFilter 231 | 232 | Filter plugin to modify messages' values along with pattern 233 | matching and filter them. 234 | 235 | Note that filter version of rewrite plugin does not have append/add tags functionality. 236 | 237 | Thus, this filter version does not able to specify `append_to_tag`, `tag`, and `fallback` rules. 238 | 239 | ## Synopsis 240 | 241 | ``` 242 | 243 | @type rewrite 244 | 245 | 246 | key path 247 | pattern \\?.+$ 248 | replace 249 | 250 | 251 | key path 252 | pattern (/[^/]+)\\?([^=]+)=(\\d) 253 | replace \\1/\\2/\\3 254 | 255 | 256 | key status 257 | pattern ^500$ 258 | ignore true 259 | 260 | 261 | ``` 262 | 263 | ## Configuration 264 | 265 | Note: This filter version of rewrite plugin does not have `remove_prefix` and `add_prefix` configuration. 266 | 267 | ### rule: replace 268 | 269 | Same as OutputRewrite section's [rule: replace](#rule-replace). 270 | 271 | ### rule: ignore 272 | 273 | Same as OutputRewrite section's [rule: ignore](#rule-ignore). 274 | 275 | ### rule: last 276 | 277 | Same as OutputRewrite section's [rule: last](#rule-last). 278 | 279 | ## Installation 280 | 281 | Add this line to your application's Gemfile: 282 | 283 | gem 'fluent-plugin-rewrite' 284 | 285 | And then execute: 286 | 287 | $ bundle 288 | 289 | Or install it yourself as: 290 | 291 | $ gem install fluent-plugin-rewrite 292 | 293 | ## Contributing 294 | 295 | 1. Fork it 296 | 2. Create your feature branch (`git checkout -b my-new-feature`) 297 | 3. Commit your changes (`git commit -am 'Added some feature'`) 298 | 4. Push to the branch (`git push origin my-new-feature`) 299 | 5. Create new Pull Request 300 | 301 | ## Copyright 302 | 303 | ### Copyright 304 | 305 | Copyright (c) 2012- Kentaro Kuribayashi (@kentaro) 306 | 307 | ### License 308 | 309 | Apache License, Version 2.0 310 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env rake 2 | require "bundler/gem_tasks" 3 | require 'rake/testtask' 4 | 5 | Rake::TestTask.new(:test) do |test| 6 | test.libs << 'lib' << 'test' 7 | test.pattern = 'test/**/test_*.rb' 8 | test.verbose = true 9 | end 10 | -------------------------------------------------------------------------------- /fluent-plugin-rewrite.gemspec: -------------------------------------------------------------------------------- 1 | Gem::Specification.new do |gem| 2 | gem.name = "fluent-plugin-rewrite" 3 | gem.version = '0.1.1' 4 | gem.authors = ["Kentaro Kuribayashi"] 5 | gem.email = ["kentarok@gmail.com"] 6 | gem.homepage = "http://github.com/kentaro/fluent-plugin-rewrite" 7 | gem.description = %q{Fluentd plugin to rewrite tags/values along with pattern matching and re-emit them.} 8 | gem.summary = %q{Fluentd plugin to rewrite tags/values along with pattern matching and re-emit them.} 9 | gem.license = 'MIT' 10 | 11 | gem.files = `git ls-files`.split($\) 12 | gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) } 13 | gem.test_files = gem.files.grep(%r{^(test|spec|features)/}) 14 | gem.require_paths = ["lib"] 15 | 16 | gem.add_development_dependency "rake" 17 | gem.add_development_dependency "test-unit", "~> 3.1" 18 | gem.add_development_dependency "appraisal" 19 | gem.add_runtime_dependency "fluentd", [">= 0.14.8", "< 2"] 20 | end 21 | -------------------------------------------------------------------------------- /gemfiles/fluentd_0.14.gemfile: -------------------------------------------------------------------------------- 1 | # This file was generated by Appraisal 2 | 3 | source "https://rubygems.org" 4 | 5 | gem "fluentd", "~> 0.14.0" 6 | 7 | gemspec :path => "../" 8 | -------------------------------------------------------------------------------- /lib/fluent/plugin/filter_rewrite.rb: -------------------------------------------------------------------------------- 1 | module Fluent::Plugin 2 | class RewriteFilter < Filter 3 | Fluent::Plugin.register_filter('rewrite', self) 4 | 5 | attr_reader :rewrite_rule 6 | 7 | def configure(conf) 8 | require 'fluent/plugin/rewrite_rule' 9 | 10 | super 11 | 12 | @rewrite_rule = Fluent::RewriteRule.new(self, conf) 13 | end 14 | 15 | def filter_stream(tag, es) 16 | new_es = Fluent::MultiEventStream.new 17 | 18 | es.each do |time, record| 19 | record = @rewrite_rule.rewrite(record) 20 | new_es.add(time, record) if record 21 | end 22 | 23 | new_es 24 | end 25 | end 26 | end 27 | -------------------------------------------------------------------------------- /lib/fluent/plugin/out_rewrite.rb: -------------------------------------------------------------------------------- 1 | require 'fluent/plugin/output' 2 | 3 | module Fluent::Plugin 4 | class RewriteOutput < Output 5 | Fluent::Plugin.register_output('rewrite', self) 6 | 7 | helpers :event_emitter 8 | 9 | config_param :remove_prefix, :string, :default => nil, 10 | deprecated: "use @label instead for event routing" 11 | config_param :add_prefix, :string, :default => nil, 12 | deprecated: "use @label instead for event routing" 13 | config_param :enable_warnings, :bool, :default => false 14 | 15 | attr_reader :rewrite_rule 16 | 17 | def configure(conf) 18 | require 'fluent/plugin/rewrite_rule' 19 | 20 | super 21 | 22 | if @remove_prefix 23 | @removed_prefix_string = @remove_prefix + '.' 24 | @removed_length = @removed_prefix_string.length 25 | end 26 | if @add_prefix 27 | @added_prefix_string = @add_prefix + '.' 28 | end 29 | 30 | @rewrite_rule = Fluent::RewriteRule.new(self, conf) 31 | end 32 | 33 | def process(tag, es) 34 | _tag = tag.clone 35 | 36 | if @remove_prefix and 37 | ((tag.start_with?(@removed_prefix_string) && tag.length > @removed_length) || tag == @remove_prefix) 38 | tag = tag[@removed_length..-1] || '' 39 | end 40 | 41 | if @add_prefix 42 | tag = tag && tag.length > 0 ? @added_prefix_string + tag : @add_prefix 43 | end 44 | 45 | es.each do |time, record| 46 | filtered_tag, record = @rewrite_rule.rewrite(tag, record) 47 | if filtered_tag && record && _tag != filtered_tag 48 | router.emit(filtered_tag, time, record) 49 | else 50 | if @enable_warnings 51 | $log.warn "Can not emit message because the tag(#{tag}) has not changed. Dropped record #{record}" 52 | end 53 | end 54 | end 55 | end 56 | end 57 | end 58 | -------------------------------------------------------------------------------- /lib/fluent/plugin/rewrite_rule.rb: -------------------------------------------------------------------------------- 1 | module Fluent 2 | class RewriteRule 3 | attr_reader :rules 4 | 5 | def initialize(plugin, conf) 6 | @plugin = plugin 7 | @rules = conf.elements.select {|element| element.name == 'rule' }.map do |element| 8 | rule = {} 9 | element.keys.each do |key| 10 | # read and throw away to supress unread configuration warning 11 | rule[key] = element[key] 12 | end 13 | rule["regex"] = Regexp.new(element["pattern"]) if element.has_key?("pattern") 14 | rule 15 | end 16 | end 17 | 18 | def rewrite(tag=nil, record) 19 | @rules.each do |rule| 20 | tag, record, last = apply_rule(rule, tag, record) 21 | 22 | break if last 23 | if @plugin.is_a?(Fluent::Plugin::Output) 24 | return if !tag && !record 25 | else 26 | return if !record 27 | end 28 | end 29 | 30 | return [record] if not @plugin.is_a?(Fluent::Plugin::Output) 31 | return [tag, record] if @plugin.is_a?(Fluent::Plugin::Output) 32 | end 33 | 34 | def apply_rule(rule, tag=nil, record) 35 | tag_prefix = tag && tag.length > 0 ? "." : "" 36 | key = rule["key"] 37 | pattern = rule["pattern"] 38 | last = nil 39 | 40 | return [tag, record] if !key || !record.has_key?(key) 41 | return [tag, record] unless pattern 42 | 43 | if matched = record[key].match(rule["regex"]) 44 | return if rule["ignore"] 45 | 46 | if rule["replace"] 47 | replace = rule["replace"] 48 | record[key] = record[key].gsub(rule["regex"], replace) 49 | end 50 | 51 | if rule["append_to_tag"] && @plugin.is_a?(Fluent::Plugin::Output) 52 | if rule["tag"] 53 | tag += (tag_prefix + rule["tag"]) 54 | else 55 | matched.captures.each do |m| 56 | tag += (tag_prefix + "#{m}") 57 | end 58 | end 59 | end 60 | 61 | if rule["last"] 62 | last = true 63 | end 64 | else 65 | if rule["append_to_tag"] && rule["fallback"] && @plugin.is_a?(Fluent::Plugin::Output) 66 | tag += (tag_prefix + rule["fallback"]) 67 | end 68 | end 69 | 70 | return [tag, record, last] 71 | end 72 | end 73 | end 74 | -------------------------------------------------------------------------------- /test/plugin/test_filter_rewrite.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | require 'fluent/test/driver/filter' 3 | 4 | class RewriteFilterTest < Test::Unit::TestCase 5 | def setup 6 | Fluent::Test.setup 7 | end 8 | 9 | def create_driver(conf) 10 | Fluent::Test::Driver::Filter.new(Fluent::Plugin::RewriteFilter).configure(conf) 11 | end 12 | 13 | def test_configure 14 | d = create_driver(%[ 15 | 16 | key foo 17 | 18 | 19 | key bar 20 | 21 | 22 | key baz 23 | 24 | ]) 25 | 26 | assert_equal 2, d.instance.rewrite_rule.rules.size 27 | end 28 | 29 | class TestRewrite < self 30 | def test_replace 31 | d = create_driver(%[ 32 | 33 | key path 34 | pattern \\?.+$ 35 | replace 36 | 37 | ]) 38 | 39 | assert_equal( 40 | [ { "path" => "/foo" } ], 41 | d.instance.rewrite_rule.rewrite({ "path" => "/foo?bar=1" }) 42 | ) 43 | end 44 | 45 | def test_replace_with_capture 46 | d = create_driver(%[ 47 | 48 | key path 49 | pattern (/[^/]+)\\?([^=]+)=(\\d) 50 | replace \\1/\\2/\\3 51 | 52 | ]) 53 | 54 | assert_equal( 55 | [ { "path" => "/foo/bar/1" } ], 56 | d.instance.rewrite_rule.rewrite({ "path" => "/foo?bar=1" }) 57 | ) 58 | end 59 | end 60 | 61 | class TestRewriteIgnore < self 62 | def test_pattern 63 | d = create_driver(%[ 64 | 65 | key status 66 | pattern ^500$ 67 | ignore true 68 | 69 | ]) 70 | 71 | assert_equal( 72 | nil, 73 | d.instance.rewrite_rule.rewrite({ "status" => "500" }) 74 | ) 75 | end 76 | 77 | def test_negate_pattern 78 | d = create_driver(%[ 79 | 80 | key status 81 | pattern ^(?!200)\\d+$ 82 | ignore true 83 | 84 | ]) 85 | 86 | assert_equal( 87 | [ { "status" => "200" } ], 88 | d.instance.rewrite_rule.rewrite({ "status" => "200" }) 89 | ) 90 | %w[301 404 500].each do |status| 91 | assert_equal( 92 | nil, 93 | d.instance.rewrite_rule.rewrite({ "status" => status }) 94 | ) 95 | end 96 | end 97 | 98 | def test_entire_ignore 99 | d = create_driver(%[ 100 | 101 | key flag 102 | pattern ^$ 103 | ignore true 104 | 105 | ]) 106 | 107 | assert_equal( 108 | nil, 109 | d.instance.rewrite_rule.rewrite({ "flag" => "" }) 110 | ) 111 | end 112 | end 113 | 114 | def test_last 115 | d = create_driver(%[ 116 | 117 | key path 118 | pattern ^/foo$ 119 | replace /bar 120 | last true 121 | 122 | 123 | key path 124 | pattern ^/bar$ 125 | replace /baz 126 | 127 | ]) 128 | 129 | assert_equal( 130 | [ { "path" => "/bar" } ], 131 | d.instance.rewrite_rule.rewrite({ "path" => "/foo" }) 132 | ) 133 | assert_equal( 134 | [ { "path" => "/baz" } ], 135 | d.instance.rewrite_rule.rewrite({ "path" => "/bar" }) 136 | ) 137 | end 138 | 139 | def test_rewrite_rules 140 | d = create_driver(%[ 141 | 142 | key path 143 | pattern \\?.+$ 144 | replace 145 | 146 | 147 | key status 148 | pattern ^500$ 149 | ignore true 150 | 151 | ]) 152 | 153 | assert_equal( 154 | [ { "path" => "/foo" } ], 155 | d.instance.rewrite_rule.rewrite({ "path" => "/foo?bar=1" }) 156 | ) 157 | assert_equal( 158 | [ { "path" => "/users/antipop" } ], 159 | d.instance.rewrite_rule.rewrite({ "path" => "/users/antipop?hoge=1" }) 160 | ) 161 | assert_equal( 162 | nil, 163 | d.instance.rewrite_rule.rewrite({ "path" => "/foo?bar=1", "status" => "500" }) 164 | ) 165 | end 166 | 167 | class TestFilter < self 168 | def test_with_multiple_rules 169 | d = create_driver(%[ 170 | 171 | key path 172 | pattern \\?.+$ 173 | replace 174 | 175 | 176 | key status 177 | pattern ^500$ 178 | ignore true 179 | 180 | ]) 181 | 182 | d.run(default_tag: 'test') do 183 | d.feed({ "path" => "/foo?bar=1" }) 184 | d.feed({ "path" => "/foo?bar=1", "status" => "500" }) 185 | d.feed({ "path" => "/users/antipop" }) 186 | d.feed({ "path" => "/users/kentaro" }) 187 | end 188 | filtered = d.filtered 189 | 190 | assert_equal 3, filtered.size 191 | assert_equal([{ "path" => "/foo" }], filtered[0][1]) 192 | assert_equal([{ "path" => "/users/antipop" }], filtered[1][1]) # nothing to do 193 | assert_equal([{ "path" => "/users/kentaro" }], filtered[2][1]) # nothing to do 194 | end 195 | 196 | def test_remove_query_params 197 | d = create_driver(%[ 198 | 199 | key path 200 | pattern \\?.+$ 201 | replace 202 | 203 | ]) 204 | d.run(default_tag: 'test') do 205 | d.feed({ "path" => "/foo?bar=1" }) 206 | end 207 | filtered = d.filtered 208 | 209 | assert_equal 1, filtered.size 210 | assert_equal([{ "path" => "/foo" }], filtered[0][1]) 211 | end 212 | end 213 | end 214 | -------------------------------------------------------------------------------- /test/plugin/test_out_rewrite.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | require 'fluent/test/driver/output' 3 | 4 | class RewriteOutputTest < Test::Unit::TestCase 5 | def setup 6 | Fluent::Test.setup 7 | end 8 | 9 | def create_driver(conf) 10 | Fluent::Test::Driver::Output.new(Fluent::Plugin::RewriteOutput).configure(conf) 11 | end 12 | 13 | def test_configure 14 | d = create_driver(%[ 15 | remove_prefix test 16 | add_prefix filtered 17 | 18 | 19 | key foo 20 | 21 | 22 | key bar 23 | 24 | 25 | key baz 26 | 27 | ]) 28 | 29 | assert_equal "test", d.instance.remove_prefix 30 | assert_equal "filtered", d.instance.add_prefix 31 | assert_equal 2, d.instance.rewrite_rule.rules.size 32 | end 33 | 34 | def test_rewrite_replace 35 | d1 = create_driver(%[ 36 | 37 | key path 38 | pattern \\?.+$ 39 | replace 40 | 41 | ]) 42 | 43 | assert_equal( 44 | [ "test", { "path" => "/foo" } ], 45 | d1.instance.rewrite_rule.rewrite("test", { "path" => "/foo?bar=1" }) 46 | ) 47 | 48 | d2 = create_driver(%[ 49 | 50 | key path 51 | pattern (/[^/]+)\\?([^=]+)=(\\d) 52 | replace \\1/\\2/\\3 53 | 54 | ]) 55 | 56 | assert_equal( 57 | [ "test", { "path" => "/foo/bar/1" } ], 58 | d2.instance.rewrite_rule.rewrite("test", { "path" => "/foo?bar=1" }) 59 | ) 60 | end 61 | 62 | def test_rewrite_ignore 63 | d1 = create_driver(%[ 64 | 65 | key status 66 | pattern ^500$ 67 | ignore true 68 | 69 | ]) 70 | 71 | assert_equal( 72 | nil, 73 | d1.instance.rewrite_rule.rewrite("test", { "status" => "500" }) 74 | ) 75 | 76 | d2 = create_driver(%[ 77 | 78 | key status 79 | pattern ^(?!200)\\d+$ 80 | ignore true 81 | 82 | ]) 83 | 84 | assert_equal( 85 | [ "test", { "status" => "200" } ], 86 | d2.instance.rewrite_rule.rewrite("test", { "status" => "200" }) 87 | ) 88 | %w[301 404 500].each do |status| 89 | assert_equal( 90 | nil, 91 | d2.instance.rewrite_rule.rewrite("test", { "status" => status }) 92 | ) 93 | end 94 | 95 | d3 = create_driver(%[ 96 | 97 | key flag 98 | pattern ^$ 99 | ignore true 100 | 101 | ]) 102 | 103 | assert_equal( 104 | nil, 105 | d3.instance.rewrite_rule.rewrite("test", { "flag" => "" }) 106 | ) 107 | end 108 | 109 | def test_rewrite_append_tag 110 | d1 = create_driver(%[ 111 | 112 | key path 113 | pattern ^\/(users|entries) 114 | append_to_tag true 115 | 116 | ]) 117 | 118 | assert_equal( 119 | [ "test.users", { "path" => "/users/antipop" } ], 120 | d1.instance.rewrite_rule.rewrite("test", { "path" => "/users/antipop" }) 121 | ) 122 | assert_equal( 123 | [ "test", { "path" => "/unmatched/path" } ], 124 | d1.instance.rewrite_rule.rewrite("test", { "path" => "/unmatched/path" }) 125 | ) 126 | 127 | d2 = create_driver(%[ 128 | 129 | key path 130 | pattern ^\/(users|entries) 131 | append_to_tag true 132 | fallback others 133 | 134 | ]) 135 | 136 | assert_equal( 137 | [ "test.users", { "path" => "/users/antipop" } ], 138 | d2.instance.rewrite_rule.rewrite("test", { "path" => "/users/antipop" }) 139 | ) 140 | assert_equal( 141 | [ "test.others", { "path" => "/unmatched/path" } ], 142 | d2.instance.rewrite_rule.rewrite("test", { "path" => "/unmatched/path" }) 143 | ) 144 | 145 | d3 = create_driver(%[ 146 | 147 | key is_logged_in 148 | pattern 1 149 | append_to_tag true 150 | tag user 151 | 152 | ]) 153 | 154 | assert_equal( 155 | [ "test.user", { "is_logged_in" => "1" } ], 156 | d3.instance.rewrite_rule.rewrite("test", { "is_logged_in" => "1" }) 157 | ) 158 | assert_equal( 159 | [ "test", { "is_logged_in" => "0" } ], 160 | d3.instance.rewrite_rule.rewrite("test", { "is_logged_in" => "0" }) 161 | ) 162 | 163 | d4 = create_driver(%[ 164 | 165 | key path 166 | pattern ^\/(users|entries) 167 | append_to_tag true 168 | 169 | ]) 170 | 171 | assert_equal( 172 | [ "test.users", { "path" => "/users/antipop" } ], 173 | d4.instance.rewrite_rule.rewrite("test", { "path" => "/users/antipop" }) 174 | ) 175 | 176 | d5 = create_driver(%[ 177 | 178 | key is_logged_in 179 | pattern 1 180 | append_to_tag true 181 | tag user 182 | 183 | ]) 184 | 185 | assert_equal( 186 | [ "test.user", { "is_logged_in" => "1" } ], 187 | d5.instance.rewrite_rule.rewrite("test", { "is_logged_in" => "1" }) 188 | ) 189 | end 190 | 191 | def test_last 192 | d = create_driver(%[ 193 | 194 | key path 195 | pattern ^/foo$ 196 | replace /bar 197 | last true 198 | 199 | 200 | key path 201 | pattern ^/bar$ 202 | replace /baz 203 | 204 | ]) 205 | 206 | assert_equal( 207 | [ "test", { "path" => "/bar" } ], 208 | d.instance.rewrite_rule.rewrite("test", { "path" => "/foo" }) 209 | ) 210 | assert_equal( 211 | [ "test", { "path" => "/baz" } ], 212 | d.instance.rewrite_rule.rewrite("test", { "path" => "/bar" }) 213 | ) 214 | end 215 | 216 | def test_rewrite_rules 217 | d = create_driver(%[ 218 | 219 | key path 220 | pattern \\?.+$ 221 | replace 222 | 223 | 224 | key status 225 | pattern ^500$ 226 | ignore true 227 | 228 | 229 | key path 230 | pattern ^\/(users|entries) 231 | append_to_tag true 232 | fallback others 233 | 234 | ]) 235 | 236 | assert_equal( 237 | [ "test.others", { "path" => "/foo" } ], 238 | d.instance.rewrite_rule.rewrite("test", { "path" => "/foo?bar=1" }) 239 | ) 240 | assert_equal( 241 | [ "test.users", { "path" => "/users/antipop" } ], 242 | d.instance.rewrite_rule.rewrite("test", { "path" => "/users/antipop?hoge=1" }) 243 | ) 244 | assert_equal( 245 | nil, 246 | d.instance.rewrite_rule.rewrite("test", { "path" => "/foo?bar=1", "status" => "500" }) 247 | ) 248 | end 249 | 250 | def test_emit 251 | d1 = create_driver(%[ 252 | remove_prefix test 253 | add_prefix filtered 254 | 255 | 256 | key path 257 | pattern \\?.+$ 258 | replace 259 | 260 | 261 | key status 262 | pattern ^500$ 263 | ignore true 264 | 265 | 266 | key path 267 | pattern ^\/(users|entries) 268 | append_to_tag true 269 | fallback others 270 | 271 | ]) 272 | 273 | d1.run(default_tag: 'test') do 274 | d1.feed({ "path" => "/foo?bar=1" }) 275 | d1.feed({ "path" => "/foo?bar=1", "status" => "500" }) 276 | d1.feed({ "path" => "/users/antipop" }) 277 | d1.feed({ "path" => "/users/kentaro" }) 278 | d1.feed({ "path" => "/entries/1" }) 279 | end 280 | events = d1.events 281 | 282 | assert_equal 4, events.size 283 | assert_equal('filtered.others', events[0][0]) 284 | assert_equal({ "path" => "/foo" }, events[0][2]) 285 | assert_equal('filtered.users', events[1][0]) 286 | assert_equal({ "path" => "/users/antipop" }, events[1][2]) 287 | assert_equal('filtered.users', events[2][0]) 288 | assert_equal({ "path" => "/users/kentaro" }, events[2][2]) 289 | assert_equal('filtered.entries', events[3][0]) 290 | assert_equal({ "path" => "/entries/1" }, events[3][2]) 291 | 292 | d2 = create_driver(%[ 293 | add_prefix filtered 294 | 295 | 296 | key path 297 | pattern \\?.+$ 298 | replace 299 | 300 | ]) 301 | d2.run(default_tag: 'test') do 302 | d2.feed({ "path" => "/foo?bar=1" }) 303 | end 304 | events = d2.events 305 | 306 | assert_equal 1, events.size 307 | assert_equal('filtered.test', events[0][0]) 308 | assert_equal({ "path" => "/foo" }, events[0][2]) 309 | 310 | # Test for not emit if the tag has not changed. 311 | d3 = create_driver(%[ 312 | 313 | key path 314 | pattern \\?.+$ 315 | replace 316 | 317 | ]) 318 | d3.run(default_tag: 'test') do 319 | d3.feed({ "path" => "/foo?bar=1" }) 320 | end 321 | 322 | assert_equal 0, d3.events.size 323 | 324 | # Emit message if the rewrite tag rules have been defined, even if (add|remove)_prefix option is not set. 325 | d4 = create_driver(%[ 326 | 327 | key path 328 | pattern ^\/(users|entries) 329 | append_to_tag true 330 | 331 | ]) 332 | d4.run(default_tag: 'test') do 333 | d4.feed({ "path" => "/users/studio3104" }) 334 | end 335 | events = d4.events 336 | 337 | assert_equal 1, events.size 338 | assert_equal('test.users', events[0][0]) 339 | assert_equal({ "path" => "/users/studio3104" }, events[0][2]) 340 | end 341 | end 342 | -------------------------------------------------------------------------------- /test/test_helper.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'bundler' 3 | 4 | begin 5 | Bundler.setup(:default, :development) 6 | rescue Bundler::BundlerError => e 7 | $stderr.puts e.message 8 | $stderr.puts "Run `bundle install` to install missing gems" 9 | exit e.status_code 10 | end 11 | 12 | require 'test/unit' 13 | 14 | $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib')) 15 | $LOAD_PATH.unshift(File.dirname(__FILE__)) 16 | 17 | require 'fluent/test' 18 | 19 | unless ENV.has_key?('VERBOSE') 20 | nulllogger = Object.new 21 | nulllogger.instance_eval {|obj| 22 | def method_missing(method, *args) 23 | # pass 24 | end 25 | } 26 | $log = nulllogger 27 | end 28 | 29 | require 'fluent/plugin/out_rewrite' 30 | require 'fluent/plugin/filter_rewrite' 31 | 32 | class Test::Unit::TestCase 33 | end 34 | --------------------------------------------------------------------------------