├── .github ├── ISSUE_TEMPLATE.md └── workflows │ └── linux.yml ├── .gitignore ├── CHANGELOG.md ├── Gemfile ├── README.md ├── Rakefile ├── VERSION ├── fluent-plugin-redis.gemspec ├── gemfiles ├── fluentd_v0.10.gemfile ├── fluentd_v0.12.gemfile └── fluentd_v0.14.gemfile ├── lib └── fluent │ └── plugin │ └── out_redis.rb └── test └── plugin └── test_out_redis.rb /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | #### Problem 2 | 3 | ... 4 | 5 | #### Steps to replicate 6 | 7 | Provide example config and message 8 | 9 | #### Expected Behavior 10 | 11 | ... 12 | 13 | #### Your environment 14 | 15 | * OS version 16 | * paste result of ``fluentd --version`` or ``td-agent --version`` 17 | * plugin version 18 | * paste boot log of fluentd or td-agent 19 | * paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock 20 | -------------------------------------------------------------------------------- /.github/workflows/linux.yml: -------------------------------------------------------------------------------- 1 | name: Testing on Ubuntu 2 | on: 3 | - push 4 | - pull_request 5 | jobs: 6 | build: 7 | runs-on: ${{ matrix.os }} 8 | services: 9 | redis: 10 | image: redis 11 | ports: 12 | - 6379:6379 13 | strategy: 14 | fail-fast: false 15 | matrix: 16 | ruby: [ '2.5', '2.6', '2.7', '3.0' ] 17 | os: 18 | - ubuntu-latest 19 | name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }} 20 | steps: 21 | - uses: actions/checkout@v2 22 | - uses: ruby/setup-ruby@v1 23 | with: 24 | ruby-version: ${{ matrix.ruby }} 25 | - name: unit testing 26 | env: 27 | CI: true 28 | run: | 29 | gem install bundler rake 30 | bundle install --jobs 4 --retry 3 31 | bundle exec rake test 32 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | .bundle 3 | Gemfile.lock 4 | pkg/* 5 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## 0.3.5 2 | * Confirm Ruby 3.0 ready 3 | 4 | ## 0.3.4 5 | * Pass chunk directly into builtin placeholders 6 | 7 | ## 0.3.3 8 | * Add multi workers environment considaration 9 | 10 | ## 0.3.2 11 | * Add TTL support 12 | 13 | ## 0.3.1 14 | 15 | * Use public msgpack APIs instead of its internal APIs 16 | * Remove needless rescue block 17 | 18 | ## 0.3.0 19 | 20 | * Migrate v0.14 API based plugin 21 | * Allow duplicate insert key to support update values functionality 22 | * Use v0.14's built-in placeholder functionality 23 | * Enabled to specify more flexible tag and time format for identifier 24 | 25 | ## 0.2.2 26 | 27 | * Use redis-rb 3.2.x 28 | * Support requirepass authentication 29 | 30 | ## 0.2.1 31 | 32 | * Use config_param to clarify supported/default parameters. 33 | * Use log instead of $log if avaliable. 34 | 35 | ## 0.2.0 36 | 37 | * Support Fluentd v0.10. 38 | 39 | ## 0.1.3 40 | 41 | * Colored the log message added on v0.1.2. 42 | 43 | ## 0.1.2 44 | 45 | * Shows a warning message when trying to use namespace option. 46 | 47 | ## 0.1.1 48 | 49 | * Disabled Redis::Namespace operation. 50 | 51 | ## 0.1.0 52 | 53 | * Specified the versions of dependencies(fluent, redis and redis-namespace). 54 | 55 | ## 0.0.1 56 | 57 | * This initial plugin is released. 58 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source "http://rubygems.org" 2 | 3 | # Specify your gem's dependencies in fluent-plugin-redis.gemspec 4 | gemspec 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Redis output plugin for Fluent 2 | 3 | ![Testing on Ubuntu](https://github.com/fluent-plugins-nursery/fluent-plugin-redis/workflows/Testing%20on%20Ubuntu/badge.svg) 4 | 5 | fluent-plugin-redis is a fluent plugin to output to redis. 6 | 7 | ## Requirements 8 | 9 | |fluent-plugin-redis| fluentd | ruby | 10 | |-------------------|------------------|--------| 11 | | >= 0.3.0 | >= 0.14.8 | >= 2.1 | 12 | | == 0.2.3 | ~> 0.12.0 * | >= 1.9 | 13 | | < 0.2.3 | >= 0.10.0, < 2 * | >= 1.9 | 14 | 15 | * May not support all future fluentd features 16 | 17 | ## Installation 18 | 19 | What you have to do is only installing like this: 20 | 21 | gem install fluent-plugin-redis 22 | 23 | Then fluent automatically loads the plugin installed. 24 | 25 | ## Configuration 26 | 27 | ### Example 28 | 29 | 30 | @type redis 31 | 32 | # host localhost 33 | # port 6379 34 | # db_number 0 35 | # password hogefuga 36 | # insert_key_prefix '${tag}' 37 | # strftime_format "%s" 38 | # allow_duplicate_key false 39 | # ttl 300 40 | 41 | 42 | ### Parameter 43 | 44 | |parameter|description|default| 45 | |---|---|---| 46 | |host|The hostname of Redis server|`localhost`| 47 | |port|The port number of Redis server|`6379`| 48 | |db_number|The number of database|`0`| 49 | |password|The password of Redis. If requirepass is set, please specify this|nil| 50 | |insert\_key\_prefix|Users can set '${tag}' or ${tag[0]}.${tag[1]} or ...?|`${tag}`| 51 | |strftime\_format|Users can set strftime format.
"%s" will be replaced into unixtime. "%Y%m%d.%H%M%S" will be replaced like as 20161202.112335|`"%s"`| 52 | |allow\_duplicate\_key|Allow duplicated insert key. It will work as update values|`false`| 53 | |ttl|The value of TTL. If 0 or negative value is set, ttl is not set in each key|`-1`| 54 | 55 | 56 | 57 | 58 | 59 | ### With multi workers 60 | 61 | fluent-plugin-redis can handle multi workers. 62 | This feature can be enabled with the following configuration: 63 | 64 | 65 | workers n # where n >= 2. 66 | 67 | 68 | ### Notice 69 | 70 | insert_key_prefix, strftime_format, and allow_duplicate_key are newly added config parameters. 71 | 72 | They can use v0.3.0 or later. To use this parameters, users must update Fluentd to v0.14 or later and this plugin to v0.3.0 or later. 73 | 74 | multi workers are newly introduced feature in Fluentd v0.14. 75 | 76 | It can use this feature in this plugin in v0.3.3 or later. 77 | 78 | ## Contributing to fluent-plugin-redis 79 | 80 | * Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet 81 | * Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it 82 | * Fork the project 83 | * Start a feature/bugfix branch 84 | * Commit and push until you are happy with your contribution 85 | * Make sure to add tests for it. This is important so I don't break it in a future version unintentionally. 86 | * Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it. 87 | 88 | ## Copyright 89 | 90 | Copyright (c) 2011- Yuki Nishijima 91 | 92 | ## License 93 | 94 | Apache License, Version 2.0 95 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | require "bundler/gem_tasks" 2 | 3 | require 'rake/testtask' 4 | Rake::TestTask.new(:test) do |test| 5 | test.libs << 'lib' << 'test' 6 | test.pattern = 'test/**/test_*.rb' 7 | test.verbose = true 8 | end 9 | 10 | task default: :test 11 | -------------------------------------------------------------------------------- /VERSION: -------------------------------------------------------------------------------- 1 | 0.3.1 2 | -------------------------------------------------------------------------------- /fluent-plugin-redis.gemspec: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | $:.push File.expand_path("../lib", __FILE__) 3 | 4 | Gem::Specification.new do |s| 5 | s.name = "fluent-plugin-redis" 6 | s.version = "0.3.5" 7 | s.platform = Gem::Platform::RUBY 8 | s.authors = ["Yuki Nishijima", "Hiroshi Hatake", "Kenji Okimoto"] 9 | s.date = %q{2019-02-26} 10 | s.email = ["mail@yukinishijima.net", "fluent@clear-code.com"] 11 | s.homepage = "https://github.com/fluent-plugins-nursery/fluent-plugin-redis" 12 | s.summary = "Redis output plugin for Fluent" 13 | 14 | s.files = `git ls-files`.split("\n") 15 | s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n") 16 | s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) } 17 | s.require_paths = ["lib"] 18 | 19 | s.add_dependency %q, [">= 0.14.22", "< 2"] 20 | s.add_dependency %q, ["~> 3.3.0"] 21 | s.add_development_dependency %q, [">= 11.3.0"] 22 | s.add_development_dependency %q 23 | s.add_development_dependency %q, ["~> 3.3.9"] 24 | s.add_development_dependency %q, ["~> 2.1.0"] 25 | s.add_development_dependency %q 26 | end 27 | -------------------------------------------------------------------------------- /gemfiles/fluentd_v0.10.gemfile: -------------------------------------------------------------------------------- 1 | # This file was generated by Appraisal 2 | 3 | source "http://rubygems.org" 4 | 5 | gem "fluentd", "~> 0.10.0" 6 | 7 | gemspec :path => "../" 8 | -------------------------------------------------------------------------------- /gemfiles/fluentd_v0.12.gemfile: -------------------------------------------------------------------------------- 1 | # This file was generated by Appraisal 2 | 3 | source "http://rubygems.org" 4 | 5 | gem "fluentd", "~> 0.12.0" 6 | 7 | gemspec :path => "../" 8 | -------------------------------------------------------------------------------- /gemfiles/fluentd_v0.14.gemfile: -------------------------------------------------------------------------------- 1 | # This file was generated by Appraisal 2 | 3 | source "http://rubygems.org" 4 | 5 | gem "fluentd", "~> 0.14.0" 6 | 7 | gemspec :path => "../" 8 | -------------------------------------------------------------------------------- /lib/fluent/plugin/out_redis.rb: -------------------------------------------------------------------------------- 1 | require 'redis' 2 | require 'msgpack' 3 | require 'fluent/plugin/output' 4 | 5 | module Fluent::Plugin 6 | class RedisOutput < Output 7 | Fluent::Plugin.register_output('redis', self) 8 | 9 | helpers :compat_parameters, :inject 10 | 11 | DEFAULT_BUFFER_TYPE = "memory" 12 | DEFAULT_TTL_VALUE = -1 13 | 14 | attr_reader :redis 15 | 16 | config_param :host, :string, default: 'localhost' 17 | config_param :port, :integer, default: 6379 18 | config_param :db_number, :integer, default: 0 19 | config_param :password, :string, default: nil, secret: true 20 | config_param :insert_key_prefix, :string, default: "${tag}" 21 | config_param :strftime_format, :string, default: "%s" 22 | config_param :allow_duplicate_key, :bool, default: false 23 | config_param :ttl, :integer, default: DEFAULT_TTL_VALUE 24 | 25 | config_section :buffer do 26 | config_set_default :@type, DEFAULT_BUFFER_TYPE 27 | config_set_default :chunk_keys, ['tag', 'time'] 28 | config_set_default :timekey, 60 29 | end 30 | 31 | def configure(conf) 32 | compat_parameters_convert(conf, :buffer, :inject) 33 | @running_multi_workers = system_config.workers > 1 34 | super 35 | 36 | if conf.has_key?('namespace') 37 | log.warn "namespace option has been removed from fluent-plugin-redis 0.1.3. Please add or remove the namespace '#{conf['namespace']}' manually." 38 | end 39 | raise Fluent::ConfigError, "'tag' in chunk_keys is required." if not @chunk_key_tag 40 | raise Fluent::ConfigError, "'time' in chunk_keys is required." if not @chunk_key_time 41 | @unpacker = Fluent::Engine.msgpack_factory.unpacker 42 | end 43 | 44 | def start 45 | super 46 | 47 | options = { 48 | host: @host, 49 | port: @port, 50 | thread_safe: true, 51 | db: @db_number 52 | } 53 | options[:password] = @password if @password 54 | 55 | @redis = Redis.new(options) 56 | end 57 | 58 | def shutdown 59 | @redis.quit 60 | super 61 | end 62 | 63 | def format(tag, time, record) 64 | record = inject_values_to_record(tag, time, record) 65 | [tag, time, record].to_msgpack 66 | end 67 | 68 | def formatted_to_msgpack_binary 69 | true 70 | end 71 | 72 | def multi_workers_ready? 73 | true 74 | end 75 | 76 | def write(chunk) 77 | tag, time = expand_placeholders(chunk) 78 | @redis.pipelined { 79 | unless @allow_duplicate_key 80 | stream = chunk.to_msgpack_stream 81 | @unpacker.feed_each(stream).with_index { |record, index| 82 | identifier = if @running_multi_workers 83 | [tag, time, fluentd_worker_id].join(".") 84 | else 85 | [tag, time].join(".") 86 | end 87 | @redis.multi do 88 | @redis.mapped_hmset "#{identifier}.#{index}", record[2] 89 | @redis.expire "#{identifier}.#{index}", @ttl if @ttl > 0 90 | end 91 | } 92 | else 93 | chunk.each do |_tag, _time, record| 94 | @redis.multi do 95 | @redis.mapped_hmset "#{tag}", record 96 | @redis.expire "#{tag}", @ttl if @ttl > 0 97 | end 98 | end 99 | end 100 | } 101 | end 102 | 103 | private 104 | 105 | def expand_placeholders(chunk) 106 | tag = extract_placeholders(@insert_key_prefix, chunk) 107 | time = extract_placeholders(@strftime_format, chunk) 108 | return tag, time 109 | end 110 | end 111 | end 112 | -------------------------------------------------------------------------------- /test/plugin/test_out_redis.rb: -------------------------------------------------------------------------------- 1 | require 'fluent/test' 2 | require 'fluent/test/helpers' 3 | require 'fluent/test/driver/output' 4 | require 'fluent/plugin/out_redis' 5 | require 'fluent/time' # Fluent::TimeFormatter 6 | 7 | class FileOutputTest < Test::Unit::TestCase 8 | include Fluent::Test::Helpers 9 | 10 | CONFIG = %[ 11 | host localhost 12 | port 6379 13 | db_number 1 14 | ] 15 | 16 | def setup 17 | Fluent::Test.setup 18 | 19 | @d = create_driver 20 | @time = event_time("2011-01-02 13:14:15 UTC") 21 | end 22 | 23 | def create_driver(conf = CONFIG) 24 | Fluent::Test::Driver::Output.new(Fluent::Plugin::RedisOutput).configure(conf) 25 | end 26 | 27 | def time_formatter(inject_config) 28 | Fluent::TimeFormatter.new(inject_config.time_format, inject_config.localtime, inject_config.timezone) 29 | end 30 | 31 | def test_configure 32 | assert_equal 'localhost', @d.instance.host 33 | assert_equal 6379, @d.instance.port 34 | assert_equal 1, @d.instance.db_number 35 | assert_nil @d.instance.password 36 | assert_equal '${tag}', @d.instance.insert_key_prefix 37 | assert_equal '%s', @d.instance.strftime_format 38 | assert_false @d.instance.allow_duplicate_key 39 | assert_equal Fluent::Plugin::RedisOutput::DEFAULT_TTL_VALUE, @d.instance.ttl 40 | end 41 | 42 | def test_configure_with_password 43 | d = create_driver CONFIG + %[ 44 | password testpass 45 | ] 46 | assert_equal 'localhost', d.instance.host 47 | assert_equal 6379, d.instance.port 48 | assert_equal 1, d.instance.db_number 49 | assert_equal 'testpass', d.instance.password 50 | end 51 | 52 | def test_configure_without_tag_chunk_key 53 | config = config_element('ROOT', '', { 54 | "host" => "localhost", 55 | "port" => 6379, 56 | "db_number" => 1, 57 | }, [ 58 | config_element('buffer', 'time', { 59 | 'chunk_keys' => 'time', 60 | }) 61 | ]) 62 | assert_raise Fluent::ConfigError do 63 | create_driver(config) 64 | end 65 | end 66 | 67 | def test_format 68 | @d.run(default_tag: 'test') do 69 | @d.feed(@time, {"a"=>1}) 70 | end 71 | assert_equal [["test", @time, {"a"=>1}].to_msgpack], @d.formatted 72 | end 73 | 74 | class InjectTest < self 75 | def test_format_inject_tag_keys 76 | d = create_driver CONFIG + %[ 77 | include_tag_key true 78 | ] 79 | d.run(default_tag: 'test') do 80 | d.feed(@time, {"a"=>1}) 81 | end 82 | assert_equal [["test", @time, {"a"=>1, "tag" => "test"}].to_msgpack], d.formatted 83 | end 84 | 85 | def test_format_inject_time_keys 86 | d = create_driver CONFIG + %[ 87 | include_time_key true 88 | ] 89 | timef = time_formatter(d.instance.inject_config) 90 | d.run(default_tag: 'test') do 91 | d.feed(@time, {"a"=>1}) 92 | end 93 | assert_equal [["test", @time, {"a"=>1, "time" => timef.call(@time)}].to_msgpack], d.formatted 94 | end 95 | end 96 | 97 | class WriteTest < self 98 | def test_write 99 | d = create_driver 100 | time = event_time("2011-01-02 13:14:00 UTC") 101 | d.run(default_tag: 'test') do 102 | d.feed(time, {"a"=>2}) 103 | d.feed(time, {"a"=>3}) 104 | end 105 | 106 | assert_equal "2", d.instance.redis.hget("test.#{time}.0", "a") 107 | assert_equal "3", d.instance.redis.hget("test.#{time}.1", "a") 108 | end 109 | 110 | def test_write_with_insert_key_prefix 111 | d = create_driver CONFIG + %[ 112 | insert_key_prefix "${tag[1]}.${tag[2]}" 113 | ] 114 | time = event_time("2011-01-02 13:14:00 UTC") 115 | d.run(default_tag: 'prefix.insert.test') do 116 | d.feed(time, {"a"=>2}) 117 | d.feed(time, {"a"=>3}) 118 | end 119 | 120 | assert_equal "2", d.instance.redis.hget("test.#{time}.0", "a") 121 | assert_equal "3", d.instance.redis.hget("test.#{time}.1", "a") 122 | end 123 | 124 | def test_write_with_ttl 125 | ttl = 2 126 | d = create_driver CONFIG + %[ 127 | ttl #{ttl} 128 | allow_duplicate_key true 129 | ] 130 | time = event_time("2011-01-02 13:14:00 UTC") 131 | d.run(default_tag: 'ttl.insert.test') do 132 | d.feed(time, {"a"=>2}) 133 | end 134 | 135 | assert_in_delta 2.0, d.instance.redis.ttl("ttl.insert.test"), 1.0 136 | 137 | sleep ttl+1 138 | assert_equal -2, d.instance.redis.ttl("ttl.insert.test") 139 | end 140 | 141 | def test_write_with_custom_strftime_format 142 | d = create_driver CONFIG + %[ 143 | strftime_format "%Y%m%d.%H%M%S" 144 | ] 145 | time = event_time("2011-01-02 13:14:00 UTC") 146 | strtime = Time.at(time.to_r).strftime("%Y%m%d.%H%M%S") 147 | d.run(default_tag: 'test') do 148 | d.feed(time, {"a"=>4}) 149 | d.feed(time, {"a"=>5}) 150 | end 151 | 152 | assert_equal "4", d.instance.redis.hget("test.#{strtime}.0", "a") 153 | assert_equal "5", d.instance.redis.hget("test.#{strtime}.1", "a") 154 | end 155 | 156 | def test_write_with_allow_duplicate 157 | d = create_driver CONFIG + %[ 158 | allow_duplicate_key true 159 | ] 160 | time = event_time("2011-01-02 13:14:00 UTC") 161 | d.run(default_tag: 'test.duplicate') do 162 | d.feed(time, {"a"=>6}) 163 | d.feed(time, {"a"=>7}) 164 | end 165 | 166 | assert_equal "7", d.instance.redis.hget("test.duplicate", "a") 167 | end 168 | end 169 | end 170 | --------------------------------------------------------------------------------