├── .gitignore ├── CHANGELOG.md ├── CONTRIBUTORS ├── DEVELOPER.md ├── Gemfile ├── LICENSE ├── NOTICE.TXT ├── README.md ├── Rakefile ├── lib └── logstash │ └── filters │ └── ipip.rb ├── logstash-filter-ipip.gemspec └── spec ├── filters └── ipip_spec.rb └── spec_helper.rb /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | Gemfile.lock 3 | Gemfile.bak 4 | .bundle 5 | vendor 6 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## 2.0.0 2 | - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 3 | instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895 4 | - Dependency on logstash-core update to 2.0 5 | 6 | -------------------------------------------------------------------------------- /CONTRIBUTORS: -------------------------------------------------------------------------------- 1 | The following is a list of people who have contributed ideas, code, bug 2 | reports, or in general have helped logstash along its way. 3 | 4 | Contributors: 5 | * Aaron Mildenstein (untergeek) 6 | * Pier-Hugues Pellerin (ph) 7 | 8 | Note: If you've sent us patches, bug reports, or otherwise contributed to 9 | Logstash, and you aren't on the list above and want to be, please let us know 10 | and we'll make sure you're here. Contributions from folks like you are what make 11 | open source awesome. 12 | -------------------------------------------------------------------------------- /DEVELOPER.md: -------------------------------------------------------------------------------- 1 | # logstash-filter-example 2 | Example filter plugin. This should help bootstrap your effort to write your own filter plugin! 3 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://gems.ruby-china.org/' 2 | gemspec 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012–2016 Elasticsearch 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. 14 | -------------------------------------------------------------------------------- /NOTICE.TXT: -------------------------------------------------------------------------------- 1 | Elasticsearch 2 | Copyright 2012-2015 Elasticsearch 3 | 4 | This product includes software developed by The Apache Software 5 | Foundation (http://www.apache.org/). -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Logstash Plugin 2 | 3 | [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-filter-example.svg)](https://travis-ci.org/logstash-plugins/logstash-filter-example) 4 | 5 | This is a plugin for [Logstash](https://github.com/elastic/logstash). 6 | 7 | It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way. 8 | 9 | ## Documentation 10 | 11 | Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/). 12 | 13 | - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive 14 | - For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide 15 | 16 | ## Need Help? 17 | 18 | Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum. 19 | 20 | ## Developing 21 | 22 | ### 1. Plugin Developement and Testing 23 | 24 | #### Code 25 | - To get started, you'll need JRuby with the Bundler gem installed. 26 | 27 | - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example). 28 | 29 | - Install dependencies 30 | ```sh 31 | bundle install 32 | ``` 33 | 34 | #### Test 35 | 36 | - Update your dependencies 37 | 38 | ```sh 39 | bundle install 40 | ``` 41 | 42 | - Run tests 43 | 44 | ```sh 45 | bundle exec rspec 46 | ``` 47 | 48 | ### 2. Running your unpublished Plugin in Logstash 49 | 50 | #### 2.1 Run in a local Logstash clone 51 | 52 | - Edit Logstash `Gemfile` and add the local plugin path, for example: 53 | ```ruby 54 | gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome" 55 | ``` 56 | - Install plugin 57 | ```sh 58 | # Logstash 2.3 and higher 59 | bin/logstash-plugin install --no-verify 60 | 61 | # Prior to Logstash 2.3 62 | bin/plugin install --no-verify 63 | 64 | ``` 65 | - Run Logstash with your plugin 66 | ```sh 67 | bin/logstash -e 'filter {awesome {}}' 68 | ``` 69 | At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash. 70 | 71 | #### 2.2 Run in an installed Logstash 72 | 73 | You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using: 74 | 75 | - Build your plugin gem 76 | ```sh 77 | gem build logstash-filter-awesome.gemspec 78 | ``` 79 | - Install the plugin from the Logstash home 80 | ```sh 81 | # Logstash 2.3 and higher 82 | bin/logstash-plugin install --no-verify 83 | 84 | # Prior to Logstash 2.3 85 | bin/plugin install --no-verify 86 | 87 | ``` 88 | - Start Logstash and proceed to test the plugin 89 | 90 | ## Contributing 91 | 92 | All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin. 93 | 94 | Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here. 95 | 96 | It is more important to the community that you are able to contribute. 97 | 98 | For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file. -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | require "logstash/devutils/rake" 2 | -------------------------------------------------------------------------------- /lib/logstash/filters/ipip.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | require "logstash/filters/base" 3 | require "logstash/namespace" 4 | 5 | 6 | module SeventeenMon 7 | class IPDBX 8 | 9 | private_class_method :new 10 | 11 | def ip_db_path 12 | @ip_db_path ||= File.expand_path'../../../../vendor/ipip.datx', __FILE__ 13 | end 14 | 15 | def ip_db 16 | @ip_db ||= File.open ip_db_path, 'rb' 17 | end 18 | 19 | def offset 20 | @offset ||= ip_db.read(4).unpack("Nlen")[0] 21 | end 22 | 23 | def index 24 | @index ||= ip_db.read(offset - 4) 25 | end 26 | 27 | def max_comp_length 28 | @max_comp_length ||= offset - 262144 - 4 29 | end 30 | 31 | def self.instance 32 | @instance ||= self.send :new 33 | end 34 | 35 | def seek(_offset, length) 36 | IO.read(ip_db_path, length, offset + _offset - 262144).split "\t" 37 | end 38 | end 39 | 40 | class IP 41 | attr_reader :ip 42 | 43 | # Initialize IP object 44 | # 45 | # == parameters: 46 | # params:: 47 | # Might contain address(hostname) and protocol, or just IP 48 | # 49 | # == Returns: 50 | # self 51 | # 52 | def initialize(params = {}) 53 | @ip = params[:ip] || 54 | Socket.getaddrinfo(params[:address], params[:protocol])[0][3] 55 | end 56 | 57 | def four_number 58 | @four_number ||= begin 59 | fn = ip.split(".").map(&:to_i) 60 | raise "ip is no valid" if fn.length != 4 || fn.any?{ |d| d < 0 || d > 255} 61 | fn 62 | end 63 | end 64 | 65 | def ip2long 66 | @ip2long ||= ::IPAddr.new(ip).to_i 67 | end 68 | 69 | def packed_ip 70 | @packed_ip ||= [ ip2long ].pack 'N' 71 | end 72 | 73 | def find 74 | tmp_offset = (four_number[0] * 256 + four_number[1]) * 4 75 | start = IPDBX.instance.index[tmp_offset..(tmp_offset + 3)].unpack("V")[0] * 9 + 262144 76 | 77 | index_offset = -1 78 | 79 | while start < IPDBX.instance.max_comp_length 80 | if IPDBX.instance.index[start..(start + 3)] >= packed_ip 81 | index_offset = "#{IPDBX.instance.index[(start + 4)..(start + 6)]}\x0".unpack("V")[0] 82 | index_length = IPDBX.instance.index[(start + 8)].unpack("C")[0] 83 | break 84 | end 85 | start += 9 86 | end 87 | 88 | return "N/A" unless index_offset 89 | 90 | result = IPDBX.instance.seek(index_offset, index_length).map do |str| 91 | str.encode("UTF-8", "UTF-8") 92 | end 93 | 94 | { 95 | country: result[0], 96 | province: result[1], 97 | city: result[2], 98 | carrier: result[4] 99 | } 100 | end 101 | end 102 | end 103 | 104 | module SeventeenMon 105 | require "socket" 106 | require "ipaddr" 107 | 108 | def self.find_by_ip(_ip) 109 | IP.new(ip: _ip).find 110 | end 111 | end 112 | 113 | SM = SeventeenMon 114 | 115 | # This example filter will replace the contents of the default 116 | # message field with whatever you specify in the configuration. 117 | # 118 | # It is only intended to be used as an example. 119 | class LogStash::Filters::IPIP < LogStash::Filters::Base 120 | 121 | # Setting the config_name here is required. This is how you 122 | # configure this filter from your Logstash config. 123 | # 124 | # filter { 125 | # ipip { 126 | # message => "My message..." 127 | # } 128 | # } 129 | # 130 | config_name "ipip" 131 | 132 | # The field containing the IP address to map via ipip. 133 | config :source, :validate => :string, :require => true 134 | 135 | # An array of geoip fields to be included in the event. 136 | # 137 | # Possible fields depend on the database type. By default, all ipip fields 138 | # are included in the event. 139 | # 140 | # For the built-in GeoLiteCity database, the following are available: 141 | # `city_name`, `continent_code`, `country_code2`, `country_code3`, `country_name`, 142 | # `dma_code`, `ip`, `latitude`, `longitude`, `postal_code`, `region_name` and `timezone`. 143 | #config :fields, :validate => :array 144 | 145 | # Specify the field into which Logstash should store the geoip data. 146 | # This can be useful, for example, if you have `src\_ip` and `dst\_ip` fields and 147 | # would like the GeoIP information of both IPs. 148 | # 149 | # If you save the data to a target field other than `geoip` and want to use the 150 | # `geo\_point` related functions in Elasticsearch, you need to alter the template 151 | # provided with the Elasticsearch output and configure the output to use the 152 | # new template. 153 | # 154 | # Even if you don't use the `geo\_point` mapping, the `[target][location]` field 155 | # is still valid GeoJSON. 156 | config :target, :validate => :string, :default => 'ipip' 157 | 158 | 159 | public 160 | def register 161 | # nothing need here 162 | end # def register 163 | 164 | public 165 | def filter(event) 166 | return unless filter?(event) 167 | ipip_data = nil 168 | 169 | begin 170 | ip = event[@source] 171 | ip = ip.first if ip.is_a? Array 172 | ipip_data = SM.find_by_ip ip 173 | rescue Exception => e 174 | @logger.error("Unknown error while looking up IPIP data", :exception => e, :field => @field, :event => event) 175 | end 176 | 177 | event[@target] = {} if event[@target].nil? 178 | 179 | ipip_data.each do |key, value| 180 | event[@target][key.to_s] = value 181 | end 182 | 183 | # filter_matched should go in the last line of our successful code 184 | filter_matched(event) 185 | end # def filter 186 | end # class LogStash::Filters::IPIP 187 | -------------------------------------------------------------------------------- /logstash-filter-ipip.gemspec: -------------------------------------------------------------------------------- 1 | Gem::Specification.new do |s| 2 | s.name = 'logstash-filter-ipip' 3 | s.version = '2.0.0' 4 | s.licenses = ['Apache License (2.0)'] 5 | s.summary = "This ipip filter adds information about the geographical location of IP addresses, based on data from the database from ipip.net." 6 | s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program" 7 | s.authors = ["bittopaz", "vinsonzou"] 8 | s.email = 'vinsoncou@gmail.com' 9 | s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html" 10 | s.require_paths = ["lib"] 11 | 12 | # Files 13 | s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT'] 14 | # Tests 15 | s.test_files = s.files.grep(%r{^(test|spec|features)/}) 16 | 17 | # Special flag to let us know this is actually a logstash plugin 18 | s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" } 19 | 20 | # Gem dependencies 21 | s.add_runtime_dependency "logstash-core-plugin-api", "< 2.0.0" 22 | s.add_development_dependency 'logstash-devutils' 23 | end 24 | -------------------------------------------------------------------------------- /spec/filters/ipip_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | require 'spec_helper' 3 | require "logstash/filters/ipip" 4 | 5 | describe LogStash::Filters::IPIP do 6 | describe "Test ip 8.8.8.8" do 7 | let(:config) do <<-CONFIG 8 | filter { 9 | ipip { 10 | source => "ip" 11 | } 12 | } 13 | CONFIG 14 | end 15 | 16 | sample("ip" => "8.8.8.8") do 17 | insist { subject }.include?("ipip") 18 | 19 | expected_fileds = %w(country province city carrier) 20 | expected_fileds.each do |f| 21 | insist { subject["ipip"] }.include?(f) 22 | end 23 | end 24 | end 25 | 26 | describe "Specify the target" do 27 | let(:config) do <<-CONFIG 28 | filter { 29 | ipip { 30 | source => "ip" 31 | target => src_ip 32 | } 33 | } 34 | CONFIG 35 | end 36 | 37 | sample("ipip" => "8.8.8.8") do 38 | insist { subject }.include?("src_ip") 39 | 40 | expected_fileds = %w(country province city carrier) 41 | expected_fileds.each do |f| 42 | insist { subject["src_ip"] }.include?(f) 43 | end 44 | end 45 | end 46 | end 47 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | require "logstash/devutils/rspec/spec_helper" 3 | --------------------------------------------------------------------------------