├── README.md ├── examples ├── grok.txt ├── json.txt └── sample.txt ├── jlogstash └── conf │ ├── grok.yaml │ ├── json.yaml │ └── sample.yaml └── logstash └── conf ├── grok.conf ├── json.conf └── sample.conf /README.md: -------------------------------------------------------------------------------- 1 | java logstash和 ruby logstash 做一个对比(4cores 4g 虚拟机) 2 | 3 | 4 | 5 | |app |filter |filteworkers |cpu used(avg)|data example |data numbers |time consuming（测试10次的avg）| 6 | |:------------- |:----- |:----------------- | -----------:|:------------- | -----------:| ----------------------------:| 7 | |rubylogstash |grok |4(默认cpu核数) |280% |apache |1000w |1625s| 8 | |javalogstash |grok |4(默认cpu核数) |330% |apache |1000w |543s| 9 | |rubylogstash |null |4(默认cpu核数) |200% |字符串(test 01) |1000w |705s| 10 | |javalogstash |null |4(默认cpu核数) |180% |字符串(test 01) |1000w |170s| 11 | |rubylogstash |json |4(默认cpu核数) |260% |json 字符串 |1000w |1504s| 12 | |java logstash |json |4(默认cpu核数) |310% |json 字符串 |1000w |430s| 13 | 14 | 以上三种场景的处理效率，javalogstash是rubylogstash的倍数分别是2.99倍 4.15倍 3.49倍 15 | 16 | -------------------------------------------------------------------------------- /examples/grok.txt: -------------------------------------------------------------------------------- 1 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 2 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 3 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 4 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 5 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 6 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 7 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 8 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 9 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 10 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 11 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 12 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 13 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 14 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 15 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 16 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 17 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 18 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 19 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 20 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 21 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 22 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 23 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 24 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 25 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 26 | 83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\" 27 | -------------------------------------------------------------------------------- /examples/json.txt: -------------------------------------------------------------------------------- 1 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 2 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 3 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 4 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 5 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 6 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 7 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 8 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 9 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 10 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 11 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 12 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 13 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 14 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 15 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 16 | {"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } } 17 | -------------------------------------------------------------------------------- /examples/sample.txt: -------------------------------------------------------------------------------- 1 | test 01 2 | test 01 3 | test 01 4 | test 01 5 | test 01 6 | test 01 7 | test 01 8 | test 01 9 | test 01 10 | test 01 11 | test 01 12 | test 01 13 | test 01 14 | test 01 15 | test 01 16 | test 01 17 | test 01 18 | test 01 19 | test 01 20 | test 01 21 | test 01 22 | test 01 23 | test 01 24 | test 01 25 | test 01 26 | test 01 27 | test 01 28 | test 01 29 | test 01 30 | test 01 31 | test 01 32 | test 01 33 | test 01 34 | test 01 35 | test 01 36 | test 01 37 | -------------------------------------------------------------------------------- /jlogstash/conf/grok.yaml: -------------------------------------------------------------------------------- 1 | inputs: 2 | - Stdin: 3 | 4 | filters: 5 | - JGrok: 6 | srcs: ["message"] 7 | patterns: {"%{COMBINEDAPACHELOG}":""} 8 | outputs: 9 | - Stdout: 10 | codec: json_lines 11 | -------------------------------------------------------------------------------- /jlogstash/conf/json.yaml: -------------------------------------------------------------------------------- 1 | inputs: 2 | - Stdin: 3 | 4 | filters: 5 | - Json 6 | fields: ["message":""] 7 | outputs: 8 | - Stdout: 9 | codec: json_lines 10 | -------------------------------------------------------------------------------- /jlogstash/conf/sample.yaml: -------------------------------------------------------------------------------- 1 | inputs: 2 | - Stdin: 3 | 4 | filters: 5 | outputs: 6 | - Stdout: 7 | codec: line 8 | -------------------------------------------------------------------------------- /logstash/conf/grok.conf: -------------------------------------------------------------------------------- 1 | input { 2 | stdin { type => "apache" } 3 | } 4 | 5 | filter { 6 | grok { 7 | match => {"message" => "%{COMBINEDAPACHELOG}"} 8 | } 9 | } 10 | 11 | output { 12 | stdout { codec => json_lines } 13 | } 14 | -------------------------------------------------------------------------------- /logstash/conf/json.conf: -------------------------------------------------------------------------------- 1 | input { 2 | stdin {} 3 | } 4 | 5 | filter { 6 | json { source => "message" } 7 | } 8 | 9 | output { 10 | stdout { codec => json_lines } 11 | } 12 | -------------------------------------------------------------------------------- /logstash/conf/sample.conf: -------------------------------------------------------------------------------- 1 | input { 2 | stdin {} 3 | } 4 | 5 | filter { 6 | } 7 | 8 | output { 9 | stdout { codec => line } 10 | } 11 | --------------------------------------------------------------------------------