[OUTPUT]
name file
path /var/log/fluent
format plain
match *
<match pattern>
@type file
path /var/log/fluent/myapp
</match> 47 | 48 |
[^ ]*) (?[^ ]*)(?: "(?(?:[^\"]|\\.)*)" "(?(?:[^\"]|\\.)*)")?$/ 15 | time_format %d/%b/%Y:%H:%M:%S %z 16 | ``` 17 | 18 | `host`, `user`, `method`, `path`, `code`, `size`, `referer` and `agent` are included in the event record. `time` is used for the event time. 19 | 20 | `code` and `size` fields are converted to integer type automatically. And, if the field value is `-`, it is interpreted as `nil`. See **Result Example**. 21 | 22 | ## Example 23 | 24 | This incoming event 25 | 26 | ```text 27 | 192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" 28 | ``` 29 | 30 | is parsed as: 31 | 32 | ```text 33 | time: 34 | 1362020400 (28/Feb/2013:12:00:00 +0900) 35 | 36 | record: 37 | { 38 | "user" : nil, 39 | "method" : "GET", 40 | "code" : 200, 41 | "size" : 777, 42 | "host" : "192.168.0.1", 43 | "path" : "/", 44 | "referer": nil, 45 | "agent" : "Opera/12.0" 46 | } 47 | ``` 48 | 49 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 50 | 51 | -------------------------------------------------------------------------------- /parser/apache_error.md: -------------------------------------------------------------------------------- 1 | # apache\_error 2 | 3 | The `apache_error` parser plugin parses apache error logs. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ## Regexp patterns 10 | 11 | This is regexp pattern of this plugin: 12 | 13 | ```text 14 | expression /^\[[^ ]* (?[^\]]*)\] \[(?[^\]]*)\](?: \[pid (?[^\]]*)\])? \[client (?[^\]]*)\] (?.*)$/ 15 | ``` 16 | 17 | `level`, `pid`, `client` and `message` are included in the event record. `time` is used for the event time. 18 | 19 | ## Example 20 | 21 | This incoming event: 22 | 23 | ```text 24 | [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration 25 | ``` 26 | 27 | is parsed as: 28 | 29 | ```text 30 | time: 31 | 971242372 (Wed Oct 11 14:32:52 2000) 32 | 33 | record: 34 | { 35 | "level" : "error", 36 | "client" : "127.0.0.1", 37 | "message": "client denied by server configuration" 38 | } 39 | ``` 40 | 41 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 42 | 43 | -------------------------------------------------------------------------------- /parser/csv.md: -------------------------------------------------------------------------------- 1 | # csv 2 | 3 | The `csv` parser plugin parses CSV format. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ### `keys` 10 | 11 | | type | default | version | 12 | | :--- | :--- | :--- | 13 | | array of string | required parameter | 0.14.9 | 14 | 15 | The array of names for fields on each line. 16 | 17 | ### `delimiter` 18 | 19 | | type | default | version | 20 | | :--- | :--- | :--- | 21 | | string | , | 0.14.2 | 22 | 23 | The delimiter \(character or string\) separating CSV values. 24 | 25 | ### `parser_type` 26 | 27 | | type | default | available values | version | 28 | | :--- | :--- | :--- | :--- | 29 | | enum | normal | normal/fast | 1.7.0 | 30 | 31 | The parser type used to parse the log line. 32 | 33 | * `normal` uses Ruby's 34 | 35 | [`CSV.parse_line`](http://ruby-doc.org/stdlib-2.4.1/libdoc/csv/rdoc/CSV.html#method-c-parse_line) 36 | 37 | method. 38 | 39 | * `fast` uses its own lightweight implementation. This parser is several times 40 | 41 | faster than `normal` but it supports only typical patterns. 42 | 43 | Supported CSV formats by `fast`: 44 | 45 | ```text 46 | # non-quoted 47 | value1,value2,value3,value4,value5 48 | 49 | # quoted 50 | "value1","val,ue2","va,lu,e3","val ue4","" 51 | 52 | # escaped 53 | "message","mes""sage","""message""",,"""""" 54 | 55 | # mixed 56 | message,"mes,sage","me,ssa,ge",mess age,"" 57 | ``` 58 | 59 | If your CSV format is not matched with the above patterns, use `normal` parser instead. 60 | 61 | ## Example 62 | 63 | With this configuration: 64 | 65 | ```text 66 | 67 | @type csv 68 | keys time,host,req_id,user 69 | time_key time 70 | 71 | ``` 72 | 73 | This incoming event: 74 | 75 | ```text 76 | 2013/02/28 12:00:00,192.168.0.1,111,- 77 | ``` 78 | 79 | is parsed as: 80 | 81 | ```text 82 | time: 83 | 1362020400 (2013/02/28/ 12:00:00) 84 | 85 | record: 86 | { 87 | "host" : "192.168.0.1", 88 | "req_id" : "111", 89 | "user" : "-" 90 | } 91 | ``` 92 | 93 | If you set `null_value_pattern '-'` in the configuration, `user` field becomes `nil` instead of `"-"`. 94 | 95 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 96 | 97 | -------------------------------------------------------------------------------- /parser/json.md: -------------------------------------------------------------------------------- 1 | # json 2 | 3 | The `json` parser plugin parses JSON logs. One JSON map per line. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ### `json_parser` 10 | 11 | | type | default | available | version | 12 | | :--- | :--- | :--- | :--- | 13 | | enum | oj | oj, yajl, json | 0.14.0 | 14 | 15 | Sets the JSON parser. If you have a problem with the configured parser, check the other available parser types. 16 | 17 | NOTE: If you want to enable `json_parser oj` by default, The `oj` gem must be installed separately. This is because `oj` gem is not required from fluentd by default. If `oj` gem is not installed, `json` is used as a fallback. (The fallback has been changed from `yajl` to `json` since v1.19.0.) 18 | 19 | Here is a simple comparison: 20 | 21 | * `oj`: Faster json parser 22 | * `yajl`: Mainly for stream parsing 23 | * `json`: Standard bundled library 24 | 25 | ### `stream_buffer_size` 26 | 27 | | type | default | version | 28 | | :--- | :--- | :--- | 29 | | integer | 8192 | 1.0.0 | 30 | 31 | Set the buffer size that Yajl will use when parsing streaming input. 32 | 33 | If you specify the smaller buffer size, you could get outcome response instantaneously in exchange for performance penalty. 34 | 35 | See also: [Method: Yajl::Parser\#parse](https://www.rubydoc.info/github/brianmario/yajl-ruby/Yajl%2FParser:parse) 36 | 37 | ### `time_type` 38 | 39 | `json` parser changes the default value of `time_type` to `float`. If you want to parse string field, set `time_type` and `time_format` like this: 40 | 41 | ```text 42 | # conf 43 | @type json 44 | time_type string 45 | time_format %d/%b/%Y:%H:%M:%S %z 46 | 47 | # record example 48 | {"key":"value","time":"28/Feb/2013:12:00:00 +0900"} 49 | ``` 50 | 51 | See also [`parse` Section](../configuration/parse-section.md#time-parameters) article. 52 | 53 | ## Example 54 | 55 | This incoming event: 56 | 57 | ```text 58 | {"time":1362020400,"host":"192.168.0.1","size":777,"method":"PUT"} 59 | ``` 60 | 61 | is parsed as: 62 | 63 | ```text 64 | time: 65 | 1362020400 (2013-02-28 12:00:00 +0900) 66 | 67 | record: 68 | { 69 | "host" : "192.168.0.1", 70 | "size" : 777, 71 | "method": "PUT", 72 | } 73 | ``` 74 | 75 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 76 | 77 | -------------------------------------------------------------------------------- /parser/ltsv.md: -------------------------------------------------------------------------------- 1 | # ltsv 2 | 3 | The `ltsv` parser plugin parses [LTSV](http://ltsv.org/) format. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ### `delimiter` 10 | 11 | | type | default | version | 12 | | :--- | :--- | :--- | 13 | | string | `\t` | 0.14.0 | 14 | 15 | The delimiter \(character or string\) separating LTSV values. 16 | 17 | ### `delimiter_pattern` 18 | 19 | | type | default | version | 20 | | :--- | :--- | :--- | 21 | | regexp | nil | 1.2.0 | 22 | 23 | The delimiter pattern of TSV values. This parameter overwrites `delimiter` parameter if specified. 24 | 25 | `delimiter_pattern` is string type before 1.2.0. 26 | 27 | ### `label_delimiter` 28 | 29 | | type | default | version | 30 | | :--- | :--- | :--- | 31 | | string | : | 0.14.0 | 32 | 33 | The delimiter character between field name and value. 34 | 35 | ## Example for LTSV 36 | 37 | This incoming event: 38 | 39 | ```text 40 | time:2013/02/28 12:00:00\thost:192.168.0.1\treq_id:111\tuser:- 41 | ``` 42 | 43 | is parsed as: 44 | 45 | ```text 46 | time: 47 | 1362020400 (2013/02/28/ 12:00:00) 48 | 49 | record: 50 | { 51 | "host" : "192.168.0.1", 52 | "req_id" : "111", 53 | "user" : "-" 54 | } 55 | ``` 56 | 57 | If you set `null_value_pattern '-'` in the configuration, `user` field becomes `nil` instead of `"-"`. 58 | 59 | ## Example with `delimiter_pattern` 60 | 61 | With this configuration: 62 | 63 | ```text 64 | 65 | @type ltsv 66 | delimiter_pattern /\s+/ 67 | label_delimiter = 68 | 69 | ``` 70 | 71 | This incoming event: 72 | 73 | ```text 74 | timestamp=1362020400 host=192.168.0.1 req_id=111 user=- 75 | ``` 76 | 77 | is parsed as: 78 | 79 | ```text 80 | record: 81 | { 82 | "timestamp": "1362020400", 83 | "host" : "192.168.0.1", 84 | "req_id" : "111", 85 | "user" : "-" 86 | } 87 | ``` 88 | 89 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 90 | 91 | -------------------------------------------------------------------------------- /parser/msgpack.md: -------------------------------------------------------------------------------- 1 | # msgpack 2 | 3 | The `msgpack` parser plugin parses the MessagePack data. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ## Example 10 | 11 | This incoming event: 12 | 13 | ```text 14 | \x82\xA7message\xADHello msgpack\xA3numd 15 | ``` 16 | 17 | is parsed as: 18 | 19 | ```text 20 | time: 21 | 1582648111.269613 (current time) 22 | 23 | record: 24 | {"message":"Hello msgpack", "num":100} 25 | ``` 26 | 27 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 28 | 29 | -------------------------------------------------------------------------------- /parser/nginx.md: -------------------------------------------------------------------------------- 1 | # nginx 2 | 3 | The `nginx` parser plugin parses the default Nginx logs. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ## Regexp Patterns 10 | 11 | Here is the regexp and time format patterns of this plugin: 12 | 13 | ```text 14 | expression /^(?[^ ]*) (?[^ ]*) (?[^ ]*) \[(?[^\]]*)\] "(?\S+)(?: +(?[^\"]*?)(?: +\S*)?)?" (?[^ ]*) (?[^ ]*)(?: "(?[^\"]*)" "(?[^\"]*)"(?:\s+(?[^ ]+))?)?$/ 15 | time_format %d/%b/%Y:%H:%M:%S %z 16 | ``` 17 | 18 | `remote`, `user`, `method`, `path`, `code`, `size`, `referer`, `agent` and `http_x_forwarded_for` are included in the event record. `time` is used for the event time. 19 | 20 | ## Example 21 | 22 | This incoming event: 23 | 24 | ```text 25 | 127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" - 26 | ``` 27 | 28 | is parsed as: 29 | 30 | ```text 31 | time: 32 | 1362020400 (28/Feb/2013:12:00:00 +0900) 33 | 34 | record: 35 | { 36 | "remote" : "127.0.0.1", 37 | "host" : "192.168.0.1", 38 | "user" : "-", 39 | "method" : "GET", 40 | "path" : "/", 41 | "code" : "200", 42 | "size" : "777", 43 | "referer" : "-", 44 | "agent" : "Opera/12.0", 45 | "http_x_forwarded_for": "-" 46 | } 47 | ``` 48 | 49 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 50 | 51 | -------------------------------------------------------------------------------- /parser/none.md: -------------------------------------------------------------------------------- 1 | # none 2 | 3 | The `none` parser plugin parses the line as-is with the single field. This format is to defer the parsing/structuring of the data. 4 | 5 | This parser is often used in conjunction with [`single_value`](../formatter/single_value.md) format in output plugin. 6 | 7 | ## Parameters 8 | 9 | See [Parse Section Configurations](../configuration/parse-section.md). 10 | 11 | ### `message_key` 12 | 13 | | type | default | version | 14 | | :--- | :--- | :--- | 15 | | string | message | 0.14.0 | 16 | 17 | Specifies the field name to contain logs. 18 | 19 | ## Example 20 | 21 | This incoming event: 22 | 23 | ```text 24 | Hello world. I am a line of log! 25 | ``` 26 | 27 | is parsed as: 28 | 29 | ```text 30 | time: 31 | 1362020400 (current time) 32 | 33 | record: 34 | {"message":"Hello world. I am a line of log!"} 35 | ``` 36 | 37 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 38 | 39 | -------------------------------------------------------------------------------- /parser/tsv.md: -------------------------------------------------------------------------------- 1 | # tsv 2 | 3 | The `tsv` parser plugin parses the TSV format. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ### `keys` 10 | 11 | | type | default | version | 12 | | :--- | :--- | :--- | 13 | | array of string | required parameter | 0.14.9 | 14 | 15 | The array of names for fields on each line. 16 | 17 | ### `delimiter` 18 | 19 | | type | default | version | 20 | | :--- | :--- | :--- | 21 | | string | `\t` | 0.14.0 | 22 | 23 | The delimiter \(character or string\) separating TSV values. 24 | 25 | ## Example 26 | 27 | With this configuration: 28 | 29 | ```text 30 | 31 | @type tsv 32 | keys time,host,req_id,user 33 | time_key time 34 | 35 | ``` 36 | 37 | This incoming event: 38 | 39 | ```text 40 | 2013/02/28 12:00:00\t192.168.0.1\t111\t- 41 | ``` 42 | 43 | is parsed as: 44 | 45 | ```text 46 | time: 47 | 1362020400 (2013/02/28/ 12:00:00) 48 | 49 | record: 50 | { 51 | "host" : "192.168.0.1", 52 | "req_id" : "111", 53 | "user" : "-" 54 | } 55 | ``` 56 | 57 | If you set `null_value_pattern '-'` in the configuration, `user` field becomes `nil` instead of `"-"`. 58 | 59 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 60 | 61 | -------------------------------------------------------------------------------- /plugin-development/api-plugin-buffer.md: -------------------------------------------------------------------------------- 1 | # How to Write Buffer Plugin 2 | 3 | TODO: Write 4 | 5 | ## Methods 6 | 7 | ### evacuate_chunk 8 | 9 | You can override this method to add feature to evacuate chunks before clearing the queue when reaching the retry limit. 10 | See [Buffer - Handling Successive Failures](../buffer/README.md#handling-successive-failures) for details. 11 | 12 | ```ruby 13 | def evacuate_chunk(chunk) 14 | unless chunk.is_a?(Fluent::Plugin::Buffer::FileChunk) 15 | raise ArgumentError, "The chunk must be FileChunk, but it was #{chunk.class}." 16 | end 17 | 18 | backup_dir = File.join(backup_base_dir, 'buffer', safe_owner_id) 19 | FileUtils.mkdir_p(backup_dir, mode: system_config.dir_permission || Fluent::DEFAULT_DIR_PERMISSION) unless Dir.exist?(backup_dir) 20 | 21 | FileUtils.copy([chunk.path, chunk.meta_path], backup_dir) 22 | log.warn "chunk files are evacuated to #{backup_dir}.", chunk_id: dump_unique_id_hex(chunk.unique_id) 23 | rescue => e 24 | log.error "unexpected error while evacuating chunk files.", error: e 25 | end 26 | ``` 27 | 28 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 29 | 30 | -------------------------------------------------------------------------------- /plugin-development/api-plugin-service_discovery.md: -------------------------------------------------------------------------------- 1 | # How to Write Service Discovery Plugin 2 | 3 | TODO: Write 4 | 5 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 6 | 7 | -------------------------------------------------------------------------------- /plugin-development/api-plugin-storage.md: -------------------------------------------------------------------------------- 1 | # How to Write Storage Plugin 2 | 3 | TODO: Write 4 | 5 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 6 | 7 | -------------------------------------------------------------------------------- /plugin-helper-overview/README.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper API 2 | 3 | Fluentd provides plugin helpers to encapsulate and make commonly implemented features available such as timer, threading, formatting, parsing, ensuring configuration syntax's backward compatibility, etc. 4 | 5 | ## How To Use 6 | 7 | To use plugin helpers, call `helpers(*snake_case_symbols)` method to make them available. 8 | 9 | Example: 10 | 11 | ```ruby 12 | helpers :timer, :storage, :compat_parameters 13 | ``` 14 | 15 | It will include `Timer`, `Storage`. and `CompatParameters` plugin helpers. 16 | 17 | ## Built-in Plugin Helpers 18 | 19 | * [`child_process`](api-plugin-helper-child_process.md) 20 | * [`compat_parameters`](api-plugin-helper-compat_parameters.md) 21 | * [`event_emitter`](api-plugin-helper-event_emitter.md) 22 | * [`event_loop`](api-plugin-helper-event_loop.md) 23 | * [`extract`](api-plugin-helper-extract.md) 24 | * [`formatter`](api-plugin-helper-formatter.md) 25 | * [`inject`](api-plugin-helper-inject.md) 26 | * [`metrics`](api-plugin-helper-metrics.md) 27 | * [`parser`](api-plugin-helper-parser.md) 28 | * [`record_accessor`](api-plugin-helper-record_accessor.md) 29 | * [`server`](api-plugin-helper-server.md) 30 | * [`socket`](api-plugin-helper-socket.md) 31 | * [`storage`](api-plugin-helper-storage.md) 32 | * [`thread`](api-plugin-helper-thread.md) 33 | * [`timer`](api-plugin-helper-timer.md) 34 | * [`http_server`](api-plugin-helper-http_server.md) 35 | 36 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 37 | 38 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-child_process.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Child Process 2 | 3 | The `child_process` helper manages the child processes' lifecycle. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput < Output 12 | Fluent::Plugin.register_output('example', self) 13 | 14 | # 1. Load child_process helper 15 | helpers :child_process 16 | 17 | # Omit configure, shutdown and other plugin API 18 | 19 | def start 20 | super 21 | # 2. Execute child process with unique name 22 | child_process_execute(:exec_external_command, 'external_command', immediate: true, mode: [:read]) do 23 | # ... 24 | end 25 | end 26 | end 27 | end 28 | ``` 29 | 30 | The launched child process is managed by the plugin helper which automatically `shutdown`s it. 31 | 32 | ## Methods 33 | 34 | ### `child_process_execute(title, sub_process_name, arguments: nil, subprocess_name: nil, interval: nil, immediate: false, parallel: false, mode: [:read, :write], stderr: :discard, env: {}, unsetenv: false, chdir: nil, internal_encoding: 'utf-8', external_encoding: 'ascii-8bit', scrub: true, replace_string: nil, wait_timeout: nil, on_exit_callback: nil, &block)` 35 | 36 | This method executes `child_process` with the given parameters and routine. 37 | 38 | * `title`: unique symbol value 39 | * `sub_process_name`: sub process name value 40 | * `interval`: Second unit `integer`/`float` value. 41 | * `immediate`: `true`/`false`. Default is `false`. 42 | * `parallel`: `true`/`false`. Default is `false`. 43 | * `mode`: \[`:read`, `:write`\]. Default is \[`:read`, `:write`\]. 44 | * `stderr`: Connect stderr or not. Default is `:discard`. 45 | * `env`: Environment valuables. Default is {}. 46 | * `unsetenv`: `true`/`false`. Default is `false` 47 | * `chdir`: Working directory. Default is `nil`. 48 | * `internal_encoding`: Internal character encoding. Default is `utf-8`. 49 | * `external_encoding`: External character encoding. Default is `ascii-8bit`' 50 | * `scrub`: `true`/`false`. Default is `true`. 51 | * `replace_string`: Replace the invalid code point with the specified character. 52 | 53 | Default is `nil`. 54 | 55 | * `wait_timeout`: Set timeout seconds. Default is `nil`. 56 | * `on_exit_callback`: Set callback function. Default is `nil` 57 | 58 | Code examples: 59 | 60 | ```ruby 61 | # Pass block directly. block is executed with 10 second interval. 62 | child_process_execute(:exec_awesome_command, @command, interval: 10, mode: [:read]) do |io| 63 | # ... 64 | end 65 | 66 | # Pass block with existing method. block is executed once. 67 | child_process_execute(:exec_awesome_command, @command, immediate: true, mode: [:read], &method(:run)) 68 | 69 | def run(io) 70 | # ... 71 | end 72 | ``` 73 | 74 | ## Plugins using `child_process` 75 | 76 | * [`in_exec`](../input/exec.md) 77 | * [`out_exec`](../output/exec.md) 78 | * [`out_exec_filter`](../output/exec_filter.md) 79 | 80 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 81 | 82 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-event_emitter.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Event Emitter 2 | 3 | The `event_emitter` plugin helper introduces the `router` method to the plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class RelabelOutput < Output 12 | Fluent::Plugin.register_output('relabel', self) 13 | helpers :event_emitter 14 | 15 | def multi_workers_ready? 16 | true 17 | end 18 | 19 | def process(tag, es) 20 | router.emit_stream(tag, es) 21 | end 22 | end 23 | end 24 | ``` 25 | 26 | ## Methods 27 | 28 | ### `router` 29 | 30 | This method returns a `Fluent::EventRouter` instance. 31 | 32 | Code example: 33 | 34 | ```ruby 35 | # emit event 36 | router.emit(tag, time, record) 37 | 38 | # emit event stream 39 | router.emit_stream(tag, es) 40 | 41 | # emit error event 42 | begin 43 | # do something 44 | rescue => error 45 | # Route event to @ERROR label or 46 | # log error message when @ERROR label is not defined in configuration 47 | router.emit_error_event(tag, time, record, error) 48 | end 49 | ``` 50 | 51 | ## Plugins using `event_emitter` 52 | 53 | * All input plugins 54 | * All filter plugins 55 | * [`out_exec`](../output/exec.md) 56 | * [`out_relabel`](../output/relabel.md) 57 | * [`out_copy`](../output/copy.md) 58 | * [`out_roundrobin`](../output/roundrobin.md) 59 | 60 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 61 | 62 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-event_loop.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Event Loop 2 | 3 | The `event_loop` plugin helper manages event loops. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load `event_loop_helper` 15 | helpers :event_loop 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def start 20 | super 21 | 22 | # 2. Attach watcher 23 | watcher = Coolio::TCPServer.new(...) 24 | event_loop_attach(watcher) 25 | end 26 | end 27 | end 28 | ``` 29 | 30 | The attached watcher is managed by the plugin. No need of watcher detach code in plugin's `shutdown`. The plugin shutdowns the attached watchers automatically. 31 | 32 | ## Methods 33 | 34 | ### `event_loop_attach(watcher)` 35 | 36 | This method attaches watcher to the event loop. 37 | 38 | * `watcher`: `Coolio::Watcher` instances 39 | 40 | ## Plugins using `event_loop` 41 | 42 | * [`in_http`](../input/http.md) 43 | * [`in_tail`](../input/tail.md) 44 | 45 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 46 | 47 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-extract.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Extract 2 | 3 | The `extract` plugin helper extracts `tag` or `time` from the event `record` according to the configuration. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput > Output 12 | Fluent::Plugin.register_output('Example') 13 | 14 | def process(tag, es) 15 | es.each do |time, record| 16 | new_tag = extract_tag_from_record(record) 17 | new_time = extract_time_from_record(record) 18 | end 19 | # ... 20 | end 21 | end 22 | ``` 23 | 24 | For more details, see [Extract section](../configuration/extract-section.md). 25 | 26 | ## Methods 27 | 28 | ### `extract_tag_from_record(record)` 29 | 30 | This method extracts `tag` from the given record. 31 | 32 | * `record`: event record 33 | 34 | Example: 35 | 36 | ```ruby 37 | new_tag = extract_tag_from_record(record) 38 | ``` 39 | 40 | ### `extract_time_from_record(record)` 41 | 42 | This method extracts `time` from the given record. 43 | 44 | * `record`: event record 45 | 46 | Example: 47 | 48 | ```ruby 49 | new_time = extract_time_from_record(record) 50 | ``` 51 | 52 | ## Plugins using `extract` 53 | 54 | * [`in_exec`](../input/exec.md) 55 | * [`in_tcp`](../input/tcp.md) 56 | * [`in_udp`](../input/udp.md) 57 | * [`out_exec_filter`](../output/exec_filter.md) 58 | 59 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 60 | 61 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-formatter.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Formatter 2 | 3 | The `formatter` plugin helper manages the lifecycle of the formatter plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput < Output 12 | Fluent::Plugin.register_output('example', self) 13 | 14 | # 1. Load formatter helper 15 | helpers :formatter 16 | 17 | # Omit `shutdown` and other plugin APIs 18 | 19 | def configure(conf) 20 | super 21 | 22 | # 2. Call `formatter_create` to create object 23 | @formatter = formatter_create(usage: 'awesome_formatter') 24 | end 25 | 26 | def format(tag, time, record) 27 | # 3. Call `format` method to format `record` 28 | @formatter.format(tag, time, record) 29 | end 30 | end 31 | end 32 | ``` 33 | 34 | For more details, see the following articles: 35 | 36 | * [Formatter Plugin Overview](../formatter/) 37 | * [Writing Formatter Plugins](../plugin-development/api-plugin-formatter.md) 38 | * [Format Section](../configuration/format-section.md) 39 | 40 | ## Methods 41 | 42 | ### `formatter_create(usage: "", type: nil, conf: nil, default_type: nil)` 43 | 44 | * `usage`: unique name required for multiple formatters 45 | * `type`: formatter type 46 | * `conf`: formatter plugin configuration 47 | * `default_type`: default formatter type 48 | 49 | **Examples**: 50 | 51 | ```ruby 52 | def configure(conf) 53 | super 54 | # Create formatter plugin instance using section in `fluent.conf` during configure phase 55 | @formatter = formatter_create 56 | end 57 | 58 | def format(tag, time, record) 59 | @formatter.format(tag, time, record) 60 | end 61 | ``` 62 | 63 | JSON formatter example: 64 | 65 | ```ruby 66 | # Create JSON formatter 67 | def configure(conf) 68 | super 69 | @json_formatter = formatter_create(usage: 'formatter_in_example_json', type: 'json') 70 | end 71 | 72 | def format(tag, time, record) 73 | @json_formatter.format(tag, time, record) 74 | end 75 | ``` 76 | 77 | MsgPack formatter example: 78 | 79 | ```ruby 80 | # Create MsgPack formatter 81 | def configure(conf) 82 | super 83 | @msgpack_formatter = formatter_create(usage: 'formatter_in_example_msgpack', type: 'msgpack') 84 | end 85 | 86 | def format(tag, time, record) 87 | @msgpack_formatter.format(tag, time, record) 88 | end 89 | ``` 90 | 91 | ## Plugins using `formatter` 92 | 93 | * [`filter_stdout`](../filter/stdout.md) 94 | * [`out_stdout`](../output/stdout.md) 95 | * [`out_exec`](../output/exec.md) 96 | * [`out_exec_filter`](../output/exec_filter.md) 97 | * [`out_file`](../output/file.md) 98 | * [`out_s3`](../output/s3.md) 99 | 100 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 101 | 102 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-inject.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Inject 2 | 3 | The `inject` plugin helper injects values into events according to the configuration. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/filter' 9 | 10 | module Fluent::Plugin 11 | class ExampleFilter < Filter 12 | Fluent::Plugin.register_filter('example', self) 13 | 14 | # 1. Load inject helper 15 | helpers :inject 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def filter(tag, time, record) 20 | # 2. Inject values into `record` 21 | new_record = inject_values_to_record(tag, time, record) 22 | # edit new_record ... 23 | new_record 24 | end 25 | end 26 | end 27 | ``` 28 | 29 | For more details, see [Inject section](../configuration/inject-section.md). 30 | 31 | ## Methods 32 | 33 | ### `inject_values_to_record(tag, time, record)` 34 | 35 | This method injects values to the given record and returns the new record. 36 | 37 | * `tag`: the tag of the event 38 | * `time`: event timestamp 39 | * `record`: event record 40 | 41 | Example: 42 | 43 | ```ruby 44 | def filter(tag, time, record) 45 | new_record = inject_values_to_record(tag, time, record) 46 | # edit new_record ... 47 | new_record 48 | end 49 | ``` 50 | 51 | ### `inject_values_to_event_stream(tag, es)` 52 | 53 | This method injects values into the given event stream and returns the new event stream. 54 | 55 | * `tag`: the tag of the event 56 | * `es`: event stream 57 | 58 | Example: 59 | 60 | ```ruby 61 | def process(tag, es) 62 | new_es = inject_values_to_event_stream(tag, es) 63 | # do something using new_es 64 | end 65 | ``` 66 | 67 | ## Plugins using `inject` 68 | 69 | * [`filter_stdout`](../filter/stdout.md) 70 | * [`out_exec`](../output/exec.md) 71 | * [`out_exec_filter`](../output/exec_filter.md) 72 | * [`out_file`](../output/file.md) 73 | * [`out_stdout`](../output/stdout.md) 74 | 75 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 76 | 77 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-metrics.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Metrics 2 | 3 | The `metrics` plugin helper manages the metrics values in plugins. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load metrics helper 15 | helpers :metrics 16 | 17 | def configure(conf) 18 | super 19 | 20 | # 2. Create parser plugin instance 21 | @metrics = metrics_create(namespace: "fluentd", subsystem: "input", name: "example", help_text: "Example metrics") 22 | end 23 | 24 | def start 25 | super 26 | 27 | # 3. Increase metrics value 28 | @metrics.inc 29 | 30 | end 31 | 32 | def statistics 33 | stats = super 34 | 35 | # 4. Retrieve metrics value 36 | stats = { 37 | 'input' => stats["input"].merge({ 'example' => @metrics.get }) 38 | } 39 | stats 40 | end 41 | end 42 | ``` 43 | 44 | For more details, see the following articles: 45 | 46 | * [Metrics Plugins Overview](../metrics/) 47 | 48 | ## Methods 49 | 50 | ### `metrics_create(namespace: "fluentd", subsystem: "metrics", name:, help_text:, labels: {}, prefer_gauge: false)` 51 | 52 | This method creates a metrics instance. 53 | 54 | * `namespace`: The namespace for the metrics. 55 | * `subsystem`: The names that represent specific functions or components. 56 | * `name`: The metrics name. 57 | * `help_text`: The description for metrics. 58 | * `labels`: The key/value pair for metrics labels. 59 | * `prefer_gauge`: Use gauge instead of counter for the metrics if `true`. 60 | 61 | Since 1.19.0, `metrics_create` method generates a getter method with the specified `name` on the calling instance. 62 | 63 | ## Plugins using `metrics` 64 | 65 | * [`buffer`](../buffer/) 66 | * [`filter`](../filter/) 67 | * [`input`](../input/) 68 | * [`output`](../output/) 69 | 70 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 71 | 72 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-parser.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Parser 2 | 3 | The `parser` plugin helper manages the lifecycle of the parser plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load parser helper 15 | helpers :parser 16 | 17 | # Omit `shutdown` and other plugin APIs 18 | 19 | def configure(conf) 20 | super 21 | 22 | # 2. Create parser plugin instance 23 | @parser = parser_create 24 | end 25 | 26 | def start 27 | super 28 | 29 | # Use parser helper in combination usually with other plugin helpers 30 | timer_execute(:example_timer, 10) do 31 | read_raw_data do |text| 32 | # 3. Call `@parser.parse(text)` to parse raw data 33 | @parser.parse(text) do |time, record| 34 | router.emit(tag, time, record) 35 | end 36 | end 37 | end 38 | end 39 | end 40 | end 41 | ``` 42 | 43 | For more details, see the following articles: 44 | 45 | * [Parser Plugin Overview](../parser/) 46 | * [Writing Parser Plugins](../plugin-development/api-plugin-parser.md) 47 | * [Parser Plugin](../configuration/parse-section.md) 48 | 49 | ## Methods 50 | 51 | ### `parser_create(usage: "", type: nil, conf: nil, default_type: nil)` 52 | 53 | This method creates a parser plugin instance with the given parameters. 54 | 55 | * `usage`: unique name required for multiple parsers 56 | * `type`: parser type 57 | * `conf`: parser plugin configuration 58 | * `default_type`: default parser type 59 | 60 | **Examples** 61 | 62 | ```ruby 63 | # Create parser plugin instance using section in fluent.conf during configure phase 64 | @parser = parser_create 65 | @parser.parse(text) do |time, record| 66 | # ... 67 | end 68 | 69 | # Create JSON parser 70 | @json_parser = parser_create(usage: 'parser_in_example_json', type: 'json') 71 | @json_parser.parse(json) do |time, record| 72 | # ... 73 | end 74 | 75 | # Create MessagePack parser 76 | @msgpack_parser = parser_create(usage: 'parser_in_example_msgpack', type: 'msgpack') 77 | @msgpack_parser.parse(msgpack_binary) do |time, record| 78 | # ... 79 | end 80 | ``` 81 | 82 | ## Plugins using `parser` 83 | 84 | * [`filter_parser`](../filter/parser.md) 85 | * [`in_exec`](../input/exec.md) 86 | * [`in_http`](../input/http.md) 87 | * [`in_syslog`](../input/syslog.md) 88 | * [`in_tail`](../input/tail.md) 89 | * [`in_tcp`](../input/tcp.md) 90 | * [`in_udp`](../input/udp.md) 91 | * [`out_exec_filter`](../output/exec_filter.md) 92 | 93 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 94 | 95 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-record_accessor.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Record Accessor 2 | 3 | The `record_accessor` plugin helper provides unified access to the event record. It uses `jsonpath` like syntax for the target field. With this helper, you can easily access/delete a nested field in the plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/filter' 9 | 10 | module Fluent::Plugin 11 | class ExampleFilter < Filter 12 | Fluent::Plugin.register_filter('example', self) 13 | 14 | # 1. Load record_accessor helper 15 | helpers :record_accessor 16 | 17 | def configure(conf) 18 | super 19 | 20 | # 2. Call `record_accessor_create` to create object 21 | @accessor = record_accessor_create('$.user.name') 22 | end 23 | 24 | # Omit `super`, `shutdown` and other plugin APIs 25 | 26 | def filter(tag, time, record) 27 | # 3. Call `call` method to get value 28 | value = @accessor.call(record) # With `$.user.name`, access to record["user"]["name"] 29 | # ... 30 | end 31 | end 32 | end 33 | ``` 34 | 35 | ## Syntax 36 | 37 | * dot notation: `$.` is the starting parameter. Chain fields with dots `.`. 38 | 39 | For example: 40 | 41 | `$.event.level` for `record["event"]["level"]` 42 | 43 | `$.key1[0].key2` for `record["key1"][0]["key2"]` 44 | 45 | * bracket notation: `$[` starting parameter. Chain fields with `[]`. 46 | 47 | Useful for special characters, `.`, whitespace, etc. 48 | 49 | `$['dot.key'][0]['space key']` for `record["dot.key"][0]["space key"]` 50 | 51 | If you set non `$.` or `$[` starting value, e.g. `key log`, it is the same as `record["log"]`. So, using `record_accessor` does not break the existing plugin behavior. 52 | 53 | ## Methods 54 | 55 | ### `record_accessor_create(param)` 56 | 57 | This method returns the accessor object of the event record. 58 | 59 | The `param` is a `String`. 60 | 61 | See the "Syntax" section for more details. 62 | 63 | ```ruby 64 | record_accessor_create("log") 65 | record_accessor_create("$.key1.key2") 66 | record_accessor_create("$['key1'][0]['key2']") 67 | ``` 68 | 69 | After creating an object, call `call`/`delete`/`set` method with the record object. 70 | 71 | ```ruby 72 | accessor.call(record) # get record field 73 | accessor.delete(record) # delete record field 74 | accessor.set(record, value) # set new value to record field 75 | ``` 76 | 77 | NOTE: `set` method is supported since v1.10.3 78 | 79 | ## Plugins using `record_accessor` 80 | 81 | * [`filter_grep`](../filter/grep.md) 82 | * [`filter_parser`](../filter/parser.md) 83 | * [`filter_record_transformer`](../filter/record_transformer.md) 84 | 85 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 86 | 87 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-thread.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Thread 2 | 3 | The `thread` plugin helper manages threads and these threads are integrated with plugins. No need manual run or shutdown in the plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput < Output 12 | Fluent::Plugin.register_output('example', self) 13 | 14 | # 1. Load thread helper 15 | helpers :thread 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def start 20 | super 21 | 22 | # 2. Create and run thread with unique name. 23 | thread_create(:example_thread_run, &method(:run)) 24 | end 25 | 26 | def run 27 | # ... 28 | end 29 | end 30 | end 31 | ``` 32 | 33 | The launched thread is managed by the plugin helper. No need of thread shutdown code in plugin's `shutdown` method. It shutdowns the launched threads automatically. 34 | 35 | ## Methods 36 | 37 | ### `thread_create(title)` 38 | 39 | This method creates a thread and runs it with the given routine. 40 | 41 | `title` must be unique. 42 | 43 | ```ruby 44 | # Pass block directly 45 | thread_create(:example_plugin_main) { 46 | # ... 47 | } 48 | 49 | # Pass method object with existing method 50 | thread_create(:foo_plugin_body, &method(:run)) 51 | def run 52 | # ... 53 | end 54 | ``` 55 | 56 | ### `thread_current_running?` 57 | 58 | Checks whether the current thread is running or not. This method is available in the running block. 59 | 60 | ```ruby 61 | thread_create(:example_plugin_main) { 62 | while thread_current_running? 63 | # ... 64 | end 65 | } 66 | ``` 67 | 68 | ## Plugins using `thread` 69 | 70 | * [`out_forward`](../output/forward.md) 71 | * [`in_monitor_agent`](../input/monitor_agent.md) 72 | * [`in_sample`](../input/sample.md) 73 | 74 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 75 | 76 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-timer.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Timer 2 | 3 | The `timer` plugin helper manages event timers. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load timer helper 15 | helpers :timer 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def start 20 | super 21 | 22 | # 2. Execute timer with unique name and second unit interval 23 | timer_execute(:example_timer, 10) { 24 | # ... 25 | } 26 | end 27 | end 28 | end 29 | ``` 30 | 31 | The launched timer is managed by the plugin helper. No need of timer shutdown code in plugin's `shutdown` method. The plugin shutdowns the launched timers automatically. 32 | 33 | ## Methods 34 | 35 | ### `timer_execute(title, interval, repeat: true, &block)` 36 | 37 | This method executes the timer with the given parameters and routine. 38 | 39 | * `title`: unique symbol value 40 | * `interval`: second unit `integer`/`float` value. 41 | * `repeat`: `true`/`false` \(default: `true`\). If `false`, timer is one-shot. 42 | 43 | Code examples: 44 | 45 | ```ruby 46 | # Pass block directly. block is executed in 10 second interval. 47 | timer_execute(:example_timer, 10) { 48 | # ... 49 | } 50 | 51 | # Pass block with existing method. block is executed in 5 second and one-shot. 52 | timer_execute(:example_timer_run, 5s, repeat: false, &method(:run)) 53 | def run 54 | # ... 55 | end 56 | ``` 57 | 58 | ## Plugins using `timer` 59 | 60 | * [`out_forward`](../output/forward.md) 61 | * [`in_monitor_agent`](../input/monitor_agent.md) 62 | * [`in_sample`](../input/sample.md) 63 | 64 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 65 | 66 | -------------------------------------------------------------------------------- /plugins/input/dummy.md: -------------------------------------------------------------------------------- 1 | # `dummy` Input Plugin 2 | 3 | `in_dummy` has been renamed `in_sample` since Fluentd v1.11.12. 4 | See [`sample` input plugin](/input/sample.md) article for more details. 5 | 6 | You can keep to use following configuration in v1: 7 | 8 | ```text 9 | 10 | @type dummy 11 | dummy {"hello":"world"} 12 | 13 | ``` 14 | 15 | Fluentd v2 will remove this plugin name. 16 | 17 | 18 | ------------------------------------------------------------------------ 19 | 20 | If this article is incorrect or outdated, or omits critical information, please 21 | [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). 22 | [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native 23 | Computing Foundation (CNCF)](https://cncf.io/). All components are available 24 | under the Apache 2 License. 25 | -------------------------------------------------------------------------------- /quickstart/README.md: -------------------------------------------------------------------------------- 1 | # Overview 2 | 3 | Let's get started with **Fluentd**! **Fluentd** is a fully free and fully open-source log collector that instantly enables you to have a '**Log Everything**' architecture with [125+ types of systems](https://www.fluentd.org/plugins). 4 | 5 |  6 | 7 | Fluentd treats logs as JSON, a popular machine-readable format. It is written primarily in C with a thin-Ruby wrapper that gives users flexibility. 8 | 9 | Fluentd's scalability has been proven in the field: its largest user currently collects logs from **50,000+ servers**. 10 | 11 | ## Step 1: Installing Fluentd 12 | 13 | You can install Fluentd as a package, Ruby gem, or from source. 14 | 15 | Please see the following article for details. 16 | 17 | * [Installation](../installation/) 18 | 19 | ## Step 2: Use Cases 20 | 21 | The following articles cover the typical use cases of Fluentd: 22 | 23 | * Use Cases 24 | * [Data Search like Splunk](../how-to-guides/free-alternative-to-splunk-by-fluentd.md) 25 | * [Data Filtering and Alerting](../how-to-guides/splunk-like-grep-and-alert-email.md) 26 | * [Data Analytics with Treasure Data](../how-to-guides/http-to-td.md) 27 | * [Data Collection to MongoDB](../how-to-guides/apache-to-mongodb.md) 28 | * [Data Collection to HDFS](../how-to-guides/http-to-hdfs.md) 29 | * [Data Archiving to Amazon S3](../how-to-guides/apache-to-s3.md) 30 | * Basic Configuration 31 | * [Config File](../configuration/config-file.md) 32 | * Application Logs 33 | * [Ruby](../language-bindings/ruby.md), [Java](../language-bindings/java.md), [Python](../language-bindings/python.md), [PHP](../language-bindings/php.md), 34 | 35 | [Perl](../language-bindings/perl.md), [Node.js](../language-bindings/nodejs.md), [Scala](../language-bindings/scala.md) 36 | * Happy Users :\) 37 | * [Users](https://www.fluentd.org/testimonials) 38 | 39 | ## Step 3: Learn More 40 | 41 | The following articles will provide detailed information about Fluentd: 42 | 43 | * [Architecture Overview](https://www.fluentd.org/architecture) 44 | * [Lifecycle of a Fluentd Event](life-of-a-fluentd-event.md) 45 | * Plugin Overview 46 | * [Input Plugins](../input/) 47 | * [Output Plugins](../output/) 48 | * [Buffer Plugins](../buffer/) 49 | * [Filter Plugins](../filter/) 50 | * [Parser Plugins](../parser/) 51 | * [Formatter Plugins](../formatter/) 52 | * [High Availability Configuration](../deployment/high-availability.md) 53 | * [FAQ](faq.md) 54 | 55 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0) 56 | 57 | -------------------------------------------------------------------------------- /quickstart/logo.md: -------------------------------------------------------------------------------- 1 | # Logo 2 | 3 | Feel free to use these logos on your slides, blog posts, etc. 4 | 5 | ## Square 6 | 7 | [](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_square.png) 8 | 9 | [\[PNG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_square.png) [\[SVG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_square.svg) 10 | 11 | ## Horizontal 12 | 13 | [](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_horizontal.png) 14 | 15 | [\[PNG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_horizontal.png) [\[SVG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_horizontal.svg) 16 | 17 | ## Icon 18 | 19 | [](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_icon.png) 20 | 21 | [\[PNG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_icon.png) [\[SVG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_icon.svg) 22 | 23 | ## Links 24 | 25 | * [Fluentd.ai](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd.ai) 26 | * [Complete List of Logos](https://github.com/fluent/fluentd-docs-gitbook/tree/1.0/images/logo) 27 | 28 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0) 29 | 30 | -------------------------------------------------------------------------------- /quickstart/support.md: -------------------------------------------------------------------------------- 1 | # Support 2 | 3 | ## Community Forum \(GitHub Discussions\) 4 | 5 | Got a question? Have some cool Fluentd-related projects? The GitHub discussion is the place to be: 6 | 7 | * [Fluentd Discussions](https://github.com/fluent/fluentd/discussions) 8 | 9 | ## Slack 10 | 11 | Join our Slack channel and talk with Fluentd developers and users directly, now. 12 | 13 | * [Slack](https://slack.fluentd.org/) 14 | 15 | ## Enterprise Support 16 | 17 | If you need support for running in production from enterprise companies you can find providers at the following link 18 | 19 | * [Enterprise Support providers](https://www.fluentd.org/enterprise_services) 20 | 21 | ## Community Resources 22 | 23 | Here are other community resources: 24 | 25 | * [StackOverflow](https://stackoverflow.com/questions/tagged/fluentd?sort=newest) 26 | * [Twitter @fluentd](https://www.twitter.com/fluentd) 27 | * [Facebook Page](https://www.facebook.com/pages/Fluentd-Log-Everything-in-JSON/196064987183037) 28 | 29 | ## Change Logs 30 | 31 | If you are interested in what's new in each version, please refer to the following links: 32 | 33 | * [Fluentd's ChangeLog](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md). 34 | * [fluent-package's ChangeLog](https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG.md) (Successor of td-agent package) 35 | * [td-agent's ChangeLog](https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG-v4.md) (End of Life in Dec, 2023) 36 | 37 | {% hint style='danger' %} 38 | The series of td-agent had already reached End of Life (EOL). td-agent should not be newly installed because no support, no new release and no security updates anymore. 39 | Use fluent-package instead. See [Upgrade to fluent-package v5](https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5) for migration. 40 | {% endhint %} 41 | 42 | ## Development 43 | 44 | If you find bugs, please report to the GitHub Issue. 45 | 46 | * [Bug/Feature Tracker](https://github.com/fluent/fluentd/issues) 47 | 48 | The source code is available at GitHub: 49 | 50 | * [Fluentd](https://github.com/fluent/fluentd/) 51 | 52 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0) 53 | 54 | -------------------------------------------------------------------------------- /service_discovery/README.md: -------------------------------------------------------------------------------- 1 | # Service Discovery Plugins 2 | 3 | Fluentd has nine \(9\) types of plugins: 4 | 5 | * [Input](../input/) 6 | * [Parser](../parser/) 7 | * [Filter](../filter/) 8 | * [Output](../output/) 9 | * [Formatter](../formatter/) 10 | * [Storage](../storage/) 11 | * [Service Discovery](./) 12 | * [Buffer](../buffer/) 13 | * [Metrics](../metrics/) 14 | 15 | This article gives an overview of the Service Discovery Plugin. 16 | 17 | ## Overview 18 | 19 | Some plugins support `` \(e.g. [`out_forward`](../output/forward.md)\). Sometimes, the service discovery for an output plugin does not meet one's needs. Fluentd has a pluggable system called Service Discovery that lets the user extend and reuse custom output service discovery. 20 | 21 | ## How To Use 22 | 23 | Here is a simple example to update target by reading file \(`/etc/fluentd/sd.yaml`\) with `out_forward` and `service_discovery`: 24 | 25 | ```text 26 | 27 | @type forward 28 | 29 | 30 | @type file 31 | path "/etc/fluentd/sd.yaml" 32 | 33 | 34 | ``` 35 | 36 | ## List of Built-in Service Discovery 37 | 38 | * [`static`](static.md) 39 | * [`file`](file.md) 40 | * [`srv`](srv.md) 41 | 42 | ## List of Core Output Plugins with Service Discovery support 43 | 44 | * [`out_forward`](../output/forward.md) 45 | 46 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 47 | 48 | -------------------------------------------------------------------------------- /service_discovery/file.md: -------------------------------------------------------------------------------- 1 | # file 2 | 3 | The `file` service discovery plugin updates the targets by reading the local file. YAML and JSON are the allowed file formats. 4 | 5 | ## Example Configuration 6 | 7 | Here is an example with `out_forward` updating targets by sending data: 8 | 9 | ```text 10 | 11 | @type forward 12 | 13 | 14 | @type file 15 | path "/etc/fluentd/sd.yaml" 16 | 17 |
[^ ]*) (?[^ ]*)(?: "(?[^\"]*)" "(?[^\"]*)"(?:\s+(?[^ ]+))?)?$/ 15 | time_format %d/%b/%Y:%H:%M:%S %z 16 | ``` 17 | 18 | `remote`, `user`, `method`, `path`, `code`, `size`, `referer`, `agent` and `http_x_forwarded_for` are included in the event record. `time` is used for the event time. 19 | 20 | ## Example 21 | 22 | This incoming event: 23 | 24 | ```text 25 | 127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" - 26 | ``` 27 | 28 | is parsed as: 29 | 30 | ```text 31 | time: 32 | 1362020400 (28/Feb/2013:12:00:00 +0900) 33 | 34 | record: 35 | { 36 | "remote" : "127.0.0.1", 37 | "host" : "192.168.0.1", 38 | "user" : "-", 39 | "method" : "GET", 40 | "path" : "/", 41 | "code" : "200", 42 | "size" : "777", 43 | "referer" : "-", 44 | "agent" : "Opera/12.0", 45 | "http_x_forwarded_for": "-" 46 | } 47 | ``` 48 | 49 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 50 | 51 | -------------------------------------------------------------------------------- /parser/none.md: -------------------------------------------------------------------------------- 1 | # none 2 | 3 | The `none` parser plugin parses the line as-is with the single field. This format is to defer the parsing/structuring of the data. 4 | 5 | This parser is often used in conjunction with [`single_value`](../formatter/single_value.md) format in output plugin. 6 | 7 | ## Parameters 8 | 9 | See [Parse Section Configurations](../configuration/parse-section.md). 10 | 11 | ### `message_key` 12 | 13 | | type | default | version | 14 | | :--- | :--- | :--- | 15 | | string | message | 0.14.0 | 16 | 17 | Specifies the field name to contain logs. 18 | 19 | ## Example 20 | 21 | This incoming event: 22 | 23 | ```text 24 | Hello world. I am a line of log! 25 | ``` 26 | 27 | is parsed as: 28 | 29 | ```text 30 | time: 31 | 1362020400 (current time) 32 | 33 | record: 34 | {"message":"Hello world. I am a line of log!"} 35 | ``` 36 | 37 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 38 | 39 | -------------------------------------------------------------------------------- /parser/tsv.md: -------------------------------------------------------------------------------- 1 | # tsv 2 | 3 | The `tsv` parser plugin parses the TSV format. 4 | 5 | ## Parameters 6 | 7 | See [Parse Section Configurations](../configuration/parse-section.md). 8 | 9 | ### `keys` 10 | 11 | | type | default | version | 12 | | :--- | :--- | :--- | 13 | | array of string | required parameter | 0.14.9 | 14 | 15 | The array of names for fields on each line. 16 | 17 | ### `delimiter` 18 | 19 | | type | default | version | 20 | | :--- | :--- | :--- | 21 | | string | `\t` | 0.14.0 | 22 | 23 | The delimiter \(character or string\) separating TSV values. 24 | 25 | ## Example 26 | 27 | With this configuration: 28 | 29 | ```text 30 | 31 | @type tsv 32 | keys time,host,req_id,user 33 | time_key time 34 | 35 | ``` 36 | 37 | This incoming event: 38 | 39 | ```text 40 | 2013/02/28 12:00:00\t192.168.0.1\t111\t- 41 | ``` 42 | 43 | is parsed as: 44 | 45 | ```text 46 | time: 47 | 1362020400 (2013/02/28/ 12:00:00) 48 | 49 | record: 50 | { 51 | "host" : "192.168.0.1", 52 | "req_id" : "111", 53 | "user" : "-" 54 | } 55 | ``` 56 | 57 | If you set `null_value_pattern '-'` in the configuration, `user` field becomes `nil` instead of `"-"`. 58 | 59 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 60 | 61 | -------------------------------------------------------------------------------- /plugin-development/api-plugin-buffer.md: -------------------------------------------------------------------------------- 1 | # How to Write Buffer Plugin 2 | 3 | TODO: Write 4 | 5 | ## Methods 6 | 7 | ### evacuate_chunk 8 | 9 | You can override this method to add feature to evacuate chunks before clearing the queue when reaching the retry limit. 10 | See [Buffer - Handling Successive Failures](../buffer/README.md#handling-successive-failures) for details. 11 | 12 | ```ruby 13 | def evacuate_chunk(chunk) 14 | unless chunk.is_a?(Fluent::Plugin::Buffer::FileChunk) 15 | raise ArgumentError, "The chunk must be FileChunk, but it was #{chunk.class}." 16 | end 17 | 18 | backup_dir = File.join(backup_base_dir, 'buffer', safe_owner_id) 19 | FileUtils.mkdir_p(backup_dir, mode: system_config.dir_permission || Fluent::DEFAULT_DIR_PERMISSION) unless Dir.exist?(backup_dir) 20 | 21 | FileUtils.copy([chunk.path, chunk.meta_path], backup_dir) 22 | log.warn "chunk files are evacuated to #{backup_dir}.", chunk_id: dump_unique_id_hex(chunk.unique_id) 23 | rescue => e 24 | log.error "unexpected error while evacuating chunk files.", error: e 25 | end 26 | ``` 27 | 28 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 29 | 30 | -------------------------------------------------------------------------------- /plugin-development/api-plugin-service_discovery.md: -------------------------------------------------------------------------------- 1 | # How to Write Service Discovery Plugin 2 | 3 | TODO: Write 4 | 5 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 6 | 7 | -------------------------------------------------------------------------------- /plugin-development/api-plugin-storage.md: -------------------------------------------------------------------------------- 1 | # How to Write Storage Plugin 2 | 3 | TODO: Write 4 | 5 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 6 | 7 | -------------------------------------------------------------------------------- /plugin-helper-overview/README.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper API 2 | 3 | Fluentd provides plugin helpers to encapsulate and make commonly implemented features available such as timer, threading, formatting, parsing, ensuring configuration syntax's backward compatibility, etc. 4 | 5 | ## How To Use 6 | 7 | To use plugin helpers, call `helpers(*snake_case_symbols)` method to make them available. 8 | 9 | Example: 10 | 11 | ```ruby 12 | helpers :timer, :storage, :compat_parameters 13 | ``` 14 | 15 | It will include `Timer`, `Storage`. and `CompatParameters` plugin helpers. 16 | 17 | ## Built-in Plugin Helpers 18 | 19 | * [`child_process`](api-plugin-helper-child_process.md) 20 | * [`compat_parameters`](api-plugin-helper-compat_parameters.md) 21 | * [`event_emitter`](api-plugin-helper-event_emitter.md) 22 | * [`event_loop`](api-plugin-helper-event_loop.md) 23 | * [`extract`](api-plugin-helper-extract.md) 24 | * [`formatter`](api-plugin-helper-formatter.md) 25 | * [`inject`](api-plugin-helper-inject.md) 26 | * [`metrics`](api-plugin-helper-metrics.md) 27 | * [`parser`](api-plugin-helper-parser.md) 28 | * [`record_accessor`](api-plugin-helper-record_accessor.md) 29 | * [`server`](api-plugin-helper-server.md) 30 | * [`socket`](api-plugin-helper-socket.md) 31 | * [`storage`](api-plugin-helper-storage.md) 32 | * [`thread`](api-plugin-helper-thread.md) 33 | * [`timer`](api-plugin-helper-timer.md) 34 | * [`http_server`](api-plugin-helper-http_server.md) 35 | 36 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 37 | 38 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-child_process.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Child Process 2 | 3 | The `child_process` helper manages the child processes' lifecycle. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput < Output 12 | Fluent::Plugin.register_output('example', self) 13 | 14 | # 1. Load child_process helper 15 | helpers :child_process 16 | 17 | # Omit configure, shutdown and other plugin API 18 | 19 | def start 20 | super 21 | # 2. Execute child process with unique name 22 | child_process_execute(:exec_external_command, 'external_command', immediate: true, mode: [:read]) do 23 | # ... 24 | end 25 | end 26 | end 27 | end 28 | ``` 29 | 30 | The launched child process is managed by the plugin helper which automatically `shutdown`s it. 31 | 32 | ## Methods 33 | 34 | ### `child_process_execute(title, sub_process_name, arguments: nil, subprocess_name: nil, interval: nil, immediate: false, parallel: false, mode: [:read, :write], stderr: :discard, env: {}, unsetenv: false, chdir: nil, internal_encoding: 'utf-8', external_encoding: 'ascii-8bit', scrub: true, replace_string: nil, wait_timeout: nil, on_exit_callback: nil, &block)` 35 | 36 | This method executes `child_process` with the given parameters and routine. 37 | 38 | * `title`: unique symbol value 39 | * `sub_process_name`: sub process name value 40 | * `interval`: Second unit `integer`/`float` value. 41 | * `immediate`: `true`/`false`. Default is `false`. 42 | * `parallel`: `true`/`false`. Default is `false`. 43 | * `mode`: \[`:read`, `:write`\]. Default is \[`:read`, `:write`\]. 44 | * `stderr`: Connect stderr or not. Default is `:discard`. 45 | * `env`: Environment valuables. Default is {}. 46 | * `unsetenv`: `true`/`false`. Default is `false` 47 | * `chdir`: Working directory. Default is `nil`. 48 | * `internal_encoding`: Internal character encoding. Default is `utf-8`. 49 | * `external_encoding`: External character encoding. Default is `ascii-8bit`' 50 | * `scrub`: `true`/`false`. Default is `true`. 51 | * `replace_string`: Replace the invalid code point with the specified character. 52 | 53 | Default is `nil`. 54 | 55 | * `wait_timeout`: Set timeout seconds. Default is `nil`. 56 | * `on_exit_callback`: Set callback function. Default is `nil` 57 | 58 | Code examples: 59 | 60 | ```ruby 61 | # Pass block directly. block is executed with 10 second interval. 62 | child_process_execute(:exec_awesome_command, @command, interval: 10, mode: [:read]) do |io| 63 | # ... 64 | end 65 | 66 | # Pass block with existing method. block is executed once. 67 | child_process_execute(:exec_awesome_command, @command, immediate: true, mode: [:read], &method(:run)) 68 | 69 | def run(io) 70 | # ... 71 | end 72 | ``` 73 | 74 | ## Plugins using `child_process` 75 | 76 | * [`in_exec`](../input/exec.md) 77 | * [`out_exec`](../output/exec.md) 78 | * [`out_exec_filter`](../output/exec_filter.md) 79 | 80 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 81 | 82 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-event_emitter.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Event Emitter 2 | 3 | The `event_emitter` plugin helper introduces the `router` method to the plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class RelabelOutput < Output 12 | Fluent::Plugin.register_output('relabel', self) 13 | helpers :event_emitter 14 | 15 | def multi_workers_ready? 16 | true 17 | end 18 | 19 | def process(tag, es) 20 | router.emit_stream(tag, es) 21 | end 22 | end 23 | end 24 | ``` 25 | 26 | ## Methods 27 | 28 | ### `router` 29 | 30 | This method returns a `Fluent::EventRouter` instance. 31 | 32 | Code example: 33 | 34 | ```ruby 35 | # emit event 36 | router.emit(tag, time, record) 37 | 38 | # emit event stream 39 | router.emit_stream(tag, es) 40 | 41 | # emit error event 42 | begin 43 | # do something 44 | rescue => error 45 | # Route event to @ERROR label or 46 | # log error message when @ERROR label is not defined in configuration 47 | router.emit_error_event(tag, time, record, error) 48 | end 49 | ``` 50 | 51 | ## Plugins using `event_emitter` 52 | 53 | * All input plugins 54 | * All filter plugins 55 | * [`out_exec`](../output/exec.md) 56 | * [`out_relabel`](../output/relabel.md) 57 | * [`out_copy`](../output/copy.md) 58 | * [`out_roundrobin`](../output/roundrobin.md) 59 | 60 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 61 | 62 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-event_loop.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Event Loop 2 | 3 | The `event_loop` plugin helper manages event loops. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load `event_loop_helper` 15 | helpers :event_loop 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def start 20 | super 21 | 22 | # 2. Attach watcher 23 | watcher = Coolio::TCPServer.new(...) 24 | event_loop_attach(watcher) 25 | end 26 | end 27 | end 28 | ``` 29 | 30 | The attached watcher is managed by the plugin. No need of watcher detach code in plugin's `shutdown`. The plugin shutdowns the attached watchers automatically. 31 | 32 | ## Methods 33 | 34 | ### `event_loop_attach(watcher)` 35 | 36 | This method attaches watcher to the event loop. 37 | 38 | * `watcher`: `Coolio::Watcher` instances 39 | 40 | ## Plugins using `event_loop` 41 | 42 | * [`in_http`](../input/http.md) 43 | * [`in_tail`](../input/tail.md) 44 | 45 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 46 | 47 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-extract.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Extract 2 | 3 | The `extract` plugin helper extracts `tag` or `time` from the event `record` according to the configuration. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput > Output 12 | Fluent::Plugin.register_output('Example') 13 | 14 | def process(tag, es) 15 | es.each do |time, record| 16 | new_tag = extract_tag_from_record(record) 17 | new_time = extract_time_from_record(record) 18 | end 19 | # ... 20 | end 21 | end 22 | ``` 23 | 24 | For more details, see [Extract section](../configuration/extract-section.md). 25 | 26 | ## Methods 27 | 28 | ### `extract_tag_from_record(record)` 29 | 30 | This method extracts `tag` from the given record. 31 | 32 | * `record`: event record 33 | 34 | Example: 35 | 36 | ```ruby 37 | new_tag = extract_tag_from_record(record) 38 | ``` 39 | 40 | ### `extract_time_from_record(record)` 41 | 42 | This method extracts `time` from the given record. 43 | 44 | * `record`: event record 45 | 46 | Example: 47 | 48 | ```ruby 49 | new_time = extract_time_from_record(record) 50 | ``` 51 | 52 | ## Plugins using `extract` 53 | 54 | * [`in_exec`](../input/exec.md) 55 | * [`in_tcp`](../input/tcp.md) 56 | * [`in_udp`](../input/udp.md) 57 | * [`out_exec_filter`](../output/exec_filter.md) 58 | 59 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 60 | 61 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-formatter.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Formatter 2 | 3 | The `formatter` plugin helper manages the lifecycle of the formatter plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput < Output 12 | Fluent::Plugin.register_output('example', self) 13 | 14 | # 1. Load formatter helper 15 | helpers :formatter 16 | 17 | # Omit `shutdown` and other plugin APIs 18 | 19 | def configure(conf) 20 | super 21 | 22 | # 2. Call `formatter_create` to create object 23 | @formatter = formatter_create(usage: 'awesome_formatter') 24 | end 25 | 26 | def format(tag, time, record) 27 | # 3. Call `format` method to format `record` 28 | @formatter.format(tag, time, record) 29 | end 30 | end 31 | end 32 | ``` 33 | 34 | For more details, see the following articles: 35 | 36 | * [Formatter Plugin Overview](../formatter/) 37 | * [Writing Formatter Plugins](../plugin-development/api-plugin-formatter.md) 38 | * [Format Section](../configuration/format-section.md) 39 | 40 | ## Methods 41 | 42 | ### `formatter_create(usage: "", type: nil, conf: nil, default_type: nil)` 43 | 44 | * `usage`: unique name required for multiple formatters 45 | * `type`: formatter type 46 | * `conf`: formatter plugin configuration 47 | * `default_type`: default formatter type 48 | 49 | **Examples**: 50 | 51 | ```ruby 52 | def configure(conf) 53 | super 54 | # Create formatter plugin instance using section in `fluent.conf` during configure phase 55 | @formatter = formatter_create 56 | end 57 | 58 | def format(tag, time, record) 59 | @formatter.format(tag, time, record) 60 | end 61 | ``` 62 | 63 | JSON formatter example: 64 | 65 | ```ruby 66 | # Create JSON formatter 67 | def configure(conf) 68 | super 69 | @json_formatter = formatter_create(usage: 'formatter_in_example_json', type: 'json') 70 | end 71 | 72 | def format(tag, time, record) 73 | @json_formatter.format(tag, time, record) 74 | end 75 | ``` 76 | 77 | MsgPack formatter example: 78 | 79 | ```ruby 80 | # Create MsgPack formatter 81 | def configure(conf) 82 | super 83 | @msgpack_formatter = formatter_create(usage: 'formatter_in_example_msgpack', type: 'msgpack') 84 | end 85 | 86 | def format(tag, time, record) 87 | @msgpack_formatter.format(tag, time, record) 88 | end 89 | ``` 90 | 91 | ## Plugins using `formatter` 92 | 93 | * [`filter_stdout`](../filter/stdout.md) 94 | * [`out_stdout`](../output/stdout.md) 95 | * [`out_exec`](../output/exec.md) 96 | * [`out_exec_filter`](../output/exec_filter.md) 97 | * [`out_file`](../output/file.md) 98 | * [`out_s3`](../output/s3.md) 99 | 100 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 101 | 102 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-inject.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Inject 2 | 3 | The `inject` plugin helper injects values into events according to the configuration. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/filter' 9 | 10 | module Fluent::Plugin 11 | class ExampleFilter < Filter 12 | Fluent::Plugin.register_filter('example', self) 13 | 14 | # 1. Load inject helper 15 | helpers :inject 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def filter(tag, time, record) 20 | # 2. Inject values into `record` 21 | new_record = inject_values_to_record(tag, time, record) 22 | # edit new_record ... 23 | new_record 24 | end 25 | end 26 | end 27 | ``` 28 | 29 | For more details, see [Inject section](../configuration/inject-section.md). 30 | 31 | ## Methods 32 | 33 | ### `inject_values_to_record(tag, time, record)` 34 | 35 | This method injects values to the given record and returns the new record. 36 | 37 | * `tag`: the tag of the event 38 | * `time`: event timestamp 39 | * `record`: event record 40 | 41 | Example: 42 | 43 | ```ruby 44 | def filter(tag, time, record) 45 | new_record = inject_values_to_record(tag, time, record) 46 | # edit new_record ... 47 | new_record 48 | end 49 | ``` 50 | 51 | ### `inject_values_to_event_stream(tag, es)` 52 | 53 | This method injects values into the given event stream and returns the new event stream. 54 | 55 | * `tag`: the tag of the event 56 | * `es`: event stream 57 | 58 | Example: 59 | 60 | ```ruby 61 | def process(tag, es) 62 | new_es = inject_values_to_event_stream(tag, es) 63 | # do something using new_es 64 | end 65 | ``` 66 | 67 | ## Plugins using `inject` 68 | 69 | * [`filter_stdout`](../filter/stdout.md) 70 | * [`out_exec`](../output/exec.md) 71 | * [`out_exec_filter`](../output/exec_filter.md) 72 | * [`out_file`](../output/file.md) 73 | * [`out_stdout`](../output/stdout.md) 74 | 75 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 76 | 77 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-metrics.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Metrics 2 | 3 | The `metrics` plugin helper manages the metrics values in plugins. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load metrics helper 15 | helpers :metrics 16 | 17 | def configure(conf) 18 | super 19 | 20 | # 2. Create parser plugin instance 21 | @metrics = metrics_create(namespace: "fluentd", subsystem: "input", name: "example", help_text: "Example metrics") 22 | end 23 | 24 | def start 25 | super 26 | 27 | # 3. Increase metrics value 28 | @metrics.inc 29 | 30 | end 31 | 32 | def statistics 33 | stats = super 34 | 35 | # 4. Retrieve metrics value 36 | stats = { 37 | 'input' => stats["input"].merge({ 'example' => @metrics.get }) 38 | } 39 | stats 40 | end 41 | end 42 | ``` 43 | 44 | For more details, see the following articles: 45 | 46 | * [Metrics Plugins Overview](../metrics/) 47 | 48 | ## Methods 49 | 50 | ### `metrics_create(namespace: "fluentd", subsystem: "metrics", name:, help_text:, labels: {}, prefer_gauge: false)` 51 | 52 | This method creates a metrics instance. 53 | 54 | * `namespace`: The namespace for the metrics. 55 | * `subsystem`: The names that represent specific functions or components. 56 | * `name`: The metrics name. 57 | * `help_text`: The description for metrics. 58 | * `labels`: The key/value pair for metrics labels. 59 | * `prefer_gauge`: Use gauge instead of counter for the metrics if `true`. 60 | 61 | Since 1.19.0, `metrics_create` method generates a getter method with the specified `name` on the calling instance. 62 | 63 | ## Plugins using `metrics` 64 | 65 | * [`buffer`](../buffer/) 66 | * [`filter`](../filter/) 67 | * [`input`](../input/) 68 | * [`output`](../output/) 69 | 70 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 71 | 72 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-parser.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Parser 2 | 3 | The `parser` plugin helper manages the lifecycle of the parser plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load parser helper 15 | helpers :parser 16 | 17 | # Omit `shutdown` and other plugin APIs 18 | 19 | def configure(conf) 20 | super 21 | 22 | # 2. Create parser plugin instance 23 | @parser = parser_create 24 | end 25 | 26 | def start 27 | super 28 | 29 | # Use parser helper in combination usually with other plugin helpers 30 | timer_execute(:example_timer, 10) do 31 | read_raw_data do |text| 32 | # 3. Call `@parser.parse(text)` to parse raw data 33 | @parser.parse(text) do |time, record| 34 | router.emit(tag, time, record) 35 | end 36 | end 37 | end 38 | end 39 | end 40 | end 41 | ``` 42 | 43 | For more details, see the following articles: 44 | 45 | * [Parser Plugin Overview](../parser/) 46 | * [Writing Parser Plugins](../plugin-development/api-plugin-parser.md) 47 | * [Parser Plugin](../configuration/parse-section.md) 48 | 49 | ## Methods 50 | 51 | ### `parser_create(usage: "", type: nil, conf: nil, default_type: nil)` 52 | 53 | This method creates a parser plugin instance with the given parameters. 54 | 55 | * `usage`: unique name required for multiple parsers 56 | * `type`: parser type 57 | * `conf`: parser plugin configuration 58 | * `default_type`: default parser type 59 | 60 | **Examples** 61 | 62 | ```ruby 63 | # Create parser plugin instance using section in fluent.conf during configure phase 64 | @parser = parser_create 65 | @parser.parse(text) do |time, record| 66 | # ... 67 | end 68 | 69 | # Create JSON parser 70 | @json_parser = parser_create(usage: 'parser_in_example_json', type: 'json') 71 | @json_parser.parse(json) do |time, record| 72 | # ... 73 | end 74 | 75 | # Create MessagePack parser 76 | @msgpack_parser = parser_create(usage: 'parser_in_example_msgpack', type: 'msgpack') 77 | @msgpack_parser.parse(msgpack_binary) do |time, record| 78 | # ... 79 | end 80 | ``` 81 | 82 | ## Plugins using `parser` 83 | 84 | * [`filter_parser`](../filter/parser.md) 85 | * [`in_exec`](../input/exec.md) 86 | * [`in_http`](../input/http.md) 87 | * [`in_syslog`](../input/syslog.md) 88 | * [`in_tail`](../input/tail.md) 89 | * [`in_tcp`](../input/tcp.md) 90 | * [`in_udp`](../input/udp.md) 91 | * [`out_exec_filter`](../output/exec_filter.md) 92 | 93 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 94 | 95 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-record_accessor.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Record Accessor 2 | 3 | The `record_accessor` plugin helper provides unified access to the event record. It uses `jsonpath` like syntax for the target field. With this helper, you can easily access/delete a nested field in the plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/filter' 9 | 10 | module Fluent::Plugin 11 | class ExampleFilter < Filter 12 | Fluent::Plugin.register_filter('example', self) 13 | 14 | # 1. Load record_accessor helper 15 | helpers :record_accessor 16 | 17 | def configure(conf) 18 | super 19 | 20 | # 2. Call `record_accessor_create` to create object 21 | @accessor = record_accessor_create('$.user.name') 22 | end 23 | 24 | # Omit `super`, `shutdown` and other plugin APIs 25 | 26 | def filter(tag, time, record) 27 | # 3. Call `call` method to get value 28 | value = @accessor.call(record) # With `$.user.name`, access to record["user"]["name"] 29 | # ... 30 | end 31 | end 32 | end 33 | ``` 34 | 35 | ## Syntax 36 | 37 | * dot notation: `$.` is the starting parameter. Chain fields with dots `.`. 38 | 39 | For example: 40 | 41 | `$.event.level` for `record["event"]["level"]` 42 | 43 | `$.key1[0].key2` for `record["key1"][0]["key2"]` 44 | 45 | * bracket notation: `$[` starting parameter. Chain fields with `[]`. 46 | 47 | Useful for special characters, `.`, whitespace, etc. 48 | 49 | `$['dot.key'][0]['space key']` for `record["dot.key"][0]["space key"]` 50 | 51 | If you set non `$.` or `$[` starting value, e.g. `key log`, it is the same as `record["log"]`. So, using `record_accessor` does not break the existing plugin behavior. 52 | 53 | ## Methods 54 | 55 | ### `record_accessor_create(param)` 56 | 57 | This method returns the accessor object of the event record. 58 | 59 | The `param` is a `String`. 60 | 61 | See the "Syntax" section for more details. 62 | 63 | ```ruby 64 | record_accessor_create("log") 65 | record_accessor_create("$.key1.key2") 66 | record_accessor_create("$['key1'][0]['key2']") 67 | ``` 68 | 69 | After creating an object, call `call`/`delete`/`set` method with the record object. 70 | 71 | ```ruby 72 | accessor.call(record) # get record field 73 | accessor.delete(record) # delete record field 74 | accessor.set(record, value) # set new value to record field 75 | ``` 76 | 77 | NOTE: `set` method is supported since v1.10.3 78 | 79 | ## Plugins using `record_accessor` 80 | 81 | * [`filter_grep`](../filter/grep.md) 82 | * [`filter_parser`](../filter/parser.md) 83 | * [`filter_record_transformer`](../filter/record_transformer.md) 84 | 85 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 86 | 87 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-thread.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Thread 2 | 3 | The `thread` plugin helper manages threads and these threads are integrated with plugins. No need manual run or shutdown in the plugin. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/output' 9 | 10 | module Fluent::Plugin 11 | class ExampleOutput < Output 12 | Fluent::Plugin.register_output('example', self) 13 | 14 | # 1. Load thread helper 15 | helpers :thread 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def start 20 | super 21 | 22 | # 2. Create and run thread with unique name. 23 | thread_create(:example_thread_run, &method(:run)) 24 | end 25 | 26 | def run 27 | # ... 28 | end 29 | end 30 | end 31 | ``` 32 | 33 | The launched thread is managed by the plugin helper. No need of thread shutdown code in plugin's `shutdown` method. It shutdowns the launched threads automatically. 34 | 35 | ## Methods 36 | 37 | ### `thread_create(title)` 38 | 39 | This method creates a thread and runs it with the given routine. 40 | 41 | `title` must be unique. 42 | 43 | ```ruby 44 | # Pass block directly 45 | thread_create(:example_plugin_main) { 46 | # ... 47 | } 48 | 49 | # Pass method object with existing method 50 | thread_create(:foo_plugin_body, &method(:run)) 51 | def run 52 | # ... 53 | end 54 | ``` 55 | 56 | ### `thread_current_running?` 57 | 58 | Checks whether the current thread is running or not. This method is available in the running block. 59 | 60 | ```ruby 61 | thread_create(:example_plugin_main) { 62 | while thread_current_running? 63 | # ... 64 | end 65 | } 66 | ``` 67 | 68 | ## Plugins using `thread` 69 | 70 | * [`out_forward`](../output/forward.md) 71 | * [`in_monitor_agent`](../input/monitor_agent.md) 72 | * [`in_sample`](../input/sample.md) 73 | 74 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 75 | 76 | -------------------------------------------------------------------------------- /plugin-helper-overview/api-plugin-helper-timer.md: -------------------------------------------------------------------------------- 1 | # Plugin Helper: Timer 2 | 3 | The `timer` plugin helper manages event timers. 4 | 5 | Here is an example: 6 | 7 | ```ruby 8 | require 'fluent/plugin/input' 9 | 10 | module Fluent::Plugin 11 | class ExampleInput < Input 12 | Fluent::Plugin.register_input('example', self) 13 | 14 | # 1. Load timer helper 15 | helpers :timer 16 | 17 | # Omit `configure`, `shutdown` and other plugin APIs 18 | 19 | def start 20 | super 21 | 22 | # 2. Execute timer with unique name and second unit interval 23 | timer_execute(:example_timer, 10) { 24 | # ... 25 | } 26 | end 27 | end 28 | end 29 | ``` 30 | 31 | The launched timer is managed by the plugin helper. No need of timer shutdown code in plugin's `shutdown` method. The plugin shutdowns the launched timers automatically. 32 | 33 | ## Methods 34 | 35 | ### `timer_execute(title, interval, repeat: true, &block)` 36 | 37 | This method executes the timer with the given parameters and routine. 38 | 39 | * `title`: unique symbol value 40 | * `interval`: second unit `integer`/`float` value. 41 | * `repeat`: `true`/`false` \(default: `true`\). If `false`, timer is one-shot. 42 | 43 | Code examples: 44 | 45 | ```ruby 46 | # Pass block directly. block is executed in 10 second interval. 47 | timer_execute(:example_timer, 10) { 48 | # ... 49 | } 50 | 51 | # Pass block with existing method. block is executed in 5 second and one-shot. 52 | timer_execute(:example_timer_run, 5s, repeat: false, &method(:run)) 53 | def run 54 | # ... 55 | end 56 | ``` 57 | 58 | ## Plugins using `timer` 59 | 60 | * [`out_forward`](../output/forward.md) 61 | * [`in_monitor_agent`](../input/monitor_agent.md) 62 | * [`in_sample`](../input/sample.md) 63 | 64 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 65 | 66 | -------------------------------------------------------------------------------- /plugins/input/dummy.md: -------------------------------------------------------------------------------- 1 | # `dummy` Input Plugin 2 | 3 | `in_dummy` has been renamed `in_sample` since Fluentd v1.11.12. 4 | See [`sample` input plugin](/input/sample.md) article for more details. 5 | 6 | You can keep to use following configuration in v1: 7 | 8 | ```text 9 | 10 | @type dummy 11 | dummy {"hello":"world"} 12 | 13 | ``` 14 | 15 | Fluentd v2 will remove this plugin name. 16 | 17 | 18 | ------------------------------------------------------------------------ 19 | 20 | If this article is incorrect or outdated, or omits critical information, please 21 | [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). 22 | [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native 23 | Computing Foundation (CNCF)](https://cncf.io/). All components are available 24 | under the Apache 2 License. 25 | -------------------------------------------------------------------------------- /quickstart/README.md: -------------------------------------------------------------------------------- 1 | # Overview 2 | 3 | Let's get started with **Fluentd**! **Fluentd** is a fully free and fully open-source log collector that instantly enables you to have a '**Log Everything**' architecture with [125+ types of systems](https://www.fluentd.org/plugins). 4 | 5 |  6 | 7 | Fluentd treats logs as JSON, a popular machine-readable format. It is written primarily in C with a thin-Ruby wrapper that gives users flexibility. 8 | 9 | Fluentd's scalability has been proven in the field: its largest user currently collects logs from **50,000+ servers**. 10 | 11 | ## Step 1: Installing Fluentd 12 | 13 | You can install Fluentd as a package, Ruby gem, or from source. 14 | 15 | Please see the following article for details. 16 | 17 | * [Installation](../installation/) 18 | 19 | ## Step 2: Use Cases 20 | 21 | The following articles cover the typical use cases of Fluentd: 22 | 23 | * Use Cases 24 | * [Data Search like Splunk](../how-to-guides/free-alternative-to-splunk-by-fluentd.md) 25 | * [Data Filtering and Alerting](../how-to-guides/splunk-like-grep-and-alert-email.md) 26 | * [Data Analytics with Treasure Data](../how-to-guides/http-to-td.md) 27 | * [Data Collection to MongoDB](../how-to-guides/apache-to-mongodb.md) 28 | * [Data Collection to HDFS](../how-to-guides/http-to-hdfs.md) 29 | * [Data Archiving to Amazon S3](../how-to-guides/apache-to-s3.md) 30 | * Basic Configuration 31 | * [Config File](../configuration/config-file.md) 32 | * Application Logs 33 | * [Ruby](../language-bindings/ruby.md), [Java](../language-bindings/java.md), [Python](../language-bindings/python.md), [PHP](../language-bindings/php.md), 34 | 35 | [Perl](../language-bindings/perl.md), [Node.js](../language-bindings/nodejs.md), [Scala](../language-bindings/scala.md) 36 | * Happy Users :\) 37 | * [Users](https://www.fluentd.org/testimonials) 38 | 39 | ## Step 3: Learn More 40 | 41 | The following articles will provide detailed information about Fluentd: 42 | 43 | * [Architecture Overview](https://www.fluentd.org/architecture) 44 | * [Lifecycle of a Fluentd Event](life-of-a-fluentd-event.md) 45 | * Plugin Overview 46 | * [Input Plugins](../input/) 47 | * [Output Plugins](../output/) 48 | * [Buffer Plugins](../buffer/) 49 | * [Filter Plugins](../filter/) 50 | * [Parser Plugins](../parser/) 51 | * [Formatter Plugins](../formatter/) 52 | * [High Availability Configuration](../deployment/high-availability.md) 53 | * [FAQ](faq.md) 54 | 55 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0) 56 | 57 | -------------------------------------------------------------------------------- /quickstart/logo.md: -------------------------------------------------------------------------------- 1 | # Logo 2 | 3 | Feel free to use these logos on your slides, blog posts, etc. 4 | 5 | ## Square 6 | 7 | [](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_square.png) 8 | 9 | [\[PNG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_square.png) [\[SVG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_square.svg) 10 | 11 | ## Horizontal 12 | 13 | [](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_horizontal.png) 14 | 15 | [\[PNG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_horizontal.png) [\[SVG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_horizontal.svg) 16 | 17 | ## Icon 18 | 19 | [](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_icon.png) 20 | 21 | [\[PNG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_icon.png) [\[SVG\]](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd_icon.svg) 22 | 23 | ## Links 24 | 25 | * [Fluentd.ai](https://github.com/fluent/fluentd-docs-gitbook/tree/53020426cdcfcb5a5f722031838ee1cb95b5a7a2/images/logo/Fluentd.ai) 26 | * [Complete List of Logos](https://github.com/fluent/fluentd-docs-gitbook/tree/1.0/images/logo) 27 | 28 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0) 29 | 30 | -------------------------------------------------------------------------------- /quickstart/support.md: -------------------------------------------------------------------------------- 1 | # Support 2 | 3 | ## Community Forum \(GitHub Discussions\) 4 | 5 | Got a question? Have some cool Fluentd-related projects? The GitHub discussion is the place to be: 6 | 7 | * [Fluentd Discussions](https://github.com/fluent/fluentd/discussions) 8 | 9 | ## Slack 10 | 11 | Join our Slack channel and talk with Fluentd developers and users directly, now. 12 | 13 | * [Slack](https://slack.fluentd.org/) 14 | 15 | ## Enterprise Support 16 | 17 | If you need support for running in production from enterprise companies you can find providers at the following link 18 | 19 | * [Enterprise Support providers](https://www.fluentd.org/enterprise_services) 20 | 21 | ## Community Resources 22 | 23 | Here are other community resources: 24 | 25 | * [StackOverflow](https://stackoverflow.com/questions/tagged/fluentd?sort=newest) 26 | * [Twitter @fluentd](https://www.twitter.com/fluentd) 27 | * [Facebook Page](https://www.facebook.com/pages/Fluentd-Log-Everything-in-JSON/196064987183037) 28 | 29 | ## Change Logs 30 | 31 | If you are interested in what's new in each version, please refer to the following links: 32 | 33 | * [Fluentd's ChangeLog](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md). 34 | * [fluent-package's ChangeLog](https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG.md) (Successor of td-agent package) 35 | * [td-agent's ChangeLog](https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG-v4.md) (End of Life in Dec, 2023) 36 | 37 | {% hint style='danger' %} 38 | The series of td-agent had already reached End of Life (EOL). td-agent should not be newly installed because no support, no new release and no security updates anymore. 39 | Use fluent-package instead. See [Upgrade to fluent-package v5](https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5) for migration. 40 | {% endhint %} 41 | 42 | ## Development 43 | 44 | If you find bugs, please report to the GitHub Issue. 45 | 46 | * [Bug/Feature Tracker](https://github.com/fluent/fluentd/issues) 47 | 48 | The source code is available at GitHub: 49 | 50 | * [Fluentd](https://github.com/fluent/fluentd/) 51 | 52 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0) 53 | 54 | -------------------------------------------------------------------------------- /service_discovery/README.md: -------------------------------------------------------------------------------- 1 | # Service Discovery Plugins 2 | 3 | Fluentd has nine \(9\) types of plugins: 4 | 5 | * [Input](../input/) 6 | * [Parser](../parser/) 7 | * [Filter](../filter/) 8 | * [Output](../output/) 9 | * [Formatter](../formatter/) 10 | * [Storage](../storage/) 11 | * [Service Discovery](./) 12 | * [Buffer](../buffer/) 13 | * [Metrics](../metrics/) 14 | 15 | This article gives an overview of the Service Discovery Plugin. 16 | 17 | ## Overview 18 | 19 | Some plugins support `` \(e.g. [`out_forward`](../output/forward.md)\). Sometimes, the service discovery for an output plugin does not meet one's needs. Fluentd has a pluggable system called Service Discovery that lets the user extend and reuse custom output service discovery. 20 | 21 | ## How To Use 22 | 23 | Here is a simple example to update target by reading file \(`/etc/fluentd/sd.yaml`\) with `out_forward` and `service_discovery`: 24 | 25 | ```text 26 | 27 | @type forward 28 | 29 | 30 | @type file 31 | path "/etc/fluentd/sd.yaml" 32 | 33 | 34 | ``` 35 | 36 | ## List of Built-in Service Discovery 37 | 38 | * [`static`](static.md) 39 | * [`file`](file.md) 40 | * [`srv`](srv.md) 41 | 42 | ## List of Core Output Plugins with Service Discovery support 43 | 44 | * [`out_forward`](../output/forward.md) 45 | 46 | If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. 47 | 48 | -------------------------------------------------------------------------------- /service_discovery/file.md: -------------------------------------------------------------------------------- 1 | # file 2 | 3 | The `file` service discovery plugin updates the targets by reading the local file. YAML and JSON are the allowed file formats. 4 | 5 | ## Example Configuration 6 | 7 | Here is an example with `out_forward` updating targets by sending data: 8 | 9 | ```text 10 | 11 | @type forward 12 | 13 | 14 | @type file 15 | path "/etc/fluentd/sd.yaml" 16 | 17 |