├── .github ├── CODEOWNERS ├── PULL_REQUEST_TEMPLATE.md ├── dependabot.yml └── workflows │ └── build.yml ├── .gitignore ├── CHANGELOG.md ├── CODEOWNERS ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── NOTICE ├── README.md ├── THIRD-PARTY ├── VERSION ├── firehose ├── firehose.go ├── firehose_test.go ├── generate_mock.go └── mock_firehose │ └── mock.go ├── fluent-bit-firehose.go ├── go.mod ├── go.sum ├── plugins ├── plugins.go ├── plugins_test.go └── user_agent.go └── scripts └── mockgen.sh /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @aws/aws-firelens 2 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | - package-ecosystem: gomod 4 | directory: "/" # Location of package manifests 5 | schedule: 6 | interval: "weekly" 7 | -------------------------------------------------------------------------------- /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | name: Build 2 | 3 | on: 4 | push: 5 | branches: [ mainline ] 6 | pull_request: 7 | branches: [ mainline ] 8 | 9 | jobs: 10 | 11 | build: 12 | name: Build 13 | runs-on: ubuntu-latest 14 | steps: 15 | 16 | - name: Set up Go 1.20 17 | uses: actions/setup-go@v2 18 | with: 19 | go-version: "1.20" 20 | id: go 21 | 22 | - name: Install cross-compiler for Windows 23 | run: sudo apt-get install -y gcc-multilib gcc-mingw-w64 24 | 25 | - name: Check out code into the Go module directory 26 | uses: actions/checkout@v2 27 | 28 | - name: golint 29 | run: go install golang.org/x/lint/golint@latest 30 | 31 | - name: Build 32 | run: make build windows-release test 33 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Binaries for programs and plugins 3 | *.exe 4 | *.exe~ 5 | *.dll 6 | *.so 7 | *.dylib 8 | 9 | # Test binary, built with `go test -c` 10 | *.test 11 | 12 | # build output dir 13 | bin 14 | 15 | # Output of the go coverage tool, specifically when used with LiteIDE 16 | *.out 17 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## 1.7.2 4 | * Enhancement - upgrade Go version to 1.20 5 | 6 | ## 1.7.1 7 | * Enhancement - Added different base user agent for Linux and Windows 8 | 9 | ## 1.7.0 10 | * Feature - Add support for building this plugin on Windows. *Note that this is only support in this plugin repo for Windows compilation.* 11 | 12 | ## 1.6.1 13 | * Enhancement - upgrade Go version to 1.17 14 | 15 | ## 1.6.0 16 | * Feature - Add new option `simple_aggregation` to send multiple log events per record (#12) 17 | 18 | ## 1.5.0 19 | * Feature - Add new option `replace_dots` to replace dots in key names (#46) 20 | 21 | ## 1.4.2 22 | * Bug - Truncate record to max size (#58) 23 | 24 | ## 1.4.0 25 | * Feature - Add `log_key` option for firehose output plugin (#33) 26 | * Bug - Check for empty batch before sending (#27) 27 | 28 | ## 1.3.0 29 | * Feature - Add `sts_endpoint` param for custom STS API endpoint (#31) 30 | 31 | ## 1.2.1 32 | * Bug - Remove exponential backoff code (#23) 33 | 34 | ## 1.2.0 35 | * Feature - Add `time_key` and `time_key_format` config options to add timestamp to records (#9) 36 | 37 | ## 1.1.0 38 | * Feature - Support IAM Roles for Service Accounts in Amazon EKS (#17) 39 | * Enhancement - Change the log severity from `error` to `warning` for retryable API errors (#18) 40 | 41 | 42 | ## 1.0.0 43 | Initial versioned release of the Amazon Kinesis Data Firehose for Fluent Bit Plugin 44 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | / @aws/aws-firelens 2 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/awslabs/amazon-kinesis-for-fluent-bit/issues), or [recently closed](https://github.com/awslabs/amazon-kinesis-for-fluent-bit/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/amazon-kinesis-for-fluent-bit/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/awslabs/amazon-kinesis-for-fluent-bit/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"). You 4 | # may not use this file except in compliance with the License. A copy of 5 | # the License is located at 6 | # 7 | # http://aws.amazon.com/apache2.0/ 8 | # 9 | # or in the "license" file accompanying this file. This file is 10 | # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 11 | # ANY KIND, either express or implied. See the License for the specific 12 | # language governing permissions and limitations under the License. 13 | 14 | # Build settings. 15 | GOARCH ?= amd64 16 | COMPILER ?= x86_64-w64-mingw32-gcc # Cross-compiler for Windows 17 | 18 | ROOT := $(shell pwd) 19 | 20 | all: build 21 | 22 | SCRIPT_PATH := $(ROOT)/scripts/:${PATH} 23 | SOURCES := $(shell find . -name '*.go') 24 | PLUGIN_BINARY := ./bin/firehose.so 25 | 26 | .PHONY: build 27 | build: $(PLUGIN_BINARY) 28 | 29 | $(PLUGIN_BINARY): $(SOURCES) 30 | PATH=${PATH} golint ./plugins ./firehose 31 | mkdir -p ./bin 32 | go build -buildmode c-shared -o ./bin/firehose.so ./ 33 | @echo "Built Amazon Kinesis Data Firehose Fluent Bit Plugin" 34 | 35 | .PHONY: release 36 | release: 37 | mkdir -p ./bin 38 | go build -buildmode c-shared -o ./bin/firehose.so ./ 39 | @echo "Built Amazon Kinesis Data Firehose Fluent Bit Plugin" 40 | 41 | .PHONY: windows-release 42 | windows-release: 43 | mkdir -p ./bin 44 | GOOS=windows GOARCH=$(GOARCH) CGO_ENABLED=1 CC=$(COMPILER) go build -buildmode c-shared -o ./bin/firehose.dll ./ 45 | @echo "Built Amazon Kinesis Data Firehose Fluent Bit Plugin for Windows" 46 | 47 | .PHONY: generate 48 | generate: $(SOURCES) 49 | PATH=$(SCRIPT_PATH) go generate ./... 50 | 51 | 52 | .PHONY: test 53 | test: 54 | go test -timeout=120s -v -cover ./... 55 | 56 | .PHONY: clean 57 | clean: 58 | rm -rf ./bin/* 59 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Fluent Bit Plugin for Amazon Kinesis Firehose 2 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Test Actions Status](https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit/workflows/Build/badge.svg)](https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit/actions) 2 | ## Fluent Bit Plugin for Amazon Kinesis Firehose 3 | 4 | **NOTE: A new higher performance Fluent Bit Firehose Plugin has been released.** Check out our [official guidance](#new-higher-performance-core-fluent-bit-plugin). 5 | 6 | A Fluent Bit output plugin for Amazon Kinesis Data Firehose. 7 | 8 | #### Security disclosures 9 | 10 | If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions [here](https://aws.amazon.com/security/vulnerability-reporting/) or email AWS security directly at [aws-security@amazon.com](mailto:aws-security@amazon.com). 11 | 12 | ### Usage 13 | 14 | Run `make` to build `./bin/firehose.so`. Then use with Fluent Bit: 15 | ``` 16 | ./fluent-bit -e ./firehose.so -i cpu \ 17 | -o firehose \ 18 | -p "region=us-west-2" \ 19 | -p "delivery_stream=example-stream" 20 | ``` 21 | 22 | For building Windows binaries, we need to install `mingw-64w` for cross-compilation. The same can be done using- 23 | ``` 24 | sudo apt-get install -y gcc-multilib gcc-mingw-w64 25 | ``` 26 | After this step, run `make windows-release` to build `./bin/firehose.dll`. Then use with Fluent Bit on Windows: 27 | ``` 28 | ./fluent-bit.exe -e ./firehose.dll -i dummy ` 29 | -o firehose ` 30 | -p "region=us-west-2" ` 31 | -p "delivery_stream=example-stream" 32 | ``` 33 | ### Plugin Options 34 | 35 | * `region`: The region which your Firehose delivery stream(s) is/are in. 36 | * `delivery_stream`: The name of the delivery stream that you want log records sent to. 37 | * `data_keys`: By default, the whole log record will be sent to Kinesis. If you specify a key name(s) with this option, then only those keys and values will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify `data_keys log` and only the log message will be sent to Kinesis. If you specify multiple keys, they should be comma delimited. 38 | * `log_key`: By default, the whole log record will be sent to Firehose. If you specify a key name with this option, then only the value of that key will be sent to Firehose. For example, if you are using the Fluentd Docker log driver, you can specify `log_key log` and only the log message will be sent to Firehose. 39 | * `role_arn`: ARN of an IAM role to assume (for cross account access). 40 | * `endpoint`: Specify a custom endpoint for the Kinesis Firehose API. 41 | * `sts_endpoint`: Specify a custom endpoint for the STS API; used to assume your custom role provided with `role_arn`. 42 | * `time_key`: Add the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis. The timestamp inserted comes from the timestamp that Fluent Bit associates with the log record, which is set by the input that collected it. For example, if you are reading a log file with the [tail input](https://docs.fluentbit.io/manual/pipeline/inputs/tail), then the timestamp for each log line/record can be obtained/parsed by using a Fluent Bit parser on the log line. 43 | * `time_key_format`: [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html) compliant format string for the timestamp; for example, `%Y-%m-%dT%H:%M:%S%z`. This option is used with `time_key`. You can also use `%L` for milliseconds and `%f` for microseconds. Remember that the `time_key` option only inserts the timestamp Fluent Bit has for each record into the record. So the record must have been collected with a timestamp with precision in order to use sub-second precision formatters. If you are using ECS FireLens, make sure you are running Amazon ECS Container Agent v1.42.0 or later, otherwise the timestamps associated with your stdout & stderr container logs will only have second precision. 44 | * `replace_dots`: Replace dot characters in key names with the value of this option. For example, if you add `replace_dots _` in your config then all occurrences of `.` will be replaced with an underscore. By default, dots will not be replaced. 45 | * `simple_aggregation`: Option to allow plugin send multiple log events in the same record if current record not exceed the maximumRecordSize (1 MiB). It joins together as many log records as possible into a single Firehose record and delimits them with newline. It's good to enable if your destination supports aggregation like S3. Default to be `false`, set to `true` to enable this option. 46 | ### Permissions 47 | 48 | The plugin requires `firehose:PutRecordBatch` permissions. 49 | 50 | ### Credentials 51 | 52 | This plugin uses the AWS SDK Go, and uses its [default credential provider chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html). If you are using the plugin on Amazon EC2 or Amazon ECS or Amazon EKS, the plugin will use your EC2 instance role or [ECS Task role permissions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) or [EKS IAM Roles for Service Accounts for pods](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). The plugin can also retrieve credentials from a [shared credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html), or from the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` environment variables. 53 | 54 | ### Environment Variables 55 | 56 | * `FLB_LOG_LEVEL`: Set the log level for the plugin. Valid values are: `debug`, `info`, and `error` (case insensitive). Default is `info`. **Note**: Setting log level in the Fluent Bit Configuration file using the Service key will not affect the plugin log level (because the plugin is external). 57 | * `SEND_FAILURE_TIMEOUT`: Allows you to configure a timeout if the plugin can not send logs to Firehose. The timeout is specified as a [Golang duration](https://golang.org/pkg/time/#ParseDuration), for example: `5m30s`. If the plugin has failed to make any progress for the given period of time, then it will exit and kill Fluent Bit. This is useful in scenarios where you want your logging solution to fail fast if it has been misconfigured (i.e. network or credentials have not been set up to allow it to send to Firehose). 58 | 59 | 60 | ### New Higher Performance Core Fluent Bit Plugin 61 | 62 | In the summer of 2020, we released a [new higher performance Kinesis Firehose plugin](https://docs.fluentbit.io/manual/pipeline/outputs/firehose) named `kinesis_firehose`. 63 | 64 | That plugin has almost all of the features of this older, lower performance and less efficient plugin. Check out its [documentation](https://docs.fluentbit.io/manual/pipeline/outputs/firehose). 65 | 66 | #### Do you plan to deprecate this older plugin? 67 | 68 | This plugin will continue to be supported. However, we are pausing development on it and will focus on the high performance version instead. 69 | 70 | #### Which plugin should I use? 71 | 72 | If the features of the higher performance plugin are sufficient for your use cases, please use it. It can achieve higher throughput and will consume less CPU and memory. 73 | 74 | As time goes on we expect new features to be added to the C plugin only, however, this is determined on a case by case basis. There is a small feature gap between the two plugins. Please consult the [C plugin documentation](https://docs.fluentbit.io/manual/pipeline/outputs/firehose) and this document for the features offered by each plugin. 75 | 76 | #### How can I migrate to the higher performance plugin? 77 | 78 | For many users, you can simply replace the plugin name `firehose` with the new name `kinesis_firehose`. At the time of writing, the only feature missing from the high performance version is the `replace_dots` option. Check out its [documentation](https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch). 79 | 80 | #### Do you accept contributions to both plugins? 81 | 82 | Yes. The high performance plugin is written in C, and this plugin is written in Golang. We understand that Go is an easier language for amateur contributors to write code in- that is the primary reason we are continuing to maintain this repo. 83 | 84 | However, if you can write code in C, please consider contributing new features to the [higher performance plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/out_kinesis_firehose). 85 | 86 | ### Fluent Bit Versions 87 | 88 | This plugin has been tested with Fluent Bit 1.2.0+. It may not work with older Fluent Bit versions. We recommend using the latest version of Fluent Bit as it will contain the newest features and bug fixes. 89 | 90 | ### Example Fluent Bit Config File 91 | 92 | ``` 93 | [INPUT] 94 | Name forward 95 | Listen 0.0.0.0 96 | Port 24224 97 | 98 | [OUTPUT] 99 | Name firehose 100 | Match * 101 | region us-west-2 102 | delivery_stream my-stream 103 | replace_dots _ 104 | ``` 105 | 106 | ### AWS for Fluent Bit 107 | 108 | We distribute a container image with Fluent Bit and these plugins. 109 | 110 | ##### GitHub 111 | 112 | [github.com/aws/aws-for-fluent-bit](https://github.com/aws/aws-for-fluent-bit) 113 | 114 | ##### Amazon ECR Public Gallery 115 | 116 | [aws-for-fluent-bit](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) 117 | 118 | Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command: 119 | 120 | ``` 121 | docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit: 122 | ``` 123 | 124 | For example, you can pull the image with latest version by: 125 | 126 | ``` 127 | docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest 128 | ``` 129 | 130 | If you see errors for image pull limits, try log into public ECR with your AWS credentials: 131 | 132 | ``` 133 | aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws 134 | ``` 135 | 136 | You can check the [Amazon ECR Public official doc](https://docs.aws.amazon.com/AmazonECR/latest/public/get-set-up-for-amazon-ecr.html) for more details. 137 | 138 | ##### Docker Hub 139 | 140 | [amazon/aws-for-fluent-bit](https://hub.docker.com/r/amazon/aws-for-fluent-bit/tags) 141 | 142 | ##### Amazon ECR 143 | 144 | You can use our SSM Public Parameters to find the Amazon ECR image URI in your region: 145 | 146 | ``` 147 | aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/ 148 | ``` 149 | 150 | For more see [our docs](https://github.com/aws/aws-for-fluent-bit#public-images). 151 | 152 | ## License 153 | 154 | This library is licensed under the Apache 2.0 License. 155 | -------------------------------------------------------------------------------- /THIRD-PARTY: -------------------------------------------------------------------------------- 1 | ** github.com/aws/aws-sdk-go; version v1.20.6 -- 2 | https://github.com/aws/aws-sdk-go 3 | Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | Copyright 2014-2015 Stripe, Inc. 5 | ** github.com/fluent/fluent-bit-go; version fc386d263885 -- 6 | https://github.com/fluent/fluent-bit-go 7 | Copyright (C) 2015-2017 Treasure Data Inc. 8 | ** github.com/golang/mock; version 1.3.1 -- https://github.com/golang/mock 9 | Copyright 2010 Google Inc. 10 | ** github.com/jmespath/go-jmespath; version c2b33e8439af -- 11 | https://github.com/jmespath/go-jmespath 12 | Copyright 2015 James Saryerwinnie 13 | ** github.com/modern-go/concurrent; version bacd9c7ef1dd -- 14 | https://github.com/modern-go/concurrent 15 | None 16 | ** github.com/modern-go/reflect2; version v1.0.1 -- 17 | https://github.com/modern-go/reflect2 18 | None 19 | 20 | Apache License 21 | 22 | Version 2.0, January 2004 23 | 24 | http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND 25 | DISTRIBUTION 26 | 27 | 1. Definitions. 28 | 29 | "License" shall mean the terms and conditions for use, reproduction, and 30 | distribution as defined by Sections 1 through 9 of this document. 31 | 32 | "Licensor" shall mean the copyright owner or entity authorized by the 33 | copyright owner that is granting the License. 34 | 35 | "Legal Entity" shall mean the union of the acting entity and all other 36 | entities that control, are controlled by, or are under common control 37 | with that entity. For the purposes of this definition, "control" means 38 | (i) the power, direct or indirect, to cause the direction or management 39 | of such entity, whether by contract or otherwise, or (ii) ownership of 40 | fifty percent (50%) or more of the outstanding shares, or (iii) 41 | beneficial ownership of such entity. 42 | 43 | "You" (or "Your") shall mean an individual or Legal Entity exercising 44 | permissions granted by this License. 45 | 46 | "Source" form shall mean the preferred form for making modifications, 47 | including but not limited to software source code, documentation source, 48 | and configuration files. 49 | 50 | "Object" form shall mean any form resulting from mechanical 51 | transformation or translation of a Source form, including but not limited 52 | to compiled object code, generated documentation, and conversions to 53 | other media types. 54 | 55 | "Work" shall mean the work of authorship, whether in Source or Object 56 | form, made available under the License, as indicated by a copyright 57 | notice that is included in or attached to the work (an example is 58 | provided in the Appendix below). 59 | 60 | "Derivative Works" shall mean any work, whether in Source or Object form, 61 | that is based on (or derived from) the Work and for which the editorial 62 | revisions, annotations, elaborations, or other modifications represent, 63 | as a whole, an original work of authorship. For the purposes of this 64 | License, Derivative Works shall not include works that remain separable 65 | from, or merely link (or bind by name) to the interfaces of, the Work and 66 | Derivative Works thereof. 67 | 68 | "Contribution" shall mean any work of authorship, including the original 69 | version of the Work and any modifications or additions to that Work or 70 | Derivative Works thereof, that is intentionally submitted to Licensor for 71 | inclusion in the Work by the copyright owner or by an individual or Legal 72 | Entity authorized to submit on behalf of the copyright owner. For the 73 | purposes of this definition, "submitted" means any form of electronic, 74 | verbal, or written communication sent to the Licensor or its 75 | representatives, including but not limited to communication on electronic 76 | mailing lists, source code control systems, and issue tracking systems 77 | that are managed by, or on behalf of, the Licensor for the purpose of 78 | discussing and improving the Work, but excluding communication that is 79 | conspicuously marked or otherwise designated in writing by the copyright 80 | owner as "Not a Contribution." 81 | 82 | "Contributor" shall mean Licensor and any individual or Legal Entity on 83 | behalf of whom a Contribution has been received by Licensor and 84 | subsequently incorporated within the Work. 85 | 86 | 2. Grant of Copyright License. Subject to the terms and conditions of this 87 | License, each Contributor hereby grants to You a perpetual, worldwide, 88 | non-exclusive, no-charge, royalty-free, irrevocable copyright license to 89 | reproduce, prepare Derivative Works of, publicly display, publicly perform, 90 | sublicense, and distribute the Work and such Derivative Works in Source or 91 | Object form. 92 | 93 | 3. Grant of Patent License. Subject to the terms and conditions of this 94 | License, each Contributor hereby grants to You a perpetual, worldwide, 95 | non-exclusive, no-charge, royalty-free, irrevocable (except as stated in 96 | this section) patent license to make, have made, use, offer to sell, sell, 97 | import, and otherwise transfer the Work, where such license applies only to 98 | those patent claims licensable by such Contributor that are necessarily 99 | infringed by their Contribution(s) alone or by combination of their 100 | Contribution(s) with the Work to which such Contribution(s) was submitted. 101 | If You institute patent litigation against any entity (including a 102 | cross-claim or counterclaim in a lawsuit) alleging that the Work or a 103 | Contribution incorporated within the Work constitutes direct or contributory 104 | patent infringement, then any patent licenses granted to You under this 105 | License for that Work shall terminate as of the date such litigation is 106 | filed. 107 | 108 | 4. Redistribution. You may reproduce and distribute copies of the Work or 109 | Derivative Works thereof in any medium, with or without modifications, and 110 | in Source or Object form, provided that You meet the following conditions: 111 | 112 | (a) You must give any other recipients of the Work or Derivative Works a 113 | copy of this License; and 114 | 115 | (b) You must cause any modified files to carry prominent notices stating 116 | that You changed the files; and 117 | 118 | (c) You must retain, in the Source form of any Derivative Works that You 119 | distribute, all copyright, patent, trademark, and attribution notices 120 | from the Source form of the Work, excluding those notices that do not 121 | pertain to any part of the Derivative Works; and 122 | 123 | (d) If the Work includes a "NOTICE" text file as part of its 124 | distribution, then any Derivative Works that You distribute must include 125 | a readable copy of the attribution notices contained within such NOTICE 126 | file, excluding those notices that do not pertain to any part of the 127 | Derivative Works, in at least one of the following places: within a 128 | NOTICE text file distributed as part of the Derivative Works; within the 129 | Source form or documentation, if provided along with the Derivative 130 | Works; or, within a display generated by the Derivative Works, if and 131 | wherever such third-party notices normally appear. The contents of the 132 | NOTICE file are for informational purposes only and do not modify the 133 | License. You may add Your own attribution notices within Derivative Works 134 | that You distribute, alongside or as an addendum to the NOTICE text from 135 | the Work, provided that such additional attribution notices cannot be 136 | construed as modifying the License. 137 | 138 | You may add Your own copyright statement to Your modifications and may 139 | provide additional or different license terms and conditions for use, 140 | reproduction, or distribution of Your modifications, or for any such 141 | Derivative Works as a whole, provided Your use, reproduction, and 142 | distribution of the Work otherwise complies with the conditions stated in 143 | this License. 144 | 145 | 5. Submission of Contributions. Unless You explicitly state otherwise, any 146 | Contribution intentionally submitted for inclusion in the Work by You to the 147 | Licensor shall be under the terms and conditions of this License, without 148 | any additional terms or conditions. Notwithstanding the above, nothing 149 | herein shall supersede or modify the terms of any separate license agreement 150 | you may have executed with Licensor regarding such Contributions. 151 | 152 | 6. Trademarks. This License does not grant permission to use the trade 153 | names, trademarks, service marks, or product names of the Licensor, except 154 | as required for reasonable and customary use in describing the origin of the 155 | Work and reproducing the content of the NOTICE file. 156 | 157 | 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in 158 | writing, Licensor provides the Work (and each Contributor provides its 159 | Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 160 | KIND, either express or implied, including, without limitation, any 161 | warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or 162 | FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining 163 | the appropriateness of using or redistributing the Work and assume any risks 164 | associated with Your exercise of permissions under this License. 165 | 166 | 8. Limitation of Liability. In no event and under no legal theory, whether 167 | in tort (including negligence), contract, or otherwise, unless required by 168 | applicable law (such as deliberate and grossly negligent acts) or agreed to 169 | in writing, shall any Contributor be liable to You for damages, including 170 | any direct, indirect, special, incidental, or consequential damages of any 171 | character arising as a result of this License or out of the use or inability 172 | to use the Work (including but not limited to damages for loss of goodwill, 173 | work stoppage, computer failure or malfunction, or any and all other 174 | commercial damages or losses), even if such Contributor has been advised of 175 | the possibility of such damages. 176 | 177 | 9. Accepting Warranty or Additional Liability. While redistributing the Work 178 | or Derivative Works thereof, You may choose to offer, and charge a fee for, 179 | acceptance of support, warranty, indemnity, or other liability obligations 180 | and/or rights consistent with this License. However, in accepting such 181 | obligations, You may act only on Your own behalf and on Your sole 182 | responsibility, not on behalf of any other Contributor, and only if You 183 | agree to indemnify, defend, and hold each Contributor harmless for any 184 | liability incurred by, or claims asserted against, such Contributor by 185 | reason of your accepting any such warranty or additional liability. END OF 186 | TERMS AND CONDITIONS 187 | 188 | APPENDIX: How to apply the Apache License to your work. 189 | 190 | To apply the Apache License to your work, attach the following boilerplate 191 | notice, with the fields enclosed by brackets "[]" replaced with your own 192 | identifying information. (Don't include the brackets!) The text should be 193 | enclosed in the appropriate comment syntax for the file format. We also 194 | recommend that a file or class name and description of purpose be included on 195 | the same "printed page" as the copyright notice for easier identification 196 | within third-party archives. 197 | 198 | Copyright [yyyy] [name of copyright owner] 199 | 200 | Licensed under the Apache License, Version 2.0 (the "License"); 201 | 202 | you may not use this file except in compliance with the License. 203 | 204 | You may obtain a copy of the License at 205 | 206 | http://www.apache.org/licenses/LICENSE-2.0 207 | 208 | Unless required by applicable law or agreed to in writing, software 209 | 210 | distributed under the License is distributed on an "AS IS" BASIS, 211 | 212 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 213 | 214 | See the License for the specific language governing permissions and 215 | 216 | limitations under the License. 217 | 218 | * For github.com/aws/aws-sdk-go see also this required NOTICE: 219 | Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved. 220 | Copyright 2014-2015 Stripe, Inc. 221 | * For github.com/fluent/fluent-bit-go see also this required NOTICE: 222 | Copyright (C) 2015-2017 Treasure Data Inc. 223 | * For github.com/golang/mock see also this required NOTICE: 224 | Copyright 2010 Google Inc. 225 | * For github.com/jmespath/go-jmespath see also this required NOTICE: 226 | Copyright 2015 James Saryerwinnie 227 | * For github.com/modern-go/concurrent see also this required NOTICE: 228 | None 229 | * For github.com/modern-go/reflect2 see also this required NOTICE: 230 | None 231 | 232 | ------ 233 | 234 | ** golang.org; version go1.12 -- https://golang.org/ 235 | Copyright (c) 2009 The Go Authors. All rights reserved. 236 | 237 | Copyright (c) 2009 The Go Authors. All rights reserved. 238 | 239 | Redistribution and use in source and binary forms, with or without 240 | modification, are permitted provided that the following conditions are 241 | met: 242 | 243 | * Redistributions of source code must retain the above copyright 244 | notice, this list of conditions and the following disclaimer. 245 | * Redistributions in binary form must reproduce the above 246 | copyright notice, this list of conditions and the following disclaimer 247 | in the documentation and/or other materials provided with the 248 | distribution. 249 | * Neither the name of Google Inc. nor the names of its 250 | contributors may be used to endorse or promote products derived from 251 | this software without specific prior written permission. 252 | 253 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 254 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 255 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 256 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 257 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 258 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 259 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 260 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 261 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 262 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 263 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 264 | 265 | ------ 266 | 267 | ** github.com/pmezard/go-difflib; version v1.0.0 -- 268 | https://github.com/pmezard/go-difflib 269 | Copyright (c) 2013, Patrick Mezard 270 | 271 | Copyright (c) 2013, Patrick Mezard 272 | All rights reserved. 273 | 274 | Redistribution and use in source and binary forms, with or without 275 | modification, are permitted provided that the following conditions are 276 | met: 277 | 278 | Redistributions of source code must retain the above copyright 279 | notice, this list of conditions and the following disclaimer. 280 | Redistributions in binary form must reproduce the above copyright 281 | notice, this list of conditions and the following disclaimer in the 282 | documentation and/or other materials provided with the distribution. 283 | The names of its contributors may not be used to endorse or promote 284 | products derived from this software without specific prior written 285 | permission. 286 | 287 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 288 | IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 289 | TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 290 | PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 291 | HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 292 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED 293 | TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 294 | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 295 | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 296 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 297 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 298 | 299 | ------ 300 | 301 | ** github.com/davecgh/go-spew; version v1.1.1 -- 302 | https://github.com/davecgh/go-spew 303 | Copyright (c) 2012-2016 Dave Collins 304 | ** github.com/davecgh/go-spew; version v1.1.1 -- 305 | https://github.com/davecgh/go-spew 306 | Copyright (c) 2012-2016 Dave Collins 307 | 308 | ISC License 309 | 310 | Copyright (c) 2012-2016 Dave Collins 311 | 312 | Permission to use, copy, modify, and/or distribute this software for any 313 | purpose with or without fee is hereby granted, provided that the above 314 | copyright notice and this permission notice appear in all copies. 315 | 316 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 317 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 318 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 319 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 320 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 321 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 322 | OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 323 | 324 | ------ 325 | 326 | ** github.com/stretchr/testify; version v1.3.0 -- 327 | https://github.com/stretchr/testify 328 | Copyright (c) 2012-2018 Mat Ryer and Tyler Bunnell 329 | 330 | MIT License 331 | 332 | Copyright (c) 2012-2018 Mat Ryer and Tyler Bunnell 333 | 334 | Permission is hereby granted, free of charge, to any person obtaining a copy 335 | of this software and associated documentation files (the "Software"), to deal 336 | in the Software without restriction, including without limitation the rights 337 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 338 | copies of the Software, and to permit persons to whom the Software is 339 | furnished to do so, subject to the following conditions: 340 | 341 | The above copyright notice and this permission notice shall be included in all 342 | copies or substantial portions of the Software. 343 | 344 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 345 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 346 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 347 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 348 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 349 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 350 | SOFTWARE. 351 | 352 | ------ 353 | 354 | ** github.com/sirupsen/logrus; version v1.4.2 -- 355 | https://github.com/sirupsen/logrus 356 | Copyright (c) 2014 Simon Eskildsen 357 | 358 | The MIT License (MIT) 359 | 360 | Copyright (c) 2014 Simon Eskildsen 361 | 362 | Permission is hereby granted, free of charge, to any person obtaining a copy 363 | of this software and associated documentation files (the "Software"), to deal 364 | in the Software without restriction, including without limitation the rights 365 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 366 | copies of the Software, and to permit persons to whom the Software is 367 | furnished to do so, subject to the following conditions: 368 | 369 | The above copyright notice and this permission notice shall be included in 370 | all copies or substantial portions of the Software. 371 | 372 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 373 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 374 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 375 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 376 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 377 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 378 | THE SOFTWARE. 379 | 380 | ------ 381 | 382 | ** github.com/konsorten/go-windows-terminal-sequences; version v1.0.1 -- 383 | https://github.com/konsorten/go-windows-terminal-sequences 384 | Copyright (c) 2017 marvin + konsorten GmbH (open-source@konsorten.de) 385 | 386 | (The MIT License) 387 | 388 | Copyright (c) 2017 marvin + konsorten GmbH (open-source@konsorten.de) 389 | 390 | Permission is hereby granted, free of charge, to any person obtaining a copy of 391 | this software and associated documentation files (the 'Software'), to deal in 392 | the Software without restriction, including without limitation the rights to 393 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 394 | of the Software, and to permit persons to whom the Software is furnished to do 395 | so, subject to the following conditions: 396 | 397 | The above copyright notice and this permission notice shall be included in all 398 | copies or substantial portions of the Software. 399 | 400 | THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 401 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 402 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 403 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 404 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 405 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 406 | SOFTWARE. 407 | 408 | ------ 409 | 410 | ** github.com/ugorji/go; version v1.1.4 -- https://github.com/ugorji/go 411 | Copyright (c) 2012-2015 Ugorji Nwoke. 412 | 413 | The MIT License (MIT) 414 | 415 | Copyright (c) 2012-2015 Ugorji Nwoke. 416 | All rights reserved. 417 | 418 | Permission is hereby granted, free of charge, to any person obtaining a copy 419 | of this software and associated documentation files (the "Software"), to deal 420 | in the Software without restriction, including without limitation the rights 421 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 422 | copies of the Software, and to permit persons to whom the Software is 423 | furnished to do so, subject to the following conditions: 424 | 425 | The above copyright notice and this permission notice shall be included in all 426 | copies or substantial portions of the Software. 427 | 428 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 429 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 430 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 431 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 432 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 433 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 434 | SOFTWARE. 435 | 436 | ------ 437 | 438 | ** github.com/json-iterator/go; version v1.1.6 -- 439 | https://github.com/json-iterator/go 440 | Copyright (c) 2016 json-iterator 441 | 442 | MIT License 443 | 444 | Copyright (c) 2016 json-iterator 445 | 446 | Permission is hereby granted, free of charge, to any person obtaining a copy 447 | of this software and associated documentation files (the "Software"), to deal 448 | in the Software without restriction, including without limitation the rights 449 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 450 | copies of the Software, and to permit persons to whom the Software is 451 | furnished to do so, subject to the following conditions: 452 | 453 | The above copyright notice and this permission notice shall be included in all 454 | copies or substantial portions of the Software. 455 | 456 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 457 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 458 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 459 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 460 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 461 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 462 | SOFTWARE. 463 | 464 | ------ 465 | 466 | ** github.com/cenkalti/backoff; version v2.1.1 -- 467 | https://github.com/cenkalti/backoff 468 | Copyright (c) 2014 Cenk Altı 469 | 470 | The MIT License (MIT) 471 | 472 | Copyright (c) 2014 Cenk Altı 473 | 474 | Permission is hereby granted, free of charge, to any person obtaining a copy of 475 | this software and associated documentation files (the "Software"), to deal in 476 | the Software without restriction, including without limitation the rights to 477 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 478 | of 479 | the Software, and to permit persons to whom the Software is furnished to do so, 480 | subject to the following conditions: 481 | 482 | The above copyright notice and this permission notice shall be included in all 483 | copies or substantial portions of the Software. 484 | 485 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 486 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 487 | FITNESS 488 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 489 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 490 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 491 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 492 | 493 | ------ 494 | 495 | ** github.com/stretchr/objx; version v0.1.1 -- https://github.com/stretchr/objx 496 | Copyright (c) 2014 Stretchr, Inc. 497 | Copyright (c) 2017-2018 objx contributors 498 | 499 | The MIT License 500 | 501 | Copyright (c) 2014 Stretchr, Inc. 502 | Copyright (c) 2017-2018 objx contributors 503 | 504 | Permission is hereby granted, free of charge, to any person obtaining a copy 505 | of this software and associated documentation files (the "Software"), to deal 506 | in the Software without restriction, including without limitation the rights 507 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 508 | copies of the Software, and to permit persons to whom the Software is 509 | furnished to do so, subject to the following conditions: 510 | 511 | The above copyright notice and this permission notice shall be included in all 512 | copies or substantial portions of the Software. 513 | 514 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 515 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 516 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 517 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 518 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 519 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 520 | SOFTWARE. 521 | -------------------------------------------------------------------------------- /VERSION: -------------------------------------------------------------------------------- 1 | 1.7.2 2 | -------------------------------------------------------------------------------- /firehose/firehose.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | // Package firehose containers the OutputPlugin which sends log records to Firehose 15 | package firehose 16 | 17 | import ( 18 | "bytes" 19 | "fmt" 20 | "os" 21 | "strings" 22 | "time" 23 | 24 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/plugins" 25 | "github.com/aws/aws-sdk-go/aws" 26 | "github.com/aws/aws-sdk-go/aws/awserr" 27 | "github.com/aws/aws-sdk-go/aws/credentials/stscreds" 28 | "github.com/aws/aws-sdk-go/aws/endpoints" 29 | "github.com/aws/aws-sdk-go/aws/session" 30 | "github.com/aws/aws-sdk-go/service/firehose" 31 | fluentbit "github.com/fluent/fluent-bit-go/output" 32 | jsoniter "github.com/json-iterator/go" 33 | "github.com/lestrrat-go/strftime" 34 | "github.com/sirupsen/logrus" 35 | ) 36 | 37 | const ( 38 | // Firehose API Limit https://docs.aws.amazon.com/firehose/latest/dev/limits.html 39 | maximumRecordsPerPut = 500 40 | maximumPutRecordBatchSize = 4194304 // 4 MiB 41 | maximumRecordSize = 1024000 // 1000 KiB 42 | truncatedSuffix = "[Truncated...]" 43 | ) 44 | 45 | const ( 46 | // We use strftime format specifiers because this will one day be re-written in C 47 | defaultTimeFmt = "%Y-%m-%dT%H:%M:%S" 48 | ) 49 | 50 | // PutRecordBatcher contains the firehose PutRecordBatch method call 51 | type PutRecordBatcher interface { 52 | PutRecordBatch(input *firehose.PutRecordBatchInput) (*firehose.PutRecordBatchOutput, error) 53 | } 54 | 55 | // OutputPlugin sends log records to firehose 56 | type OutputPlugin struct { 57 | region string 58 | deliveryStream string 59 | dataKeys string 60 | timeKey string 61 | fmtStrftime *strftime.Strftime 62 | logKey string 63 | client PutRecordBatcher 64 | records []*firehose.Record 65 | dataLength int 66 | timer *plugins.Timeout 67 | PluginID int 68 | replaceDots string 69 | simpleAggregation bool 70 | } 71 | 72 | // NewOutputPlugin creates an OutputPlugin object 73 | func NewOutputPlugin(region, deliveryStream, dataKeys, roleARN, firehoseEndpoint, stsEndpoint, timeKey, timeFmt, logKey, replaceDots string, pluginID int, simpleAggregation bool) (*OutputPlugin, error) { 74 | client, err := newPutRecordBatcher(roleARN, region, firehoseEndpoint, stsEndpoint, pluginID) 75 | if err != nil { 76 | return nil, err 77 | } 78 | 79 | records := make([]*firehose.Record, 0, maximumRecordsPerPut) 80 | 81 | timer, err := plugins.NewTimeout(func(d time.Duration) { 82 | logrus.Errorf("[firehose %d] timeout threshold reached: Failed to send logs for %s\n", pluginID, d.String()) 83 | logrus.Errorf("[firehose %d] Quitting Fluent Bit", pluginID) 84 | os.Exit(1) 85 | }) 86 | 87 | if err != nil { 88 | return nil, err 89 | } 90 | 91 | var timeFormatter *strftime.Strftime 92 | if timeKey != "" { 93 | if timeFmt == "" { 94 | timeFmt = defaultTimeFmt 95 | } 96 | timeFormatter, err = strftime.New(timeFmt, strftime.WithMilliseconds('L'), strftime.WithMicroseconds('f')) 97 | if err != nil { 98 | logrus.Errorf("[firehose %d] Issue with strftime format in 'time_key_format'", pluginID) 99 | return nil, err 100 | } 101 | } 102 | 103 | return &OutputPlugin{ 104 | region: region, 105 | deliveryStream: deliveryStream, 106 | client: client, 107 | records: records, 108 | dataKeys: dataKeys, 109 | timer: timer, 110 | timeKey: timeKey, 111 | fmtStrftime: timeFormatter, 112 | logKey: logKey, 113 | PluginID: pluginID, 114 | replaceDots: replaceDots, 115 | simpleAggregation: simpleAggregation, 116 | }, nil 117 | } 118 | 119 | func newPutRecordBatcher(roleARN, region, firehoseEndpoint, stsEndpoint string, pluginID int) (*firehose.Firehose, error) { 120 | customResolverFn := func(service, region string, optFns ...func(*endpoints.Options)) (endpoints.ResolvedEndpoint, error) { 121 | if service == endpoints.FirehoseServiceID && firehoseEndpoint != "" { 122 | return endpoints.ResolvedEndpoint{ 123 | URL: firehoseEndpoint, 124 | }, nil 125 | } else if service == endpoints.StsServiceID && stsEndpoint != "" { 126 | return endpoints.ResolvedEndpoint{ 127 | URL: stsEndpoint, 128 | }, nil 129 | } 130 | return endpoints.DefaultResolver().EndpointFor(service, region, optFns...) 131 | } 132 | 133 | // Fetching base credentials 134 | baseConfig := &aws.Config{ 135 | Region: aws.String(region), 136 | EndpointResolver: endpoints.ResolverFunc(customResolverFn), 137 | CredentialsChainVerboseErrors: aws.Bool(true), 138 | } 139 | 140 | sess, err := session.NewSession(baseConfig) 141 | if err != nil { 142 | return nil, err 143 | } 144 | 145 | var svcSess = sess 146 | var svcConfig = baseConfig 147 | eksRole := os.Getenv("EKS_POD_EXECUTION_ROLE") 148 | if eksRole != "" { 149 | logrus.Debugf("[firehose %d] Fetching EKS pod credentials.\n", pluginID) 150 | eksConfig := &aws.Config{} 151 | creds := stscreds.NewCredentials(svcSess, eksRole) 152 | eksConfig.Credentials = creds 153 | eksConfig.Region = aws.String(region) 154 | svcConfig = eksConfig 155 | 156 | svcSess, err = session.NewSession(svcConfig) 157 | if err != nil { 158 | return nil, err 159 | } 160 | } 161 | if roleARN != "" { 162 | logrus.Debugf("[firehose %d] Fetching credentials for %s\n", pluginID, roleARN) 163 | stsConfig := &aws.Config{} 164 | creds := stscreds.NewCredentials(svcSess, roleARN) 165 | stsConfig.Credentials = creds 166 | stsConfig.Region = aws.String(region) 167 | svcConfig = stsConfig 168 | 169 | svcSess, err = session.NewSession(svcConfig) 170 | if err != nil { 171 | return nil, err 172 | } 173 | } 174 | 175 | client := firehose.New(svcSess, svcConfig) 176 | client.Handlers.Build.PushBackNamed(plugins.CustomUserAgentHandler()) 177 | return client, nil 178 | } 179 | 180 | // AddRecord accepts a record and adds it to the buffer, flushing the buffer if it is full 181 | // the return value is one of: FLB_OK FLB_RETRY 182 | // API Errors lead to an FLB_RETRY, and all other errors are logged, the record is discarded and FLB_OK is returned 183 | func (output *OutputPlugin) AddRecord(record map[interface{}]interface{}, timeStamp *time.Time) int { 184 | if output.timeKey != "" { 185 | buf := new(bytes.Buffer) 186 | err := output.fmtStrftime.Format(buf, *timeStamp) 187 | if err != nil { 188 | logrus.Errorf("[firehose %d] Could not create timestamp %v\n", output.PluginID, err) 189 | return fluentbit.FLB_ERROR 190 | } 191 | record[output.timeKey] = buf.String() 192 | } 193 | data, err := output.processRecord(record) 194 | if err != nil { 195 | logrus.Errorf("[firehose %d] %v\n", output.PluginID, err) 196 | // discard this single bad record instead and let the batch continue 197 | return fluentbit.FLB_OK 198 | } 199 | 200 | newDataSize := len(data) 201 | 202 | if len(output.records) == maximumRecordsPerPut || (output.dataLength+newDataSize) > maximumPutRecordBatchSize { 203 | retCode, err := output.sendCurrentBatch() 204 | if err != nil { 205 | logrus.Errorf("[firehose %d] %v\n", output.PluginID, err) 206 | } 207 | if retCode != fluentbit.FLB_OK { 208 | return retCode 209 | } 210 | } 211 | 212 | if output.simpleAggregation && len(output.records) > 0 && len(output.records[len(output.records)-1].Data) + newDataSize <= maximumRecordSize { 213 | output.records[len(output.records)-1].Data = append(output.records[len(output.records)-1].Data, data...) 214 | } else { 215 | output.records = append(output.records, &firehose.Record{ 216 | Data: data, 217 | }) 218 | } 219 | output.dataLength += newDataSize 220 | return fluentbit.FLB_OK 221 | } 222 | 223 | // Flush sends the current buffer of records 224 | // Returns FLB_OK, FLB_RETRY, FLB_ERROR 225 | func (output *OutputPlugin) Flush() int { 226 | retCode, err := output.sendCurrentBatch() 227 | if err != nil { 228 | logrus.Errorf("[firehose %d] %v\n", output.PluginID, err) 229 | } 230 | return retCode 231 | } 232 | 233 | func replaceDots(obj map[interface{}]interface{}, replacement string) map[interface{}]interface{} { 234 | for k, v := range obj { 235 | var curK = k 236 | switch kt := k.(type) { 237 | case string: 238 | curK = strings.ReplaceAll(kt, ".", replacement) 239 | } 240 | delete(obj, k) 241 | switch vt := v.(type) { 242 | case map[interface{}]interface{}: 243 | v = replaceDots(vt, replacement) 244 | } 245 | 246 | obj[curK] = v 247 | } 248 | 249 | return obj 250 | } 251 | 252 | func (output *OutputPlugin) processRecord(record map[interface{}]interface{}) ([]byte, error) { 253 | if output.dataKeys != "" { 254 | record = plugins.DataKeys(output.dataKeys, record) 255 | } 256 | 257 | var err error 258 | record, err = plugins.DecodeMap(record) 259 | if err != nil { 260 | logrus.Debugf("[firehose %d] Failed to decode record: %v\n", output.PluginID, record) 261 | return nil, err 262 | } 263 | 264 | if output.replaceDots != "" { 265 | record = replaceDots(record, output.replaceDots) 266 | } 267 | 268 | var json = jsoniter.ConfigCompatibleWithStandardLibrary 269 | var data []byte 270 | 271 | if output.logKey != "" { 272 | log, err := plugins.LogKey(record, output.logKey) 273 | if err != nil { 274 | return nil, err 275 | } 276 | 277 | data, err = plugins.EncodeLogKey(log) 278 | } else { 279 | data, err = json.Marshal(record) 280 | } 281 | 282 | if err != nil { 283 | logrus.Debugf("[firehose %d] Failed to marshal record: %v\n", output.PluginID, record) 284 | return nil, err 285 | } 286 | 287 | // append newline 288 | data = append(data, []byte("\n")...) 289 | 290 | if len(data) > maximumRecordSize { 291 | logrus.Warnf("[firehose %d] Found record with %d bytes, truncating to 1000Kib, stream=%s\n", output.PluginID, len(data), output.deliveryStream) 292 | data = data[:maximumRecordSize-len(truncatedSuffix)] 293 | data = append(data, []byte(truncatedSuffix)...) 294 | } 295 | 296 | return data, nil 297 | } 298 | 299 | func (output *OutputPlugin) sendCurrentBatch() (int, error) { 300 | // return if the batch is empty 301 | if len(output.records) == 0 { 302 | return fluentbit.FLB_OK, nil 303 | } 304 | 305 | output.timer.Check() 306 | 307 | response, err := output.client.PutRecordBatch(&firehose.PutRecordBatchInput{ 308 | DeliveryStreamName: aws.String(output.deliveryStream), 309 | Records: output.records, 310 | }) 311 | if err != nil { 312 | logrus.Errorf("[firehose %d] PutRecordBatch failed with %v", output.PluginID, err) 313 | output.timer.Start() 314 | if aerr, ok := err.(awserr.Error); ok { 315 | if aerr.Code() == firehose.ErrCodeServiceUnavailableException { 316 | logrus.Warnf("[firehose %d] Throughput limits for the delivery stream may have been exceeded.", output.PluginID) 317 | } 318 | } 319 | return fluentbit.FLB_RETRY, err 320 | } 321 | logrus.Debugf("[firehose %d] Sent %d events to Firehose\n", output.PluginID, len(output.records)) 322 | 323 | return output.processAPIResponse(response) 324 | } 325 | 326 | // processAPIResponse processes the successful and failed records 327 | // it returns an error iff no records succeeded (i.e.) no progress has been made 328 | func (output *OutputPlugin) processAPIResponse(response *firehose.PutRecordBatchOutput) (int, error) { 329 | if aws.Int64Value(response.FailedPutCount) > 0 { 330 | // start timer if all records failed (no progress has been made) 331 | if aws.Int64Value(response.FailedPutCount) == int64(len(output.records)) { 332 | output.timer.Start() 333 | return fluentbit.FLB_RETRY, fmt.Errorf("PutRecordBatch request returned with no records successfully recieved") 334 | } 335 | 336 | logrus.Warnf("[firehose %d] %d records failed to be delivered. Will retry.\n", output.PluginID, aws.Int64Value(response.FailedPutCount)) 337 | failedRecords := make([]*firehose.Record, 0, aws.Int64Value(response.FailedPutCount)) 338 | // try to resend failed records 339 | for i, record := range response.RequestResponses { 340 | if record.ErrorMessage != nil { 341 | logrus.Debugf("[firehose %d] Record failed to send with error: %s\n", output.PluginID, aws.StringValue(record.ErrorMessage)) 342 | failedRecords = append(failedRecords, output.records[i]) 343 | } 344 | if aws.StringValue(record.ErrorCode) == firehose.ErrCodeServiceUnavailableException { 345 | logrus.Warnf("[firehose %d] Throughput limits for the delivery stream may have been exceeded.", output.PluginID) 346 | return fluentbit.FLB_RETRY, nil 347 | } 348 | } 349 | 350 | output.records = output.records[:0] 351 | output.records = append(output.records, failedRecords...) 352 | output.dataLength = 0 353 | for _, record := range output.records { 354 | output.dataLength += len(record.Data) 355 | } 356 | 357 | } else { 358 | // request fully succeeded 359 | output.timer.Reset() 360 | output.records = output.records[:0] 361 | output.dataLength = 0 362 | } 363 | 364 | return fluentbit.FLB_OK, nil 365 | } 366 | -------------------------------------------------------------------------------- /firehose/firehose_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package firehose 15 | 16 | import ( 17 | "encoding/json" 18 | "os" 19 | "testing" 20 | "time" 21 | 22 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/firehose/mock_firehose" 23 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/plugins" 24 | "github.com/aws/aws-sdk-go/aws" 25 | "github.com/aws/aws-sdk-go/service/firehose" 26 | fluentbit "github.com/fluent/fluent-bit-go/output" 27 | "github.com/golang/mock/gomock" 28 | "github.com/sirupsen/logrus" 29 | "github.com/stretchr/testify/assert" 30 | ) 31 | 32 | func TestAddRecord(t *testing.T) { 33 | timer, _ := plugins.NewTimeout(func(d time.Duration) { 34 | logrus.Errorf("[firehose] timeout threshold reached: Failed to send logs for %v\n", d) 35 | logrus.Error("[firehose] Quitting Fluent Bit") 36 | os.Exit(1) 37 | }) 38 | output := OutputPlugin{ 39 | region: "us-east-1", 40 | deliveryStream: "stream", 41 | dataKeys: "", 42 | client: nil, 43 | records: make([]*firehose.Record, 0, 500), 44 | timer: timer, 45 | } 46 | 47 | record := map[interface{}]interface{}{ 48 | "somekey": []byte("some value"), 49 | } 50 | 51 | timeStamp := time.Now() 52 | retCode := output.AddRecord(record, &timeStamp) 53 | 54 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 55 | assert.Len(t, output.records, 1, "Expected output to contain 1 record") 56 | } 57 | 58 | func TestAddRecordWithSimpleAggregationEnable(t *testing.T) { 59 | timer, _ := plugins.NewTimeout(func(d time.Duration) { 60 | logrus.Errorf("[firehose] timeout threshold reached: Failed to send logs for %v\n", d) 61 | logrus.Error("[firehose] Quitting Fluent Bit") 62 | os.Exit(1) 63 | }) 64 | output := OutputPlugin{ 65 | region: "us-east-1", 66 | deliveryStream: "stream", 67 | dataKeys: "", 68 | client: nil, 69 | records: make([]*firehose.Record, 0, 500), 70 | timer: timer, 71 | simpleAggregation: true, 72 | } 73 | 74 | record := map[interface{}]interface{}{ 75 | "somekey": []byte("some value"), 76 | } 77 | 78 | timeStamp1 := time.Now() 79 | retCode1 := output.AddRecord(record, &timeStamp1) 80 | timeStamp2 := time.Now() 81 | retCode2 := output.AddRecord(record, &timeStamp2) 82 | 83 | assert.Equal(t, retCode1, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 84 | assert.Equal(t, retCode2, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 85 | assert.Len(t, output.records, 1, "Expected output to contain 1 record") 86 | } 87 | 88 | func TestTruncateLargeLogEvent(t *testing.T) { 89 | timer, _ := plugins.NewTimeout(func(d time.Duration) { 90 | logrus.Errorf("[firehose] timeout threshold reached: Failed to send logs for %v\n", d) 91 | logrus.Error("[firehose] Quitting Fluent Bit") 92 | os.Exit(1) 93 | }) 94 | output := OutputPlugin{ 95 | region: "us-east-1", 96 | deliveryStream: "stream", 97 | dataKeys: "", 98 | client: nil, 99 | records: make([]*firehose.Record, 0, 500), 100 | timer: timer, 101 | } 102 | 103 | record := map[interface{}]interface{}{ 104 | "somekey": make([]byte, 1024005), 105 | } 106 | 107 | timeStamp := time.Now() 108 | retCode := output.AddRecord(record, &timeStamp) 109 | actualData, err := output.processRecord(record) 110 | 111 | if err != nil { 112 | logrus.Debugf("[firehose %d] Failed to marshal record: %v\n", output.PluginID, record) 113 | } 114 | 115 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 116 | assert.Len(t, output.records, 1, "Expected output to contain 1 record") 117 | assert.Len(t, actualData, 1024000, "Expected length is 1024000") 118 | } 119 | 120 | func TestAddRecordAndFlush(t *testing.T) { 121 | record := map[interface{}]interface{}{ 122 | "somekey": []byte("some value"), 123 | } 124 | 125 | ctrl := gomock.NewController(t) 126 | mockFirehose := mock_firehose.NewMockPutRecordBatcher(ctrl) 127 | 128 | mockFirehose.EXPECT().PutRecordBatch(gomock.Any()).Return(&firehose.PutRecordBatchOutput{ 129 | FailedPutCount: aws.Int64(0), 130 | }, nil) 131 | 132 | timer, _ := plugins.NewTimeout(func(d time.Duration) { 133 | logrus.Errorf("[firehose] timeout threshold reached: Failed to send logs for %v\n", d) 134 | logrus.Error("[firehose] Quitting Fluent Bit") 135 | os.Exit(1) 136 | }) 137 | 138 | output := OutputPlugin{ 139 | region: "us-east-1", 140 | deliveryStream: "stream", 141 | dataKeys: "", 142 | client: mockFirehose, 143 | records: make([]*firehose.Record, 0, 500), 144 | timer: timer, 145 | } 146 | 147 | timeStamp := time.Now() 148 | retCode := output.AddRecord(record, &timeStamp) 149 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 150 | 151 | retCode = output.Flush() 152 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 153 | 154 | } 155 | 156 | func TestSendCurrentBatch(t *testing.T) { 157 | output := OutputPlugin{ 158 | region: "us-east-1", 159 | deliveryStream: "stream", 160 | dataKeys: "", 161 | client: nil, 162 | records: nil, 163 | } 164 | 165 | retCode, err := output.sendCurrentBatch() 166 | 167 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 168 | assert.Nil(t, err) 169 | 170 | output.records = make([]*firehose.Record, 0, 500) 171 | retCode, err = output.sendCurrentBatch() 172 | 173 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 174 | assert.Nil(t, err) 175 | 176 | } 177 | 178 | func TestDotReplace(t *testing.T) { 179 | timer, _ := plugins.NewTimeout(func(d time.Duration) { 180 | logrus.Errorf("[firehose] timeout threshold reached: Failed to send logs for %v\n", d) 181 | logrus.Error("[firehose] Quitting Fluent Bit") 182 | os.Exit(1) 183 | }) 184 | output := OutputPlugin{ 185 | region: "us-east-1", 186 | deliveryStream: "stream", 187 | dataKeys: "", 188 | client: nil, 189 | records: make([]*firehose.Record, 0, 500), 190 | timer: timer, 191 | replaceDots: "-", 192 | } 193 | 194 | record := map[interface{}]interface{}{ 195 | "message.key": map[interface{}]interface{}{ 196 | "messagevalue": []byte("some.message"), 197 | "message.value/one": []byte("some message"), 198 | "message.value/two": []byte("some message"), 199 | }, 200 | "kubernetes": map[interface{}]interface{}{ 201 | "app": []byte("test app label"), 202 | "app.kubernetes.io/name": []byte("test key with dots"), 203 | }, 204 | } 205 | 206 | timeStamp := time.Now() 207 | retCode := output.AddRecord(record, &timeStamp) 208 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 209 | assert.Len(t, output.records, 1, "Expected output to contain 1 record") 210 | 211 | data := output.records[0].Data 212 | 213 | var log map[string]map[string]interface{} 214 | json.Unmarshal(data, &log) 215 | 216 | assert.Equal(t, "test app label", log["kubernetes"]["app"]) 217 | assert.Equal(t, "test key with dots", log["kubernetes"]["app-kubernetes-io/name"]) 218 | assert.Equal(t, "some.message", log["message-key"]["messagevalue"]) 219 | assert.Equal(t, "some message", log["message-key"]["message-value/one"]) 220 | assert.Equal(t, "some message", log["message-key"]["message-value/two"]) 221 | } 222 | -------------------------------------------------------------------------------- /firehose/generate_mock.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package firehose 15 | 16 | //go:generate mockgen.sh github.com/aws/amazon-kinesis-firehose-for-fluent-bit/firehose PutRecordBatcher mock_firehose/mock.go 17 | -------------------------------------------------------------------------------- /firehose/mock_firehose/mock.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | // Code generated by MockGen. DO NOT EDIT. 15 | // Source: github.com/aws/amazon-kinesis-firehose-for-fluent-bit/firehose (interfaces: PutRecordBatcher) 16 | 17 | // Package mock_firehose is a generated GoMock package. 18 | package mock_firehose 19 | 20 | import ( 21 | reflect "reflect" 22 | 23 | firehose "github.com/aws/aws-sdk-go/service/firehose" 24 | gomock "github.com/golang/mock/gomock" 25 | ) 26 | 27 | // MockPutRecordBatcher is a mock of PutRecordBatcher interface 28 | type MockPutRecordBatcher struct { 29 | ctrl *gomock.Controller 30 | recorder *MockPutRecordBatcherMockRecorder 31 | } 32 | 33 | // MockPutRecordBatcherMockRecorder is the mock recorder for MockPutRecordBatcher 34 | type MockPutRecordBatcherMockRecorder struct { 35 | mock *MockPutRecordBatcher 36 | } 37 | 38 | // NewMockPutRecordBatcher creates a new mock instance 39 | func NewMockPutRecordBatcher(ctrl *gomock.Controller) *MockPutRecordBatcher { 40 | mock := &MockPutRecordBatcher{ctrl: ctrl} 41 | mock.recorder = &MockPutRecordBatcherMockRecorder{mock} 42 | return mock 43 | } 44 | 45 | // EXPECT returns an object that allows the caller to indicate expected use 46 | func (m *MockPutRecordBatcher) EXPECT() *MockPutRecordBatcherMockRecorder { 47 | return m.recorder 48 | } 49 | 50 | // PutRecordBatch mocks base method 51 | func (m *MockPutRecordBatcher) PutRecordBatch(arg0 *firehose.PutRecordBatchInput) (*firehose.PutRecordBatchOutput, error) { 52 | ret := m.ctrl.Call(m, "PutRecordBatch", arg0) 53 | ret0, _ := ret[0].(*firehose.PutRecordBatchOutput) 54 | ret1, _ := ret[1].(error) 55 | return ret0, ret1 56 | } 57 | 58 | // PutRecordBatch indicates an expected call of PutRecordBatch 59 | func (mr *MockPutRecordBatcherMockRecorder) PutRecordBatch(arg0 interface{}) *gomock.Call { 60 | return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PutRecordBatch", reflect.TypeOf((*MockPutRecordBatcher)(nil).PutRecordBatch), arg0) 61 | } 62 | -------------------------------------------------------------------------------- /fluent-bit-firehose.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package main 15 | 16 | import ( 17 | "C" 18 | "unsafe" 19 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/firehose" 20 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/plugins" 21 | "github.com/fluent/fluent-bit-go/output" 22 | 23 | "github.com/sirupsen/logrus" 24 | ) 25 | import ( 26 | "fmt" 27 | "time" 28 | ) 29 | 30 | var ( 31 | pluginInstances []*firehose.OutputPlugin 32 | ) 33 | 34 | func addPluginInstance(ctx unsafe.Pointer) error { 35 | pluginID := len(pluginInstances) 36 | output.FLBPluginSetContext(ctx, pluginID) 37 | instance, err := newFirehoseOutput(ctx, pluginID) 38 | if err != nil { 39 | return err 40 | } 41 | 42 | pluginInstances = append(pluginInstances, instance) 43 | return nil 44 | } 45 | 46 | func getPluginInstance(ctx unsafe.Pointer) *firehose.OutputPlugin { 47 | pluginID := output.FLBPluginGetContext(ctx).(int) 48 | return pluginInstances[pluginID] 49 | } 50 | 51 | // The "export" comments have syntactic meaning 52 | // This is how the compiler knows a function should be callable from the C code 53 | 54 | //export FLBPluginRegister 55 | func FLBPluginRegister(ctx unsafe.Pointer) int { 56 | return output.FLBPluginRegister(ctx, "firehose", "Amazon Kinesis Data Firehose Fluent Bit Plugin.") 57 | } 58 | 59 | func newFirehoseOutput(ctx unsafe.Pointer, pluginID int) (*firehose.OutputPlugin, error) { 60 | deliveryStream := output.FLBPluginConfigKey(ctx, "delivery_stream") 61 | logrus.Infof("[firehose %d] plugin parameter delivery_stream = '%s'", pluginID, deliveryStream) 62 | region := output.FLBPluginConfigKey(ctx, "region") 63 | logrus.Infof("[firehose %d] plugin parameter region = '%s'", pluginID, region) 64 | dataKeys := output.FLBPluginConfigKey(ctx, "data_keys") 65 | logrus.Infof("[firehose %d] plugin parameter data_keys = '%s'", pluginID, dataKeys) 66 | roleARN := output.FLBPluginConfigKey(ctx, "role_arn") 67 | logrus.Infof("[firehose %d] plugin parameter role_arn = '%s'", pluginID, roleARN) 68 | firehoseEndpoint := output.FLBPluginConfigKey(ctx, "endpoint") 69 | logrus.Infof("[firehose %d] plugin parameter endpoint = '%s'", pluginID, firehoseEndpoint) 70 | stsEndpoint := output.FLBPluginConfigKey(ctx, "sts_endpoint") 71 | logrus.Infof("[firehose %d] plugin parameter sts_endpoint = '%s'", pluginID, stsEndpoint) 72 | timeKey := output.FLBPluginConfigKey(ctx, "time_key") 73 | logrus.Infof("[firehose %d] plugin parameter time_key = '%s'", pluginID, timeKey) 74 | timeKeyFmt := output.FLBPluginConfigKey(ctx, "time_key_format") 75 | logrus.Infof("[firehose %d] plugin parameter time_key_format = '%s'", pluginID, timeKeyFmt) 76 | logKey := output.FLBPluginConfigKey(ctx, "log_key") 77 | logrus.Infof("[firehose %d] plugin parameter log_key = '%s'", pluginID, logKey) 78 | replaceDots := output.FLBPluginConfigKey(ctx, "replace_dots") 79 | logrus.Infof("[firehose %d] plugin parameter replace_dots = '%s'", pluginID, replaceDots) 80 | simpleAggregation := plugins.GetBoolParam(output.FLBPluginConfigKey(ctx, "simple_aggregation"), false) 81 | logrus.Infof("[firehose %d] plugin parameter simple_aggregation = '%v'", 82 | pluginID, simpleAggregation) 83 | if deliveryStream == "" || region == "" { 84 | return nil, fmt.Errorf("[firehose %d] delivery_stream and region are required configuration parameters", pluginID) 85 | } 86 | 87 | return firehose.NewOutputPlugin(region, deliveryStream, dataKeys, roleARN, firehoseEndpoint, stsEndpoint, timeKey, 88 | timeKeyFmt, logKey, replaceDots, pluginID, simpleAggregation) 89 | } 90 | 91 | //export FLBPluginInit 92 | func FLBPluginInit(ctx unsafe.Pointer) int { 93 | plugins.SetupLogger() 94 | 95 | logrus.Info("A new higher performance Firehose plugin has been released; " + 96 | "you are using the old plugin. Check out the new plugin's documentation and " + 97 | "consider migrating.\n" + 98 | "https://docs.fluentbit.io/manual/pipeline/outputs/firehose") 99 | 100 | err := addPluginInstance(ctx) 101 | if err != nil { 102 | logrus.Errorf("[firehose] Failed to initialize plugin: %v", err) 103 | return output.FLB_ERROR 104 | } 105 | return output.FLB_OK 106 | } 107 | 108 | //export FLBPluginFlushCtx 109 | func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int { 110 | var count int 111 | var ret int 112 | var ts interface{} 113 | var timestamp time.Time 114 | var record map[interface{}]interface{} 115 | 116 | // Create Fluent Bit decoder 117 | dec := output.NewDecoder(data, int(length)) 118 | 119 | firehoseOutput := getPluginInstance(ctx) 120 | fluentTag := C.GoString(tag) 121 | logrus.Debugf("[firehose %d] Found logs with tag: %s", firehoseOutput.PluginID, fluentTag) 122 | 123 | for { 124 | // Extract Record 125 | ret, ts, record = output.GetRecord(dec) 126 | if ret != 0 { 127 | break 128 | } 129 | 130 | switch tts := ts.(type) { 131 | case output.FLBTime: 132 | timestamp = tts.Time 133 | case uint64: 134 | // when ts is of type uint64 it appears to 135 | // be the amount of seconds since unix epoch. 136 | timestamp = time.Unix(int64(tts), 0) 137 | default: 138 | timestamp = time.Now() 139 | } 140 | 141 | retCode := firehoseOutput.AddRecord(record, ×tamp) 142 | if retCode != output.FLB_OK { 143 | return retCode 144 | } 145 | count++ 146 | } 147 | retCode := firehoseOutput.Flush() 148 | if retCode != output.FLB_OK { 149 | return retCode 150 | } 151 | logrus.Debugf("[firehose %d] Processed %d events with tag %s", firehoseOutput.PluginID, count, fluentTag) 152 | 153 | return output.FLB_OK 154 | } 155 | 156 | //export FLBPluginExit 157 | func FLBPluginExit() int { 158 | // Before final exit, call Flush() for all the instances of the Output Plugin 159 | for i := range pluginInstances { 160 | pluginInstances[i].Flush() 161 | } 162 | 163 | return output.FLB_OK 164 | } 165 | 166 | func main() { 167 | } 168 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/aws/amazon-kinesis-firehose-for-fluent-bit 2 | 3 | go 1.20 4 | 5 | require ( 6 | github.com/aws/aws-sdk-go v1.44.229 7 | github.com/fluent/fluent-bit-go v0.0.0-20201210173045-3fd1e0486df2 8 | github.com/golang/mock v1.6.0 9 | github.com/json-iterator/go v1.1.12 10 | github.com/lestrrat-go/strftime v1.0.6 11 | github.com/sirupsen/logrus v1.9.0 12 | github.com/stretchr/testify v1.8.2 13 | ) 14 | 15 | require ( 16 | github.com/davecgh/go-spew v1.1.1 // indirect 17 | github.com/jmespath/go-jmespath v0.4.0 // indirect 18 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect 19 | github.com/modern-go/reflect2 v1.0.2 // indirect 20 | github.com/pkg/errors v0.9.1 // indirect 21 | github.com/pmezard/go-difflib v1.0.0 // indirect 22 | github.com/ugorji/go/codec v1.1.7 // indirect 23 | golang.org/x/sys v0.1.0 // indirect 24 | gopkg.in/yaml.v3 v3.0.1 // indirect 25 | ) 26 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/aws/aws-sdk-go v1.44.229 h1:lku0ZSHRzj/qtFVM//QE8VjV6kvJ6CFijDZSsjNaD9A= 2 | github.com/aws/aws-sdk-go v1.44.229/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= 3 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 4 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 5 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 6 | github.com/fluent/fluent-bit-go v0.0.0-20201210173045-3fd1e0486df2 h1:G57WNyWS0FQf43hjRXLy5JT1V5LWVsSiEpkUcT67Ugk= 7 | github.com/fluent/fluent-bit-go v0.0.0-20201210173045-3fd1e0486df2/go.mod h1:L92h+dgwElEyUuShEwjbiHjseW410WIcNz+Bjutc8YQ= 8 | github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc= 9 | github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs= 10 | github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 11 | github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= 12 | github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= 13 | github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8= 14 | github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U= 15 | github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= 16 | github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= 17 | github.com/lestrrat-go/envload v0.0.0-20180220234015-a3eb8ddeffcc h1:RKf14vYWi2ttpEmkA4aQ3j4u9dStX2t4M8UM6qqNsG8= 18 | github.com/lestrrat-go/envload v0.0.0-20180220234015-a3eb8ddeffcc/go.mod h1:kopuH9ugFRkIXf3YoqHKyrJ9YfUFsckUU9S7B+XP+is= 19 | github.com/lestrrat-go/strftime v1.0.6 h1:CFGsDEt1pOpFNU+TJB0nhz9jl+K0hZSLE205AhTIGQQ= 20 | github.com/lestrrat-go/strftime v1.0.6/go.mod h1:f7jQKgV5nnJpYgdEasS+/y7EsTb8ykN2z68n3TtcTaw= 21 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 h1:ZqeYNhU3OHLH3mGKHDcjJRFFRrJa6eAM5H+CtDdOsPc= 22 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 23 | github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= 24 | github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= 25 | github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= 26 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 27 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 28 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 29 | github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= 30 | github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= 31 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 32 | github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= 33 | github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= 34 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 35 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 36 | github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 37 | github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= 38 | github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= 39 | github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= 40 | github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo= 41 | github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw= 42 | github.com/ugorji/go/codec v1.1.7 h1:2SvQaVZ1ouYrrKKwoSk2pzd4A9evlKJb9oTL+OaLUSs= 43 | github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY= 44 | github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= 45 | github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= 46 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 47 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 48 | golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= 49 | golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 50 | golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= 51 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 52 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 53 | golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= 54 | golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= 55 | golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= 56 | golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= 57 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 58 | golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 59 | golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 60 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 61 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 62 | golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 63 | golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 64 | golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 65 | golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 66 | golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 67 | golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 68 | golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 69 | golang.org/x/sys v0.1.0 h1:kunALQeHf1/185U1i0GOB/fy1IPRDDpuoOOqRReG57U= 70 | golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 71 | golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= 72 | golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= 73 | golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= 74 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 75 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 76 | golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= 77 | golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= 78 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 79 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 80 | golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= 81 | golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= 82 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 83 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 84 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 85 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 86 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 87 | gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10= 88 | gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 89 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 90 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 91 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 92 | -------------------------------------------------------------------------------- /plugins/plugins.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | // Package plugins contains functions that are useful across fluent bit plugins. 15 | // This package will be imported by the CloudWatch Logs and Kinesis Data Streams plugins. 16 | package plugins 17 | 18 | import ( 19 | "encoding/json" 20 | "fmt" 21 | "os" 22 | "path" 23 | "runtime" 24 | "strings" 25 | "time" 26 | 27 | "github.com/sirupsen/logrus" 28 | ) 29 | 30 | const ( 31 | fluentBitLogLevelEnvVar = "FLB_LOG_LEVEL" 32 | sendFailureTimeoutEnvVar = "SEND_FAILURE_TIMEOUT" 33 | ) 34 | 35 | // Timeout is a simple timeout for single-threaded programming 36 | // (Goroutines are expensive in Cgo) 37 | type Timeout struct { 38 | timeoutFunc func(time.Duration) 39 | duration time.Duration 40 | stopTime time.Time 41 | ticking bool 42 | enabled bool 43 | } 44 | 45 | // Start the timer 46 | // this method has no effect if the timer has already been started 47 | func (t *Timeout) Start() { 48 | if t.enabled && !t.ticking { 49 | t.ticking = true 50 | t.stopTime = time.Now().Add(t.duration) 51 | } 52 | } 53 | 54 | // Reset the timer 55 | func (t *Timeout) Reset() { 56 | t.ticking = false 57 | } 58 | 59 | // Check the timer to see if its timed out 60 | func (t *Timeout) Check() { 61 | if t.enabled && t.ticking { 62 | if t.stopTime.Before(time.Now()) { 63 | // run the timeout function 64 | t.timeoutFunc(t.duration) 65 | } 66 | } 67 | } 68 | 69 | // EncodeLogKey prepares a record to be sent when the log_key parameter is used 70 | func EncodeLogKey(log *interface{}) ([]byte, error) { 71 | switch t := (*log).(type) { 72 | case []byte: 73 | return t, nil 74 | case string: 75 | return []byte(t), nil 76 | default: 77 | return json.Marshal(log) 78 | } 79 | } 80 | 81 | // LogKey returns the value associated with the input key from the record map, or an error if the key is not found. 82 | func LogKey(record map[interface{}]interface{}, logKey string) (*interface{}, error) { 83 | for key, val := range record { 84 | var currentKey string 85 | switch t := key.(type) { 86 | case []byte: 87 | currentKey = string(t) 88 | case string: 89 | currentKey = t 90 | default: 91 | logrus.Debugf("[go plugin]: Unable to determine type of key %v\n", t) 92 | continue 93 | } 94 | 95 | if logKey == currentKey { 96 | return &val, nil 97 | } 98 | 99 | } 100 | 101 | return nil, fmt.Errorf("Failed to find key %s specified by log_key option in log record: %v", logKey, record) 102 | } 103 | 104 | // NewTimeout returns a new timeout object 105 | // with a duration set from the env var 106 | // if the env var is not set, then a timer is returned that is disabled (it doesn't do anything) 107 | func NewTimeout(timeoutFunc func(duration time.Duration)) (*Timeout, error) { 108 | if timeout := os.Getenv(sendFailureTimeoutEnvVar); timeout != "" { 109 | duration, err := time.ParseDuration(timeout) 110 | if err != nil { 111 | return nil, err 112 | } 113 | return &Timeout{ 114 | timeoutFunc: timeoutFunc, 115 | duration: duration, 116 | ticking: false, 117 | enabled: true, 118 | }, nil 119 | } 120 | 121 | // timeout not enabled 122 | return &Timeout{ 123 | timeoutFunc: timeoutFunc, 124 | ticking: false, 125 | enabled: false, 126 | }, nil 127 | } 128 | 129 | // SetupLogger sets up Logrus with the log level determined by the Fluent Bit Env Var 130 | func SetupLogger() { 131 | logrus.SetOutput(os.Stdout) 132 | 133 | switch strings.ToUpper(os.Getenv(fluentBitLogLevelEnvVar)) { 134 | default: 135 | logrus.SetLevel(logrus.InfoLevel) 136 | case "DEBUG": 137 | logrus.SetLevel(logrus.DebugLevel) 138 | logrus.SetReportCaller(true) 139 | logrus.SetFormatter(&logrus.TextFormatter{ 140 | CallerPrettyfier: func(f *runtime.Frame) (string, string) { 141 | return f.Function + "()", fmt.Sprintf("%s:%d", path.Base(f.File), f.Line) 142 | }, 143 | }) 144 | case "INFO": 145 | logrus.SetLevel(logrus.InfoLevel) 146 | case "ERROR": 147 | logrus.SetLevel(logrus.ErrorLevel) 148 | } 149 | } 150 | 151 | // DecodeMap prepares a record for JSON marshalling 152 | // Any []byte will be base64 encoded when marshaled to JSON, so we must directly cast all []byte to string 153 | func DecodeMap(record map[interface{}]interface{}) (map[interface{}]interface{}, error) { 154 | for k, v := range record { 155 | switch t := v.(type) { 156 | case []byte: 157 | // convert all byte slices to strings 158 | record[k] = string(t) 159 | case map[interface{}]interface{}: 160 | decoded, err := DecodeMap(t) 161 | if err != nil { 162 | return nil, err 163 | } 164 | record[k] = decoded 165 | case []interface{}: 166 | decoded, err := decodeSlice(t) 167 | if err != nil { 168 | return nil, err 169 | } 170 | record[k] = decoded 171 | } 172 | } 173 | return record, nil 174 | } 175 | 176 | // DataKeys allows users to specify a list of keys in the record which they want to be sent 177 | // all others are discarded 178 | func DataKeys(input string, record map[interface{}]interface{}) map[interface{}]interface{} { 179 | input = strings.TrimSpace(input) 180 | keys := strings.Split(input, ",") 181 | 182 | for k := range record { 183 | var currentKey string 184 | switch t := k.(type) { 185 | case []byte: 186 | currentKey = string(t) 187 | case string: 188 | currentKey = t 189 | default: 190 | logrus.Debugf("[go plugin]: Unable to determine type of key %v\n", t) 191 | continue 192 | } 193 | 194 | if !contains(keys, currentKey) { 195 | delete(record, k) 196 | } 197 | } 198 | 199 | return record 200 | } 201 | 202 | func decodeSlice(record []interface{}) ([]interface{}, error) { 203 | for i, v := range record { 204 | switch t := v.(type) { 205 | case []byte: 206 | // convert all byte slices to strings 207 | record[i] = string(t) 208 | case map[interface{}]interface{}: 209 | decoded, err := DecodeMap(t) 210 | if err != nil { 211 | return nil, err 212 | } 213 | record[i] = decoded 214 | case []interface{}: 215 | decoded, err := decodeSlice(t) 216 | if err != nil { 217 | return nil, err 218 | } 219 | record[i] = decoded 220 | } 221 | } 222 | return record, nil 223 | } 224 | 225 | func contains(s []string, e string) bool { 226 | for _, a := range s { 227 | if a == e { 228 | return true 229 | } 230 | } 231 | return false 232 | } 233 | 234 | // GetBoolParam is used for boolean config setup 235 | func GetBoolParam(param string, defaultVal bool) bool { 236 | val := strings.ToLower(param) 237 | if val == "true" { 238 | return true 239 | } else if val == "false" { 240 | return false 241 | } else { 242 | return defaultVal 243 | } 244 | } 245 | -------------------------------------------------------------------------------- /plugins/plugins_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package plugins 15 | 16 | import ( 17 | "testing" 18 | 19 | "github.com/stretchr/testify/assert" 20 | ) 21 | 22 | func TestDecodeMap(t *testing.T) { 23 | sliceVal := []interface{}{ 24 | []byte("Seattle is"), 25 | []byte("rainy"), 26 | } 27 | innerMap := map[interface{}]interface{}{ 28 | "clyde": []byte("best dog"), 29 | "slice_value": sliceVal, 30 | } 31 | record := map[interface{}]interface{}{ 32 | "somekey": []byte("some value"), 33 | "key-with-nested-value": innerMap, 34 | } 35 | 36 | var err error 37 | record, err = DecodeMap(record) 38 | 39 | assert.NoError(t, err, "Unexpected error calling DecodeMap") 40 | 41 | assertTypeIsString(t, record["somekey"]) 42 | assertTypeIsString(t, innerMap["clyde"]) 43 | assertTypeIsString(t, sliceVal[0]) 44 | assertTypeIsString(t, sliceVal[1]) 45 | 46 | } 47 | 48 | func assertTypeIsString(t *testing.T, val interface{}) { 49 | _, ok := val.(string) 50 | assert.True(t, ok, "Expected value to be a string after call to DecodeMap") 51 | } 52 | 53 | func TestDataKeys(t *testing.T) { 54 | record := map[interface{}]interface{}{ 55 | "this": "is a test", 56 | "this is only": "a test", 57 | "dumpling": "is a dog", 58 | "pudding": "is a dog", 59 | "sushi": "is a dog", 60 | "why do": "people name their dogs after food...", 61 | } 62 | 63 | record = DataKeys("dumpling,pudding", record) 64 | 65 | assert.Len(t, record, 2, "Expected record to contain 2 keys") 66 | assert.Equal(t, record["pudding"], "is a dog", "Expected data key to have correct value") 67 | assert.Equal(t, record["dumpling"], "is a dog", "Expected data key to have correct value") 68 | } 69 | 70 | func TestGetBoolParam(t *testing.T) { 71 | value1 := GetBoolParam("true", false) 72 | assert.Equal(t, value1, true, "Expected option value is true") 73 | value2 := GetBoolParam("false", false) 74 | assert.Equal(t, value2, false, "Expected option value is false") 75 | value3 := GetBoolParam("fakeString", false) 76 | assert.Equal(t, value3, false, "Expected option value is false") 77 | } 78 | -------------------------------------------------------------------------------- /plugins/user_agent.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package plugins 15 | 16 | import ( 17 | "fmt" 18 | "runtime" 19 | 20 | "github.com/aws/aws-sdk-go/aws/request" 21 | ) 22 | 23 | const ( 24 | userAgentHeader = "User-Agent" 25 | // linuxBaseUserAgent is the base user agent string used for Linux. 26 | linuxBaseUserAgent = "aws-fluent-bit-plugin" 27 | // windowsBaseUserAgent is the base user agent string used for Windows. 28 | windowsBaseUserAgent = "aws-fluent-bit-plugin-windows" 29 | ) 30 | 31 | // CustomUserAgentHandler returns a http request handler that sets a custom user agent to all aws requests 32 | func CustomUserAgentHandler() request.NamedHandler { 33 | baseUserAgent := linuxBaseUserAgent 34 | if runtime.GOOS == "windows" { 35 | baseUserAgent = windowsBaseUserAgent 36 | } 37 | 38 | return request.NamedHandler{ 39 | Name: "ECSLocalEndpointsAgentHandler", 40 | Fn: func(r *request.Request) { 41 | currentAgent := r.HTTPRequest.Header.Get(userAgentHeader) 42 | r.HTTPRequest.Header.Set(userAgentHeader, 43 | fmt.Sprintf("%s (%s) %s", baseUserAgent, runtime.GOOS, currentAgent)) 44 | }, 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /scripts/mockgen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the 5 | # "License"). You may not use this file except in compliance 6 | # with the License. A copy of the License is located at 7 | # 8 | # http://aws.amazon.com/apache2.0/ 9 | # 10 | # or in the "license" file accompanying this file. This file is 11 | # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR 12 | # CONDITIONS OF ANY KIND, either express or implied. See the 13 | # License for the specific language governing permissions and 14 | # limitations under the License. 15 | # 16 | # This script wraps the mockgen tool and inserts licensing information. 17 | 18 | set -e 19 | package=${1?Must provide package} 20 | interfaces=${2?Must provide interface names} 21 | outputfile=${3?Must provide an output file} 22 | 23 | export PATH="${GOPATH//://bin:}/bin:$PATH" 24 | 25 | data=$( 26 | cat << EOF 27 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 28 | // 29 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 30 | // not use this file except in compliance with the License. A copy of the 31 | // License is located at 32 | // 33 | // http://aws.amazon.com/apache2.0/ 34 | // 35 | // or in the "license" file accompanying this file. This file is distributed 36 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 37 | // express or implied. See the License for the specific language governing 38 | // permissions and limitations under the License. 39 | 40 | $(mockgen "$package" "$interfaces") 41 | EOF 42 | ) 43 | 44 | echo "$data" | goimports > "${outputfile}" 45 | --------------------------------------------------------------------------------