├── .github ├── CODEOWNERS ├── PULL_REQUEST_TEMPLATE.md ├── dependabot.yml └── workflows │ └── build.yml ├── .gitignore ├── CHANGELOG.md ├── CODEOWNERS ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── NOTICE ├── README.md ├── THIRD-PARTY ├── VERSION ├── cloudwatch ├── cloudwatch.go ├── cloudwatch_test.go ├── generate_mock.go ├── handlers.go ├── handlers_test.go ├── helpers.go ├── helpers_test.go └── mock_cloudwatch │ └── mock.go ├── fluent-bit-cloudwatch.go ├── go.mod ├── go.sum └── scripts └── mockgen.sh /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @aws/aws-firelens 2 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | - package-ecosystem: gomod 4 | directory: "/" # Location of package manifests 5 | schedule: 6 | interval: "weekly" 7 | -------------------------------------------------------------------------------- /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | name: Build 2 | 3 | on: 4 | push: 5 | branches: [ mainline ] 6 | pull_request: 7 | branches: [ mainline ] 8 | 9 | jobs: 10 | 11 | build: 12 | name: Build 13 | runs-on: ubuntu-latest 14 | steps: 15 | 16 | - name: Set up Go 1.20 17 | uses: actions/setup-go@v2 18 | with: 19 | go-version: "1.20" 20 | id: go 21 | 22 | - name: Install cross-compiler for Windows 23 | run: sudo apt-get update && sudo apt-get install -y -o Acquire::Retries=3 gcc-multilib gcc-mingw-w64 24 | 25 | - name: Check out code into the Go module directory 26 | uses: actions/checkout@v2 27 | 28 | - name: golint 29 | run: go install golang.org/x/lint/golint@latest 30 | 31 | - name: Build 32 | run: make build windows-release test 33 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Binaries for programs and plugins 3 | *.exe 4 | *.exe~ 5 | *.dll 6 | *.so 7 | *.dylib 8 | 9 | # Test binary, built with `go test -c` 10 | *.test 11 | 12 | # build output dir 13 | bin 14 | 15 | # Output of the go coverage tool, specifically when used with LiteIDE 16 | *.out 17 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## 1.9.4 4 | * Bug - Fix utf-8 calculation of payload length to account for invalid unicode bytes that will be replaced with the 3 byte unicode replacement character. This bug can lead to an `InvalidParameterException` from CloudWatch when the payload sent is calculated to be over the limit due to character replacement. 5 | 6 | ## 1.9.3 7 | * Enhancement - Upgrade Go version to 1.20 8 | 9 | ## 1.9.2 10 | * Bug - Fixed Log Loss can occur when log group creation or retention policy API calls fail. (#314) 11 | 12 | ## 1.9.1 13 | * Enhancement - Added different base user agent for Linux and Windows 14 | 15 | ## 1.9.0 16 | * Feature - Add support for building this plugin on Windows. *Note that this is only support in this plugin repo for Windows compilation.* 17 | 18 | ## 1.8.0 19 | * Feature - Add `auto_create_stream ` option (#257) 20 | * Bug - Allow recovery from a stream being deleted and created by a user (#257) 21 | 22 | ## 1.7.0 23 | * Feature - Add support for external_id (#226) 24 | 25 | ## 1.6.4 26 | * Bug - Remove corrupted unicode fragments on truncation (#208) 27 | 28 | ## 1.6.3 29 | * Enhancement - Upgrade Go version to 1.17 30 | 31 | ## 1.6.2 32 | * Enhancement - Add validation to stop accepting both of `log_stream_name` and `log_stream_prefix` together (#190) 33 | 34 | ## 1.6.1 35 | * Enhancement - Delete debug messages which make log info useless (#146) 36 | 37 | ## 1.6.0 38 | * Enhancement - Add support for updating the retention policy of existing log groups (#121) 39 | 40 | ## 1.5.0 41 | * Feature - Automatically re-create CloudWatch log groups and log streams if they are deleted (#95) 42 | * Feature - Add default fallback log group and stream names (#99) 43 | * Feature - Add support for ECS Metadata and UUID via special variables in log stream and group names (#108) 44 | * Enhancement - Remove invalid characters in log stream and log group names (#103) 45 | 46 | ## 1.4.1 47 | * Bug - Add back `auto_create_group` option (#96) 48 | * Bug - Truncate log events to max size (#85) 49 | 50 | ## 1.4.0 51 | * Feature - Add support for dynamic log group names (#46) 52 | * Feature - Add support for dynamic log stream names (#16) 53 | * Feature - Support tagging of newly created log groups (#51) 54 | * Feature - Support setting log group retention policies (#50) 55 | 56 | ## 1.3.1 57 | * Bug - Check for empty logEvents before calling PutLogEvents (#66) 58 | 59 | ## 1.3.0 60 | * Feature - Add sts_endpoint param for custom STS API endpoint (#55) 61 | 62 | ## 1.2.0 63 | * Feature - Add support for Embedded Metric Format (#27) 64 | 65 | ## 1.1.1 66 | * Bug - Discard and do not send empty messages (#40) 67 | 68 | ## 1.1.0 69 | * Bug - A single CloudWatch Logs PutLogEvents request can not contain logs that span more than 24 hours (#29) 70 | * Feature - Add `credentials_endpoint` option (#36) 71 | * Feature - Support IAM Roles for Service Accounts in Amazon EKS (#33) 72 | 73 | ## 1.0.0 74 | Initial versioned release of the Amazon CloudWatch Logs for Fluent Bit Plugin 75 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | / @aws/aws-firelens 2 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/awslabs/cloudwatch-logs-for-fluent-bit/issues), or [recently closed](https://github.com/awslabs/cloudwatch-logs-for-fluent-bit/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/cloudwatch-logs-for-fluent-bit/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/awslabs/cloudwatch-logs-for-fluent-bit/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"). You 4 | # may not use this file except in compliance with the License. A copy of 5 | # the License is located at 6 | # 7 | # http://aws.amazon.com/apache2.0/ 8 | # 9 | # or in the "license" file accompanying this file. This file is 10 | # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 11 | # ANY KIND, either express or implied. See the License for the specific 12 | # language governing permissions and limitations under the License. 13 | 14 | # Build settings. 15 | GOARCH ?= amd64 16 | COMPILER ?= x86_64-w64-mingw32-gcc # Cross-compiler for Windows 17 | 18 | ROOT := $(shell pwd) 19 | 20 | all: build 21 | 22 | SCRIPT_PATH := $(ROOT)/scripts/:${PATH} 23 | SOURCES := $(shell find . -name '*.go') 24 | PLUGIN_BINARY := ./bin/cloudwatch.so 25 | PLUGIN_BINARY_WINDOWS := ./bin/cloudwatch.dll 26 | 27 | .PHONY: build 28 | build: $(PLUGIN_BINARY) 29 | 30 | $(PLUGIN_BINARY): $(SOURCES) 31 | PATH=${PATH} golint ./cloudwatch 32 | mkdir -p ./bin 33 | go build -buildmode c-shared -o $(PLUGIN_BINARY) ./ 34 | @echo "Built Amazon CloudWatch Logs Fluent Bit Plugin" 35 | 36 | .PHONY: release 37 | release: 38 | mkdir -p ./bin 39 | go build -buildmode c-shared -o $(PLUGIN_BINARY) ./ 40 | @echo "Built Amazon CloudWatch Logs Fluent Bit Plugin" 41 | 42 | .PHONY: windows-release 43 | windows-release: 44 | mkdir -p ./bin 45 | GOOS=windows GOARCH=$(GOARCH) CGO_ENABLED=1 CC=$(COMPILER) go build -buildmode c-shared -o $(PLUGIN_BINARY_WINDOWS) ./ 46 | @echo "Built Amazon CloudWatch Logs Fluent Bit Plugin for Windows" 47 | 48 | .PHONY: generate 49 | generate: $(SOURCES) 50 | PATH=$(SCRIPT_PATH) go generate ./... 51 | 52 | 53 | .PHONY: test 54 | test: 55 | go test -timeout=120s -v -cover ./... 56 | 57 | .PHONY: clean 58 | clean: 59 | rm -rf ./bin/* 60 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Fluent Bit Plugin for CloudWatch Logs 2 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Test Actions Status](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/workflows/Build/badge.svg)](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/actions) 2 | 3 | ## Fluent Bit Plugin for CloudWatch Logs 4 | 5 | **NOTE: A new higher performance Fluent Bit CloudWatch Logs Plugin has been released.** Check out our [official guidance](#new-higher-performance-core-fluent-bit-plugin). 6 | 7 | A Fluent Bit output plugin for CloudWatch Logs 8 | 9 | #### Security disclosures 10 | 11 | If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions [here](https://aws.amazon.com/security/vulnerability-reporting/) or email AWS security directly at [aws-security@amazon.com](mailto:aws-security@amazon.com). 12 | 13 | ### Usage 14 | 15 | Run `make` to build `./bin/cloudwatch.so`. Then use with Fluent Bit: 16 | ``` 17 | ./fluent-bit -e ./cloudwatch.so -i cpu \ 18 | -o cloudwatch \ 19 | -p "region=us-west-2" \ 20 | -p "log_group_name=fluent-bit-cloudwatch" \ 21 | -p "log_stream_name=testing" \ 22 | -p "auto_create_group=true" 23 | ``` 24 | 25 | For building Windows binaries, we need to install `mingw-w64` for cross-compilation. The same can be done using- 26 | ``` 27 | sudo apt-get install -y gcc-multilib gcc-mingw-w64 28 | ``` 29 | After this step, run `make windows-release`. Then use with Fluent Bit on Windows: 30 | ``` 31 | ./fluent-bit.exe -e ./cloudwatch.dll -i dummy ` 32 | -o cloudwatch ` 33 | -p "region=us-west-2" ` 34 | -p "log_group_name=fluent-bit-cloudwatch" ` 35 | -p "log_stream_name=testing" ` 36 | -p "auto_create_group=true" 37 | ``` 38 | 39 | ### Plugin Options 40 | 41 | * `region`: The AWS region. 42 | * `log_group_name`: The name of the CloudWatch Log Group that you want log records sent to. This value allows a template in the form of `$(variable)`. See section [Templating Log Group and Stream Names](#templating-log-group-and-stream-names) for more. Fluent Bit will create missing log groups if `auto_create_group` is set, and will throw an error if it does not have permission. 43 | * `log_stream_name`: The name of the CloudWatch Log Stream that you want log records sent to. This value allows a template in the form of `$(variable)`. See section [Templating Log Group and Stream Names](#templating-log-group-and-stream-names) for more. 44 | * `default_log_group_name`: This required variable is the fallback in case any variables in `log_group_name` fails to parse. Defaults to `fluentbit-default`. 45 | * `default_log_stream_name`: This required variable is the fallback in case any variables in `log_stream_name` fails to parse. Defaults to `/fluentbit-default`. 46 | * `log_stream_prefix`: (deprecated) Prefix for the Log Stream name. Setting this to `prefix-` is the same as setting `log_stream_name = prefix-$(tag)`. 47 | * `log_key`: By default, the whole log record will be sent to CloudWatch. If you specify a key name with this option, then only the value of that key will be sent to CloudWatch. For example, if you are using the Fluentd Docker log driver, you can specify `log_key log` and only the log message will be sent to CloudWatch. 48 | * `log_format`: An optional parameter that can be used to tell CloudWatch the format of the data. A value of `json/emf` enables CloudWatch to extract custom metrics embedded in a JSON payload. See the [Embedded Metric Format](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html). 49 | * `role_arn`: ARN of an IAM role to assume (for cross account access). 50 | * `auto_create_group`: Automatically create log groups (and add tags). Valid values are "true" or "false" (case insensitive). Defaults to false. If you use dynamic variables in your log group name, you may need this to be `true`. 51 | * `auto_create_stream`: Automatically create log streams. Valid values are "true" or "false" (case insensitive). Defaults to true. 52 | * `new_log_group_tags`: Comma/equal delimited string of tags to include with _auto created_ log groups. Example: `"tag=val,cooltag2=my other value"` 53 | * `log_retention_days`: If set to a number greater than zero, and newly create log group's retention policy is set to this many days. 54 | * `endpoint`: Specify a custom endpoint for the CloudWatch Logs API. 55 | * `sts_endpoint`: Specify a custom endpoint for the STS API; used to assume your custom role provided with `role_arn`. 56 | * `credentials_endpoint`: Specify a custom HTTP endpoint to pull credentials from. The HTTP response body should look like the following: 57 | ``` 58 | { 59 | "AccessKeyId": "ACCESS_KEY_ID", 60 | "Expiration": "EXPIRATION_DATE", 61 | "SecretAccessKey": "SECRET_ACCESS_KEY", 62 | "Token": "SECURITY_TOKEN_STRING" 63 | } 64 | ``` 65 | 66 | **Note**: The plugin will always create the log stream, if it does not exist. 67 | 68 | ### Permissions 69 | 70 | This plugin requires the following permissions: 71 | * CreateLogGroup (useful when using dynamic groups) 72 | * CreateLogStream 73 | * DescribeLogStreams 74 | * PutLogEvents 75 | * PutRetentionPolicy (if `log_retention_days` is set > 0) 76 | 77 | ### Credentials 78 | 79 | This plugin uses the AWS SDK Go, and uses its [default credential provider chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html). If you are using the plugin on Amazon EC2 or Amazon ECS or Amazon EKS, the plugin will use your EC2 instance role or [ECS Task role permissions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) or [EKS IAM Roles for Service Accounts for pods](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). The plugin can also retrieve credentials from a [shared credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html), or from the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` environment variables. 80 | 81 | ### Environment Variables 82 | 83 | * `FLB_LOG_LEVEL`: Set the log level for the plugin. Valid values are: `debug`, `info`, and `error` (case insensitive). Default is `info`. **Note**: Setting log level in the Fluent Bit Configuration file using the Service key will not affect the plugin log level (because the plugin is external). 84 | * `SEND_FAILURE_TIMEOUT`: Allows you to configure a timeout if the plugin can not send logs to CloudWatch. The timeout is specified as a [Golang duration](https://golang.org/pkg/time/#ParseDuration), for example: `5m30s`. If the plugin has failed to make any progress for the given period of time, then it will exit and kill Fluent Bit. This is useful in scenarios where you want your logging solution to fail fast if it has been misconfigured (i.e. network or credentials have not been set up to allow it to send to CloudWatch). 85 | 86 | ### Retries and Buffering 87 | 88 | Buffering and retries are managed by the Fluent Bit core engine, not by the plugin. Whenever the plugin encounters any error, it returns a retry to the engine which schedules a retry. This means that log group creation, log stream creation or log retention policy calls can consume a retry if they fail. 89 | 90 | * [Fluent Bit upstream documentation on retries](https://docs.fluentbit.io/manual/administration/scheduling-and-retries) 91 | * [Fluent Bit upstream documentation on buffering](https://docs.fluentbit.io/manual/administration/buffering-and-storage) 92 | * [FireLens OOMKill prevent example for buffering](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/oomkill-prevention) 93 | 94 | ### Templating Log Group and Stream Names 95 | 96 | A template in the form of `$(variable)` can be set in `log_group_name` or `log_stream_name`. `variable` can be a map key name in the log message. To access sub-values in the map use the form `$(variable['subkey'])`. Also, it can be replaced with special values to insert the tag, ECS metadata or a random string in the name. 97 | 98 | Special Values: 99 | * `$(tag)` references the full tag name, `$(tag[0])` and `$(tag[1])` are the first and second values of log tag split on periods. You may access any member by index, 0 through 9. 100 | * `$(uuid)` will insert a random string in the names. The random string is generated automatically with format: 4 bytes of time (seconds) + 16 random bytes. It is created when the plugin starts up and uniquely identifies the output - which means that until Fluent Bit is restarted, it will be the same. If you have multiple CloudWatch outputs, each one will get a unique UUID. 101 | * If your container is running in ECS, `$(variable)` can be set as `$(ecs_task_id)`, `$(ecs_cluster)` or `$(ecs_task_arn)`. It will set ECS metadata into `log_group_name` or `log_stream_name`. 102 | 103 | Here is an example for `fluent-bit.conf`: 104 | 105 | ``` 106 | [INPUT] 107 | Name dummy 108 | Tag dummy.data 109 | Dummy {"pam": {"item": "soup", "item2":{"subitem": "rice"}}} 110 | 111 | [OUTPUT] 112 | Name cloudwatch 113 | Match * 114 | region us-east-1 115 | log_group_name fluent-bit-cloudwatch-$(uuid)-$(tag) 116 | log_stream_name from-fluent-bit-$(pam['item2']['subitem'])-$(ecs_task_id)-$(ecs_cluster) 117 | auto_create_group true 118 | ``` 119 | 120 | And here is the resulting log stream name and log group name: 121 | 122 | ``` 123 | log_group_name fluent-bit-cloudwatch-1jD7P6bbSRtbc9stkWjJZYerO6s-dummy.data 124 | log_stream_name from-fluent-bit-rice-37e873f6-37b4-42a7-af47-eac7275c6152-ecs-local-cluster 125 | ``` 126 | 127 | #### Templating Log Group and Stream Names based on Kubernetes metadata 128 | 129 | If you enable the kubernetes filter, then metadata like the following will be added to each log: 130 | 131 | ``` 132 | kubernetes: { 133 | annotations: { 134 | "kubernetes.io/psp": "eks.privileged" 135 | }, 136 | container_hash: "", 137 | container_name: "myapp", 138 | docker_id: "", 139 | host: "ip-10-1-128-166.us-east-2.compute.internal", 140 | labels: { 141 | app: "myapp", 142 | "pod-template-hash": "" 143 | }, 144 | namespace_name: "default", 145 | pod_id: "198f7dd2-2270-11ea-be47-0a5d932f5920", 146 | pod_name: "myapp-5468c5d4d7-n2swr" 147 | } 148 | ``` 149 | 150 | For help setting up Fluent Bit with kubernetes please see [Kubernetes Logging Powered by AWS for Fluent Bit](https://aws.amazon.com/blogs/containers/kubernetes-logging-powered-by-aws-for-fluent-bit/) or [Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html). 151 | 152 | The kubernetes metadata can be referenced just like any other keys using the templating feature, for example, the following will result in a log group name which is `/eks/{namespace_name}/{pod_name}`. 153 | 154 | ``` 155 | [OUTPUT] 156 | Name cloudwatch 157 | Match kube.* 158 | region us-east-1 159 | log_group_name /eks/$(kubernetes['namespace_name'])/$(kubernetes['pod_name']) 160 | log_stream_name $(kubernetes['namespace_name'])/$(kubernetes['container_name']) 161 | auto_create_group true 162 | ``` 163 | 164 | ### New Higher Performance Core Fluent Bit Plugin 165 | 166 | In the summer of 2020, we released a [new higher performance CloudWatch Logs plugin](https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch) named `cloudwatch_logs`. 167 | 168 | That plugin has a core subset of the features of this older, lower performance and less efficient plugin. Check out its [documentation](https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch). 169 | 170 | #### Do you plan to deprecate this older plugin? 171 | 172 | At this time, we do not. This plugin will continue to be supported. It contains features that have not been ported to the higher performance version. Specifically, the feature for [templating of log group name and streams with ECS Metadata or values in the logs](#templating-log-group-and-stream-names). While [simple templating support](https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch#log-stream-and-group-name-templating-using-record_accessor-syntax) now exists in the high performance plugin, it does not have all of the features of the plugin in this repo. Some users will continue to need the features in this repo. 173 | 174 | #### Which plugin should I use? 175 | 176 | If the features of the higher performance plugin are sufficient for your use cases, please use it. It can achieve higher throughput and will consume less CPU and memory. 177 | 178 | #### How can I migrate to the higher performance plugin? 179 | 180 | It supports a subset of the options of this plugin. For many users, you can simply replace the plugin name `cloudwatch` with the new name `cloudwatch_logs`. Check out its [documentation](https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch). 181 | 182 | #### Do you accept contributions to both plugins? 183 | 184 | Yes. The high performance plugin is written in C, and this plugin is written in Golang. We understand that Go is an easier language for amateur contributors to write code in- that is a key reason why we are continuing to maintain it. 185 | 186 | However, if you can write code in C, please consider contributing new features to the [higher performance plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/out_cloudwatch_logs). 187 | 188 | ### Fluent Bit Versions 189 | 190 | This plugin has been tested with Fluent Bit 1.2.0+. It may not work with older Fluent Bit versions. We recommend using the latest version of Fluent Bit as it will contain the newest features and bug fixes. 191 | 192 | ### Example Fluent Bit Config File 193 | 194 | ``` 195 | [INPUT] 196 | Name forward 197 | Listen 0.0.0.0 198 | Port 24224 199 | 200 | [OUTPUT] 201 | Name cloudwatch 202 | Match * 203 | region us-east-1 204 | log_group_name fluent-bit-cloudwatch 205 | log_stream_prefix from-fluent-bit- 206 | auto_create_group true 207 | ``` 208 | 209 | ### AWS for Fluent Bit 210 | 211 | We distribute a container image with Fluent Bit and these plugins. 212 | 213 | ##### GitHub 214 | 215 | [github.com/aws/aws-for-fluent-bit](https://github.com/aws/aws-for-fluent-bit) 216 | 217 | ##### Amazon ECR Public Gallery 218 | 219 | [aws-for-fluent-bit](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) 220 | 221 | Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command: 222 | 223 | ``` 224 | docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit: 225 | ``` 226 | 227 | For example, you can pull the image with latest version by: 228 | 229 | ``` 230 | docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest 231 | ``` 232 | 233 | If you see errors for image pull limits, try log into public ECR with your AWS credentials: 234 | 235 | ``` 236 | aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws 237 | ``` 238 | 239 | You can check the [Amazon ECR Public official doc](https://docs.aws.amazon.com/AmazonECR/latest/public/get-set-up-for-amazon-ecr.html) for more details. 240 | 241 | ##### Docker Hub 242 | 243 | [amazon/aws-for-fluent-bit](https://hub.docker.com/r/amazon/aws-for-fluent-bit/tags) 244 | 245 | ##### Amazon ECR 246 | 247 | You can use our SSM Public Parameters to find the Amazon ECR image URI in your region: 248 | 249 | ``` 250 | aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/ 251 | ``` 252 | 253 | For more see [our docs](https://github.com/aws/aws-for-fluent-bit#public-images). 254 | 255 | ## License 256 | 257 | This library is licensed under the Apache 2.0 License. 258 | -------------------------------------------------------------------------------- /THIRD-PARTY: -------------------------------------------------------------------------------- 1 | ** github.com/aws/amazon-kinesis-firehose-for-fluent-bit; version c41b42995068 2 | -- https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit 3 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | ** github.com/aws/aws-sdk-go; version v1.20.6 -- 5 | https://github.com/aws/aws-sdk-go 6 | Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved. 7 | Copyright 2014-2015 Stripe, Inc. 8 | ** github.com/fluent/fluent-bit-go; version fc386d263885 -- 9 | https://github.com/fluent/fluent-bit-go 10 | Copyright (C) 2015-2017 Treasure Data Inc. 11 | ** github.com/golang/mock; version 1.3.1 -- https://github.com/golang/mock 12 | Copyright 2010 Google Inc. 13 | ** github.com/jmespath/go-jmespath; version c2b33e8439af -- 14 | https://github.com/jmespath/go-jmespath 15 | Copyright 2015 James Saryerwinnie 16 | ** github.com/modern-go/concurrent; version bacd9c7ef1dd -- 17 | https://github.com/modern-go/concurrent 18 | None 19 | ** github.com/modern-go/reflect2; version v1.0.1 -- 20 | https://github.com/modern-go/reflect2 21 | None 22 | 23 | Apache License 24 | 25 | Version 2.0, January 2004 26 | 27 | http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND 28 | DISTRIBUTION 29 | 30 | 1. Definitions. 31 | 32 | "License" shall mean the terms and conditions for use, reproduction, and 33 | distribution as defined by Sections 1 through 9 of this document. 34 | 35 | "Licensor" shall mean the copyright owner or entity authorized by the 36 | copyright owner that is granting the License. 37 | 38 | "Legal Entity" shall mean the union of the acting entity and all other 39 | entities that control, are controlled by, or are under common control 40 | with that entity. For the purposes of this definition, "control" means 41 | (i) the power, direct or indirect, to cause the direction or management 42 | of such entity, whether by contract or otherwise, or (ii) ownership of 43 | fifty percent (50%) or more of the outstanding shares, or (iii) 44 | beneficial ownership of such entity. 45 | 46 | "You" (or "Your") shall mean an individual or Legal Entity exercising 47 | permissions granted by this License. 48 | 49 | "Source" form shall mean the preferred form for making modifications, 50 | including but not limited to software source code, documentation source, 51 | and configuration files. 52 | 53 | "Object" form shall mean any form resulting from mechanical 54 | transformation or translation of a Source form, including but not limited 55 | to compiled object code, generated documentation, and conversions to 56 | other media types. 57 | 58 | "Work" shall mean the work of authorship, whether in Source or Object 59 | form, made available under the License, as indicated by a copyright 60 | notice that is included in or attached to the work (an example is 61 | provided in the Appendix below). 62 | 63 | "Derivative Works" shall mean any work, whether in Source or Object form, 64 | that is based on (or derived from) the Work and for which the editorial 65 | revisions, annotations, elaborations, or other modifications represent, 66 | as a whole, an original work of authorship. For the purposes of this 67 | License, Derivative Works shall not include works that remain separable 68 | from, or merely link (or bind by name) to the interfaces of, the Work and 69 | Derivative Works thereof. 70 | 71 | "Contribution" shall mean any work of authorship, including the original 72 | version of the Work and any modifications or additions to that Work or 73 | Derivative Works thereof, that is intentionally submitted to Licensor for 74 | inclusion in the Work by the copyright owner or by an individual or Legal 75 | Entity authorized to submit on behalf of the copyright owner. For the 76 | purposes of this definition, "submitted" means any form of electronic, 77 | verbal, or written communication sent to the Licensor or its 78 | representatives, including but not limited to communication on electronic 79 | mailing lists, source code control systems, and issue tracking systems 80 | that are managed by, or on behalf of, the Licensor for the purpose of 81 | discussing and improving the Work, but excluding communication that is 82 | conspicuously marked or otherwise designated in writing by the copyright 83 | owner as "Not a Contribution." 84 | 85 | "Contributor" shall mean Licensor and any individual or Legal Entity on 86 | behalf of whom a Contribution has been received by Licensor and 87 | subsequently incorporated within the Work. 88 | 89 | 2. Grant of Copyright License. Subject to the terms and conditions of this 90 | License, each Contributor hereby grants to You a perpetual, worldwide, 91 | non-exclusive, no-charge, royalty-free, irrevocable copyright license to 92 | reproduce, prepare Derivative Works of, publicly display, publicly perform, 93 | sublicense, and distribute the Work and such Derivative Works in Source or 94 | Object form. 95 | 96 | 3. Grant of Patent License. Subject to the terms and conditions of this 97 | License, each Contributor hereby grants to You a perpetual, worldwide, 98 | non-exclusive, no-charge, royalty-free, irrevocable (except as stated in 99 | this section) patent license to make, have made, use, offer to sell, sell, 100 | import, and otherwise transfer the Work, where such license applies only to 101 | those patent claims licensable by such Contributor that are necessarily 102 | infringed by their Contribution(s) alone or by combination of their 103 | Contribution(s) with the Work to which such Contribution(s) was submitted. 104 | If You institute patent litigation against any entity (including a 105 | cross-claim or counterclaim in a lawsuit) alleging that the Work or a 106 | Contribution incorporated within the Work constitutes direct or contributory 107 | patent infringement, then any patent licenses granted to You under this 108 | License for that Work shall terminate as of the date such litigation is 109 | filed. 110 | 111 | 4. Redistribution. You may reproduce and distribute copies of the Work or 112 | Derivative Works thereof in any medium, with or without modifications, and 113 | in Source or Object form, provided that You meet the following conditions: 114 | 115 | (a) You must give any other recipients of the Work or Derivative Works a 116 | copy of this License; and 117 | 118 | (b) You must cause any modified files to carry prominent notices stating 119 | that You changed the files; and 120 | 121 | (c) You must retain, in the Source form of any Derivative Works that You 122 | distribute, all copyright, patent, trademark, and attribution notices 123 | from the Source form of the Work, excluding those notices that do not 124 | pertain to any part of the Derivative Works; and 125 | 126 | (d) If the Work includes a "NOTICE" text file as part of its 127 | distribution, then any Derivative Works that You distribute must include 128 | a readable copy of the attribution notices contained within such NOTICE 129 | file, excluding those notices that do not pertain to any part of the 130 | Derivative Works, in at least one of the following places: within a 131 | NOTICE text file distributed as part of the Derivative Works; within the 132 | Source form or documentation, if provided along with the Derivative 133 | Works; or, within a display generated by the Derivative Works, if and 134 | wherever such third-party notices normally appear. The contents of the 135 | NOTICE file are for informational purposes only and do not modify the 136 | License. You may add Your own attribution notices within Derivative Works 137 | that You distribute, alongside or as an addendum to the NOTICE text from 138 | the Work, provided that such additional attribution notices cannot be 139 | construed as modifying the License. 140 | 141 | You may add Your own copyright statement to Your modifications and may 142 | provide additional or different license terms and conditions for use, 143 | reproduction, or distribution of Your modifications, or for any such 144 | Derivative Works as a whole, provided Your use, reproduction, and 145 | distribution of the Work otherwise complies with the conditions stated in 146 | this License. 147 | 148 | 5. Submission of Contributions. Unless You explicitly state otherwise, any 149 | Contribution intentionally submitted for inclusion in the Work by You to the 150 | Licensor shall be under the terms and conditions of this License, without 151 | any additional terms or conditions. Notwithstanding the above, nothing 152 | herein shall supersede or modify the terms of any separate license agreement 153 | you may have executed with Licensor regarding such Contributions. 154 | 155 | 6. Trademarks. This License does not grant permission to use the trade 156 | names, trademarks, service marks, or product names of the Licensor, except 157 | as required for reasonable and customary use in describing the origin of the 158 | Work and reproducing the content of the NOTICE file. 159 | 160 | 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in 161 | writing, Licensor provides the Work (and each Contributor provides its 162 | Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 163 | KIND, either express or implied, including, without limitation, any 164 | warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or 165 | FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining 166 | the appropriateness of using or redistributing the Work and assume any risks 167 | associated with Your exercise of permissions under this License. 168 | 169 | 8. Limitation of Liability. In no event and under no legal theory, whether 170 | in tort (including negligence), contract, or otherwise, unless required by 171 | applicable law (such as deliberate and grossly negligent acts) or agreed to 172 | in writing, shall any Contributor be liable to You for damages, including 173 | any direct, indirect, special, incidental, or consequential damages of any 174 | character arising as a result of this License or out of the use or inability 175 | to use the Work (including but not limited to damages for loss of goodwill, 176 | work stoppage, computer failure or malfunction, or any and all other 177 | commercial damages or losses), even if such Contributor has been advised of 178 | the possibility of such damages. 179 | 180 | 9. Accepting Warranty or Additional Liability. While redistributing the Work 181 | or Derivative Works thereof, You may choose to offer, and charge a fee for, 182 | acceptance of support, warranty, indemnity, or other liability obligations 183 | and/or rights consistent with this License. However, in accepting such 184 | obligations, You may act only on Your own behalf and on Your sole 185 | responsibility, not on behalf of any other Contributor, and only if You 186 | agree to indemnify, defend, and hold each Contributor harmless for any 187 | liability incurred by, or claims asserted against, such Contributor by 188 | reason of your accepting any such warranty or additional liability. END OF 189 | TERMS AND CONDITIONS 190 | 191 | APPENDIX: How to apply the Apache License to your work. 192 | 193 | To apply the Apache License to your work, attach the following boilerplate 194 | notice, with the fields enclosed by brackets "[]" replaced with your own 195 | identifying information. (Don't include the brackets!) The text should be 196 | enclosed in the appropriate comment syntax for the file format. We also 197 | recommend that a file or class name and description of purpose be included on 198 | the same "printed page" as the copyright notice for easier identification 199 | within third-party archives. 200 | 201 | Copyright [yyyy] [name of copyright owner] 202 | 203 | Licensed under the Apache License, Version 2.0 (the "License"); 204 | 205 | you may not use this file except in compliance with the License. 206 | 207 | You may obtain a copy of the License at 208 | 209 | http://www.apache.org/licenses/LICENSE-2.0 210 | 211 | Unless required by applicable law or agreed to in writing, software 212 | 213 | distributed under the License is distributed on an "AS IS" BASIS, 214 | 215 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 216 | 217 | See the License for the specific language governing permissions and 218 | 219 | limitations under the License. 220 | 221 | * For github.com/aws/amazon-kinesis-firehose-for-fluent-bit see also this 222 | required NOTICE: 223 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 224 | * For github.com/aws/aws-sdk-go see also this required NOTICE: 225 | Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved. 226 | Copyright 2014-2015 Stripe, Inc. 227 | * For github.com/fluent/fluent-bit-go see also this required NOTICE: 228 | Copyright (C) 2015-2017 Treasure Data Inc. 229 | * For github.com/golang/mock see also this required NOTICE: 230 | Copyright 2010 Google Inc. 231 | * For github.com/jmespath/go-jmespath see also this required NOTICE: 232 | Copyright 2015 James Saryerwinnie 233 | * For github.com/modern-go/concurrent see also this required NOTICE: 234 | None 235 | * For github.com/modern-go/reflect2 see also this required NOTICE: 236 | None 237 | 238 | ------ 239 | 240 | ** golang.org; version go1.12 -- https://golang.org/ 241 | Copyright (c) 2009 The Go Authors. All rights reserved. 242 | 243 | Copyright (c) 2009 The Go Authors. All rights reserved. 244 | 245 | Redistribution and use in source and binary forms, with or without 246 | modification, are permitted provided that the following conditions are 247 | met: 248 | 249 | * Redistributions of source code must retain the above copyright 250 | notice, this list of conditions and the following disclaimer. 251 | * Redistributions in binary form must reproduce the above 252 | copyright notice, this list of conditions and the following disclaimer 253 | in the documentation and/or other materials provided with the 254 | distribution. 255 | * Neither the name of Google Inc. nor the names of its 256 | contributors may be used to endorse or promote products derived from 257 | this software without specific prior written permission. 258 | 259 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 260 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 261 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 262 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 263 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 264 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 265 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 266 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 267 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 268 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 269 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 270 | 271 | ------ 272 | 273 | ** github.com/pmezard/go-difflib; version v1.0.0 -- 274 | https://github.com/pmezard/go-difflib 275 | Copyright (c) 2013, Patrick Mezard 276 | 277 | Copyright (c) 2013, Patrick Mezard 278 | All rights reserved. 279 | 280 | Redistribution and use in source and binary forms, with or without 281 | modification, are permitted provided that the following conditions are 282 | met: 283 | 284 | Redistributions of source code must retain the above copyright 285 | notice, this list of conditions and the following disclaimer. 286 | Redistributions in binary form must reproduce the above copyright 287 | notice, this list of conditions and the following disclaimer in the 288 | documentation and/or other materials provided with the distribution. 289 | The names of its contributors may not be used to endorse or promote 290 | products derived from this software without specific prior written 291 | permission. 292 | 293 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 294 | IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 295 | TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 296 | PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 297 | HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 298 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED 299 | TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 300 | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 301 | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 302 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 303 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 304 | 305 | ------ 306 | 307 | ** github.com/davecgh/go-spew; version v1.1.1 -- 308 | https://github.com/davecgh/go-spew 309 | Copyright (c) 2012-2016 Dave Collins 310 | ** github.com/davecgh/go-spew; version v1.1.1 -- 311 | https://github.com/davecgh/go-spew 312 | Copyright (c) 2012-2016 Dave Collins 313 | 314 | ISC License 315 | 316 | Copyright (c) 2012-2016 Dave Collins 317 | 318 | Permission to use, copy, modify, and/or distribute this software for any 319 | purpose with or without fee is hereby granted, provided that the above 320 | copyright notice and this permission notice appear in all copies. 321 | 322 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 323 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 324 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 325 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 326 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 327 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 328 | OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 329 | 330 | ------ 331 | 332 | ** github.com/stretchr/testify; version v1.3.0 -- 333 | https://github.com/stretchr/testify 334 | Copyright (c) 2012-2018 Mat Ryer and Tyler Bunnell 335 | 336 | MIT License 337 | 338 | Copyright (c) 2012-2018 Mat Ryer and Tyler Bunnell 339 | 340 | Permission is hereby granted, free of charge, to any person obtaining a copy 341 | of this software and associated documentation files (the "Software"), to deal 342 | in the Software without restriction, including without limitation the rights 343 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 344 | copies of the Software, and to permit persons to whom the Software is 345 | furnished to do so, subject to the following conditions: 346 | 347 | The above copyright notice and this permission notice shall be included in all 348 | copies or substantial portions of the Software. 349 | 350 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 351 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 352 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 353 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 354 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 355 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 356 | SOFTWARE. 357 | 358 | ------ 359 | 360 | ** github.com/sirupsen/logrus; version v1.4.2 -- 361 | https://github.com/sirupsen/logrus 362 | Copyright (c) 2014 Simon Eskildsen 363 | 364 | The MIT License (MIT) 365 | 366 | Copyright (c) 2014 Simon Eskildsen 367 | 368 | Permission is hereby granted, free of charge, to any person obtaining a copy 369 | of this software and associated documentation files (the "Software"), to deal 370 | in the Software without restriction, including without limitation the rights 371 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 372 | copies of the Software, and to permit persons to whom the Software is 373 | furnished to do so, subject to the following conditions: 374 | 375 | The above copyright notice and this permission notice shall be included in 376 | all copies or substantial portions of the Software. 377 | 378 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 379 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 380 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 381 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 382 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 383 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 384 | THE SOFTWARE. 385 | 386 | ------ 387 | 388 | ** github.com/konsorten/go-windows-terminal-sequences; version v1.0.1 -- 389 | https://github.com/konsorten/go-windows-terminal-sequences 390 | Copyright (c) 2017 marvin + konsorten GmbH (open-source@konsorten.de) 391 | 392 | (The MIT License) 393 | 394 | Copyright (c) 2017 marvin + konsorten GmbH (open-source@konsorten.de) 395 | 396 | Permission is hereby granted, free of charge, to any person obtaining a copy of 397 | this software and associated documentation files (the 'Software'), to deal in 398 | the Software without restriction, including without limitation the rights to 399 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 400 | of the Software, and to permit persons to whom the Software is furnished to do 401 | so, subject to the following conditions: 402 | 403 | The above copyright notice and this permission notice shall be included in all 404 | copies or substantial portions of the Software. 405 | 406 | THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 407 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 408 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 409 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 410 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 411 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 412 | SOFTWARE. 413 | 414 | ------ 415 | 416 | ** github.com/ugorji/go; version v1.1.4 -- https://github.com/ugorji/go 417 | Copyright (c) 2012-2015 Ugorji Nwoke. 418 | 419 | The MIT License (MIT) 420 | 421 | Copyright (c) 2012-2015 Ugorji Nwoke. 422 | All rights reserved. 423 | 424 | Permission is hereby granted, free of charge, to any person obtaining a copy 425 | of this software and associated documentation files (the "Software"), to deal 426 | in the Software without restriction, including without limitation the rights 427 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 428 | copies of the Software, and to permit persons to whom the Software is 429 | furnished to do so, subject to the following conditions: 430 | 431 | The above copyright notice and this permission notice shall be included in all 432 | copies or substantial portions of the Software. 433 | 434 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 435 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 436 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 437 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 438 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 439 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 440 | SOFTWARE. 441 | 442 | ------ 443 | 444 | ** github.com/json-iterator/go; version v1.1.6 -- 445 | https://github.com/json-iterator/go 446 | Copyright (c) 2016 json-iterator 447 | 448 | MIT License 449 | 450 | Copyright (c) 2016 json-iterator 451 | 452 | Permission is hereby granted, free of charge, to any person obtaining a copy 453 | of this software and associated documentation files (the "Software"), to deal 454 | in the Software without restriction, including without limitation the rights 455 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 456 | copies of the Software, and to permit persons to whom the Software is 457 | furnished to do so, subject to the following conditions: 458 | 459 | The above copyright notice and this permission notice shall be included in all 460 | copies or substantial portions of the Software. 461 | 462 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 463 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 464 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 465 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 466 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 467 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 468 | SOFTWARE. 469 | 470 | ------ 471 | 472 | ** github.com/cenkalti/backoff; version v2.1.1 -- 473 | https://github.com/cenkalti/backoff 474 | Copyright (c) 2014 Cenk Altı 475 | 476 | The MIT License (MIT) 477 | 478 | Copyright (c) 2014 Cenk Altı 479 | 480 | Permission is hereby granted, free of charge, to any person obtaining a copy of 481 | this software and associated documentation files (the "Software"), to deal in 482 | the Software without restriction, including without limitation the rights to 483 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 484 | of 485 | the Software, and to permit persons to whom the Software is furnished to do so, 486 | subject to the following conditions: 487 | 488 | The above copyright notice and this permission notice shall be included in all 489 | copies or substantial portions of the Software. 490 | 491 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 492 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 493 | FITNESS 494 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 495 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 496 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 497 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 498 | 499 | ------ 500 | 501 | ** github.com/stretchr/objx; version v0.1.1 -- https://github.com/stretchr/objx 502 | Copyright (c) 2014 Stretchr, Inc. 503 | Copyright (c) 2017-2018 objx contributors 504 | 505 | The MIT License 506 | 507 | Copyright (c) 2014 Stretchr, Inc. 508 | Copyright (c) 2017-2018 objx contributors 509 | 510 | Permission is hereby granted, free of charge, to any person obtaining a copy 511 | of this software and associated documentation files (the "Software"), to deal 512 | in the Software without restriction, including without limitation the rights 513 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 514 | copies of the Software, and to permit persons to whom the Software is 515 | furnished to do so, subject to the following conditions: 516 | 517 | The above copyright notice and this permission notice shall be included in all 518 | copies or substantial portions of the Software. 519 | 520 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 521 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 522 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 523 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 524 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 525 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 526 | SOFTWARE. 527 | -------------------------------------------------------------------------------- /VERSION: -------------------------------------------------------------------------------- 1 | 1.9.4 2 | -------------------------------------------------------------------------------- /cloudwatch/cloudwatch.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package cloudwatch 15 | 16 | import ( 17 | "encoding/json" 18 | "fmt" 19 | "io/ioutil" 20 | "net/http" 21 | "os" 22 | "runtime" 23 | "sort" 24 | "strings" 25 | "time" 26 | "unicode/utf8" 27 | 28 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/plugins" 29 | "github.com/aws/aws-sdk-go/aws" 30 | "github.com/aws/aws-sdk-go/aws/arn" 31 | "github.com/aws/aws-sdk-go/aws/awserr" 32 | "github.com/aws/aws-sdk-go/aws/credentials/endpointcreds" 33 | "github.com/aws/aws-sdk-go/aws/credentials/stscreds" 34 | "github.com/aws/aws-sdk-go/aws/endpoints" 35 | "github.com/aws/aws-sdk-go/aws/request" 36 | "github.com/aws/aws-sdk-go/aws/session" 37 | "github.com/aws/aws-sdk-go/service/cloudwatchlogs" 38 | fluentbit "github.com/fluent/fluent-bit-go/output" 39 | jsoniter "github.com/json-iterator/go" 40 | "github.com/segmentio/ksuid" 41 | "github.com/sirupsen/logrus" 42 | "github.com/valyala/bytebufferpool" 43 | "github.com/valyala/fasttemplate" 44 | ) 45 | 46 | const ( 47 | // See: http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html 48 | perEventBytes = 26 49 | maximumBytesPerPut = 1048576 50 | maximumLogEventsPerPut = 10000 51 | maximumBytesPerEvent = 1024 * 256 //256KB 52 | maximumTimeSpanPerPut = time.Hour * 24 53 | truncatedSuffix = "[Truncated...]" 54 | maxGroupStreamLength = 512 55 | ) 56 | 57 | const ( 58 | // Log stream objects that are empty and inactive for longer than the timeout get cleaned up 59 | logStreamInactivityTimeout = time.Hour 60 | // Check for expired log streams every 10 minutes 61 | logStreamInactivityCheckInterval = 10 * time.Minute 62 | // linuxBaseUserAgent is the base user agent string used for Linux. 63 | linuxBaseUserAgent = "aws-fluent-bit-plugin" 64 | // windowsBaseUserAgent is the base user agent string used for Windows. 65 | windowsBaseUserAgent = "aws-fluent-bit-plugin-windows" 66 | ) 67 | 68 | // LogsClient contains the CloudWatch API calls used by this plugin 69 | type LogsClient interface { 70 | CreateLogGroup(input *cloudwatchlogs.CreateLogGroupInput) (*cloudwatchlogs.CreateLogGroupOutput, error) 71 | PutRetentionPolicy(input *cloudwatchlogs.PutRetentionPolicyInput) (*cloudwatchlogs.PutRetentionPolicyOutput, error) 72 | CreateLogStream(input *cloudwatchlogs.CreateLogStreamInput) (*cloudwatchlogs.CreateLogStreamOutput, error) 73 | DescribeLogStreams(input *cloudwatchlogs.DescribeLogStreamsInput) (*cloudwatchlogs.DescribeLogStreamsOutput, error) 74 | PutLogEvents(input *cloudwatchlogs.PutLogEventsInput) (*cloudwatchlogs.PutLogEventsOutput, error) 75 | } 76 | 77 | type logStream struct { 78 | logEvents []*cloudwatchlogs.InputLogEvent 79 | currentByteLength int 80 | currentBatchStart *time.Time 81 | currentBatchEnd *time.Time 82 | nextSequenceToken *string 83 | logStreamName string 84 | logGroupName string 85 | expiration time.Time 86 | } 87 | 88 | // Event is the input data and contains a log entry. 89 | // The group and stream are added during processing. 90 | type Event struct { 91 | TS time.Time 92 | Record map[interface{}]interface{} 93 | Tag string 94 | group string 95 | stream string 96 | } 97 | 98 | // TaskMetadata it the task metadata from ECS V3 endpoint 99 | type TaskMetadata struct { 100 | Cluster string `json:"Cluster,omitempty"` 101 | TaskARN string `json:"TaskARN,omitempty"` 102 | TaskID string `json:"TaskID,omitempty"` 103 | } 104 | 105 | type streamDoesntExistError struct { 106 | streamName string 107 | groupName string 108 | } 109 | 110 | func (stream *logStream) isExpired() bool { 111 | if len(stream.logEvents) == 0 && stream.expiration.Before(time.Now()) { 112 | return true 113 | } 114 | return false 115 | } 116 | 117 | func (stream *logStream) updateExpiration() { 118 | stream.expiration = time.Now().Add(logStreamInactivityTimeout) 119 | } 120 | 121 | type fastTemplate struct { 122 | String string 123 | *fasttemplate.Template 124 | } 125 | 126 | // OutputPlugin is the CloudWatch Logs Fluent Bit output plugin 127 | type OutputPlugin struct { 128 | logGroupName *fastTemplate 129 | defaultLogGroupName string 130 | logStreamPrefix string 131 | logStreamName *fastTemplate 132 | defaultLogStreamName string 133 | logKey string 134 | client LogsClient 135 | streams map[string]*logStream 136 | groups map[string]struct{} 137 | timer *plugins.Timeout 138 | nextLogStreamCleanUpCheckTime time.Time 139 | PluginInstanceID int 140 | logGroupTags map[string]*string 141 | logGroupRetention int64 142 | autoCreateGroup bool 143 | autoCreateStream bool 144 | bufferPool bytebufferpool.Pool 145 | ecsMetadata TaskMetadata 146 | runningInECS bool 147 | uuid string 148 | extraUserAgent string 149 | } 150 | 151 | // OutputPluginConfig is the input information used by NewOutputPlugin to create a new OutputPlugin 152 | type OutputPluginConfig struct { 153 | Region string 154 | LogGroupName string 155 | DefaultLogGroupName string 156 | LogStreamPrefix string 157 | LogStreamName string 158 | DefaultLogStreamName string 159 | LogKey string 160 | RoleARN string 161 | AutoCreateGroup bool 162 | AutoCreateStream bool 163 | NewLogGroupTags string 164 | LogRetentionDays int64 165 | CWEndpoint string 166 | STSEndpoint string 167 | ExternalID string 168 | CredsEndpoint string 169 | PluginInstanceID int 170 | LogFormat string 171 | ExtraUserAgent string 172 | } 173 | 174 | // Validate checks the configuration input for an OutputPlugin instances 175 | func (config OutputPluginConfig) Validate() error { 176 | errorStr := "%s is a required parameter" 177 | if config.Region == "" { 178 | return fmt.Errorf(errorStr, "region") 179 | } 180 | if config.LogGroupName == "" { 181 | return fmt.Errorf(errorStr, "log_group_name") 182 | } 183 | if config.LogStreamName == "" && config.LogStreamPrefix == "" { 184 | return fmt.Errorf("log_stream_name or log_stream_prefix is required") 185 | } 186 | 187 | if config.LogStreamName != "" && config.LogStreamPrefix != "" { 188 | return fmt.Errorf("either log_stream_name or log_stream_prefix can be configured. They cannot be provided together") 189 | } 190 | 191 | return nil 192 | } 193 | 194 | // NewOutputPlugin creates a OutputPlugin object 195 | func NewOutputPlugin(config OutputPluginConfig) (*OutputPlugin, error) { 196 | logrus.Debugf("[cloudwatch %d] Initializing NewOutputPlugin", config.PluginInstanceID) 197 | 198 | client, err := newCloudWatchLogsClient(config) 199 | if err != nil { 200 | return nil, err 201 | } 202 | 203 | timer, err := plugins.NewTimeout(func(d time.Duration) { 204 | logrus.Errorf("[cloudwatch %d] timeout threshold reached: Failed to send logs for %s\n", config.PluginInstanceID, d.String()) 205 | logrus.Fatalf("[cloudwatch %d] Quitting Fluent Bit", config.PluginInstanceID) // exit the plugin and kill Fluent Bit 206 | }) 207 | if err != nil { 208 | return nil, err 209 | } 210 | 211 | logGroupTemplate, err := newTemplate(config.LogGroupName) 212 | if err != nil { 213 | return nil, err 214 | } 215 | 216 | logStreamTemplate, err := newTemplate(config.LogStreamName) 217 | if err != nil { 218 | return nil, err 219 | } 220 | 221 | runningInECS := true 222 | // check if it is running in ECS 223 | if os.Getenv("ECS_CONTAINER_METADATA_URI") == "" { 224 | runningInECS = false 225 | } 226 | 227 | return &OutputPlugin{ 228 | logGroupName: logGroupTemplate, 229 | logStreamName: logStreamTemplate, 230 | logStreamPrefix: config.LogStreamPrefix, 231 | defaultLogGroupName: config.DefaultLogGroupName, 232 | defaultLogStreamName: config.DefaultLogStreamName, 233 | logKey: config.LogKey, 234 | client: client, 235 | timer: timer, 236 | streams: make(map[string]*logStream), 237 | nextLogStreamCleanUpCheckTime: time.Now().Add(logStreamInactivityCheckInterval), 238 | PluginInstanceID: config.PluginInstanceID, 239 | logGroupTags: tagKeysToMap(config.NewLogGroupTags), 240 | logGroupRetention: config.LogRetentionDays, 241 | autoCreateGroup: config.AutoCreateGroup, 242 | autoCreateStream: config.AutoCreateStream, 243 | groups: make(map[string]struct{}), 244 | ecsMetadata: TaskMetadata{}, 245 | runningInECS: runningInECS, 246 | uuid: ksuid.New().String(), 247 | extraUserAgent: config.ExtraUserAgent, 248 | }, nil 249 | } 250 | 251 | func newCloudWatchLogsClient(config OutputPluginConfig) (*cloudwatchlogs.CloudWatchLogs, error) { 252 | customResolverFn := func(service, region string, optFns ...func(*endpoints.Options)) (endpoints.ResolvedEndpoint, error) { 253 | if service == endpoints.LogsServiceID && config.CWEndpoint != "" { 254 | return endpoints.ResolvedEndpoint{ 255 | URL: config.CWEndpoint, 256 | }, nil 257 | } else if service == endpoints.StsServiceID && config.STSEndpoint != "" { 258 | return endpoints.ResolvedEndpoint{ 259 | URL: config.STSEndpoint, 260 | }, nil 261 | } 262 | return endpoints.DefaultResolver().EndpointFor(service, region, optFns...) 263 | } 264 | 265 | // Fetch base credentials 266 | baseConfig := &aws.Config{ 267 | Region: aws.String(config.Region), 268 | EndpointResolver: endpoints.ResolverFunc(customResolverFn), 269 | CredentialsChainVerboseErrors: aws.Bool(true), 270 | } 271 | 272 | if config.CredsEndpoint != "" { 273 | creds := endpointcreds.NewCredentialsClient(*baseConfig, request.Handlers{}, config.CredsEndpoint, 274 | func(provider *endpointcreds.Provider) { 275 | provider.ExpiryWindow = 5 * time.Minute 276 | }) 277 | baseConfig.Credentials = creds 278 | } 279 | 280 | sess, err := session.NewSession(baseConfig) 281 | if err != nil { 282 | return nil, err 283 | } 284 | 285 | var svcSess = sess 286 | var svcConfig = baseConfig 287 | eksRole := os.Getenv("EKS_POD_EXECUTION_ROLE") 288 | if eksRole != "" { 289 | logrus.Debugf("[cloudwatch %d] Fetching EKS pod credentials.\n", config.PluginInstanceID) 290 | eksConfig := &aws.Config{} 291 | creds := stscreds.NewCredentials(svcSess, eksRole) 292 | eksConfig.Credentials = creds 293 | eksConfig.Region = aws.String(config.Region) 294 | svcConfig = eksConfig 295 | 296 | svcSess, err = session.NewSession(svcConfig) 297 | if err != nil { 298 | return nil, err 299 | } 300 | } 301 | 302 | if config.RoleARN != "" { 303 | logrus.Debugf("[cloudwatch %d] Fetching credentials for %s\n", config.PluginInstanceID, config.RoleARN) 304 | stsConfig := &aws.Config{} 305 | creds := stscreds.NewCredentials(svcSess, config.RoleARN, func(p *stscreds.AssumeRoleProvider) { 306 | if config.ExternalID != "" { 307 | p.ExternalID = aws.String(config.ExternalID) 308 | } 309 | }) 310 | stsConfig.Credentials = creds 311 | stsConfig.Region = aws.String(config.Region) 312 | svcConfig = stsConfig 313 | 314 | svcSess, err = session.NewSession(svcConfig) 315 | if err != nil { 316 | return nil, err 317 | } 318 | } 319 | 320 | client := cloudwatchlogs.New(svcSess, svcConfig) 321 | client.Handlers.Build.PushBackNamed(customUserAgentHandler(config)) 322 | if config.LogFormat != "" { 323 | client.Handlers.Build.PushBackNamed(LogFormatHandler(config.LogFormat)) 324 | } 325 | return client, nil 326 | } 327 | 328 | // CustomUserAgentHandler returns a http request handler that sets a custom user agent to all aws requests 329 | func customUserAgentHandler(config OutputPluginConfig) request.NamedHandler { 330 | const userAgentHeader = "User-Agent" 331 | 332 | baseUserAgent := linuxBaseUserAgent 333 | if runtime.GOOS == "windows" { 334 | baseUserAgent = windowsBaseUserAgent 335 | } 336 | 337 | return request.NamedHandler{ 338 | Name: "ECSLocalEndpointsAgentHandler", 339 | Fn: func(r *request.Request) { 340 | currentAgent := r.HTTPRequest.Header.Get(userAgentHeader) 341 | if config.ExtraUserAgent != "" { 342 | r.HTTPRequest.Header.Set(userAgentHeader, 343 | fmt.Sprintf("%s-%s (%s) %s", baseUserAgent, config.ExtraUserAgent, runtime.GOOS, currentAgent)) 344 | } else { 345 | r.HTTPRequest.Header.Set(userAgentHeader, 346 | fmt.Sprintf("%s (%s) %s", baseUserAgent, runtime.GOOS, currentAgent)) 347 | } 348 | }, 349 | } 350 | } 351 | 352 | // AddEvent accepts a record and adds it to the buffer for its stream, flushing the buffer if it is full 353 | // the return value is one of: FLB_OK, FLB_RETRY 354 | // API Errors lead to an FLB_RETRY, and all other errors are logged, the record is discarded and FLB_OK is returned 355 | func (output *OutputPlugin) AddEvent(e *Event) int { 356 | // Step 1: convert the Event data to strings, and check for a log key. 357 | data, err := output.processRecord(e) 358 | if err != nil { 359 | logrus.Errorf("[cloudwatch %d] %v\n", output.PluginInstanceID, err) 360 | // discard this single bad record and let the batch continue 361 | return fluentbit.FLB_OK 362 | } 363 | 364 | // Step 2. Make sure the Event data isn't empty. 365 | eventString := logString(data) 366 | if len(eventString) == 0 { 367 | logrus.Debugf("[cloudwatch %d] Discarding an event from publishing as it is empty\n", output.PluginInstanceID) 368 | // discard this single empty record and let the batch continue 369 | return fluentbit.FLB_OK 370 | } 371 | 372 | // Step 3. Extract the Task Metadata if applicable. 373 | if output.runningInECS && output.ecsMetadata.TaskID == "" { 374 | err := output.getECSMetadata() 375 | if err != nil { 376 | logrus.Errorf("[cloudwatch %d] Failed to get ECS Task Metadata with error: %v\n", output.PluginInstanceID, err) 377 | return fluentbit.FLB_RETRY 378 | } 379 | } 380 | 381 | // Step 4. Assign a log group and log stream name to the Event. 382 | output.setGroupStreamNames(e) 383 | 384 | // Step 5. Create a missing log group for this Event. 385 | if _, ok := output.groups[e.group]; !ok { 386 | logrus.Debugf("[cloudwatch %d] Finding log group: %s", output.PluginInstanceID, e.group) 387 | 388 | if err := output.createLogGroup(e); err != nil { 389 | logrus.Error(err) 390 | return fluentbit.FLB_RETRY 391 | } 392 | 393 | output.groups[e.group] = struct{}{} 394 | } 395 | 396 | // Step 6. Create or retrieve an existing log stream for this Event. 397 | stream, err := output.getLogStream(e) 398 | if err != nil { 399 | logrus.Errorf("[cloudwatch %d] %v\n", output.PluginInstanceID, err) 400 | // an error means that the log stream was not created; this is retryable 401 | return fluentbit.FLB_RETRY 402 | } 403 | 404 | // Step 7. Check batch limits and flush buffer if any of these limits will be exeeded by this log Entry. 405 | countLimit := len(stream.logEvents) == maximumLogEventsPerPut 406 | sizeLimit := (stream.currentByteLength + cloudwatchLen(eventString)) >= maximumBytesPerPut 407 | spanLimit := stream.logBatchSpan(e.TS) >= maximumTimeSpanPerPut 408 | if countLimit || sizeLimit || spanLimit { 409 | err = output.putLogEvents(stream) 410 | if err != nil { 411 | logrus.Errorf("[cloudwatch %d] %v\n", output.PluginInstanceID, err) 412 | // send failures are retryable 413 | return fluentbit.FLB_RETRY 414 | } 415 | } 416 | 417 | // Step 8. Add this event to the running tally. 418 | stream.logEvents = append(stream.logEvents, &cloudwatchlogs.InputLogEvent{ 419 | Message: aws.String(eventString), 420 | Timestamp: aws.Int64(e.TS.UnixNano() / 1e6), // CloudWatch uses milliseconds since epoch 421 | }) 422 | stream.currentByteLength += cloudwatchLen(eventString) 423 | if stream.currentBatchStart == nil || stream.currentBatchStart.After(e.TS) { 424 | stream.currentBatchStart = &e.TS 425 | } 426 | if stream.currentBatchEnd == nil || stream.currentBatchEnd.Before(e.TS) { 427 | stream.currentBatchEnd = &e.TS 428 | } 429 | 430 | return fluentbit.FLB_OK 431 | } 432 | 433 | // This plugin tracks CW Log streams 434 | // We need to periodically delete any streams that haven't been written to in a while 435 | // Because each stream incurs some memory for its buffer of log events 436 | // (Which would be empty for an unused stream) 437 | func (output *OutputPlugin) cleanUpExpiredLogStreams() { 438 | if output.nextLogStreamCleanUpCheckTime.Before(time.Now()) { 439 | logrus.Debugf("[cloudwatch %d] Checking for expired log streams", output.PluginInstanceID) 440 | 441 | for name, stream := range output.streams { 442 | if stream.isExpired() { 443 | logrus.Debugf("[cloudwatch %d] Removing internal buffer for log stream %s in group %s; the stream has not been written to for %s", 444 | output.PluginInstanceID, stream.logStreamName, stream.logGroupName, logStreamInactivityTimeout.String()) 445 | delete(output.streams, name) 446 | } 447 | } 448 | output.nextLogStreamCleanUpCheckTime = time.Now().Add(logStreamInactivityCheckInterval) 449 | } 450 | } 451 | 452 | func (err *streamDoesntExistError) Error() string { 453 | return fmt.Sprintf("error: stream %s doesn't exist in log group %s", err.streamName, err.groupName) 454 | } 455 | 456 | func (output *OutputPlugin) getLogStream(e *Event) (*logStream, error) { 457 | stream, ok := output.streams[e.group+e.stream] 458 | if !ok { 459 | // assume the stream exists 460 | stream, err := output.existingLogStream(e) 461 | if err != nil { 462 | // if it doesn't then create it 463 | if _, ok := err.(*streamDoesntExistError); ok { 464 | return output.createStream(e) 465 | } 466 | } 467 | return stream, err 468 | } 469 | return stream, nil 470 | } 471 | 472 | func (output *OutputPlugin) existingLogStream(e *Event) (*logStream, error) { 473 | var nextToken *string 474 | var stream *logStream 475 | 476 | for stream == nil { 477 | resp, err := output.describeLogStreams(e, nextToken) 478 | if err != nil { 479 | return nil, err 480 | } 481 | 482 | for _, result := range resp.LogStreams { 483 | if aws.StringValue(result.LogStreamName) == e.stream { 484 | stream = &logStream{ 485 | logGroupName: e.group, 486 | logStreamName: e.stream, 487 | logEvents: make([]*cloudwatchlogs.InputLogEvent, 0, maximumLogEventsPerPut), 488 | nextSequenceToken: result.UploadSequenceToken, 489 | } 490 | output.streams[e.group+e.stream] = stream 491 | 492 | logrus.Debugf("[cloudwatch %d] Initializing internal buffer for exising log stream %s\n", output.PluginInstanceID, e.stream) 493 | stream.updateExpiration() // initialize 494 | 495 | break 496 | } 497 | } 498 | 499 | if stream == nil && resp.NextToken == nil { 500 | logrus.Infof("[cloudwatch %d] Log stream %s does not exist in log group %s", output.PluginInstanceID, e.stream, e.group) 501 | return nil, &streamDoesntExistError{ 502 | streamName: e.stream, 503 | groupName: e.group, 504 | } 505 | } 506 | 507 | nextToken = resp.NextToken 508 | } 509 | return stream, nil 510 | } 511 | 512 | func (output *OutputPlugin) describeLogStreams(e *Event, nextToken *string) (*cloudwatchlogs.DescribeLogStreamsOutput, error) { 513 | output.timer.Check() 514 | resp, err := output.client.DescribeLogStreams(&cloudwatchlogs.DescribeLogStreamsInput{ 515 | LogGroupName: aws.String(e.group), 516 | LogStreamNamePrefix: aws.String(e.stream), 517 | NextToken: nextToken, 518 | }) 519 | 520 | if err != nil { 521 | output.timer.Start() 522 | return nil, err 523 | } 524 | output.timer.Reset() 525 | 526 | return resp, err 527 | } 528 | 529 | // setGroupStreamNames adds the log group and log stream names to the event struct. 530 | // This happens by parsing (any) template data in either configured name. 531 | func (output *OutputPlugin) setGroupStreamNames(e *Event) { 532 | // This happens here to avoid running Split more than once per log Event. 533 | logTagSplit := strings.SplitN(e.Tag, ".", 10) 534 | s := &sanitizer{sanitize: sanitizeGroup, buf: output.bufferPool.Get()} 535 | 536 | if _, err := parseDataMapTags(e, logTagSplit, output.logGroupName, output.ecsMetadata, output.uuid, s); err != nil { 537 | e.group = output.defaultLogGroupName 538 | logrus.Errorf("[cloudwatch %d] parsing log_group_name template '%s' "+ 539 | "(using value of default_log_group_name instead): %v", 540 | output.PluginInstanceID, output.logGroupName.String, err) 541 | } else if e.group = s.buf.String(); len(e.group) == 0 { 542 | e.group = output.defaultLogGroupName 543 | } else if len(e.group) > maxGroupStreamLength { 544 | e.group = e.group[:maxGroupStreamLength] 545 | } 546 | 547 | if output.logStreamPrefix != "" { 548 | e.stream = output.logStreamPrefix + e.Tag 549 | output.bufferPool.Put(s.buf) 550 | 551 | return 552 | } 553 | 554 | s.sanitize = sanitizeStream 555 | s.buf.Reset() 556 | 557 | if _, err := parseDataMapTags(e, logTagSplit, output.logStreamName, output.ecsMetadata, output.uuid, s); err != nil { 558 | e.stream = output.defaultLogStreamName 559 | logrus.Errorf("[cloudwatch %d] parsing log_stream_name template '%s': %v", 560 | output.PluginInstanceID, output.logStreamName.String, err) 561 | } else if e.stream = s.buf.String(); len(e.stream) == 0 { 562 | e.stream = output.defaultLogStreamName 563 | } else if len(e.stream) > maxGroupStreamLength { 564 | e.stream = e.stream[:maxGroupStreamLength] 565 | } 566 | 567 | output.bufferPool.Put(s.buf) 568 | } 569 | 570 | func (output *OutputPlugin) createStream(e *Event) (*logStream, error) { 571 | if !output.autoCreateStream { 572 | return nil, fmt.Errorf("error: attempting to create log Stream %s in log group %s however auto_create_stream is disabled", e.stream, e.group) 573 | } 574 | output.timer.Check() 575 | _, err := output.client.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{ 576 | LogGroupName: aws.String(e.group), 577 | LogStreamName: aws.String(e.stream), 578 | }) 579 | 580 | if err != nil { 581 | output.timer.Start() 582 | return nil, err 583 | } 584 | output.timer.Reset() 585 | 586 | stream := &logStream{ 587 | logStreamName: e.stream, 588 | logGroupName: e.group, 589 | logEvents: make([]*cloudwatchlogs.InputLogEvent, 0, maximumLogEventsPerPut), 590 | nextSequenceToken: nil, // sequence token not required for a new log stream 591 | } 592 | output.streams[e.group+e.stream] = stream 593 | stream.updateExpiration() // initialize 594 | logrus.Infof("[cloudwatch %d] Created log stream %s in group %s", output.PluginInstanceID, e.stream, e.group) 595 | 596 | return stream, nil 597 | } 598 | 599 | func (output *OutputPlugin) createLogGroup(e *Event) error { 600 | if !output.autoCreateGroup { 601 | return nil 602 | } 603 | 604 | _, err := output.client.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{ 605 | LogGroupName: aws.String(e.group), 606 | Tags: output.logGroupTags, 607 | }) 608 | if err == nil { 609 | logrus.Infof("[cloudwatch %d] Created log group %s\n", output.PluginInstanceID, e.group) 610 | return output.setLogGroupRetention(e.group) 611 | } 612 | 613 | if awsErr, ok := err.(awserr.Error); !ok || 614 | awsErr.Code() != cloudwatchlogs.ErrCodeResourceAlreadyExistsException { 615 | return err 616 | } 617 | 618 | logrus.Infof("[cloudwatch %d] Log group %s already exists\n", output.PluginInstanceID, e.group) 619 | return output.setLogGroupRetention(e.group) 620 | } 621 | 622 | func (output *OutputPlugin) setLogGroupRetention(name string) error { 623 | if output.logGroupRetention < 1 { 624 | return nil 625 | } 626 | 627 | _, err := output.client.PutRetentionPolicy(&cloudwatchlogs.PutRetentionPolicyInput{ 628 | LogGroupName: aws.String(name), 629 | RetentionInDays: aws.Int64(output.logGroupRetention), 630 | }) 631 | if err != nil { 632 | return err 633 | } 634 | 635 | logrus.Infof("[cloudwatch %d] Set retention policy on log group %s to %dd\n", output.PluginInstanceID, name, output.logGroupRetention) 636 | 637 | return nil 638 | } 639 | 640 | // Takes the byte slice and returns a string 641 | // Also removes leading and trailing whitespace 642 | func logString(record []byte) string { 643 | return strings.TrimSpace(string(record)) 644 | } 645 | 646 | func (output *OutputPlugin) processRecord(e *Event) ([]byte, error) { 647 | var err error 648 | e.Record, err = plugins.DecodeMap(e.Record) 649 | if err != nil { 650 | logrus.Debugf("[cloudwatch %d] Failed to decode record: %v\n", output.PluginInstanceID, e.Record) 651 | return nil, err 652 | } 653 | 654 | var json = jsoniter.ConfigCompatibleWithStandardLibrary 655 | var data []byte 656 | 657 | if output.logKey != "" { 658 | log, err := plugins.LogKey(e.Record, output.logKey) 659 | if err != nil { 660 | return nil, err 661 | } 662 | 663 | data, err = plugins.EncodeLogKey(log) 664 | } else { 665 | data, err = json.Marshal(e.Record) 666 | } 667 | 668 | if err != nil { 669 | logrus.Debugf("[cloudwatch %d] Failed to marshal record: %v\nLog Key: %s\n", output.PluginInstanceID, e.Record, output.logKey) 670 | return nil, err 671 | } 672 | 673 | // append newline 674 | data = append(data, []byte("\n")...) 675 | 676 | if (len(data) + perEventBytes) > maximumBytesPerEvent { 677 | logrus.Warnf("[cloudwatch %d] Found record with %d bytes, truncating to 256KB, logGroup=%s, stream=%s\n", 678 | output.PluginInstanceID, len(data)+perEventBytes, e.group, e.stream) 679 | 680 | /* 681 | * Find last byte of trailing unicode character via efficient byte scanning 682 | * Avoids corrupting rune 683 | * 684 | * A unicode character may be composed of 1 - 4 bytes 685 | * bytes [11, 01, 00]xx xxxx: represent the first byte in a unicode character 686 | * byte 10xx xxxx: represent all bytes following the first byte. 687 | * 688 | * nextByte is the first byte that is truncated, 689 | * so nextByte should be the start of a new unicode character in first byte format. 690 | */ 691 | nextByte := (maximumBytesPerEvent - len(truncatedSuffix) - perEventBytes) 692 | for (data[nextByte]&0xc0 == 0x80) && nextByte > 0 { 693 | nextByte-- 694 | } 695 | 696 | data = data[:nextByte] 697 | data = append(data, []byte(truncatedSuffix)...) 698 | } 699 | 700 | return data, nil 701 | } 702 | 703 | func (output *OutputPlugin) getECSMetadata() error { 704 | ecsTaskMetadataEndpointV3 := os.Getenv("ECS_CONTAINER_METADATA_URI") 705 | var metadata TaskMetadata 706 | res, err := http.Get(fmt.Sprintf("%s/task", ecsTaskMetadataEndpointV3)) 707 | if err != nil { 708 | return fmt.Errorf("Failed to get endpoint response: %w", err) 709 | } 710 | response, err := ioutil.ReadAll(res.Body) 711 | if err != nil { 712 | return fmt.Errorf("Failed to read response '%v' from URL: %w", res, err) 713 | } 714 | res.Body.Close() 715 | 716 | err = json.Unmarshal(response, &metadata) 717 | if err != nil { 718 | return fmt.Errorf("Failed to unmarshal ECS metadata '%+v': %w", metadata, err) 719 | } 720 | 721 | arnInfo, err := arn.Parse(metadata.TaskARN) 722 | if err != nil { 723 | return fmt.Errorf("Failed to parse ECS TaskARN '%s': %w", metadata.TaskARN, err) 724 | } 725 | resourceID := strings.Split(arnInfo.Resource, "/") 726 | taskID := resourceID[len(resourceID)-1] 727 | metadata.TaskID = taskID 728 | 729 | output.ecsMetadata = metadata 730 | return nil 731 | } 732 | 733 | // Flush sends the current buffer of records. 734 | func (output *OutputPlugin) Flush() error { 735 | logrus.Debugf("[cloudwatch %d] Flush() Called", output.PluginInstanceID) 736 | 737 | for _, stream := range output.streams { 738 | if err := output.flushStream(stream); err != nil { 739 | return err 740 | } 741 | } 742 | 743 | return nil 744 | } 745 | 746 | func (output *OutputPlugin) flushStream(stream *logStream) error { 747 | output.cleanUpExpiredLogStreams() // will periodically clean up, otherwise is no-op 748 | return output.putLogEvents(stream) 749 | } 750 | 751 | func (output *OutputPlugin) putLogEvents(stream *logStream) error { 752 | // return in case of empty logEvents 753 | if len(stream.logEvents) == 0 { 754 | return nil 755 | } 756 | 757 | output.timer.Check() 758 | stream.updateExpiration() 759 | 760 | // Log events in a single PutLogEvents request must be in chronological order. 761 | sort.SliceStable(stream.logEvents, func(i, j int) bool { 762 | return aws.Int64Value(stream.logEvents[i].Timestamp) < aws.Int64Value(stream.logEvents[j].Timestamp) 763 | }) 764 | response, err := output.client.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{ 765 | LogEvents: stream.logEvents, 766 | LogGroupName: aws.String(stream.logGroupName), 767 | LogStreamName: aws.String(stream.logStreamName), 768 | SequenceToken: stream.nextSequenceToken, 769 | }) 770 | if err != nil { 771 | if awsErr, ok := err.(awserr.Error); ok { 772 | if awsErr.Code() == cloudwatchlogs.ErrCodeDataAlreadyAcceptedException { 773 | // already submitted, just grab the correct sequence token 774 | parts := strings.Split(awsErr.Message(), " ") 775 | stream.nextSequenceToken = &parts[len(parts)-1] 776 | stream.logEvents = stream.logEvents[:0] 777 | stream.currentByteLength = 0 778 | stream.currentBatchStart = nil 779 | stream.currentBatchEnd = nil 780 | logrus.Infof("[cloudwatch %d] Encountered error %v; data already accepted, ignoring error\n", output.PluginInstanceID, awsErr) 781 | return nil 782 | } else if awsErr.Code() == cloudwatchlogs.ErrCodeInvalidSequenceTokenException { 783 | // sequence code is bad, grab the correct one and retry 784 | parts := strings.Split(awsErr.Message(), " ") 785 | nextSequenceToken := &parts[len(parts)-1] 786 | // If this is a new stream then the error will end like "The next expected sequenceToken is: null" and sequenceToken should be nil 787 | if strings.HasPrefix(*nextSequenceToken, "null") { 788 | nextSequenceToken = nil 789 | } 790 | stream.nextSequenceToken = nextSequenceToken 791 | 792 | return output.putLogEvents(stream) 793 | } else if awsErr.Code() == cloudwatchlogs.ErrCodeResourceNotFoundException { 794 | // a log group or a log stream should be re-created after it is deleted and then retry 795 | logrus.Errorf("[cloudwatch %d] Encountered error %v; detailed information: %s\n", output.PluginInstanceID, awsErr, awsErr.Message()) 796 | if strings.Contains(awsErr.Message(), "group") { 797 | if err := output.createLogGroup(&Event{group: stream.logGroupName}); err != nil { 798 | logrus.Errorf("[cloudwatch %d] Encountered error %v\n", output.PluginInstanceID, err) 799 | return err 800 | } 801 | } else if strings.Contains(awsErr.Message(), "stream") { 802 | if _, err := output.createStream(&Event{group: stream.logGroupName, stream: stream.logStreamName}); err != nil { 803 | logrus.Errorf("[cloudwatch %d] Encountered error %v\n", output.PluginInstanceID, err) 804 | return err 805 | } 806 | } 807 | 808 | return fmt.Errorf("A Log group/stream did not exist, re-created it. Will retry PutLogEvents on next flush") 809 | } else { 810 | output.timer.Start() 811 | return err 812 | } 813 | } else { 814 | return err 815 | } 816 | } 817 | output.processRejectedEventsInfo(response) 818 | output.timer.Reset() 819 | logrus.Debugf("[cloudwatch %d] Sent %d events to CloudWatch for stream '%s' in group '%s'", 820 | output.PluginInstanceID, len(stream.logEvents), stream.logStreamName, stream.logGroupName) 821 | 822 | stream.nextSequenceToken = response.NextSequenceToken 823 | stream.logEvents = stream.logEvents[:0] 824 | stream.currentByteLength = 0 825 | stream.currentBatchStart = nil 826 | stream.currentBatchEnd = nil 827 | 828 | return nil 829 | } 830 | 831 | func (output *OutputPlugin) processRejectedEventsInfo(response *cloudwatchlogs.PutLogEventsOutput) { 832 | if response.RejectedLogEventsInfo != nil { 833 | if response.RejectedLogEventsInfo.ExpiredLogEventEndIndex != nil { 834 | logrus.Warnf("[cloudwatch %d] %d log events were marked as expired by CloudWatch\n", output.PluginInstanceID, aws.Int64Value(response.RejectedLogEventsInfo.ExpiredLogEventEndIndex)) 835 | } 836 | if response.RejectedLogEventsInfo.TooNewLogEventStartIndex != nil { 837 | logrus.Warnf("[cloudwatch %d] %d log events were marked as too new by CloudWatch\n", output.PluginInstanceID, aws.Int64Value(response.RejectedLogEventsInfo.TooNewLogEventStartIndex)) 838 | } 839 | if response.RejectedLogEventsInfo.TooOldLogEventEndIndex != nil { 840 | logrus.Warnf("[cloudwatch %d] %d log events were marked as too old by CloudWatch\n", output.PluginInstanceID, aws.Int64Value(response.RejectedLogEventsInfo.TooOldLogEventEndIndex)) 841 | } 842 | } 843 | } 844 | 845 | // counts the effective number of bytes in the string, after 846 | // UTF-8 normalization. UTF-8 normalization includes replacing bytes that do 847 | // not constitute valid UTF-8 encoded Unicode codepoints with the Unicode 848 | // replacement codepoint U+FFFD (a 3-byte UTF-8 sequence, represented in Go as 849 | // utf8.RuneError) 850 | // this works because Go range will parse the string as UTF-8 runes 851 | // copied from AWSLogs driver: https://github.com/moby/moby/commit/1e8ef386279e2e28aff199047e798fad660efbdd 852 | func cloudwatchLen(event string) int { 853 | effectiveBytes := perEventBytes 854 | for _, rune := range event { 855 | effectiveBytes += utf8.RuneLen(rune) 856 | } 857 | return effectiveBytes 858 | } 859 | 860 | func (stream *logStream) logBatchSpan(timestamp time.Time) time.Duration { 861 | if stream.currentBatchStart == nil || stream.currentBatchEnd == nil { 862 | return 0 863 | } 864 | 865 | if stream.currentBatchStart.After(timestamp) { 866 | return stream.currentBatchEnd.Sub(timestamp) 867 | } else if stream.currentBatchEnd.Before(timestamp) { 868 | return timestamp.Sub(*stream.currentBatchStart) 869 | } 870 | 871 | return stream.currentBatchEnd.Sub(*stream.currentBatchStart) 872 | } 873 | -------------------------------------------------------------------------------- /cloudwatch/cloudwatch_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package cloudwatch 15 | 16 | import ( 17 | "bytes" 18 | "errors" 19 | "fmt" 20 | "os" 21 | "strings" 22 | "testing" 23 | "time" 24 | 25 | "github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/cloudwatch/mock_cloudwatch" 26 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/plugins" 27 | "github.com/aws/aws-sdk-go/aws" 28 | "github.com/aws/aws-sdk-go/aws/awserr" 29 | "github.com/aws/aws-sdk-go/service/cloudwatchlogs" 30 | fluentbit "github.com/fluent/fluent-bit-go/output" 31 | "github.com/golang/mock/gomock" 32 | "github.com/sirupsen/logrus" 33 | "github.com/stretchr/testify/assert" 34 | ) 35 | 36 | const ( 37 | testRegion = "us-west-2" 38 | testLogGroup = "my-logs" 39 | testLogStreamPrefix = "my-prefix" 40 | testTag = "tag" 41 | testNextToken = "next-token" 42 | testSequenceToken = "sequence-token" 43 | ) 44 | 45 | type configTest struct { 46 | name string 47 | config OutputPluginConfig 48 | isValidConfig bool 49 | expectedError string 50 | } 51 | 52 | var ( 53 | configValidationTestCases = []configTest{ 54 | { 55 | name: "ValidConfiguration", 56 | config: OutputPluginConfig{ 57 | Region: testRegion, 58 | LogGroupName: testLogGroup, 59 | LogStreamPrefix: testLogStreamPrefix, 60 | }, 61 | isValidConfig: true, 62 | expectedError: "", 63 | }, 64 | { 65 | name: "MissingRegion", 66 | config: OutputPluginConfig{ 67 | LogGroupName: testLogGroup, 68 | LogStreamPrefix: testLogStreamPrefix, 69 | }, 70 | isValidConfig: false, 71 | expectedError: "region is a required parameter", 72 | }, 73 | { 74 | name: "MissingLogGroup", 75 | config: OutputPluginConfig{ 76 | Region: testRegion, 77 | LogStreamPrefix: testLogStreamPrefix, 78 | }, 79 | isValidConfig: false, 80 | expectedError: "log_group_name is a required parameter", 81 | }, 82 | { 83 | name: "OnlyLogStreamNameProvided", 84 | config: OutputPluginConfig{ 85 | Region: testRegion, 86 | LogGroupName: testLogGroup, 87 | LogStreamName: "testLogStream", 88 | }, 89 | isValidConfig: true, 90 | }, 91 | { 92 | name: "OnlyLogStreamPrefixProvided", 93 | config: OutputPluginConfig{ 94 | Region: testRegion, 95 | LogGroupName: testLogGroup, 96 | LogStreamPrefix: testLogStreamPrefix, 97 | }, 98 | isValidConfig: true, 99 | }, 100 | { 101 | name: "LogStreamAndPrefixBothProvided", 102 | config: OutputPluginConfig{ 103 | Region: testRegion, 104 | LogGroupName: testLogGroup, 105 | LogStreamName: "testLogStream", 106 | LogStreamPrefix: testLogStreamPrefix, 107 | }, 108 | isValidConfig: false, 109 | expectedError: "either log_stream_name or log_stream_prefix can be configured. They cannot be provided together", 110 | }, 111 | { 112 | name: "LogStreamAndPrefixBothMissing", 113 | config: OutputPluginConfig{ 114 | Region: testRegion, 115 | LogGroupName: testLogGroup, 116 | }, 117 | isValidConfig: false, 118 | expectedError: "log_stream_name or log_stream_prefix is required", 119 | }, 120 | } 121 | ) 122 | 123 | // helper function to make a log stream/log group name template from a string. 124 | func testTemplate(template string) *fastTemplate { 125 | t, _ := newTemplate(template) 126 | return t 127 | } 128 | 129 | func TestAddEvent(t *testing.T) { 130 | ctrl := gomock.NewController(t) 131 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 132 | 133 | gomock.InOrder( 134 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 135 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 136 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 137 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 138 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 139 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log group name to match") 140 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 141 | ) 142 | 143 | output := OutputPlugin{ 144 | logGroupName: testTemplate(testLogGroup), 145 | logStreamPrefix: testLogStreamPrefix, 146 | client: mockCloudWatch, 147 | timer: setupTimeout(), 148 | streams: make(map[string]*logStream), 149 | groups: map[string]struct{}{testLogGroup: {}}, 150 | autoCreateStream: true, 151 | } 152 | 153 | record := map[interface{}]interface{}{ 154 | "somekey": []byte("some value"), 155 | } 156 | 157 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 158 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 159 | } 160 | 161 | func TestTruncateLargeLogEvent(t *testing.T) { 162 | ctrl := gomock.NewController(t) 163 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 164 | 165 | gomock.InOrder( 166 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 167 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 168 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 169 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 170 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 171 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log group name to match") 172 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 173 | ) 174 | 175 | output := OutputPlugin{ 176 | logGroupName: testTemplate(testLogGroup), 177 | logStreamPrefix: testLogStreamPrefix, 178 | client: mockCloudWatch, 179 | timer: setupTimeout(), 180 | streams: make(map[string]*logStream), 181 | groups: map[string]struct{}{testLogGroup: {}}, 182 | autoCreateStream: true, 183 | } 184 | 185 | record := map[interface{}]interface{}{ 186 | "somekey": make([]byte, 256*1024+100), 187 | } 188 | 189 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 190 | actualData, err := output.processRecord(&Event{TS: time.Now(), Tag: testTag, Record: record}) 191 | 192 | if err != nil { 193 | logrus.Debugf("[cloudwatch %d] Failed to process record: %v\n", output.PluginInstanceID, record) 194 | } 195 | 196 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 197 | assert.Len(t, actualData, 256*1024-26, "Expected length is 256*1024-26") 198 | } 199 | 200 | func TestTruncateLargeLogEventWithSpecialCharacterOneTrailingFragments(t *testing.T) { 201 | ctrl := gomock.NewController(t) 202 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 203 | 204 | gomock.InOrder( 205 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 206 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 207 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 208 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 209 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 210 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log group name to match") 211 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 212 | ) 213 | 214 | output := OutputPlugin{ 215 | logGroupName: testTemplate(testLogGroup), 216 | logStreamPrefix: testLogStreamPrefix, 217 | client: mockCloudWatch, 218 | timer: setupTimeout(), 219 | streams: make(map[string]*logStream), 220 | groups: map[string]struct{}{testLogGroup: {}}, 221 | autoCreateStream: true, 222 | } 223 | 224 | var b bytes.Buffer 225 | for i := 0; i < 262095; i++ { 226 | b.WriteString("x") 227 | } 228 | b.WriteString("𒁈zrgchimqigtm") 229 | 230 | record := map[interface{}]interface{}{ 231 | "key": b.String(), 232 | } 233 | 234 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 235 | actualData, err := output.processRecord(&Event{TS: time.Now(), Tag: testTag, Record: record}) 236 | 237 | if err != nil { 238 | logrus.Debugf("[cloudwatch %d] Failed to process record: %v\n", output.PluginInstanceID, record) 239 | } 240 | 241 | /* invalid characters will be expanded when sent as request */ 242 | actualDataString := logString(actualData) 243 | actualDataString = fmt.Sprintf("%q", actualDataString) /* converts: -> \x */ 244 | 245 | exampleWorkingData := "{\"key\":\"x\"}" 246 | addedLength := len(fmt.Sprintf("%q", exampleWorkingData)) - len(exampleWorkingData) 247 | 248 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 249 | assert.LessOrEqual(t, len(actualDataString), 256*1024-26+addedLength, "Expected length to be less than or equal to 256*1024-26") 250 | } 251 | 252 | func TestTruncateLargeLogEventWithSpecialCharacterTwoTrailingFragments(t *testing.T) { 253 | ctrl := gomock.NewController(t) 254 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 255 | gomock.InOrder( 256 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 257 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 258 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 259 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 260 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 261 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log group name to match") 262 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 263 | ) 264 | 265 | output := OutputPlugin{ 266 | logGroupName: testTemplate(testLogGroup), 267 | logStreamPrefix: testLogStreamPrefix, 268 | client: mockCloudWatch, 269 | timer: setupTimeout(), 270 | streams: make(map[string]*logStream), 271 | groups: map[string]struct{}{testLogGroup: {}}, 272 | autoCreateStream: true, 273 | } 274 | 275 | var b bytes.Buffer 276 | for i := 0; i < 262094; i++ { 277 | b.WriteString("x") 278 | } 279 | b.WriteString("𒁈zrgchimqigtm") 280 | 281 | record := map[interface{}]interface{}{ 282 | "key": b.String(), 283 | } 284 | 285 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 286 | actualData, err := output.processRecord(&Event{TS: time.Now(), Tag: testTag, Record: record}) 287 | 288 | if err != nil { 289 | logrus.Debugf("[cloudwatch %d] Failed to process record: %v\n", output.PluginInstanceID, record) 290 | } 291 | 292 | /* invalid characters will be expanded when sent as request */ 293 | actualDataString := logString(actualData) 294 | actualDataString = fmt.Sprintf("%q", actualDataString) /* converts: -> \x */ 295 | 296 | exampleWorkingData := "{\"key\":\"x\"}" 297 | addedLength := len(fmt.Sprintf("%q", exampleWorkingData)) - len(exampleWorkingData) 298 | 299 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 300 | assert.LessOrEqual(t, len(actualDataString), 256*1024-26+addedLength, "Expected length to be less than or equal to 256*1024-26") 301 | } 302 | 303 | func TestTruncateLargeLogEventWithSpecialCharacterThreeTrailingFragments(t *testing.T) { 304 | ctrl := gomock.NewController(t) 305 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 306 | gomock.InOrder( 307 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 308 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 309 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 310 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 311 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 312 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log group name to match") 313 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 314 | ) 315 | 316 | output := OutputPlugin{ 317 | logGroupName: testTemplate(testLogGroup), 318 | logStreamPrefix: testLogStreamPrefix, 319 | client: mockCloudWatch, 320 | timer: setupTimeout(), 321 | streams: make(map[string]*logStream), 322 | groups: map[string]struct{}{testLogGroup: {}}, 323 | autoCreateStream: true, 324 | } 325 | 326 | var b bytes.Buffer 327 | for i := 0; i < 262093; i++ { 328 | b.WriteString("x") 329 | } 330 | b.WriteString("𒁈zrgchimqigtm") 331 | 332 | record := map[interface{}]interface{}{ 333 | "key": b.String(), 334 | } 335 | 336 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 337 | actualData, err := output.processRecord(&Event{TS: time.Now(), Tag: testTag, Record: record}) 338 | 339 | if err != nil { 340 | logrus.Debugf("[cloudwatch %d] Failed to process record: %v\n", output.PluginInstanceID, record) 341 | } 342 | 343 | /* invalid characters will be expanded when sent as request */ 344 | actualDataString := logString(actualData) 345 | actualDataString = fmt.Sprintf("%q", actualDataString) /* converts: -> \x */ 346 | 347 | exampleWorkingData := "{\"key\":\"x\"}" 348 | addedLength := len(fmt.Sprintf("%q", exampleWorkingData)) - len(exampleWorkingData) 349 | 350 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to be FLB_OK") 351 | assert.LessOrEqual(t, len(actualDataString), 256*1024-26+addedLength, "Expected length to be less than or equal to 256*1024-26") 352 | } 353 | 354 | func TestAddEventCreateLogGroup(t *testing.T) { 355 | ctrl := gomock.NewController(t) 356 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 357 | 358 | gomock.InOrder( 359 | mockCloudWatch.EXPECT().CreateLogGroup(gomock.Any()).Return(&cloudwatchlogs.CreateLogGroupOutput{}, nil), 360 | mockCloudWatch.EXPECT().PutRetentionPolicy(gomock.Any()).Return(&cloudwatchlogs.PutRetentionPolicyOutput{}, nil), 361 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 362 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 363 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 364 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 365 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 366 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log group name to match") 367 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 368 | ) 369 | 370 | output := OutputPlugin{ 371 | logGroupName: testTemplate(testLogGroup), 372 | logStreamPrefix: testLogStreamPrefix, 373 | client: mockCloudWatch, 374 | timer: setupTimeout(), 375 | streams: make(map[string]*logStream), 376 | groups: make(map[string]struct{}), 377 | logGroupRetention: 14, 378 | autoCreateGroup: true, 379 | autoCreateStream: true, 380 | } 381 | 382 | record := map[interface{}]interface{}{ 383 | "somekey": []byte("some value"), 384 | } 385 | 386 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 387 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 388 | 389 | } 390 | 391 | // Existing Log Stream that requires 2 API calls to find 392 | func TestAddEventExistingStream(t *testing.T) { 393 | ctrl := gomock.NewController(t) 394 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 395 | 396 | gomock.InOrder( 397 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 398 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 399 | assert.Equal(t, aws.StringValue(input.LogStreamNamePrefix), testLogStreamPrefix+testTag, "Expected log group name to match") 400 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{ 401 | LogStreams: []*cloudwatchlogs.LogStream{ 402 | &cloudwatchlogs.LogStream{ 403 | LogStreamName: aws.String("wrong stream"), 404 | }, 405 | }, 406 | NextToken: aws.String(testNextToken), 407 | }, nil), 408 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 409 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 410 | assert.Equal(t, aws.StringValue(input.LogStreamNamePrefix), testLogStreamPrefix+testTag, "Expected log group name to match") 411 | assert.Equal(t, aws.StringValue(input.NextToken), testNextToken, "Expected next token to match") 412 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{ 413 | LogStreams: []*cloudwatchlogs.LogStream{ 414 | &cloudwatchlogs.LogStream{ 415 | LogStreamName: aws.String(testLogStreamPrefix + testTag), 416 | }, 417 | }, 418 | NextToken: aws.String(testNextToken), 419 | }, nil), 420 | ) 421 | 422 | output := OutputPlugin{ 423 | logGroupName: testTemplate(testLogGroup), 424 | logStreamPrefix: testLogStreamPrefix, 425 | client: mockCloudWatch, 426 | timer: setupTimeout(), 427 | streams: make(map[string]*logStream), 428 | groups: map[string]struct{}{testLogGroup: {}}, 429 | } 430 | 431 | record := map[interface{}]interface{}{ 432 | "somekey": []byte("some value"), 433 | } 434 | 435 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 436 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 437 | 438 | } 439 | 440 | func TestAddEventDescribeStreamsException(t *testing.T) { 441 | ctrl := gomock.NewController(t) 442 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 443 | 444 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 445 | }).Return(nil, awserr.New(cloudwatchlogs.ErrCodeResourceNotFoundException, "The specified log group does not exist.", fmt.Errorf("API Error"))) 446 | 447 | output := OutputPlugin{ 448 | logGroupName: testTemplate(testLogGroup), 449 | logStreamPrefix: testLogStreamPrefix, 450 | client: mockCloudWatch, 451 | timer: setupTimeout(), 452 | streams: make(map[string]*logStream), 453 | groups: map[string]struct{}{testLogGroup: {}}, 454 | autoCreateStream: true, 455 | } 456 | 457 | record := map[interface{}]interface{}{ 458 | "somekey": []byte("some value"), 459 | } 460 | 461 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 462 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_OK") 463 | } 464 | 465 | func TestAddEventAutoCreateDisabled(t *testing.T) { 466 | ctrl := gomock.NewController(t) 467 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 468 | 469 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 470 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 471 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil) 472 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 473 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil).Times(0) 474 | 475 | output := OutputPlugin{ 476 | logGroupName: testTemplate(testLogGroup), 477 | logStreamPrefix: testLogStreamPrefix, 478 | client: mockCloudWatch, 479 | timer: setupTimeout(), 480 | streams: make(map[string]*logStream), 481 | groups: map[string]struct{}{testLogGroup: {}}, 482 | autoCreateStream: false, 483 | } 484 | 485 | record := map[interface{}]interface{}{ 486 | "somekey": []byte("some value"), 487 | } 488 | 489 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 490 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 491 | } 492 | 493 | func TestAddEventExistingStreamNotFound(t *testing.T) { 494 | ctrl := gomock.NewController(t) 495 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 496 | 497 | gomock.InOrder( 498 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 499 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 500 | assert.Equal(t, aws.StringValue(input.LogStreamNamePrefix), testLogStreamPrefix+testTag, "Expected log group name to match") 501 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{ 502 | LogStreams: []*cloudwatchlogs.LogStream{ 503 | &cloudwatchlogs.LogStream{ 504 | LogStreamName: aws.String("wrong stream"), 505 | }, 506 | }, 507 | NextToken: aws.String(testNextToken), 508 | }, nil), 509 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 510 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 511 | assert.Equal(t, aws.StringValue(input.LogStreamNamePrefix), testLogStreamPrefix+testTag, "Expected log group name to match") 512 | assert.Equal(t, aws.StringValue(input.NextToken), testNextToken, "Expected next token to match") 513 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{ 514 | LogStreams: []*cloudwatchlogs.LogStream{ 515 | &cloudwatchlogs.LogStream{ 516 | LogStreamName: aws.String("another wrong stream"), 517 | }, 518 | }, 519 | }, nil), 520 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 521 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 522 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log group name to match") 523 | }).Return(nil, awserr.New(cloudwatchlogs.ErrCodeResourceAlreadyExistsException, "Log Stream already exists", fmt.Errorf("API Error"))), 524 | ) 525 | 526 | output := OutputPlugin{ 527 | logGroupName: testTemplate(testLogGroup), 528 | logStreamPrefix: testLogStreamPrefix, 529 | client: mockCloudWatch, 530 | timer: setupTimeout(), 531 | streams: make(map[string]*logStream), 532 | groups: map[string]struct{}{testLogGroup: {}}, 533 | autoCreateStream: true, 534 | } 535 | 536 | record := map[interface{}]interface{}{ 537 | "somekey": []byte("some value"), 538 | } 539 | 540 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 541 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 542 | 543 | } 544 | 545 | func TestAddEventEmptyRecord(t *testing.T) { 546 | ctrl := gomock.NewController(t) 547 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 548 | 549 | output := OutputPlugin{ 550 | logGroupName: testTemplate(testLogGroup), 551 | logStreamPrefix: testLogStreamPrefix, 552 | client: mockCloudWatch, 553 | timer: setupTimeout(), 554 | streams: make(map[string]*logStream), 555 | logKey: "somekey", 556 | groups: map[string]struct{}{testLogGroup: {}}, 557 | } 558 | 559 | record := map[interface{}]interface{}{ 560 | "somekey": []byte(""), 561 | } 562 | 563 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 564 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 565 | 566 | } 567 | 568 | func TestAddEventAndFlush(t *testing.T) { 569 | ctrl := gomock.NewController(t) 570 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 571 | 572 | gomock.InOrder( 573 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 574 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 575 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 576 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 577 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 578 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 579 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 580 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 581 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 582 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 583 | }).Return(&cloudwatchlogs.PutLogEventsOutput{ 584 | NextSequenceToken: aws.String("token"), 585 | }, nil), 586 | ) 587 | 588 | output := OutputPlugin{ 589 | logGroupName: testTemplate(testLogGroup), 590 | logStreamPrefix: testLogStreamPrefix, 591 | client: mockCloudWatch, 592 | timer: setupTimeout(), 593 | streams: make(map[string]*logStream), 594 | groups: map[string]struct{}{testLogGroup: {}}, 595 | autoCreateStream: true, 596 | } 597 | 598 | record := map[interface{}]interface{}{ 599 | "somekey": []byte("some value"), 600 | } 601 | 602 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 603 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 604 | output.Flush() 605 | } 606 | 607 | func TestPutLogEvents(t *testing.T) { 608 | ctrl := gomock.NewController(t) 609 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 610 | 611 | output := OutputPlugin{ 612 | logGroupName: testTemplate(testLogGroup), 613 | logStreamPrefix: testLogStreamPrefix, 614 | client: mockCloudWatch, 615 | timer: setupTimeout(), 616 | streams: make(map[string]*logStream), 617 | logKey: "somekey", 618 | groups: map[string]struct{}{testLogGroup: {}}, 619 | } 620 | 621 | stream := &logStream{} 622 | err := output.putLogEvents(stream) 623 | assert.Nil(t, err) 624 | } 625 | 626 | func TestSetGroupStreamNames(t *testing.T) { 627 | record := map[interface{}]interface{}{ 628 | "ident": "cron", 629 | "msg": "my cool log message", 630 | "details": map[interface{}]interface{}{ 631 | "region": "us-west-2", 632 | "az": "a", 633 | }, 634 | } 635 | 636 | e := &Event{Tag: "syslog.0", Record: record} 637 | 638 | // Test against non-template name. 639 | output := OutputPlugin{ 640 | logStreamName: testTemplate("/aws/ecs/test-stream-name"), 641 | logGroupName: testTemplate(""), 642 | defaultLogGroupName: "fluentbit-default", 643 | defaultLogStreamName: "/fluentbit-default", 644 | } 645 | 646 | output.setGroupStreamNames(e) 647 | assert.Equal(t, "/aws/ecs/test-stream-name", e.stream, 648 | "The provided stream name must be returned exactly, without modifications.") 649 | 650 | output.logStreamName = testTemplate("") 651 | output.setGroupStreamNames(e) 652 | assert.Equal(t, output.defaultLogStreamName, e.stream, 653 | "The default stream name must be set when no stream name is provided.") 654 | 655 | // Test against a simple log stream prefix. 656 | output.logStreamPrefix = "/aws/ecs/test-stream-prefix/" 657 | output.setGroupStreamNames(e) 658 | assert.Equal(t, output.logStreamPrefix+"syslog.0", e.stream, 659 | "The provided stream prefix must be prefixed to the provided tag name.") 660 | 661 | // Test replacing items from template variables. 662 | output.logStreamPrefix = "" 663 | output.logStreamName = testTemplate("/aws/ecs/$(tag[0])/$(tag[1])/$(details['region'])/$(details['az'])/$(ident)") 664 | output.setGroupStreamNames(e) 665 | assert.Equal(t, "/aws/ecs/syslog/0/us-west-2/a/cron", e.stream, 666 | "The stream name template was not correctly parsed.") 667 | assert.Equal(t, output.defaultLogGroupName, e.group, 668 | "The default log group name must be set when no log group is provided.") 669 | 670 | // Test another bad template ] missing. 671 | output.logStreamName = testTemplate("/aws/ecs/$(details['region')") 672 | output.setGroupStreamNames(e) 673 | assert.Equal(t, "/aws/ecs/['region'", e.stream, 674 | "The provided stream name must match when the tag is incomplete.") 675 | 676 | // Make sure we get default group and stream names when their variables cannot be parsed. 677 | output.logStreamName = testTemplate("/aws/ecs/$(details['activity'])") 678 | output.logGroupName = testTemplate("$(details['activity'])") 679 | output.setGroupStreamNames(e) 680 | assert.Equal(t, output.defaultLogStreamName, e.stream, 681 | "The default stream name must return when elements are missing.") 682 | assert.Equal(t, output.defaultLogGroupName, e.group, 683 | "The default group name must return when elements are missing.") 684 | 685 | // Test that log stream and log group names get truncated to the maximum allowed. 686 | b := make([]byte, maxGroupStreamLength*2) 687 | for i := range b { // make a string twice the max 688 | b[i] = '_' 689 | } 690 | 691 | ident := string(b) 692 | assert.True(t, len(ident) > maxGroupStreamLength, "test string creation failed") 693 | 694 | e.Record = map[interface{}]interface{}{"ident": ident} // set the long string into our record. 695 | output.logStreamName = testTemplate("/aws/ecs/$(ident)") 696 | output.logGroupName = testTemplate("/aws/ecs/$(ident)") 697 | 698 | output.setGroupStreamNames(e) 699 | assert.Equal(t, maxGroupStreamLength, len(e.stream), "the stream name should be truncated to the maximum size") 700 | assert.Equal(t, maxGroupStreamLength, len(e.group), "the group name should be truncated to the maximum size") 701 | assert.Equal(t, "/aws/ecs/"+string(b[:maxGroupStreamLength-len("/aws/ecs/")]), 702 | e.stream, "the stream name was incorrectly truncated") 703 | assert.Equal(t, "/aws/ecs/"+string(b[:maxGroupStreamLength-len("/aws/ecs/")]), 704 | e.group, "the group name was incorrectly truncated") 705 | } 706 | 707 | func TestAddEventAndFlushDataAlreadyAcceptedException(t *testing.T) { 708 | ctrl := gomock.NewController(t) 709 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 710 | 711 | gomock.InOrder( 712 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 713 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 714 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 715 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 716 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 717 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 718 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 719 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 720 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 721 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 722 | }).Return(nil, awserr.New(cloudwatchlogs.ErrCodeDataAlreadyAcceptedException, "Data already accepted; The next expected sequenceToken is: "+testSequenceToken, fmt.Errorf("API Error"))), 723 | ) 724 | 725 | output := OutputPlugin{ 726 | logGroupName: testTemplate(testLogGroup), 727 | logStreamPrefix: testLogStreamPrefix, 728 | client: mockCloudWatch, 729 | timer: setupTimeout(), 730 | streams: make(map[string]*logStream), 731 | groups: map[string]struct{}{testLogGroup: {}}, 732 | autoCreateStream: true, 733 | } 734 | 735 | record := map[interface{}]interface{}{ 736 | "somekey": []byte("some value"), 737 | } 738 | 739 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 740 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 741 | output.Flush() 742 | } 743 | 744 | func TestAddEventAndFlushDataInvalidSequenceTokenException(t *testing.T) { 745 | ctrl := gomock.NewController(t) 746 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 747 | 748 | gomock.InOrder( 749 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 750 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 751 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 752 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 753 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 754 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 755 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 756 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 757 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 758 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 759 | }).Return(nil, awserr.New(cloudwatchlogs.ErrCodeInvalidSequenceTokenException, "The given sequenceToken is invalid; The next expected sequenceToken is: "+testSequenceToken, fmt.Errorf("API Error"))), 760 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 761 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 762 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 763 | assert.Equal(t, aws.StringValue(input.SequenceToken), testSequenceToken, "Expected sequence token to match response from previous error") 764 | }).Return(&cloudwatchlogs.PutLogEventsOutput{ 765 | NextSequenceToken: aws.String("token"), 766 | }, nil), 767 | ) 768 | 769 | output := OutputPlugin{ 770 | logGroupName: testTemplate(testLogGroup), 771 | logStreamPrefix: testLogStreamPrefix, 772 | client: mockCloudWatch, 773 | timer: setupTimeout(), 774 | streams: make(map[string]*logStream), 775 | groups: map[string]struct{}{testLogGroup: {}}, 776 | autoCreateStream: true, 777 | } 778 | 779 | record := map[interface{}]interface{}{ 780 | "somekey": []byte("some value"), 781 | } 782 | 783 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 784 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 785 | output.Flush() 786 | } 787 | 788 | func TestAddEventAndFlushDataInvalidSequenceTokenNextNullException(t *testing.T) { 789 | ctrl := gomock.NewController(t) 790 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 791 | 792 | gomock.InOrder( 793 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 794 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 795 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 796 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 797 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 798 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 799 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 800 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 801 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 802 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 803 | }).Return(nil, awserr.New(cloudwatchlogs.ErrCodeInvalidSequenceTokenException, "The given sequenceToken is invalid; The next expected sequenceToken is: null", fmt.Errorf("API Error"))), 804 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 805 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 806 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 807 | assert.Nil(t, input.SequenceToken, "Expected sequence token to be nil") 808 | }).Return(&cloudwatchlogs.PutLogEventsOutput{ 809 | NextSequenceToken: aws.String("token"), 810 | }, nil), 811 | ) 812 | 813 | output := OutputPlugin{ 814 | logGroupName: testTemplate(testLogGroup), 815 | logStreamPrefix: testLogStreamPrefix, 816 | client: mockCloudWatch, 817 | timer: setupTimeout(), 818 | streams: make(map[string]*logStream), 819 | groups: map[string]struct{}{testLogGroup: {}}, 820 | autoCreateStream: true, 821 | } 822 | 823 | record := map[interface{}]interface{}{ 824 | "somekey": []byte("some value"), 825 | } 826 | 827 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 828 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 829 | output.Flush() 830 | } 831 | 832 | func TestAddEventAndDataResourceNotFoundExceptionWithNoLogGroup(t *testing.T) { 833 | ctrl := gomock.NewController(t) 834 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 835 | 836 | gomock.InOrder( 837 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 838 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 839 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 840 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 841 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 842 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 843 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 844 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 845 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 846 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 847 | }).Return(nil, awserr.New(cloudwatchlogs.ErrCodeResourceNotFoundException, "The specified log group does not exist.", fmt.Errorf("API Error"))), 848 | mockCloudWatch.EXPECT().CreateLogGroup(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogGroupInput) { 849 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 850 | }).Return(&cloudwatchlogs.CreateLogGroupOutput{}, nil), 851 | ) 852 | 853 | output := OutputPlugin{ 854 | logGroupName: testTemplate(testLogGroup), 855 | logStreamPrefix: testLogStreamPrefix, 856 | client: mockCloudWatch, 857 | timer: setupTimeout(), 858 | streams: make(map[string]*logStream), 859 | groups: map[string]struct{}{testLogGroup: {}}, 860 | autoCreateStream: true, 861 | } 862 | 863 | record := map[interface{}]interface{}{ 864 | "somekey": []byte("some value"), 865 | } 866 | 867 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 868 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 869 | } 870 | 871 | func TestAddEventAndDataResourceNotFoundExceptionWithNoLogStream(t *testing.T) { 872 | ctrl := gomock.NewController(t) 873 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 874 | 875 | gomock.InOrder( 876 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 877 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 878 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 879 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 880 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 881 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 882 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 883 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Do(func(input *cloudwatchlogs.PutLogEventsInput) { 884 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 885 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 886 | }).Return(nil, awserr.New(cloudwatchlogs.ErrCodeResourceNotFoundException, "The specified log stream does not exist.", fmt.Errorf("API Error"))), 887 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 888 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 889 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 890 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 891 | ) 892 | 893 | output := OutputPlugin{ 894 | logGroupName: testTemplate(testLogGroup), 895 | logStreamPrefix: testLogStreamPrefix, 896 | client: mockCloudWatch, 897 | timer: setupTimeout(), 898 | streams: make(map[string]*logStream), 899 | groups: map[string]struct{}{testLogGroup: {}}, 900 | autoCreateStream: true, 901 | } 902 | 903 | record := map[interface{}]interface{}{ 904 | "somekey": []byte("some value"), 905 | } 906 | 907 | retCode := output.AddEvent(&Event{TS: time.Now(), Tag: testTag, Record: record}) 908 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 909 | } 910 | 911 | func TestAddEventAndBatchSpanLimit(t *testing.T) { 912 | output := setupLimitTestOutput(t, 2) 913 | 914 | record := map[interface{}]interface{}{ 915 | "somekey": []byte("some value"), 916 | } 917 | 918 | before := time.Now() 919 | start := before.Add(time.Nanosecond) 920 | end := start.Add(time.Hour*24 - time.Nanosecond) 921 | after := start.Add(time.Hour * 24) 922 | 923 | retCode := output.AddEvent(&Event{TS: start, Tag: testTag, Record: record}) 924 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 925 | 926 | retCode = output.AddEvent(&Event{TS: end, Tag: testTag, Record: record}) 927 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 928 | 929 | retCode = output.AddEvent(&Event{TS: before, Tag: testTag, Record: record}) 930 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 931 | 932 | retCode = output.AddEvent(&Event{TS: after, Tag: testTag, Record: record}) 933 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 934 | } 935 | 936 | func TestAddEventAndBatchSpanLimitOnReverseOrder(t *testing.T) { 937 | output := setupLimitTestOutput(t, 2) 938 | 939 | record := map[interface{}]interface{}{ 940 | "somekey": []byte("some value"), 941 | } 942 | 943 | before := time.Now() 944 | start := before.Add(time.Nanosecond) 945 | end := start.Add(time.Hour*24 - time.Nanosecond) 946 | after := start.Add(time.Hour * 24) 947 | 948 | retCode := output.AddEvent(&Event{TS: end, Tag: testTag, Record: record}) 949 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 950 | 951 | retCode = output.AddEvent(&Event{TS: start, Tag: testTag, Record: record}) 952 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 953 | 954 | retCode = output.AddEvent(&Event{TS: before, Tag: testTag, Record: record}) 955 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 956 | 957 | retCode = output.AddEvent(&Event{TS: after, Tag: testTag, Record: record}) 958 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 959 | } 960 | 961 | func TestAddEventAndEventsCountLimit(t *testing.T) { 962 | output := setupLimitTestOutput(t, 1) 963 | 964 | record := map[interface{}]interface{}{ 965 | "somekey": []byte("some value"), 966 | } 967 | 968 | now := time.Now() 969 | 970 | for i := 0; i < 10000; i++ { 971 | retCode := output.AddEvent(&Event{TS: now, Tag: testTag, Record: record}) 972 | assert.Equal(t, retCode, fluentbit.FLB_OK, fmt.Sprintf("Expected return code to FLB_OK on %d iteration", i)) 973 | } 974 | retCode := output.AddEvent(&Event{TS: now, Tag: testTag, Record: record}) 975 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 976 | } 977 | 978 | func TestAddEventAndBatchSizeLimit(t *testing.T) { 979 | output := setupLimitTestOutput(t, 1) 980 | 981 | record := map[interface{}]interface{}{ 982 | "somekey": []byte(strings.Repeat("some value", 100)), 983 | } 984 | 985 | now := time.Now() 986 | 987 | for i := 0; i < 104; i++ { // 104 * 10_000 < 1_048_576 988 | retCode := output.AddEvent(&Event{TS: now, Tag: testTag, Record: record}) 989 | assert.Equal(t, retCode, fluentbit.FLB_OK, "Expected return code to FLB_OK") 990 | } 991 | 992 | // 105 * 10_000 > 1_048_576 993 | retCode := output.AddEvent(&Event{TS: now.Add(time.Hour*24 + time.Nanosecond), Tag: testTag, Record: record}) 994 | assert.Equal(t, retCode, fluentbit.FLB_RETRY, "Expected return code to FLB_RETRY") 995 | } 996 | 997 | func setupLimitTestOutput(t *testing.T, times int) OutputPlugin { 998 | ctrl := gomock.NewController(t) 999 | mockCloudWatch := mock_cloudwatch.NewMockLogsClient(ctrl) 1000 | 1001 | gomock.InOrder( 1002 | mockCloudWatch.EXPECT().DescribeLogStreams(gomock.Any()).Do(func(input *cloudwatchlogs.DescribeLogStreamsInput) { 1003 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 1004 | }).Return(&cloudwatchlogs.DescribeLogStreamsOutput{}, nil), 1005 | mockCloudWatch.EXPECT().CreateLogStream(gomock.Any()).AnyTimes().Do(func(input *cloudwatchlogs.CreateLogStreamInput) { 1006 | assert.Equal(t, aws.StringValue(input.LogGroupName), testLogGroup, "Expected log group name to match") 1007 | assert.Equal(t, aws.StringValue(input.LogStreamName), testLogStreamPrefix+testTag, "Expected log stream name to match") 1008 | }).Return(&cloudwatchlogs.CreateLogStreamOutput{}, nil), 1009 | mockCloudWatch.EXPECT().PutLogEvents(gomock.Any()).Times(times).Return(nil, errors.New("should fail")), 1010 | ) 1011 | 1012 | return OutputPlugin{ 1013 | logGroupName: testTemplate(testLogGroup), 1014 | logStreamPrefix: testLogStreamPrefix, 1015 | client: mockCloudWatch, 1016 | timer: setupTimeout(), 1017 | streams: make(map[string]*logStream), 1018 | groups: map[string]struct{}{testLogGroup: {}}, 1019 | autoCreateStream: true, 1020 | } 1021 | } 1022 | 1023 | func setupTimeout() *plugins.Timeout { 1024 | timer, _ := plugins.NewTimeout(func(d time.Duration) { 1025 | logrus.Errorf("[firehose] timeout threshold reached: Failed to send logs for %v\n", d) 1026 | logrus.Error("[firehose] Quitting Fluent Bit") 1027 | os.Exit(1) 1028 | }) 1029 | return timer 1030 | } 1031 | 1032 | func TestValidate(t *testing.T) { 1033 | for _, test := range configValidationTestCases { 1034 | t.Run(test.name, func(t *testing.T) { 1035 | err := test.config.Validate() 1036 | 1037 | if test.isValidConfig { 1038 | assert.Nil(t, err) 1039 | } else { 1040 | assert.NotNil(t, err) 1041 | assert.Equal(t, err.Error(), test.expectedError) 1042 | } 1043 | }) 1044 | } 1045 | } 1046 | -------------------------------------------------------------------------------- /cloudwatch/generate_mock.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package cloudwatch 15 | 16 | //go:generate ../scripts/mockgen.sh github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/cloudwatch LogsClient mock_cloudwatch/mock.go 17 | -------------------------------------------------------------------------------- /cloudwatch/handlers.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package cloudwatch 15 | 16 | import "github.com/aws/aws-sdk-go/aws/request" 17 | 18 | const logFormatHeader = "x-amzn-logs-format" 19 | 20 | // LogFormatHandler returns an http request handler that sets an HTTP header. 21 | // The header is used to indicate the format of the logs being sent. 22 | func LogFormatHandler(format string) request.NamedHandler { 23 | return request.NamedHandler{ 24 | Name: "LogFormatHandler", 25 | Fn: func(req *request.Request) { 26 | req.HTTPRequest.Header.Set(logFormatHeader, format) 27 | }, 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /cloudwatch/handlers_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package cloudwatch 15 | 16 | import ( 17 | "net/http" 18 | "testing" 19 | 20 | "github.com/aws/aws-sdk-go/aws/request" 21 | "github.com/stretchr/testify/assert" 22 | ) 23 | 24 | func TestLogFormatHandler(t *testing.T) { 25 | httpReq, _ := http.NewRequest("POST", "", nil) 26 | r := &request.Request{ 27 | HTTPRequest: httpReq, 28 | Body: nil, 29 | } 30 | r.SetBufferBody([]byte{}) 31 | 32 | handler := LogFormatHandler("json/emf") 33 | handler.Fn(r) 34 | 35 | header := r.HTTPRequest.Header.Get(logFormatHeader) 36 | assert.Equal(t, "json/emf", header) 37 | } 38 | -------------------------------------------------------------------------------- /cloudwatch/helpers.go: -------------------------------------------------------------------------------- 1 | package cloudwatch 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | "strconv" 7 | "strings" 8 | 9 | "github.com/valyala/bytebufferpool" 10 | "github.com/valyala/fasttemplate" 11 | ) 12 | 13 | // Errors output by the help procedures. 14 | var ( 15 | ErrNoTagValue = fmt.Errorf("not enough dots in the tag to satisfy the index position") 16 | ErrMissingTagName = fmt.Errorf("tag name not found") 17 | ErrMissingSubName = fmt.Errorf("sub-tag name not found") 18 | ) 19 | 20 | // newTemplate is the only place you'll find the template start and end tags. 21 | func newTemplate(template string) (*fastTemplate, error) { 22 | t, err := fasttemplate.NewTemplate(template, "$(", ")") 23 | 24 | return &fastTemplate{Template: t, String: template}, err 25 | } 26 | 27 | // tagKeysToMap converts a raw string into a go map. 28 | // This is used by input data to create AWS tags applied to newly-created log groups. 29 | // 30 | // The input string should be match this: "key=value,key2=value2". 31 | // Spaces are trimmed, empty values are permitted, empty keys are ignored. 32 | // The final value in the input string wins in case of duplicate keys. 33 | func tagKeysToMap(tags string) map[string]*string { 34 | output := make(map[string]*string) 35 | 36 | for _, tag := range strings.Split(strings.TrimSpace(tags), ",") { 37 | split := strings.SplitN(tag, "=", 2) 38 | key := strings.TrimSpace(split[0]) 39 | value := "" 40 | 41 | if key == "" { 42 | continue 43 | } 44 | 45 | if len(split) > 1 { 46 | value = strings.TrimSpace(split[1]) 47 | } 48 | 49 | output[key] = &value 50 | } 51 | 52 | if len(output) == 0 { 53 | return nil 54 | } 55 | 56 | return output 57 | } 58 | 59 | // parseKeysTemplate takes in an interface map and a list of nested keys. It returns 60 | // the value of the final key, or the name of the first key not found in the chain. 61 | // example keys := "['level1']['level2']['level3']" 62 | // This is called by parseDataMapTags any time a nested value is found in a log Event. 63 | // This procedure checks if any of the nested values match variable identifiers in the logStream or logGroups. 64 | func parseKeysTemplate(data map[interface{}]interface{}, keys string, w io.Writer) (int64, error) { 65 | return fasttemplate.ExecuteFunc(keys, "['", "']", w, func(w io.Writer, tag string) (int, error) { 66 | switch val := data[tag].(type) { 67 | case []byte: 68 | return w.Write(val) 69 | case string: 70 | return w.Write([]byte(val)) 71 | case map[interface{}]interface{}: 72 | data = val // drill down another level. 73 | return 0, nil 74 | default: // missing 75 | return 0, fmt.Errorf("%s: %w", tag, ErrMissingSubName) 76 | } 77 | }) 78 | } 79 | 80 | // parseDataMapTags parses the provided tag values in template form, 81 | // from an interface{} map (expected to contain strings or more interface{} maps). 82 | // This runs once for every log line. 83 | // Used to fill in any template variables that may exist in the logStream or logGroup names. 84 | func parseDataMapTags(e *Event, logTags []string, t *fastTemplate, metadata TaskMetadata, uuid string, w io.Writer) (int64, error) { 85 | return t.ExecuteFunc(w, func(w io.Writer, tag string) (int, error) { 86 | switch tag { 87 | case "ecs_task_id": 88 | if metadata.TaskID != "" { 89 | return w.Write([]byte(metadata.TaskID)) 90 | } 91 | 92 | return 0, fmt.Errorf("Failed to fetch ecs_task_id; The container is not running in ECS") 93 | case "ecs_cluster": 94 | if metadata.Cluster != "" { 95 | return w.Write([]byte(metadata.Cluster)) 96 | } 97 | 98 | return 0, fmt.Errorf("Failed to fetch ecs_cluster; The container is not running in ECS") 99 | case "ecs_task_arn": 100 | if metadata.TaskARN != "" { 101 | return w.Write([]byte(metadata.TaskARN)) 102 | } 103 | 104 | return 0, fmt.Errorf("Failed to fetch ecs_task_arn; The container is not running in ECS") 105 | case "uuid": 106 | return w.Write([]byte(uuid)) 107 | } 108 | 109 | v := strings.Index(tag, "[") 110 | if v == -1 { 111 | v = len(tag) 112 | } 113 | 114 | if tag[:v] == "tag" { 115 | switch { 116 | default: // input string is either `tag` or `tag[`, so return the $tag. 117 | return w.Write([]byte(e.Tag)) 118 | case len(tag) >= 5: // input string is at least "tag[x" where x is hopefully an integer 0-9. 119 | // The index value is always in the same position: 4:5 (this is why supporting more than 0-9 is rough) 120 | if v, _ = strconv.Atoi(tag[4:5]); len(logTags) <= v { 121 | return 0, fmt.Errorf("%s: %w", tag, ErrNoTagValue) 122 | } 123 | 124 | return w.Write([]byte(logTags[v])) 125 | } 126 | } 127 | 128 | switch val := e.Record[tag[:v]].(type) { 129 | case string: 130 | return w.Write([]byte(val)) 131 | case map[interface{}]interface{}: 132 | i, err := parseKeysTemplate(val, tag[v:], w) 133 | 134 | return int(i), err 135 | case []byte: 136 | // we should never land here because the interface{} map should have already been converted to strings. 137 | return w.Write(val) 138 | default: // missing 139 | return 0, fmt.Errorf("%s: %w", tag, ErrMissingTagName) 140 | } 141 | }) 142 | } 143 | 144 | // sanitizer implements io.Writer for fasttemplate usage. 145 | // Instead of just writing bytes to a buffer, sanitize them first. 146 | type sanitizer struct { 147 | sanitize func(b []byte) []byte 148 | buf *bytebufferpool.ByteBuffer 149 | } 150 | 151 | // Write completes the io.Writer implementation. 152 | func (s *sanitizer) Write(b []byte) (int, error) { 153 | return s.buf.Write(s.sanitize(b)) 154 | } 155 | 156 | // sanitizeGroup removes special characters from the log group names bytes. 157 | // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html 158 | func sanitizeGroup(b []byte) []byte { 159 | for i, r := range b { 160 | // 45-47 = / . - 161 | // 48-57 = 0-9 162 | // 65-90 = A-Z 163 | // 95 = _ 164 | // 97-122 = a-z 165 | if r == 95 || (r > 44 && r < 58) || 166 | (r > 64 && r < 91) || (r > 96 && r < 123) { 167 | continue 168 | } 169 | 170 | b[i] = '.' 171 | } 172 | 173 | return b 174 | } 175 | 176 | // sanitizeStream removes : and * from the log stream bytes. 177 | // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-logstream.html 178 | func sanitizeStream(b []byte) []byte { 179 | for i, r := range b { 180 | if r == '*' || r == ':' { 181 | b[i] = '.' 182 | } 183 | } 184 | 185 | return b 186 | } 187 | -------------------------------------------------------------------------------- /cloudwatch/helpers_test.go: -------------------------------------------------------------------------------- 1 | package cloudwatch 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/assert" 7 | "github.com/valyala/bytebufferpool" 8 | ) 9 | 10 | func TestTagKeysToMap(t *testing.T) { 11 | t.Parallel() 12 | 13 | // Testable values. Purposely "messed up" - they should all parse out OK. 14 | values := " key1 =value , key2=value2, key3= value3 ,key4=, key5 = v5,,key7==value7," + 15 | " k8, k9,key1=value1,space key = space value" 16 | // The values above should return a map like this. 17 | expect := map[string]string{"key1": "value1", "key2": "value2", "key3": "value3", 18 | "key4": "", "key5": "v5", "key7": "=value7", "k8": "", "k9": "", "space key": "space value"} 19 | 20 | for k, v := range tagKeysToMap(values) { 21 | assert.Equal(t, *v, expect[k], "Tag key or value failed parser.") 22 | } 23 | } 24 | 25 | func TestParseDataMapTags(t *testing.T) { 26 | t.Parallel() 27 | 28 | template := testTemplate("$(ecs_task_id).$(ecs_cluster).$(ecs_task_arn).$(uuid).$(tag).$(pam['item2']['subitem2']['more']).$(pam['item']).$(pam['item2'])." + 29 | "$(pam['item2']['subitem'])-$(pam['item2']['subitem2']['more'])-$(tag[1])") 30 | data := map[interface{}]interface{}{ 31 | "pam": map[interface{}]interface{}{ 32 | "item": "soup", 33 | "item2": map[interface{}]interface{}{"subitem": []byte("SubIt3m"), 34 | "subitem2": map[interface{}]interface{}{"more": "final"}}, 35 | }, 36 | } 37 | 38 | s := &sanitizer{buf: bytebufferpool.Get(), sanitize: sanitizeGroup} 39 | defer bytebufferpool.Put(s.buf) 40 | 41 | _, err := parseDataMapTags(&Event{Record: data, Tag: "syslog.0"}, []string{"syslog", "0"}, template, TaskMetadata{Cluster: "cluster", TaskARN: "taskARN", TaskID: "taskID"}, "123", s) 42 | 43 | assert.Nil(t, err, err) 44 | assert.Equal(t, "taskID.cluster.taskARN.123.syslog.0.final.soup..SubIt3m-final-0", s.buf.String(), "Rendered string is incorrect.") 45 | 46 | // Test missing variables. These should always return an error and an empty string. 47 | s.buf.Reset() 48 | template = testTemplate("$(missing-variable).stuff") 49 | _, err = parseDataMapTags(&Event{Record: data, Tag: "syslog.0"}, []string{"syslog", "0"}, template, TaskMetadata{Cluster: "cluster", TaskARN: "taskARN", TaskID: "taskID"}, "123", s) 50 | assert.EqualError(t, err, "missing-variable: "+ErrMissingTagName.Error(), "the wrong error was returned") 51 | assert.Empty(t, s.buf.String()) 52 | 53 | s.buf.Reset() 54 | template = testTemplate("$(pam['item6']).stuff") 55 | _, err = parseDataMapTags(&Event{Record: data, Tag: "syslog.0"}, []string{"syslog", "0"}, template, TaskMetadata{}, "", s) 56 | assert.EqualError(t, err, "item6: "+ErrMissingSubName.Error(), "the wrong error was returned") 57 | assert.Empty(t, s.buf.String()) 58 | 59 | s.buf.Reset() 60 | template = testTemplate("$(tag[9]).stuff") 61 | _, err = parseDataMapTags(&Event{Record: data, Tag: "syslog.0"}, []string{"syslog", "0"}, template, TaskMetadata{}, "", s) 62 | assert.EqualError(t, err, "tag[9]: "+ErrNoTagValue.Error(), "the wrong error was returned") 63 | assert.Empty(t, s.buf.String()) 64 | } 65 | 66 | func TestSanitizeGroup(t *testing.T) { 67 | t.Parallel() 68 | 69 | tests := map[string]string{ // "send": "expect", 70 | "this.is.a.log.group.name": "this.is.a.log.group.name", 71 | "1234567890abcdefghijklmnopqrstuvwxyz": "1234567890abcdefghijklmnopqrstuvwxyz", 72 | "ABCDEFGHIJKLMNOPQRSTUVWXYZ": "ABCDEFGHIJKLMNOPQRSTUVWXYZ", 73 | `!@#$%^&*()_+}{][=-';":/.?>,<~"']}`: ".........._......-..../..........", 74 | "": "", 75 | } 76 | 77 | for send, expect := range tests { 78 | actual := sanitizeGroup([]byte(send)) 79 | assert.Equal(t, expect, string(actual), "the wrong characters were modified in sanitizeGroup") 80 | } 81 | } 82 | 83 | func TestSanitizeStream(t *testing.T) { 84 | t.Parallel() 85 | 86 | tests := map[string]string{ // "send": "expect", 87 | "this.is.a.log.group.name": "this.is.a.log.group.name", 88 | "1234567890abcdefghijklmnopqrstuvwxyz": "1234567890abcdefghijklmnopqrstuvwxyz", 89 | "ABCDEFGHIJKLMNOPQRSTUVWXYZ": "ABCDEFGHIJKLMNOPQRSTUVWXYZ", 90 | `!@#$%^&*()_+}{][=-';":/.?>,<~"']}`: `!@#$%^&.()_+}{][=-';"./.?>,<~"']}`, 91 | "": "", 92 | } 93 | 94 | for send, expect := range tests { 95 | actual := sanitizeStream([]byte(send)) 96 | assert.Equal(t, expect, string(actual), "the wrong characters were modified in sanitizeStream") 97 | } 98 | } 99 | -------------------------------------------------------------------------------- /cloudwatch/mock_cloudwatch/mock.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | // Code generated by MockGen. DO NOT EDIT. 15 | // Source: github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/cloudwatch (interfaces: LogsClient) 16 | 17 | // Package mock_cloudwatch is a generated GoMock package. 18 | package mock_cloudwatch 19 | 20 | import ( 21 | reflect "reflect" 22 | 23 | cloudwatchlogs "github.com/aws/aws-sdk-go/service/cloudwatchlogs" 24 | gomock "github.com/golang/mock/gomock" 25 | ) 26 | 27 | // MockLogsClient is a mock of LogsClient interface 28 | type MockLogsClient struct { 29 | ctrl *gomock.Controller 30 | recorder *MockLogsClientMockRecorder 31 | } 32 | 33 | // MockLogsClientMockRecorder is the mock recorder for MockLogsClient 34 | type MockLogsClientMockRecorder struct { 35 | mock *MockLogsClient 36 | } 37 | 38 | // NewMockLogsClient creates a new mock instance 39 | func NewMockLogsClient(ctrl *gomock.Controller) *MockLogsClient { 40 | mock := &MockLogsClient{ctrl: ctrl} 41 | mock.recorder = &MockLogsClientMockRecorder{mock} 42 | return mock 43 | } 44 | 45 | // EXPECT returns an object that allows the caller to indicate expected use 46 | func (m *MockLogsClient) EXPECT() *MockLogsClientMockRecorder { 47 | return m.recorder 48 | } 49 | 50 | // CreateLogGroup mocks base method 51 | func (m *MockLogsClient) CreateLogGroup(arg0 *cloudwatchlogs.CreateLogGroupInput) (*cloudwatchlogs.CreateLogGroupOutput, error) { 52 | m.ctrl.T.Helper() 53 | ret := m.ctrl.Call(m, "CreateLogGroup", arg0) 54 | ret0, _ := ret[0].(*cloudwatchlogs.CreateLogGroupOutput) 55 | ret1, _ := ret[1].(error) 56 | return ret0, ret1 57 | } 58 | 59 | // CreateLogGroup indicates an expected call of CreateLogGroup 60 | func (mr *MockLogsClientMockRecorder) CreateLogGroup(arg0 interface{}) *gomock.Call { 61 | mr.mock.ctrl.T.Helper() 62 | return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateLogGroup", reflect.TypeOf((*MockLogsClient)(nil).CreateLogGroup), arg0) 63 | } 64 | 65 | // CreateLogStream mocks base method 66 | func (m *MockLogsClient) CreateLogStream(arg0 *cloudwatchlogs.CreateLogStreamInput) (*cloudwatchlogs.CreateLogStreamOutput, error) { 67 | m.ctrl.T.Helper() 68 | ret := m.ctrl.Call(m, "CreateLogStream", arg0) 69 | ret0, _ := ret[0].(*cloudwatchlogs.CreateLogStreamOutput) 70 | ret1, _ := ret[1].(error) 71 | return ret0, ret1 72 | } 73 | 74 | // CreateLogStream indicates an expected call of CreateLogStream 75 | func (mr *MockLogsClientMockRecorder) CreateLogStream(arg0 interface{}) *gomock.Call { 76 | mr.mock.ctrl.T.Helper() 77 | return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateLogStream", reflect.TypeOf((*MockLogsClient)(nil).CreateLogStream), arg0) 78 | } 79 | 80 | // DescribeLogStreams mocks base method 81 | func (m *MockLogsClient) DescribeLogStreams(arg0 *cloudwatchlogs.DescribeLogStreamsInput) (*cloudwatchlogs.DescribeLogStreamsOutput, error) { 82 | m.ctrl.T.Helper() 83 | ret := m.ctrl.Call(m, "DescribeLogStreams", arg0) 84 | ret0, _ := ret[0].(*cloudwatchlogs.DescribeLogStreamsOutput) 85 | ret1, _ := ret[1].(error) 86 | return ret0, ret1 87 | } 88 | 89 | // DescribeLogStreams indicates an expected call of DescribeLogStreams 90 | func (mr *MockLogsClientMockRecorder) DescribeLogStreams(arg0 interface{}) *gomock.Call { 91 | mr.mock.ctrl.T.Helper() 92 | return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeLogStreams", reflect.TypeOf((*MockLogsClient)(nil).DescribeLogStreams), arg0) 93 | } 94 | 95 | // PutLogEvents mocks base method 96 | func (m *MockLogsClient) PutLogEvents(arg0 *cloudwatchlogs.PutLogEventsInput) (*cloudwatchlogs.PutLogEventsOutput, error) { 97 | m.ctrl.T.Helper() 98 | ret := m.ctrl.Call(m, "PutLogEvents", arg0) 99 | ret0, _ := ret[0].(*cloudwatchlogs.PutLogEventsOutput) 100 | ret1, _ := ret[1].(error) 101 | return ret0, ret1 102 | } 103 | 104 | // PutLogEvents indicates an expected call of PutLogEvents 105 | func (mr *MockLogsClientMockRecorder) PutLogEvents(arg0 interface{}) *gomock.Call { 106 | mr.mock.ctrl.T.Helper() 107 | return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PutLogEvents", reflect.TypeOf((*MockLogsClient)(nil).PutLogEvents), arg0) 108 | } 109 | 110 | // PutRetentionPolicy mocks base method 111 | func (m *MockLogsClient) PutRetentionPolicy(arg0 *cloudwatchlogs.PutRetentionPolicyInput) (*cloudwatchlogs.PutRetentionPolicyOutput, error) { 112 | m.ctrl.T.Helper() 113 | ret := m.ctrl.Call(m, "PutRetentionPolicy", arg0) 114 | ret0, _ := ret[0].(*cloudwatchlogs.PutRetentionPolicyOutput) 115 | ret1, _ := ret[1].(error) 116 | return ret0, ret1 117 | } 118 | 119 | // PutRetentionPolicy indicates an expected call of PutRetentionPolicy 120 | func (mr *MockLogsClientMockRecorder) PutRetentionPolicy(arg0 interface{}) *gomock.Call { 121 | mr.mock.ctrl.T.Helper() 122 | return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PutRetentionPolicy", reflect.TypeOf((*MockLogsClient)(nil).PutRetentionPolicy), arg0) 123 | } 124 | -------------------------------------------------------------------------------- /fluent-bit-cloudwatch.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 4 | // not use this file except in compliance with the License. A copy of the 5 | // License is located at 6 | // 7 | // http://aws.amazon.com/apache2.0/ 8 | // 9 | // or in the "license" file accompanying this file. This file is distributed 10 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | // express or implied. See the License for the specific language governing 12 | // permissions and limitations under the License. 13 | 14 | package main 15 | 16 | import ( 17 | "C" 18 | "fmt" 19 | "strconv" 20 | "time" 21 | "unsafe" 22 | 23 | "github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/cloudwatch" 24 | "github.com/aws/amazon-kinesis-firehose-for-fluent-bit/plugins" 25 | "github.com/fluent/fluent-bit-go/output" 26 | 27 | "github.com/sirupsen/logrus" 28 | ) 29 | import ( 30 | "strings" 31 | ) 32 | 33 | var ( 34 | pluginInstances []*cloudwatch.OutputPlugin 35 | ) 36 | 37 | func addPluginInstance(ctx unsafe.Pointer) error { 38 | pluginID := len(pluginInstances) 39 | 40 | config := getConfiguration(ctx, pluginID) 41 | err := config.Validate() 42 | if err != nil { 43 | return err 44 | } 45 | 46 | instance, err := cloudwatch.NewOutputPlugin(config) 47 | if err != nil { 48 | return err 49 | } 50 | 51 | output.FLBPluginSetContext(ctx, pluginID) 52 | pluginInstances = append(pluginInstances, instance) 53 | 54 | return nil 55 | } 56 | 57 | func getPluginInstance(ctx unsafe.Pointer) *cloudwatch.OutputPlugin { 58 | pluginID := output.FLBPluginGetContext(ctx).(int) 59 | return pluginInstances[pluginID] 60 | } 61 | 62 | //export FLBPluginRegister 63 | func FLBPluginRegister(ctx unsafe.Pointer) int { 64 | return output.FLBPluginRegister(ctx, "cloudwatch", "AWS CloudWatch Fluent Bit Plugin!") 65 | } 66 | 67 | func getConfiguration(ctx unsafe.Pointer, pluginID int) cloudwatch.OutputPluginConfig { 68 | config := cloudwatch.OutputPluginConfig{} 69 | config.PluginInstanceID = pluginID 70 | 71 | config.LogGroupName = output.FLBPluginConfigKey(ctx, "log_group_name") 72 | logrus.Infof("[cloudwatch %d] plugin parameter log_group_name = '%s'", pluginID, config.LogGroupName) 73 | 74 | config.DefaultLogGroupName = output.FLBPluginConfigKey(ctx, "default_log_group_name") 75 | if config.DefaultLogGroupName == "" { 76 | config.DefaultLogGroupName = "fluentbit-default" 77 | } 78 | 79 | logrus.Infof("[cloudwatch %d] plugin parameter default_log_group_name = '%s'", pluginID, config.DefaultLogGroupName) 80 | 81 | config.LogStreamPrefix = output.FLBPluginConfigKey(ctx, "log_stream_prefix") 82 | logrus.Infof("[cloudwatch %d] plugin parameter log_stream_prefix = '%s'", pluginID, config.LogStreamPrefix) 83 | 84 | config.LogStreamName = output.FLBPluginConfigKey(ctx, "log_stream_name") 85 | logrus.Infof("[cloudwatch %d] plugin parameter log_stream_name = '%s'", pluginID, config.LogStreamName) 86 | 87 | config.DefaultLogStreamName = output.FLBPluginConfigKey(ctx, "default_log_stream_name") 88 | if config.DefaultLogStreamName == "" { 89 | config.DefaultLogStreamName = "/fluentbit-default" 90 | } 91 | 92 | logrus.Infof("[cloudwatch %d] plugin parameter default_log_stream_name = '%s'", pluginID, config.DefaultLogStreamName) 93 | 94 | config.Region = output.FLBPluginConfigKey(ctx, "region") 95 | logrus.Infof("[cloudwatch %d] plugin parameter region = '%s'", pluginID, config.Region) 96 | 97 | config.LogKey = output.FLBPluginConfigKey(ctx, "log_key") 98 | logrus.Infof("[cloudwatch %d] plugin parameter log_key = '%s'", pluginID, config.LogKey) 99 | 100 | config.RoleARN = output.FLBPluginConfigKey(ctx, "role_arn") 101 | logrus.Infof("[cloudwatch %d] plugin parameter role_arn = '%s'", pluginID, config.RoleARN) 102 | 103 | config.AutoCreateGroup = getBoolParam(ctx, "auto_create_group", false) 104 | logrus.Infof("[cloudwatch %d] plugin parameter auto_create_group = '%v'", pluginID, config.AutoCreateGroup) 105 | 106 | config.AutoCreateStream = getBoolParam(ctx, "auto_create_stream", true) 107 | logrus.Infof("[cloudwatch %d] plugin parameter auto_create_stream = '%v'", pluginID, config.AutoCreateStream) 108 | 109 | config.NewLogGroupTags = output.FLBPluginConfigKey(ctx, "new_log_group_tags") 110 | logrus.Infof("[cloudwatch %d] plugin parameter new_log_group_tags = '%s'", pluginID, config.NewLogGroupTags) 111 | 112 | config.LogRetentionDays, _ = strconv.ParseInt(output.FLBPluginConfigKey(ctx, "log_retention_days"), 10, 64) 113 | logrus.Infof("[cloudwatch %d] plugin parameter log_retention_days = '%d'", pluginID, config.LogRetentionDays) 114 | 115 | config.CWEndpoint = output.FLBPluginConfigKey(ctx, "endpoint") 116 | logrus.Infof("[cloudwatch %d] plugin parameter endpoint = '%s'", pluginID, config.CWEndpoint) 117 | 118 | config.STSEndpoint = output.FLBPluginConfigKey(ctx, "sts_endpoint") 119 | logrus.Infof("[cloudwatch %d] plugin parameter sts_endpoint = '%s'", pluginID, config.STSEndpoint) 120 | 121 | config.ExternalID = output.FLBPluginConfigKey(ctx, "external_id") 122 | logrus.Infof("[cloudwatch %d] plugin parameter external_id = '%s'", pluginID, config.ExternalID) 123 | 124 | config.CredsEndpoint = output.FLBPluginConfigKey(ctx, "credentials_endpoint") 125 | logrus.Infof("[cloudwatch %d] plugin parameter credentials_endpoint = %s", pluginID, config.CredsEndpoint) 126 | 127 | config.LogFormat = output.FLBPluginConfigKey(ctx, "log_format") 128 | logrus.Infof("[cloudwatch %d] plugin parameter log_format = '%s'", pluginID, config.LogFormat) 129 | 130 | config.ExtraUserAgent = output.FLBPluginConfigKey(ctx, "extra_user_agent") 131 | 132 | return config 133 | } 134 | 135 | func getBoolParam(ctx unsafe.Pointer, param string, defaultVal bool) bool { 136 | val := strings.ToLower(output.FLBPluginConfigKey(ctx, param)) 137 | if val == "true" { 138 | return true 139 | } else if val == "false" { 140 | return false 141 | } else { 142 | return defaultVal 143 | } 144 | } 145 | 146 | //export FLBPluginInit 147 | func FLBPluginInit(ctx unsafe.Pointer) int { 148 | plugins.SetupLogger() 149 | 150 | logrus.Debug("A new higher performance CloudWatch Logs plugin has been released; " + 151 | "you are using the old plugin. Check out the new plugin's documentation and " + 152 | "determine if you can migrate.\n" + 153 | "https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch") 154 | 155 | err := addPluginInstance(ctx) 156 | if err != nil { 157 | logrus.Error(err) 158 | return output.FLB_ERROR 159 | } 160 | return output.FLB_OK 161 | } 162 | 163 | //export FLBPluginFlushCtx 164 | func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int { 165 | var count int 166 | var ret int 167 | var ts interface{} 168 | var record map[interface{}]interface{} 169 | 170 | // Create Fluent Bit decoder 171 | dec := output.NewDecoder(data, int(length)) 172 | 173 | cloudwatchLogs := getPluginInstance(ctx) 174 | 175 | fluentTag := C.GoString(tag) 176 | logrus.Debugf("[cloudwatch %d] Found logs with tag: %s", cloudwatchLogs.PluginInstanceID, fluentTag) 177 | 178 | for { 179 | // Extract Record 180 | ret, ts, record = output.GetRecord(dec) 181 | if ret != 0 { 182 | break 183 | } 184 | 185 | var timestamp time.Time 186 | switch tts := ts.(type) { 187 | case output.FLBTime: 188 | timestamp = tts.Time 189 | case uint64: 190 | // when ts is of type uint64 it appears to 191 | // be the amount of seconds since unix epoch. 192 | timestamp = time.Unix(int64(tts), 0) 193 | default: 194 | timestamp = time.Now() 195 | } 196 | 197 | retCode := cloudwatchLogs.AddEvent(&cloudwatch.Event{Tag: fluentTag, Record: record, TS: timestamp}) 198 | if retCode != output.FLB_OK { 199 | return retCode 200 | } 201 | count++ 202 | } 203 | err := cloudwatchLogs.Flush() 204 | if err != nil { 205 | fmt.Println(err) 206 | // TODO: Better error handling 207 | return output.FLB_RETRY 208 | } 209 | 210 | logrus.Debugf("[cloudwatch %d] Processed %d events", cloudwatchLogs.PluginInstanceID, count) 211 | 212 | // Return options: 213 | // 214 | // output.FLB_OK = data have been processed. 215 | // output.FLB_ERROR = unrecoverable error, do not try this again. Never returned by flush. 216 | // output.FLB_RETRY = retry to flush later. 217 | return output.FLB_OK 218 | } 219 | 220 | //export FLBPluginExit 221 | func FLBPluginExit() int { 222 | return output.FLB_OK 223 | } 224 | 225 | func main() { 226 | } 227 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/aws/amazon-cloudwatch-logs-for-fluent-bit 2 | 3 | go 1.20 4 | 5 | require ( 6 | github.com/aws/amazon-kinesis-firehose-for-fluent-bit v1.5.1 7 | github.com/aws/aws-sdk-go v1.44.267 8 | github.com/fluent/fluent-bit-go v0.0.0-20201210173045-3fd1e0486df2 9 | github.com/golang/mock v1.4.4 10 | github.com/json-iterator/go v1.1.12 11 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect 12 | github.com/segmentio/ksuid v1.0.4 13 | github.com/sirupsen/logrus v1.9.0 14 | github.com/stretchr/testify v1.8.3 15 | github.com/valyala/bytebufferpool v1.0.0 16 | github.com/valyala/fasttemplate v1.2.2 17 | golang.org/x/sys v0.1.0 // indirect 18 | ) 19 | 20 | require ( 21 | github.com/davecgh/go-spew v1.1.1 // indirect 22 | github.com/jmespath/go-jmespath v0.4.0 // indirect 23 | github.com/modern-go/reflect2 v1.0.2 // indirect 24 | github.com/pmezard/go-difflib v1.0.0 // indirect 25 | github.com/ugorji/go/codec v1.2.6 // indirect 26 | gopkg.in/yaml.v3 v3.0.1 // indirect 27 | ) 28 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/aws/amazon-kinesis-firehose-for-fluent-bit v1.5.1 h1:6/X+V7X2W2+e2IPCiyhbWhQIMikDkwQ5tPFlcJv2FBk= 2 | github.com/aws/amazon-kinesis-firehose-for-fluent-bit v1.5.1/go.mod h1:alkjOObhCCp4KtT96XPWFi1PRRLUY0teGD8TWgroo2E= 3 | github.com/aws/aws-sdk-go v1.36.2/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= 4 | github.com/aws/aws-sdk-go v1.44.267 h1:Asrp6EMqqRxZvjK0NjzkWcrOk15RnWtupuUrUuZMabk= 5 | github.com/aws/aws-sdk-go v1.44.267/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= 6 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 7 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 8 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 9 | github.com/fastly/go-utils v0.0.0-20180712184237-d95a45783239/go.mod h1:Gdwt2ce0yfBxPvZrHkprdPPTTS3N5rwmLE8T22KBXlw= 10 | github.com/fluent/fluent-bit-go v0.0.0-20200707230002-2a28684e2382/go.mod h1:L92h+dgwElEyUuShEwjbiHjseW410WIcNz+Bjutc8YQ= 11 | github.com/fluent/fluent-bit-go v0.0.0-20201210173045-3fd1e0486df2 h1:G57WNyWS0FQf43hjRXLy5JT1V5LWVsSiEpkUcT67Ugk= 12 | github.com/fluent/fluent-bit-go v0.0.0-20201210173045-3fd1e0486df2/go.mod h1:L92h+dgwElEyUuShEwjbiHjseW410WIcNz+Bjutc8YQ= 13 | github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc= 14 | github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4= 15 | github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 16 | github.com/jehiah/go-strftime v0.0.0-20171201141054-1d33003b3869/go.mod h1:cJ6Cj7dQo+O6GJNiMx+Pa94qKj+TG8ONdKHgMNIyyag= 17 | github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= 18 | github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= 19 | github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8= 20 | github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U= 21 | github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= 22 | github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= 23 | github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= 24 | github.com/lestrrat-go/envload v0.0.0-20180220234015-a3eb8ddeffcc/go.mod h1:kopuH9ugFRkIXf3YoqHKyrJ9YfUFsckUU9S7B+XP+is= 25 | github.com/lestrrat-go/strftime v1.0.3/go.mod h1:E1nN3pCbtMSu1yjSVeyuRFVm/U0xoR76fd03sz+Qz4g= 26 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 27 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= 28 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 29 | github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= 30 | github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= 31 | github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= 32 | github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 33 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 34 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 35 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 36 | github.com/segmentio/ksuid v1.0.4 h1:sBo2BdShXjmcugAMwjugoGUdUV0pcxY5mW4xKRn3v4c= 37 | github.com/segmentio/ksuid v1.0.4/go.mod h1:/XUiZBD3kVx5SmUOl55voK5yeAbBNNIed+2O73XgrPE= 38 | github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= 39 | github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= 40 | github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= 41 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 42 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 43 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 44 | github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 45 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 46 | github.com/stretchr/testify v1.8.3 h1:RP3t2pwF7cMEbC1dqtB6poj3niw/9gnV4Cjg5oW5gtY= 47 | github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= 48 | github.com/tebeka/strftime v0.1.5/go.mod h1:29/OidkoWHdEKZqzyDLUyC+LmgDgdHo4WAFCDT7D/Ig= 49 | github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw= 50 | github.com/ugorji/go v1.2.6/go.mod h1:anCg0y61KIhDlPZmnH+so+RQbysYVyDko0IMgJv0Nn0= 51 | github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY= 52 | github.com/ugorji/go/codec v1.2.6 h1:7kbGefxLoDBuYXOms4yD7223OpNMMPNPZxXk5TvFcyQ= 53 | github.com/ugorji/go/codec v1.2.6/go.mod h1:V6TCNZ4PHqoHGFZuSG1W8nrCzzdgA2DozYxWFFpvxTw= 54 | github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw= 55 | github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= 56 | github.com/valyala/fasttemplate v1.2.2 h1:lxLXG0uE3Qnshl9QyaK6XJxMXlQZELvChBOCmQD0Loo= 57 | github.com/valyala/fasttemplate v1.2.2/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+qRAEEKiv+SiQ= 58 | github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= 59 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 60 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 61 | golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= 62 | golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= 63 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 64 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 65 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 66 | golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 67 | golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= 68 | golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= 69 | golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= 70 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 71 | golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 72 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 73 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 74 | golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 75 | golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 76 | golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 77 | golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 78 | golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 79 | golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 80 | golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 81 | golang.org/x/sys v0.1.0 h1:kunALQeHf1/185U1i0GOB/fy1IPRDDpuoOOqRReG57U= 82 | golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 83 | golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= 84 | golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= 85 | golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= 86 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 87 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 88 | golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= 89 | golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= 90 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 91 | golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 92 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 93 | golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= 94 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 95 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 96 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 97 | gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10= 98 | gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 99 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 100 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 101 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 102 | -------------------------------------------------------------------------------- /scripts/mockgen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the 5 | # "License"). You may not use this file except in compliance 6 | # with the License. A copy of the License is located at 7 | # 8 | # http://aws.amazon.com/apache2.0/ 9 | # 10 | # or in the "license" file accompanying this file. This file is 11 | # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR 12 | # CONDITIONS OF ANY KIND, either express or implied. See the 13 | # License for the specific language governing permissions and 14 | # limitations under the License. 15 | # 16 | # This script wraps the mockgen tool and inserts licensing information. 17 | 18 | set -e 19 | package=${1?Must provide package} 20 | interfaces=${2?Must provide interface names} 21 | outputfile=${3?Must provide an output file} 22 | 23 | export PATH="${GOPATH//://bin:}/bin:$PATH" 24 | 25 | data=$( 26 | cat << EOF 27 | // Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 28 | // 29 | // Licensed under the Apache License, Version 2.0 (the "License"). You may 30 | // not use this file except in compliance with the License. A copy of the 31 | // License is located at 32 | // 33 | // http://aws.amazon.com/apache2.0/ 34 | // 35 | // or in the "license" file accompanying this file. This file is distributed 36 | // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 37 | // express or implied. See the License for the specific language governing 38 | // permissions and limitations under the License. 39 | 40 | $(mockgen "$package" "$interfaces") 41 | EOF 42 | ) 43 | 44 | echo "$data" | goimports > "${outputfile}" 45 | --------------------------------------------------------------------------------