├── .github ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── ISSUE_TEMPLATE │ ├── bug.md │ └── feature.md ├── PULL_REQUEST_TEMPLATE.md ├── SECURITY.md └── SUPPORT.md ├── LICENSE ├── README.md ├── SECURITY.md └── config.yaml /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /.github/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 4 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 5 | the rights to use your contribution. For details, visit [Microsoft CLA](https://cla.microsoft.com). 6 | 7 | When you submit a pull request, a CLA-bot will automatically determine whether you need to provide 8 | a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions 9 | provided by the bot. You will only need to do this once across all repos using our CLA. 10 | 11 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 12 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 13 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 14 | 15 | - [Code of Conduct](#coc) 16 | - [Issues and Bugs](#issue) 17 | - [Feature Requests](#feature) 18 | - [Submission Guidelines](#submit) 19 | 20 | ## Code of Conduct 21 | 22 | Help us keep this project open and inclusive. Please read and follow our [Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 23 | 24 | ## Found an Issue 25 | 26 | If you find a bug in the source code or a mistake in the documentation, you can help us by 27 | [submitting an issue](#submit-issue) to the GitHub Repository. Even better, you can 28 | [submit a Pull Request](#submit-pr) with a fix. 29 | 30 | ## Want a Feature 31 | 32 | You can *request* a new feature by [submitting an issue](#submit-issue) to the GitHub 33 | Repository. If you would like to *implement* a new feature, please submit an issue with 34 | a proposal for your work first, to be sure that we can use it. 35 | 36 | - **Small Features** can be crafted and directly [submitted as a Pull Request](#submit-pr). 37 | 38 | ## Submission Guidelines 39 | 40 | ### Submitting an Issue 41 | 42 | Before you submit an issue, search the archive, maybe your question was already answered. 43 | 44 | If your issue appears to be a bug, and hasn't been reported, open a new issue. 45 | Help us to maximize the effort we can spend fixing issues and adding new 46 | features, by not reporting duplicate issues. Providing the following information will increase the 47 | chances of your issue being dealt with quickly: 48 | 49 | - **Overview of the Issue** - if an error is being thrown a non-minified stack trace helps 50 | - **Version** - what version is affected (e.g. 0.1.2) 51 | - **Motivation for or Use Case** - explain what are you trying to do and why the current behavior is a bug for you 52 | - **Browsers and Operating System** - is this a problem with all browsers? 53 | - **Reproduce the Error** - provide a live example or a unambiguous set of steps 54 | - **Related Issues** - has a similar issue been reported before? 55 | - **Suggest a Fix** - if you can't fix the bug yourself, perhaps you can point to what might be 56 | causing the problem (line of code or commit) 57 | 58 | You can file new issues by providing the above information at the corresponding repository's issues link: `https://github.com/[organization-name]/[repository-name]/issues/new]` 59 | 60 | ### Submitting a Pull Request (PR) 61 | 62 | Before you submit your Pull Request (PR) consider the following guidelines: 63 | 64 | - Search the repository `https://github.com/[organization-name]/[repository-name]/pulls` for an open or closed PR 65 | that relates to your submission. You don't want to duplicate effort. 66 | 67 | - Make your changes in a new git fork: 68 | 69 | - Commit your changes using a descriptive commit message 70 | - Push your fork to GitHub: 71 | - In GitHub, create a pull request 72 | - If we suggest changes then: 73 | - Make the required updates. 74 | - Rebase your fork and force push to your GitHub repository (this will update your Pull Request): 75 | 76 | ```shell 77 | 78 | git rebase main -i 79 | git push -f 80 | 81 | ``` 82 | 83 | That's it! Thank you for your contribution! 84 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: For reporting an issue in code or documentation for improvement 4 | labels: Bug 5 | 6 | --- 7 | 8 | # Bug Report 9 | 10 | ## Description 11 | 12 | - A clear description of the bug 13 | 14 | ## Expected Behavior 15 | 16 | - A clear description of what you expected 17 | 18 | ## Reproduce 19 | 20 | 1. Go to '...' 21 | 2. See error 22 | 23 | ## Additional Context 24 | 25 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature Request 3 | about: Suggest an idea for this project 4 | labels: Design Review, Enhancement 5 | 6 | --- 7 | 8 | # Problem Statement 9 | 10 | - 11 | 12 | ## Proposed Solution 13 | 14 | - 15 | 16 | ## Alternative Proposals 17 | 18 | - 19 | 20 | ## Additional Context 21 | 22 | - 23 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | # Purpose of PR 2 | 3 | ## Type of PR 4 | 5 | - [ ] Documentation changes 6 | - [ ] Code changes 7 | - [ ] Test changes 8 | - [ ] CI-CD changes 9 | 10 | ## Validation 11 | 12 | - [ ] Unit tests updated and ran successfully 13 | - [ ] Documentation updated 14 | 15 | ## Issues Closed or Referenced 16 | 17 | - Closes #issue_number (this will automatically close the issue when the PR closes) 18 | - References #issue_number (this references the issue but does not close with PR) 19 | -------------------------------------------------------------------------------- /.github/SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd). 40 | 41 | -------------------------------------------------------------------------------- /.github/SUPPORT.md: -------------------------------------------------------------------------------- 1 | # Support 2 | 3 | ## How to file issues and get help 4 | 5 | This project uses GitHub Issues to track bugs and feature requests. Please search the existing issues before filing new issues to avoid duplicates. For new issues, file your bug or feature request as a new Issue. 6 | 7 | For help and questions about using this project, please open a GitHub Issue. 8 | 9 | ## Microsoft Support Policy 10 | 11 | Support for this **PROJECT** is limited to the resources listed above. 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Fluent Bit with containerd, CRI-O and JSON 2 | 3 | With `dockerd` deprecated as a Kubernetes container runtime, we moved to `containerd`. After the change, our `fluentbit` logging didn't parse our JSON logs correctly. `containerd` and `CRI-O` use the `CRI Log` format which is slightly different and requires additional parsing to parse JSON application logs. 4 | 5 | We couldn't find a good end-to-end example, so we created this from various GitHub issues. There are some features missing (like multi-line logs) and we love PRs. 6 | 7 | ## Enhancement 8 | 9 | The original version of this repo used a separate filter to parse the JSON. By changing the cri parser to use the `log` field instead of the `message` field, the `kubernetes filter` converts the JSON if `Merge_Log` is set to `On` 10 | 11 | We also had to add two `nest filters` to the config to `lift` the Kubernetes values to the root level. 12 | 13 | ## Sample Config 14 | 15 | [config.yaml](./config.yaml) contains a complete and minimal example configuration using `stdout`. We have tested with `stdout` and `Azure Log Analytics`. While not tested, it should work with `Elastic Search` and outher `output` providers as well. 16 | 17 | > You will need to change the `output` `match` from `myapp*.*` 18 | 19 | ### Config Changes 20 | 21 | > Note - there are several GitHub discussions on the challenges with multi-line CRI Logs - additional processing is necessary and not included here 22 | 23 | In [config](./config.yaml) there are three changes: 24 | 25 | - Add the CRI parser which is a regex parser that maps the CRI Log fields into `time` `stream` `logtag` and `log` 26 | - `time` and `stream` map to existing `dockerd` log fields 27 | - `log` contains the text of the message, which, in our case is JSON 28 | - The JSON is parsed and merged in the `kubernetes filter` 29 | - `Merge_Log` must be set to `On` 30 | 31 | > Note the regex expression uses `log` for the 4th field, not `message` 32 | 33 | ```yaml 34 | 35 | [PARSER] 36 | Name cri 37 | Format regex 38 | Regex ^(?