├── .github └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── .travis.yml ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── MANIFEST.in ├── NOTICE ├── README.md ├── _config.yml ├── fluentmetrics ├── __init__.py ├── buffer.py └── metric.py ├── requirements-dev.txt ├── requirements.txt ├── setup.cfg ├── setup.py ├── tests ├── __init__.py ├── test_buffer.py └── test_metric.py └── tox.ini /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Created by https://www.gitignore.io/api/python 3 | 4 | ### Python ### 5 | # Byte-compiled / optimized / DLL files 6 | __pycache__/ 7 | *.py[cod] 8 | *$py.class 9 | 10 | # C extensions 11 | *.so 12 | 13 | # Distribution / packaging 14 | .Python 15 | env/ 16 | build/ 17 | develop-eggs/ 18 | dist/ 19 | downloads/ 20 | eggs/ 21 | .eggs/ 22 | lib/ 23 | lib64/ 24 | parts/ 25 | sdist/ 26 | var/ 27 | wheels/ 28 | *.egg-info/ 29 | .installed.cfg 30 | *.egg 31 | 32 | # PyInstaller 33 | # Usually these files are written by a python script from a template 34 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 35 | *.manifest 36 | *.spec 37 | 38 | # Installer logs 39 | pip-log.txt 40 | pip-delete-this-directory.txt 41 | 42 | # Unit test / coverage reports 43 | htmlcov/ 44 | .tox/ 45 | .coverage 46 | .coverage.* 47 | .cache 48 | nosetests.xml 49 | coverage.xml 50 | *,cover 51 | .hypothesis/ 52 | 53 | # Translations 54 | *.mo 55 | *.pot 56 | 57 | # Django stuff: 58 | *.log 59 | local_settings.py 60 | 61 | # Flask stuff: 62 | instance/ 63 | .webassets-cache 64 | 65 | # Scrapy stuff: 66 | .scrapy 67 | 68 | # Sphinx documentation 69 | docs/_build/ 70 | 71 | # PyBuilder 72 | target/ 73 | 74 | # Jupyter Notebook 75 | .ipynb_checkpoints 76 | 77 | # pyenv 78 | .python-version 79 | 80 | # celery beat schedule file 81 | celerybeat-schedule 82 | 83 | # SageMath parsed files 84 | *.sage.py 85 | 86 | # dotenv 87 | .env 88 | 89 | # virtualenv 90 | .venv 91 | venv/ 92 | ENV/ 93 | 94 | # Spyder project settings 95 | .spyderproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # End of https://www.gitignore.io/api/python 101 | 102 | # Vim 103 | *.swp 104 | .pytest_cache/ 105 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | env: 3 | global: 4 | - AWS_DEFAULT_REGION: us-east-1 5 | 6 | python: 7 | - "3.5" 8 | - "3.6" 9 | - "3.7" 10 | 11 | before_install: 12 | - export BOTO_CONFIG=/dev/null 13 | install: 14 | - pip install -r requirements-dev.txt 15 | - pip install tox-travis 16 | 17 | script: 18 | - tox 19 | - coverage run --source fluentmetrics -m pytest 20 | - coverage report -m 21 | 22 | after_success: 23 | - coveralls 24 | 25 | sudo: false 26 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/awslabs/cloudwatch-fluent-metrics/issues), or [recently closed](https://github.com/awslabs/cloudwatch-fluent-metrics/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/cloudwatch-fluent-metrics/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/awslabs/cloudwatch-fluent-metrics/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | include LICENSE 3 | include NOTICE 4 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | FluentMetrics 2 | Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # FluentMetrics 2 | ## **IMPORTANT: When using unique stream IDs, you have the potential to create a large number of metrics. Please make sure to review the [current AWS CloudWatch Custom Metrics pricing]( https://aws.amazon.com/cloudwatch/pricing/) before proceeding.** 3 | ## Overview 4 | `FluentMetrics` is an easy-to-use Python module that makes logging CloudWatch custom metrics a breeze. The goal is to provide a framework for logging detailed metrics with a minimal footprint. When you look at your code logic, you want to see your actual code logic, not line after line of metrics logging. `FluentMetrics` lets you maximize your metrics footprint while minimizing your metrics code footprint. 5 | ## Installation 6 | You can install directly from PyPI: 7 | 8 | ```sh 9 | pip install cloudwatch-fluent-metrics 10 | ``` 11 | ## 'Fluent' . . . what is that? 12 | Fluent describes an easy-to-read programming style. The goal of fluent development is to make code easier to read and reduce the amount of code required to build objects. It's easier to take a look a comparison between fluent and non-fluent style. 13 | #### Non-Fluent Example 14 | ```sh 15 | g = Game() 16 | f = Frame(Name='Tom') 17 | f.add_score(7) 18 | f.add_score(3) 19 | g.add_frame(f) 20 | f = Frame(Name='Tom') 21 | f.add_strike() 22 | g.add_frame(f) 23 | ``` 24 | #### Non-Fluent Example with Constructor 25 | ```sh 26 | g = Game() 27 | g.add_frame(Frame(Name='Tom', Score1=7, Score2=3) 28 | g.add_frame(Frame(Name='Tom', Score1=10) 29 | ``` 30 | #### Fluent Example 31 | ```sh 32 | g = Game() 33 | g.add_frame(Frame().with_name('Tom').score(3).spare()) 34 | g.add_frame(Frame().with_name('Tom').strike()) 35 | ``` 36 | While the difference may seem to be nitpicking, a frame is really just a constructed object. In the first example, we're taking up three lines of code to create the object--there's nothing wrong with that. However, in the second example, we're using constructors. This is slightly more readable, but there's a great deal of logic bulked up in our constructor. In the third example, we're using fluent-style code as it starts at creating the frame and *fluently* continues until it's created the entire frame in a single line. And more importantly, *it's readable.* We're not just creating an object with a massive constructor or spending several lines of code just to create a single object. 37 | ## Terminology Quickstart 38 | #### Namespaces 39 | Every metric needs to live in a namespace. Since you are logging your own custom metrics, you need to provide a custom namespace for your metric. Click [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-namespaces.html) for a list of the standard AWS namespaces. 40 | *Example*: 41 | In this example, we're creating a simple `FluentMetric` in a namespace called `Performance`. This means that every time we log a metric with `m`, we will automatically log it to the `Performance` namespace. 42 | ```sh 43 | from fluentmetrics import FluentMetric 44 | m = FluentMetric().with_namespace('Performance') 45 | ``` 46 | #### Metric Names 47 | The metric name is the thing you are actually logging. Each value that you log must be tied to a metric name. When you log a custom metric with a new metric name, the name will automatically be created if it doesn't already exist. Click [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html) to see existing metrics that can help you define names for your custom metrics. 48 | *Example*: 49 | In this example, we're logging two metrics called `StartupTime` and `StuffTime` to the `Performance` namespace (we only needed to define the namespace once). 50 | ```sh 51 | m = FluentMetric().with_namespace('Performance') 52 | m.log(MetricName='StartupTime', Value=27, Unit='Seconds') 53 | do_stuff() 54 | m.log(MetricName='StuffTime', Value=12000, Unit='Milliseconds') 55 | ``` 56 | #### Values 57 | Obviously we need to log a value with each metric. This needs to be a number since we convert this value to a `float` before sending to CloudWatch. 58 | **IMPORTANT**: When logging multiple values for the same custom metric within a minute, CloudWatch aggregates an average over a minute. Click [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html#publishingDataPoints) for more details. 59 | #### Storage Resolution 60 | The PutMetricData function now accepts an optional StorageResolution parameter. Set this parameter to 1 to publish high-resolution metrics; omit it (or set it to 60) to publish at standard 1-minute resolution. 61 | *Example*: 62 | In this example, we're logging metric at one-second resolution: 63 | ```sh 64 | m = FluentMetric().with_namespace('Application/MyApp') 65 | .with_storage_resolution(1) 66 | m.log(MetricName='Transactions/Sec', Value=trans_count, Unit='Count/Sec') 67 | ```sh 68 | #### Dimensions 69 | A dimension defines how you want to slice and dice the metric. These are simply name-value pairs and you can define up to 10 per metric. Click [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html#usingDimensions) for more details on using dimensions. 70 | **IMPORTANT:** When you define multiple dimensions, CloudMetrics attaches all of those dimensions to the metric as a single combined dimension set--think of them as an aggregate primary key. For example, if you log a metric with the dimensions `os = 'linux'` and `flavor='ubunutu'` you will only be able to aggregate by **both** `os` and `flavor`. You **cannot** aggregate only by just `os` or just `flavor`. `FluentMetrics` solves this problem by automatically logging three metrics--one for `os`, one for `flavor` and then one for the combied dimensions, giving you maximum flexibility. 71 | *Example*: 72 | In this example, we're logging boot/restart time metrics. When this code executes, we will end up with 6 metrics: 73 | * `BootTime` and `RestartTime` for `os` 74 | * `BootTime` and `RestartTime` for `instance-id` 75 | * `BootTime` and `RestartTime` for 'os` and `instance-id` 76 | ```sh 77 | m = FluentMetric().with_namespace('Performance/EC2') \ 78 | .with_dimension('os', 'linux'). \ 79 | .with_dimension('instance-id', 'i-123456') 80 | boot_time = start_instance() 81 | m.log(MetricName='BootTime', Value=boot_time, Unit='Milliseconds') 82 | restart_time = restart_instance() 83 | m.log(MetricName='RestartTime', Value=restart_time, Unit='Milliseconds') 84 | ``` 85 | #### Units 86 | CloudWatch has built-in logic to provide meaning to the metric values. We're not just logging a value--we're logging a value of some unit. By defining the unit type, CloudWatch will know how to properly present, aggregate and compare that value with other values. For example, if you submit a value with unit `Milliseconds`, then it can properly aggregate it up to seconds, minutes or hours. This is a list of the most current valid list of units. A more up-to-date list should be available [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html) under the **Unit** section,. 87 | ```sh 88 | "Seconds"|"Microseconds"|"Milliseconds"|"Bytes"|"Kilobytes"|"Megabytes"| 89 | "Gigabytes"|"Terabytes"|"Bits"|"Kilobits"|"Megabits"|"Gigabits"|"Terabits"| 90 | "Percent"|"Count"|"Bytes/Second"|"Kilobytes/Second"|"Megabytes/Second"| 91 | "Gigabytes/Second"|"Terabytes/Second"|"Bits/Second"|"Kilobits/Second"| 92 | "Megabits/Second"|"Gigabits/Second"|"Terabits/Second"|"Count/Second"|"None" 93 | ``` 94 | ##### Unit Shortcut Methods 95 | If you don't want to type out the individual unit name, there are shortcut methods for each unit. 96 | 97 | ```sh 98 | m = FluentMetric().with_namespace('Performance/EC2') \ 99 | .with_dimension('os', 'linux'). \ 100 | .with_dimension('instance-id', 'i-123456') 101 | m.seconds(MetricName='CompletionInSeconds', Value='1000') 102 | m.microseconds(MetricName='CompletionInMicroseconds', Value='1000') 103 | m.milliseconds(MetricName='CompletionInMilliseconds', Value='1000') 104 | m.bytes(MetricName='SizeInBytes', Value='1000') 105 | m.kb(MetricName='SizeInKb', Value='1000') 106 | m.mb(MetricName='SizeInMb', Value='1000') 107 | m.gb(MetricName='SizeInGb', Value='1000') 108 | m.tb(MetricName='SizeInTb', Value='1000') 109 | m.bits(MetricName='SizeInBits', Value='1000') 110 | m.kbits(MetricName='SizeInKilobits', Value='1000') 111 | m.mbits(MetricName='SizeInMegabits', Value='1000') 112 | m.gbits(MetricName='SizeInGigabits', Value='1000') 113 | m.tbits(MetricName='SizeInTerabits', Value='1000') 114 | m.pct(MetricName='Percent', Value='20') 115 | m.count(MetricName='ItemCount', Value='20') 116 | m.bsec(MetricName='BandwidthBytesPerSecond', Value='1000') 117 | m.kbsec(MetricName='BandwidthKilobytesPerSecond', Value='1000') 118 | m.mbsec(MetricName='BandwidthMegabytesPerSecond', Value='1000') 119 | m.gbsec(MetricName='BandwidthGigabytesPerSecond', Value='1000') 120 | m.tbsec(MetricName='BandwidthTerabytesPerSecond', Value='1000') 121 | m.bitsec(MetricName='BandwidthBitsPerSecond', Value='1000') 122 | m.kbitsec(MetricName='BandwidthKilobitsPerSecond', Value='1000') 123 | m.mbitsec(MetricName='BandwidthMegabitsPerSecond', Value='1000') 124 | m.gbitsec(MetricName='BandwidthGigabitsPerSecond', Value='1000') 125 | m.tbitsec(MetricName='BandwidthTerabitsPerSecond', Value='1000') 126 | m.countsec(MetricName='ItemCountsPerSecond', Value='1000') 127 | ``` 128 | #### Timers 129 | One of the most common uses of logging is measuring performance. FluentMetrics allows you to activate multiple built-in timers by name and log the elapsed time in a single line of code. **NOTE:** The elapsed time value is automatically stored as unit `Milliseconds`. 130 | *Example*: 131 | In this example, we're starting timers `workflow` and `job1` at the same time. Timers start as soon as you create them and never stop running. When you call `elapsed`, `FluentMetrics` will log the number of elapsed milliseconds with the `MetricName`. 132 | ```sh 133 | m = FluentMetric() 134 | m.with_timer('workflow').with_timer('job1') 135 | do_job1() 136 | m.elapsed(MetricName='Job1CompletionTime', TimerName='job1') 137 | m.with_timer('job2') 138 | do_job2() 139 | m.elapsed(MetricName='Job2CompletionTime', TimerName='job2') 140 | finish_workflow() 141 | m.elapsed(MetricName='WorkflowCompletionTime', TimerName='workflow') 142 | ``` 143 | #### Metric Stream ID 144 | A key feature of `FluentMetrics` is the metric stream ID. This ID will be added as a dimension and logged with every metric. The benefit of this dimension is to provide a distinct stream of metrics for an end-to-end operation. When you create a new instance of `FluentMetric`, you can either pass in your own value or `FluentMetrics` will generate a GUID. In CloudWatch, you can then see all of the metrics for a particular stream ID in chronological order. A metric stream can be a job, or a server or any way that you want to unique group a contiguous stream of metrics. 145 | *Example*: 146 | In this example, we'll have two metrics in the `Performance` namespace, each with metric stream ID of `abc-123`. We can then go to CloudWatch and filter by that stream ID to see the entire operation performance at a glance. 147 | ```sh 148 | m = FluentMetric().with_namespace('Performance').with_stream_id('abc-123') 149 | m.log(MetricName='StartupTime', Value=100, Unit='Seconds') 150 | do_work() 151 | m.log(MetricName='WorkCompleted', Value=1000, Unit='Milliseconds') 152 | ``` 153 | ## Use Case Quickstart 154 | #### #1: Least Amount of Code Required to Log a Metric 155 | This is the minimal amount of work you need to log--create a `FluentMetric` with a namespace, then log a value. 156 | **Result**: This code will log a single value `100` for `ActiveServerCount` in the `Stats` namespace. 157 | ```sh 158 | from fluentmetrics import FluentMetric 159 | m = FluentMetric().with_namespace('Stats') 160 | m.log(MetricName='ActiveServerCount', Value='100', Unit='Count') 161 | ``` 162 | #### #2: Logging Multiple Metrics to the Same Namespace 163 | If you are logging multiple metrics to the same namespace, this is a great use case for `FluentMetrics`. You only need to create one instance of `FluentMetric` and specify a different metric name when you call `log`. 164 | **Result**: This code will log a single value `100` for `ActiveServerCount` in the `Stats` namespace. 165 | ```sh 166 | from fluentmetrics import FluentMetric 167 | m = FluentMetric().with_namespace('Stats') 168 | m.log(MetricName='ActiveServerCount', Value='10', Unit='Count') \ 169 | .log(MetricName='StoppedServerCount', Value='20', Unit='Count') \ 170 | .log(MetricName='ActiveLinuxCount', Value='50', Unit='Count') \ 171 | .log(MetricName='ActiveWindowsCount', Value='50', Unit='Count') 172 | ```` 173 | #### #3: Logging Counts 174 | In the previous example, we logged a metric and identified the unit `Count`. Instead of specifying the unit, you can specify the type of object 175 | **Result**: This code will log a single value `100` for `ActiveServerCount` in the `Stats` namespace. 176 | 177 | ```sh 178 | from fluentmetrics import FluentMetric 179 | m = FluentMetric().with_namespace('Stats') 180 | m.count(MetricName='ActiveServerCount', Value='10') 181 | ``` 182 | 183 | #### BufferedFluentMetric 184 | Normally, with FluentMetric, metrics are sent immediately when `log` is called (or `count`, `milliseconds`, etc). This 185 | can result in a lot of `put_metric_data` calls to CloudWatch that are not full. When you use `BufferedFluentMetric` 186 | instead of `FluentMetric`, it waits until it has the maximum (20) metrics before calling `put_metric_data`. This optimizes 187 | traffic to cloudwatch. 188 | 189 | In general, `BufferedFluentMetric` behaves identically to `FluentMetric`, except that now it is possible to "forget" to 190 | send some metrics. The `BufferedFluentMetric.flush()` method pushes out all metrics immediately (clears the buffer). It 191 | is often best to do this at the end of a request (or some other obviously bounded interval). 192 | 193 | Here is an example of how it works in Flask: 194 | 195 | ```python 196 | from flask import g 197 | from fluentmetrics import BufferedFluentMetric 198 | 199 | @app.before_request 200 | def start_request(): 201 | g.metrics = BufferedFluentMetric() 202 | g.metrics.with_namespace('MyApp') 203 | g.metrics.with_timer('RequestLatency') 204 | 205 | @app.after_request 206 | def end_request(response): 207 | def error_counter(hundred): 208 | if response.status_code / 100 == hundred: 209 | return 1 210 | else: 211 | return 0 212 | 213 | g.metrics.count(MetricName='4xxError', Value=error_counter(400)) 214 | g.metrics.count(MetricName='5xxError', Value=error_counter(500)) 215 | g.metrics.count(MetricName='Availability', Value=(1 - error_counter(500))) 216 | g.metrics.elapsed(MetricName='RequestLatency', TimerName='RequestLatency') 217 | 218 | # Finally, ensure that all metrics end up in CloudWatch before this request finally ends. 219 | g.metrics.flush() 220 | ``` 221 | 222 | ## License 223 | 224 | This library is licensed under the Apache 2.0 License. 225 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-merlot -------------------------------------------------------------------------------- /fluentmetrics/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: Apache-2.0 3 | 4 | from .buffer import BufferedFluentMetric # noqa: F401 5 | from .metric import FluentMetric # noqa: F401 6 | -------------------------------------------------------------------------------- /fluentmetrics/buffer.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: Apache-2.0 3 | 4 | import logging 5 | 6 | from .metric import FluentMetric 7 | 8 | log = logging.getLogger('metric') 9 | log.addHandler(logging.NullHandler()) 10 | 11 | # This is defined by CloudWatch 12 | PAGE_SIZE = 20 13 | 14 | 15 | class BufferedFluentMetric(FluentMetric): 16 | '''A FluentMetric that tries to buffer as many metrics into as few requests 17 | as possible. Usage is intended to be exactly the same as FluentMetric, but 18 | make sure you re-use the BufferedFluentMetric instance (otherwise it won't buffer!) 19 | 20 | Occassionally, you may want to call metric.flush() manually (perhaps at the end of a 21 | web request or on a timer) to ensure that data is never older than a certain age. 22 | 23 | This class is not thread safe. 24 | ''' 25 | 26 | def __init__(self, client=None, max_items=PAGE_SIZE * 5, **kwargs): 27 | FluentMetric.__init__(self, client, **kwargs) 28 | self.max_items = max_items 29 | self.buffers = {} 30 | 31 | def _record_metric(self, metric_data): 32 | size = self._size() 33 | 34 | num_allowed = self.max_items - size 35 | if num_allowed < len(metric_data): 36 | log.warn("Dropping {} out of {} metrics".format(len(metric_data) - num_allowed, len(metric_data))) 37 | 38 | buffer = self.buffers.get(self.namespace, []) 39 | self.buffers[self.namespace] = buffer # in case it wasn't set 40 | 41 | buffer += metric_data[:num_allowed] 42 | 43 | # clear as much WIP as possible 44 | self.flush(send_partial=False) 45 | 46 | def _size(self): 47 | return sum([len(buffer) for buffer in self.buffers.values()]) 48 | 49 | def flush(self, send_partial=True): 50 | '''Sends as much data as possible to CloudWatch. If send_partial is set to False, 51 | this only sends full pages. This way, it minimizes the API usage at the cost of 52 | delaying data. 53 | ''' 54 | for namespace, buffer in self.buffers.items(): 55 | full_pages = len(buffer) // PAGE_SIZE 56 | for i in range(full_pages): 57 | start = i * PAGE_SIZE 58 | end = (i + 1) * PAGE_SIZE 59 | page = buffer[start:end] 60 | 61 | # ship it 62 | FluentMetric._record_metric(self, page) 63 | 64 | start = full_pages * PAGE_SIZE 65 | end = len(buffer) % PAGE_SIZE 66 | if send_partial: 67 | # ship remaining items 68 | page = buffer[start:end] 69 | FluentMetric._record_metric(self, page) 70 | 71 | # clear buffer 72 | self.buffers[namespace] = [] 73 | 74 | # This condition isn't needed for correctness, it could be an else, it just 75 | # reduces memory churn. You should get the same result either way. 76 | elif full_pages > 0: 77 | # clear shipped items from buffer 78 | self.buffers[namespace] = buffer[start:] 79 | 80 | return self 81 | -------------------------------------------------------------------------------- /fluentmetrics/metric.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: Apache-2.0 3 | 4 | import logging 5 | import arrow 6 | import uuid 7 | import boto3 8 | import boto3.session 9 | 10 | logger = logging.getLogger('metric') 11 | logger.addHandler(logging.NullHandler()) 12 | 13 | 14 | class Timer(object): 15 | def __init__(self): 16 | self.start = arrow.utcnow() 17 | 18 | def elapsed(self): 19 | return arrow.utcnow() - self.start 20 | 21 | def elapsed_in_ms(self): 22 | return self.elapsed().total_seconds() * 1000 23 | 24 | def elapsed_in_seconds(self): 25 | return self.elapsed().total_seconds() 26 | 27 | 28 | class FluentMetric(object): 29 | def __init__(self, client=None, **kwargs): 30 | self.dimensions = [] 31 | self.timers = {} 32 | self.dimension_stack = [] 33 | self.storage_resolution = 60 34 | self.use_stream_id = kwargs.get('UseStreamId', True) 35 | if self.use_stream_id: 36 | self.stream_id = str(uuid.uuid4()) 37 | self.with_dimension('MetricStreamId', self.stream_id) 38 | else: 39 | self.stream_id = None 40 | 41 | if client: 42 | self.client = client 43 | else: 44 | profile = kwargs.get('Profile') 45 | if profile: 46 | session = boto3.session.Session(profile_name=profile) 47 | self.client = session.client('cloudwatch') 48 | else: 49 | self.client = boto3.client('cloudwatch') 50 | 51 | def with_storage_resolution(self, value): 52 | self.storage_resolution = value 53 | return self 54 | 55 | def with_stream_id(self, id): 56 | self.stream_id = id 57 | self.with_dimension('MetricStreamId', self.stream_id) 58 | return self 59 | 60 | def with_namespace(self, namespace): 61 | self.namespace = namespace 62 | return self 63 | 64 | def with_dimension(self, name, value): 65 | self.without_dimension(name) 66 | self.dimensions.append({'Name': name, 'Value': value}) 67 | return self 68 | 69 | def without_dimension(self, name): 70 | if not self.does_dimension_exist(name): 71 | return 72 | self.dimensions = \ 73 | [item for item in self.dimensions if not item['Name'] == name] 74 | return self 75 | 76 | def does_dimension_exist(self, name): 77 | d = [item for item in self.dimensions if item['Name'] == name] 78 | if d: 79 | return True 80 | else: 81 | return False 82 | 83 | def get_dimension_value(self, name): 84 | d = [item for item in self.dimensions if item['Name'] == name] 85 | if d: 86 | return d[0]['Value'] 87 | else: 88 | return None 89 | 90 | def with_timer(self, timer): 91 | self.timers[timer] = Timer() 92 | return self 93 | 94 | def without_timer(self, timer): 95 | if timer in self.timers.keys(): 96 | del self.timers[timer] 97 | return self 98 | 99 | def get_timer(self, timer): 100 | if timer in self.timers.keys(): 101 | return self.timers[timer] 102 | else: 103 | return None 104 | 105 | def push_dimensions(self): 106 | self.dimension_stack.append(self.dimensions) 107 | self.dimensions = [] 108 | if self.use_stream_id and self.stream_id: 109 | self.with_stream_id(self.stream_id) 110 | return self 111 | 112 | def pop_dimensions(self): 113 | self.dimensions = self.dimension_stack.pop() 114 | return self 115 | 116 | def elapsed(self, **kwargs): 117 | tn = kwargs.get('TimerName') 118 | mn = kwargs.get('MetricName') 119 | if tn not in self.timers.keys(): 120 | logger.warn('No timer named {}'.format(tn)) 121 | return 122 | self.log(Value=self.timers[tn].elapsed_in_ms(), 123 | Unit='Milliseconds', 124 | MetricName=mn) 125 | return self 126 | 127 | def countsec(self, **kwargs): 128 | mn = kwargs.get('MetricName') 129 | count = kwargs.get('Value', 1) 130 | self.log(Value=count, 131 | Unit='Count/Second', 132 | MetricName=mn) 133 | return self 134 | 135 | def tbitsec(self, **kwargs): 136 | mn = kwargs.get('MetricName') 137 | count = kwargs.get('Value', 1) 138 | self.log(Value=count, 139 | Unit='Terabits/Second', 140 | MetricName=mn) 141 | return self 142 | 143 | def gbitsec(self, **kwargs): 144 | mn = kwargs.get('MetricName') 145 | count = kwargs.get('Value', 1) 146 | self.log(Value=count, 147 | Unit='Gigabits/Second', 148 | MetricName=mn) 149 | return self 150 | 151 | def mbitsec(self, **kwargs): 152 | mn = kwargs.get('MetricName') 153 | count = kwargs.get('Value', 1) 154 | self.log(Value=count, 155 | Unit='Megabits/Second', 156 | MetricName=mn) 157 | return self 158 | 159 | def kbitsec(self, **kwargs): 160 | mn = kwargs.get('MetricName') 161 | count = kwargs.get('Value', 1) 162 | self.log(Value=count, 163 | Unit='Kilobits/Second', 164 | MetricName=mn) 165 | return self 166 | 167 | def bitsec(self, **kwargs): 168 | mn = kwargs.get('MetricName') 169 | count = kwargs.get('Value', 1) 170 | self.log(Value=count, 171 | Unit='Bits/Second', 172 | MetricName=mn) 173 | return self 174 | 175 | def tbsec(self, **kwargs): 176 | mn = kwargs.get('MetricName') 177 | count = kwargs.get('Value', 1) 178 | self.log(Value=count, 179 | Unit='Terabytes/Second', 180 | MetricName=mn) 181 | return self 182 | 183 | def gbsec(self, **kwargs): 184 | mn = kwargs.get('MetricName') 185 | count = kwargs.get('Value', 1) 186 | self.log(Value=count, 187 | Unit='Gigabytes/Second', 188 | MetricName=mn) 189 | return self 190 | 191 | def mbsec(self, **kwargs): 192 | mn = kwargs.get('MetricName') 193 | count = kwargs.get('Value', 1) 194 | self.log(Value=count, 195 | Unit='Megabytes/Second', 196 | MetricName=mn) 197 | return self 198 | 199 | def kbsec(self, **kwargs): 200 | mn = kwargs.get('MetricName') 201 | count = kwargs.get('Value', 1) 202 | self.log(Value=count, 203 | Unit='Kilobytes/Second', 204 | MetricName=mn) 205 | return self 206 | 207 | def bsec(self, **kwargs): 208 | mn = kwargs.get('MetricName') 209 | count = kwargs.get('Value', 1) 210 | self.log(Value=count, 211 | Unit='Bytes/Second', 212 | MetricName=mn) 213 | return self 214 | 215 | def pct(self, **kwargs): 216 | mn = kwargs.get('MetricName') 217 | count = kwargs.get('Value', 1) 218 | self.log(Value=count, 219 | Unit='Percent', 220 | MetricName=mn) 221 | return self 222 | 223 | def tbits(self, **kwargs): 224 | mn = kwargs.get('MetricName') 225 | count = kwargs.get('Value', 1) 226 | self.log(Value=count, 227 | Unit='Terabits', 228 | MetricName=mn) 229 | return self 230 | 231 | def gbits(self, **kwargs): 232 | mn = kwargs.get('MetricName') 233 | count = kwargs.get('Value', 1) 234 | self.log(Value=count, 235 | Unit='Gigabits', 236 | MetricName=mn) 237 | return self 238 | 239 | def mbits(self, **kwargs): 240 | mn = kwargs.get('MetricName') 241 | count = kwargs.get('Value', 1) 242 | self.log(Value=count, 243 | Unit='Megabits', 244 | MetricName=mn) 245 | return self 246 | 247 | def kbits(self, **kwargs): 248 | mn = kwargs.get('MetricName') 249 | count = kwargs.get('Value', 1) 250 | self.log(Value=count, 251 | Unit='Kilobits', 252 | MetricName=mn) 253 | return self 254 | 255 | def bits(self, **kwargs): 256 | mn = kwargs.get('MetricName') 257 | count = kwargs.get('Value', 1) 258 | self.log(Value=count, 259 | Unit='Bits', 260 | MetricName=mn) 261 | return self 262 | 263 | def tb(self, **kwargs): 264 | mn = kwargs.get('MetricName') 265 | count = kwargs.get('Value', 1) 266 | self.log(Value=count, 267 | Unit='Terabytes', 268 | MetricName=mn) 269 | return self 270 | 271 | def gb(self, **kwargs): 272 | mn = kwargs.get('MetricName') 273 | count = kwargs.get('Value', 1) 274 | self.log(Value=count, 275 | Unit='Gigabytes', 276 | MetricName=mn) 277 | return self 278 | 279 | def mb(self, **kwargs): 280 | mn = kwargs.get('MetricName') 281 | count = kwargs.get('Value', 1) 282 | self.log(Value=count, 283 | Unit='Megabytes', 284 | MetricName=mn) 285 | return self 286 | 287 | def kb(self, **kwargs): 288 | mn = kwargs.get('MetricName') 289 | count = kwargs.get('Value', 1) 290 | self.log(Value=count, 291 | Unit='Kilobytes', 292 | MetricName=mn) 293 | return self 294 | 295 | def bytes(self, **kwargs): 296 | mn = kwargs.get('MetricName') 297 | count = kwargs.get('Value', 1) 298 | self.log(Value=count, 299 | Unit='Bytes', 300 | MetricName=mn) 301 | return self 302 | 303 | def milliseconds(self, **kwargs): 304 | mn = kwargs.get('MetricName') 305 | count = kwargs.get('Value', 1) 306 | self.log(Value=count, 307 | Unit='Milliseconds', 308 | MetricName=mn) 309 | return self 310 | 311 | def microseconds(self, **kwargs): 312 | mn = kwargs.get('MetricName') 313 | count = kwargs.get('Value', 1) 314 | self.log(Value=count, 315 | Unit='Microseconds', 316 | MetricName=mn) 317 | return self 318 | 319 | def seconds(self, **kwargs): 320 | mn = kwargs.get('MetricName') 321 | count = kwargs.get('Value', 1) 322 | self.log(Value=count, 323 | Unit='Seconds', 324 | MetricName=mn) 325 | return self 326 | 327 | def count(self, **kwargs): 328 | mn = kwargs.get('MetricName') 329 | count = kwargs.get('Value', 1) 330 | self.log(Value=count, 331 | Unit='Count', 332 | MetricName=mn) 333 | return self 334 | 335 | def log(self, **kwargs): 336 | ts = kwargs.get('TimeStamp', arrow.utcnow() 337 | .format('YYYY-MM-DD HH:mm:ss ZZ')) 338 | value = float(kwargs.get('Value')) 339 | unit = kwargs.get('Unit') 340 | md = [] 341 | for dimension in self.dimensions: 342 | md.append({ 343 | 'MetricName': kwargs.get('MetricName'), 344 | 'Dimensions': [dimension], 345 | 'Timestamp': ts, 346 | 'Value': value, 347 | 'Unit': unit, 348 | 'StorageResolution': self.storage_resolution, 349 | } 350 | ) 351 | 352 | md.append({ 353 | 'MetricName': kwargs.get('MetricName'), 354 | 'Dimensions': self.dimensions, 355 | 'Timestamp': ts, 356 | 'Value': value, 357 | 'Unit': unit, 358 | 'StorageResolution': self.storage_resolution, 359 | }) 360 | 361 | self._record_metric(md) 362 | return self 363 | 364 | def _record_metric(self, metric_data): 365 | logger.debug('log: {}'.format(metric_data)) 366 | if metric_data: 367 | self.client.put_metric_data( 368 | Namespace=self.namespace, 369 | MetricData=metric_data, 370 | ) 371 | 372 | def get_metrics(self, **kwargs): 373 | mn = kwargs.get('MetricName') 374 | return self.client.list_metrics(Namespace=self.namespace, 375 | MetricName=mn)['Metrics'] 376 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | arrow==0.14.6 2 | boto==2.49.0 3 | boto3==1.9.220 4 | coverage==4.5.4 5 | coveralls==1.8.2 6 | flake8==3.7.8 7 | mock==3.0.5 8 | moto==1.3.13 9 | pytest==5.1.2 10 | tox==3.13.2 11 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | arrow==0.14.6 2 | boto3==1.9.220 3 | mock==3.0.5 4 | moto==1.3.13 5 | zest.releaser==6.19.0 6 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | description-file = README.md 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | import setuptools 3 | from distutils.core import setup 4 | 5 | 6 | def read(fname): 7 | return open(os.path.join(os.path.dirname(__file__), fname)).read() 8 | 9 | 10 | setup( 11 | name='cloudwatch-fluent-metrics', 12 | packages=setuptools.find_packages(), 13 | version='0.5.3.dev0', 14 | description='AWS CloudWatch Fluent Metrics', 15 | long_description=read('README.md'), 16 | long_description_content_type='text/markdown', 17 | author='troylar', 18 | author_email='troylars@amazon.com', 19 | url='https://github.com/awslabs/cloudwatch-fluent-metrics', 20 | download_url='https://github.com/awslabs/cloudwatch-fluent-metrics/cloudwatch-fluent-metrics-v0.1.tgz', # noqa: E501 21 | keywords=['metrics', 'logging', 'aws', 'cloudwatch'], 22 | license="Apache-2.0", 23 | classifiers=[ 24 | "Development Status :: 5 - Production/Stable", 25 | "Topic :: Utilities", 26 | "License :: OSI Approved :: Apache Software License", 27 | ] 28 | ) 29 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/cloudwatch-fluent-metrics/504d4a95d0f4e1a19dec3c6526f2c856e7384a55/tests/__init__.py -------------------------------------------------------------------------------- /tests/test_buffer.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import mock 3 | import unittest 4 | from moto import mock_cloudwatch 5 | from fluentmetrics import BufferedFluentMetric 6 | 7 | log = logging.getLogger('metric') 8 | Metric = BufferedFluentMetric 9 | 10 | 11 | def with_metric(*dimensions): 12 | def decorator(func): 13 | @mock_cloudwatch 14 | @mock.patch('fluentmetrics.buffer.PAGE_SIZE', 3) 15 | def wrapper(*args, **kwargs): 16 | cw = Dummy() 17 | m = Metric(cw) 18 | m.with_namespace('namespace') 19 | m.without_dimension('MetricStreamId') # probably should remove it entirely as a default 20 | for name, value in dimensions: 21 | m.with_dimension(name, value) 22 | kwargs['m'] = m 23 | kwargs['cw'] = cw 24 | func(*args, **kwargs) 25 | wrapper.__name__ = func.__name__ 26 | return wrapper 27 | return decorator 28 | 29 | 30 | class TestBuffer(unittest.TestCase): 31 | @with_metric() 32 | def test_noflush_partial_page(self, m, cw): 33 | m.count(MetricName='counter', Value=1) 34 | m.count(MetricName='counter', Value=2) 35 | assert len(cw.calls) == 0 36 | 37 | @with_metric() 38 | def test_autoflush_exactly_one_page(self, m, cw): 39 | m.count(MetricName='counter', Value=1) 40 | m.count(MetricName='counter', Value=2) 41 | m.count(MetricName='counter', Value=3) 42 | 43 | assert len(cw.calls) == 1 44 | data = cw.calls[0]['MetricData'] 45 | assert len(data) == 3 46 | values = [d['Value'] for d in data if d['MetricName'] == 'counter'] 47 | self.assertSequenceEqual(values, [1, 2, 3]) 48 | 49 | @with_metric() 50 | def test_autoflush_1_and_half_pages(self, m, cw): 51 | m.count(MetricName='counter', Value=1) 52 | m.count(MetricName='counter', Value=2) 53 | m.count(MetricName='counter', Value=3) 54 | assert len(m.buffers['namespace']) == 0 55 | m.count(MetricName='counter', Value=4) 56 | 57 | assert len(cw.calls) == 1 58 | data = cw.calls[0]['MetricData'] 59 | assert len(data) == 3 60 | values = [d['Value'] for d in data if d['MetricName'] == 'counter'] 61 | self.assertSequenceEqual(values, [1, 2, 3]) 62 | 63 | m.flush() 64 | 65 | assert len(cw.calls) == 2 66 | data = cw.calls[1]['MetricData'] 67 | assert len(data) == 1 68 | values = [d['Value'] for d in data if d['MetricName'] == 'counter'] 69 | self.assertSequenceEqual(values, [4]) 70 | 71 | @with_metric() 72 | def test_flush_3_pages(self, m, cw): 73 | # this should send 9 records for each count 74 | for dim in range(9): 75 | name = 'dim{}'.format(dim + 1) 76 | m.with_dimension(name, dim) 77 | m.count(MetricName='counter', Value=1) 78 | 79 | assert len(cw.calls) == 3 80 | 81 | for i in range(3): 82 | data = cw.calls[i]['MetricData'] 83 | assert len(data) == 3 84 | values = [d['Value'] for d in data if d['MetricName'] == 'counter'] 85 | self.assertSequenceEqual(values, [1, 1, 1]) 86 | 87 | 88 | class Dummy(object): 89 | def __init__(self): 90 | self.calls = [] 91 | 92 | def put_metric_data(self, **kwargs): 93 | self.calls.append(kwargs) 94 | -------------------------------------------------------------------------------- /tests/test_metric.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: Apache-2.0 3 | 4 | 5 | import arrow 6 | import time 7 | from fluentmetrics import FluentMetric 8 | import mock 9 | from moto import mock_cloudwatch 10 | 11 | 12 | @mock_cloudwatch 13 | def test_setting_namespace_sets_namespace(): 14 | test_value = 'test_namespace' 15 | m = FluentMetric() 16 | m.with_namespace(test_value) 17 | assert m.namespace == test_value 18 | 19 | 20 | def test_adding_dimension_adds_dimension(): 21 | test_name = 'test_name' 22 | test_value = 'test_value' 23 | m = FluentMetric() 24 | m.with_dimension(test_name, test_value) 25 | assert m.does_dimension_exist(test_name) 26 | 27 | 28 | def test_removing_dimension_removes_dimension(): 29 | test_name = 'test_name' 30 | test_value = 'test_value' 31 | m = FluentMetric() 32 | m.with_dimension(test_name, test_value) 33 | assert m.does_dimension_exist(test_name) 34 | m.without_dimension(test_name) 35 | assert not m.does_dimension_exist(test_name) 36 | 37 | 38 | def test_dimension_does_not_duplicate(): 39 | test_name = 'test_name' 40 | test_value1 = 'test_value1' 41 | test_value2 = 'test_value2' 42 | m = FluentMetric() 43 | m.with_dimension(test_name, test_value1) 44 | assert m.get_dimension_value(test_name) == test_value1 45 | m.with_dimension(test_name, test_value2) 46 | assert m.get_dimension_value(test_name) == test_value2 47 | 48 | 49 | def test_adding_timer_starts_timer(): 50 | name = 'test_timer' 51 | m = FluentMetric() 52 | m.with_timer(name) 53 | time.sleep(1) 54 | t = m.get_timer(name) 55 | assert t.start < arrow.utcnow() 56 | assert t.elapsed_in_ms() > 1000 and t.elapsed_in_ms() < 2000 57 | 58 | 59 | def test_can_add_multiple_timers(): 60 | name1 = 'test_timer_1' 61 | name2 = 'test_timer_2' 62 | m = FluentMetric() 63 | m.with_timer(name1) 64 | time.sleep(1) 65 | t = m.get_timer(name1) 66 | assert t.start < arrow.utcnow() 67 | assert t.elapsed_in_ms() > 1000 and t.elapsed_in_ms() < 2000 68 | 69 | m.with_timer(name2) 70 | time.sleep(1) 71 | u = m.get_timer(name2) 72 | assert u.start < arrow.utcnow() 73 | assert u.elapsed_in_ms() > 1000 and u.elapsed_in_ms() < 2000 74 | assert t.elapsed_in_ms() > 2000 75 | 76 | 77 | def test_removing_timer_removes_timer(): 78 | name = 'test_timer' 79 | m = FluentMetric() 80 | m.with_timer(name) 81 | time.sleep(1) 82 | t = m.get_timer(name) 83 | assert t.start < arrow.utcnow() 84 | assert t.elapsed_in_ms() > 1000 and t.elapsed_in_ms() < 2000 85 | m.without_timer(name) 86 | t = m.get_timer(name) 87 | assert not t 88 | 89 | 90 | def test_can_push_dimensions(): 91 | test_name = 'test_name' 92 | test_value = 'test_value' 93 | m = FluentMetric() 94 | m.with_dimension(test_name, test_value) 95 | assert m.does_dimension_exist(test_name) 96 | m.push_dimensions() 97 | assert len(m.dimensions) == 1 98 | m.pop_dimensions() 99 | assert len(m.dimensions) == 2 100 | 101 | 102 | @mock.patch('fluentmetrics.FluentMetric.log') 103 | def test_can_log_count(fm_log): 104 | m = FluentMetric().with_namespace('Performance') 105 | m.count(MetricName='test', Count=2) 106 | fm_log.assert_called() 107 | 108 | 109 | def test_can_set_resolution(): 110 | m = FluentMetric().with_namespace('Performance').with_storage_resolution(1) 111 | assert m.storage_resolution == 1 112 | 113 | 114 | def test_can_disable_stream_id(): 115 | m = FluentMetric(UseStreamId=False).with_namespace('Performance') 116 | assert len(m.dimensions) == 0 117 | 118 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py35,py36,py37,flake8 3 | 4 | [testenv] 5 | whitelist_externals = pytest 6 | commands = pytest -v 7 | deps = -r{toxinidir}/requirements-dev.txt 8 | passenv = 9 | TRAVIS 10 | TRAVIS_BRANCH 11 | TRAVIS_JOB_ID 12 | AWS_DEFAULT_REGION 13 | 14 | [testenv:flake8] 15 | commands = flake8 . 16 | deps = flake8 17 | 18 | [travis] 19 | python = 20 | 3.7: py37, flake8 21 | 22 | [pytest] 23 | addopts = --ignore=setup.py 24 | python_files = *.py 25 | python_functions = test_ 26 | 27 | [flake8] 28 | exclude = 29 | .git, 30 | .tox, 31 | build, 32 | dist 33 | venv 34 | ignore = E501 35 | --------------------------------------------------------------------------------