├── os_performance_tools ├── tests │ ├── __init__.py │ ├── base.py │ ├── test_queues.py │ ├── test_mysql.py │ ├── test_collectors_delta.py │ ├── test_counters2statsd.py │ └── test_collect.py ├── collectors │ ├── __init__.py │ ├── _delta.py │ ├── mysql.py │ └── queues.py ├── error.py ├── __init__.py ├── collect.py └── counters2statsd.py ├── doc ├── source │ ├── readme.rst │ ├── installation.rst │ ├── index.rst │ ├── usage.rst │ ├── contributor │ │ └── contributing.rst │ └── conf.py └── requirements.txt ├── .stestr.conf ├── bindep.txt ├── .gitreview ├── .mailmap ├── MANIFEST.in ├── .coveragerc ├── HACKING.rst ├── releasenotes └── notes │ └── drop-py-2-7-495369ce08f2b3d6.yaml ├── requirements.txt ├── test-requirements.txt ├── .zuul.yaml ├── README.rst ├── CONTRIBUTING.rst ├── .gitignore ├── setup.py ├── setup.cfg ├── tox.ini └── LICENSE /os_performance_tools/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /os_performance_tools/collectors/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /doc/source/readme.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../../README.rst 2 | -------------------------------------------------------------------------------- /.stestr.conf: -------------------------------------------------------------------------------- 1 | [DEFAULT] 2 | test_path=./os_performance_tools/tests/ 3 | top_dir=./ -------------------------------------------------------------------------------- /bindep.txt: -------------------------------------------------------------------------------- 1 | # PDF Docs package dependencies 2 | tex-gyre [doc platform:dpkg] 3 | -------------------------------------------------------------------------------- /os_performance_tools/error.py: -------------------------------------------------------------------------------- 1 | class CollectionError(Exception): 2 | pass 3 | -------------------------------------------------------------------------------- /.gitreview: -------------------------------------------------------------------------------- 1 | [gerrit] 2 | host=review.opendev.org 3 | port=29418 4 | project=openstack/os-performance-tools.git 5 | -------------------------------------------------------------------------------- /.mailmap: -------------------------------------------------------------------------------- 1 | # Format is: 2 | # 3 | # 4 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include AUTHORS 2 | include ChangeLog 3 | exclude .gitignore 4 | exclude .gitreview 5 | 6 | global-exclude *.pyc 7 | -------------------------------------------------------------------------------- /doc/requirements.txt: -------------------------------------------------------------------------------- 1 | sphinx>=2.0.0,!=2.1.0 # BSD 2 | openstackdocstheme>=2.2.1 # Apache-2.0 3 | sphinxcontrib-apidoc>=0.2.0 # BSD 4 | -------------------------------------------------------------------------------- /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | branch = True 3 | source = openstack_qa_tools 4 | omit = openstack_qa_tools/openstack/* 5 | 6 | [report] 7 | ignore_errors = True 8 | -------------------------------------------------------------------------------- /HACKING.rst: -------------------------------------------------------------------------------- 1 | os-performance-tools Style Commandments 2 | =============================================== 3 | 4 | Read the OpenStack Style Commandments https://docs.openstack.org/hacking/latest/ 5 | -------------------------------------------------------------------------------- /releasenotes/notes/drop-py-2-7-495369ce08f2b3d6.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | upgrade: 3 | - | 4 | Python 2.7 support has been dropped. The minimum version of Python now 5 | supported by os-performance-tools is Python 3.6. 6 | -------------------------------------------------------------------------------- /doc/source/installation.rst: -------------------------------------------------------------------------------- 1 | ============ 2 | Installation 3 | ============ 4 | 5 | At the command line:: 6 | 7 | $ pip install os-performance-tools 8 | 9 | Or, if you have virtualenvwrapper installed:: 10 | 11 | $ mkvirtualenv os-performance-tools 12 | $ pip install os-performance-tools 13 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # The order of packages is significant, because pip processes them in the order 2 | # of appearance. Changing the order has an impact on the overall integration 3 | # process, which may cause wedges in the gate later. 4 | 5 | pbr>=2.0,!=2.1.0 6 | PyMySQL>=0.6.2 # MIT License 7 | statsd>=1.0.0 8 | oslo.config>=1.4.0.0a3 9 | testtools>=1.4.0 10 | -------------------------------------------------------------------------------- /test-requirements.txt: -------------------------------------------------------------------------------- 1 | # The order of packages is significant, because pip processes them in the order 2 | # of appearance. Changing the order has an impact on the overall integration 3 | # process, which may cause wedges in the gate later. 4 | 5 | hacking<3.2.0,>=3.1.0 6 | 7 | coverage>=3.6 8 | python-subunit>=0.0.18 9 | 10 | oslotest>=1.10.0 # Apache-2.0 11 | stestr>=2.0.0 # Apache-2.0 12 | testscenarios>=0.4 13 | -------------------------------------------------------------------------------- /.zuul.yaml: -------------------------------------------------------------------------------- 1 | - project: 2 | templates: 3 | - check-requirements 4 | - publish-openstack-docs-pti 5 | - openstack-cover-jobs 6 | check: 7 | jobs: 8 | - openstack-tox-pep8 9 | - openstack-tox-py39 10 | - openstack-tox-py310 11 | - openstack-tox-py311 12 | - openstack-tox-py312 13 | gate: 14 | jobs: 15 | - openstack-tox-pep8 16 | - openstack-tox-py39 17 | - openstack-tox-py310 18 | - openstack-tox-py311 19 | - openstack-tox-py312 20 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | =============================== 2 | os-performance-tools 3 | =============================== 4 | 5 | Python scripts for use in OpenStack's Performance Measurement 6 | 7 | 8 | * Free software: Apache license 9 | * Documentation: http://docs.openstack.org/developer/os-performance-tools 10 | * Source: http://opendev.org/openstack/os-performance-tools 11 | * Bugs: No bug tracker as of now, report on ML or IRC #openstack-qa channel. 12 | 13 | Features 14 | -------- 15 | 16 | * Collect counters from various backends and save as JSON and/or Subunit attachments 17 | * Record delta with previous JSON counters 18 | * Push counters into statsd 19 | -------------------------------------------------------------------------------- /doc/source/index.rst: -------------------------------------------------------------------------------- 1 | ================================================ 2 | Welcome to os-performance-tools's documentation! 3 | ================================================ 4 | 5 | Contents 6 | -------- 7 | .. toctree:: 8 | :maxdepth: 2 9 | 10 | readme 11 | installation 12 | usage 13 | reference/api/modules 14 | 15 | For Contributor 16 | --------------- 17 | If you are a new contributor to os-performance-tools please refer: :doc:`contributor/contributing` 18 | 19 | .. toctree:: 20 | :hidden: 21 | 22 | contributor/contributing 23 | 24 | Indices and tables 25 | ================== 26 | 27 | * :ref:`genindex` 28 | * :ref:`modindex` 29 | * :ref:`search` 30 | 31 | -------------------------------------------------------------------------------- /os_performance_tools/__init__.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); you may 2 | # not use this file except in compliance with the License. You may obtain 3 | # a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 9 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 10 | # License for the specific language governing permissions and limitations 11 | # under the License. 12 | 13 | import pbr.version 14 | 15 | 16 | __version__ = pbr.version.VersionInfo( 17 | 'os_performance_tools').version_string() 18 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | The source repository for this project can be found at: 2 | 3 | https://opendev.org/openstack/os-performance-tools 4 | 5 | Pull requests submitted through GitHub are not monitored. 6 | 7 | To start contributing to OpenStack, follow the steps in the contribution guide 8 | to set up and use Gerrit: 9 | 10 | https://docs.openstack.org/contributors/code-and-documentation/quick-start.html 11 | 12 | Bugs should be filed on Launchpad: 13 | 14 | https://bugs.launchpad.net/tempest (We use Tempest Launchpad for os-performance-tools bugs tracking.) 15 | 16 | For more specific information about contributing to this repository, see the 17 | os-performance-tools contributor guide: 18 | 19 | https://docs.openstack.org/os-performance-tools/latest/contributor/contributing.html 20 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | .eggs 12 | eggs 13 | parts 14 | bin 15 | var 16 | sdist 17 | develop-eggs 18 | .installed.cfg 19 | lib 20 | lib64 21 | 22 | # Installer logs 23 | pip-log.txt 24 | 25 | # Unit test / coverage reports 26 | .coverage 27 | .tox 28 | nosetests.xml 29 | .stestr/ 30 | .venv 31 | 32 | # Translations 33 | *.mo 34 | 35 | # Mr Developer 36 | .mr.developer.cfg 37 | .project 38 | .pydevproject 39 | 40 | # Complexity 41 | output/*.html 42 | output/*/index.html 43 | 44 | # Sphinx 45 | doc/build 46 | doc/source/reference/ 47 | 48 | # pbr generates these 49 | AUTHORS 50 | ChangeLog 51 | 52 | # Editors 53 | *~ 54 | .*.swp 55 | .*sw? 56 | 57 | doc/source/modules.rst 58 | doc/source/os_performance_tools.* 59 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2013 Hewlett-Packard Development Company, L.P. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 12 | # implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | # THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT 17 | import setuptools 18 | 19 | setuptools.setup( 20 | setup_requires=['pbr'], 21 | pbr=True) 22 | -------------------------------------------------------------------------------- /os_performance_tools/tests/base.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-2011 OpenStack Foundation 2 | # Copyright (c) 2013 Hewlett-Packard Development Company, L.P. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may 5 | # not use this file except in compliance with the License. You may obtain 6 | # a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | # License for the specific language governing permissions and limitations 14 | # under the License. 15 | 16 | from oslotest import base 17 | 18 | 19 | class TestCase(base.BaseTestCase): 20 | 21 | """Test case base class for all unit tests.""" 22 | -------------------------------------------------------------------------------- /doc/source/usage.rst: -------------------------------------------------------------------------------- 1 | ===== 2 | Usage 3 | ===== 4 | 5 | .. _counters: 6 | 7 | os-collect-counters 8 | ------------------- 9 | 10 | This is a command line program which will try to collect data from 11 | various backends as rapidly as possible and then save a representation 12 | of these as JSON. It can optionall wrap this JSON in subunit, which is 13 | useful for appending to a subunit stream from previous tests. 14 | 15 | os_performance_tools.counters2statsd 16 | ------------------------------------ 17 | 18 | This module is useful in a subunit processing stream for saving the 19 | counters into statsd. It should be added as a target before processing 20 | the stream, and then upon reaching the end, it will send the counters into 21 | statsd. Currently this is limited to an attachment named 'counters.json' 22 | but other schemes may be implemented in the future. 23 | 24 | Please see counters_ for information on the backends and counters that 25 | are supported. 26 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = os-performance-tools 3 | summary = Python scripts for use in OpenStack's QA process 4 | description_file = 5 | README.rst 6 | author = OpenStack 7 | author_email = openstack-discuss@lists.openstack.org 8 | home_page = https://docs.openstack.org/os-performance-tools/latest/ 9 | python_requires = >=3.9 10 | classifier = 11 | Environment :: OpenStack 12 | Intended Audience :: Information Technology 13 | Intended Audience :: System Administrators 14 | License :: OSI Approved :: Apache Software License 15 | Operating System :: POSIX :: Linux 16 | Programming Language :: Python 17 | Programming Language :: Python :: 3 18 | Programming Language :: Python :: 3.9 19 | Programming Language :: Python :: 3.10 20 | Programming Language :: Python :: 3.11 21 | Programming Language :: Python :: 3.12 22 | Programming Language :: Python :: 3 :: Only 23 | Programming Language :: Python :: Implementation :: CPython 24 | 25 | 26 | [entry_points] 27 | console_scripts = 28 | os-collect-counters = os_performance_tools.collect:main 29 | subunit2sql.target = 30 | os_counters_statsd = os_performance_tools.counters2statsd:AttachmentResult 31 | 32 | [files] 33 | packages = 34 | os_performance_tools 35 | -------------------------------------------------------------------------------- /os_performance_tools/tests/test_queues.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); you may 2 | # not use this file except in compliance with the License. You may obtain 3 | # a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 9 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 10 | # License for the specific language governing permissions and limitations 11 | # under the License. 12 | 13 | """ 14 | test_collectors 15 | ---------------------------------- 16 | 17 | Tests for `os_performance_tools.collectors` 18 | """ 19 | 20 | import json 21 | from unittest import mock 22 | 23 | from os_performance_tools.collectors import queues 24 | from os_performance_tools.tests import base 25 | 26 | 27 | class TestOpenStackQaTols(base.TestCase): 28 | 29 | @mock.patch('http.client.HTTPConnection') 30 | def test_queues(self, httplib_mock): 31 | reader = mock.MagicMock(name='getresponse_reader') 32 | rval = json.dumps([{'name': 'foo', 'message_stats': {'publish': 1}}]) 33 | reader.read.return_value = rval 34 | conn = httplib_mock.return_value 35 | conn.getresponse.return_value = reader 36 | data = queues.collect() 37 | self.assertEqual({'foo_publish': 1}, data) 38 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | minversion = 3.18.0 3 | envlist = py3,pypy,pep8 4 | skipsdist = True 5 | ignore_basepython_conflict = True 6 | 7 | [testenv] 8 | basepython = python3 9 | usedevelop = True 10 | setenv = 11 | VIRTUAL_ENV={envdir} 12 | deps = 13 | -r{toxinidir}/requirements.txt 14 | -r{toxinidir}/test-requirements.txt 15 | commands = stestr run --slowest {posargs} 16 | 17 | [testenv:pep8] 18 | commands = flake8 19 | 20 | [testenv:venv] 21 | commands = {posargs} 22 | 23 | [testenv:cover] 24 | allowlist_externals = find 25 | setenv = 26 | {[testenv]setenv} 27 | PYTHON=coverage run --source os_performance_tools --parallel-mode 28 | commands = 29 | coverage erase 30 | find . -type f -name "*.pyc" -delete 31 | stestr run {posargs} 32 | coverage combine 33 | coverage html -d cover 34 | coverage xml -o cover/coverage.xml 35 | coverage report 36 | 37 | [testenv:docs] 38 | deps = 39 | -r{toxinidir}/requirements.txt 40 | -r{toxinidir}/test-requirements.txt 41 | -r{toxinidir}/doc/requirements.txt 42 | commands = 43 | sphinx-build -W -b html -d doc/build/doctrees doc/source doc/build/html 44 | 45 | [testenv:pdf-docs] 46 | deps = {[testenv:docs]deps} 47 | allowlist_externals = 48 | make 49 | commands = 50 | sphinx-build -W -b latex doc/source doc/build/pdf 51 | make -C doc/build/pdf 52 | 53 | [testenv:debug] 54 | commands = oslo_debug_helper {posargs} 55 | 56 | [flake8] 57 | # E123, E125 skipped as they are invalid PEP-8. 58 | 59 | show-source = True 60 | ignore = E123,E125 61 | builtins = _ 62 | exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build 63 | -------------------------------------------------------------------------------- /os_performance_tools/tests/test_mysql.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); you may 2 | # not use this file except in compliance with the License. You may obtain 3 | # a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 9 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 10 | # License for the specific language governing permissions and limitations 11 | # under the License. 12 | 13 | """ 14 | test_collectors 15 | ---------------------------------- 16 | 17 | Tests for `os_performance_tools.collectors` 18 | """ 19 | 20 | from unittest import mock 21 | 22 | from os_performance_tools.collectors import mysql 23 | from os_performance_tools.tests import base 24 | 25 | 26 | class TestOpenStackQaTools(base.TestCase): 27 | 28 | @mock.patch('os_performance_tools.collectors.mysql._get_config') 29 | @mock.patch('pymysql.connect') 30 | def test_mysql(self, pymysql_mock, get_config_mock): 31 | connection = mock.MagicMock() 32 | curs = mock.MagicMock() 33 | side_effect = [(k, 0) for k in mysql.COLLECT_COUNTERS] 34 | side_effect.append(None) # Instead of StopIteration pymsql uses None 35 | curs.fetchone.side_effect = side_effect 36 | connection.cursor.return_value = curs 37 | pymysql_mock.return_value = connection 38 | result = mysql.collect() 39 | self.assertEqual(sorted(mysql.COLLECT_COUNTERS), 40 | sorted(result.keys())) 41 | self.assertTrue(all([val == 0 for val in result.values()])) 42 | -------------------------------------------------------------------------------- /os_performance_tools/collectors/_delta.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); 2 | # you may not use this file except in compliance with the License. 3 | # You may obtain a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, 9 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 10 | # See the License for the specific language governing permissions and 11 | # limitations under the License. 12 | 13 | import json 14 | 15 | 16 | def delta(previous, current, meta=False): 17 | product = {} 18 | seen = set() 19 | 20 | # Old keys 21 | for k, v in previous.items(): 22 | if k not in current: 23 | continue 24 | newv = current[k] 25 | if type(v) is not type(newv): 26 | raise ValueError( 27 | 'Type of key %s changed from %s to %s' % (k, 28 | type(v), 29 | type(newv))) 30 | if k == '__meta__': 31 | meta = True 32 | if meta and k == 'delta_seconds': 33 | continue 34 | elif meta and k == 'unixtime': 35 | product['delta_seconds'] = newv - v 36 | product[k] = newv 37 | elif isinstance(v, int) or isinstance(v, float): 38 | product[k] = newv - v 39 | elif isinstance(v, dict): 40 | product[k] = delta(v, newv, meta) 41 | else: 42 | raise ValueError('Only mappings of numbers are understood') 43 | seen.add(k) 44 | # New keys 45 | for k in set(current) - seen: 46 | product[k] = current[k] 47 | return product 48 | 49 | 50 | def delta_with_file(previous_path, current_data): 51 | with open(previous_path) as previous_file: 52 | previous_data = json.loads(previous_file.read()) 53 | return delta(previous_data, current_data) 54 | -------------------------------------------------------------------------------- /doc/source/contributor/contributing.rst: -------------------------------------------------------------------------------- 1 | ============================ 2 | So You Want to Contribute... 3 | ============================ 4 | 5 | For general information on contributing to OpenStack, please check out the 6 | `contributor guide `_ to get started. 7 | It covers all the basics that are common to all OpenStack projects: the accounts 8 | you need, the basics of interacting with our Gerrit review system, how we 9 | communicate as a community, etc. 10 | 11 | Below will cover the more project specific information you need to get started 12 | with os-performance-tools. 13 | 14 | Communication 15 | ~~~~~~~~~~~~~ 16 | * IRC channel ``#openstack-qa`` at FreeNode 17 | * Mailing list (prefix subjects with ``[qa]`` for faster responses) 18 | http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss 19 | 20 | Contacting the Core Team 21 | ~~~~~~~~~~~~~~~~~~~~~~~~ 22 | Please refer to the `os-performance-tools Core Team 23 | `_ contacts. 24 | 25 | New Feature Planning 26 | ~~~~~~~~~~~~~~~~~~~~ 27 | If you want to propose a new feature please read `Feature Proposal Process`_ 28 | 29 | Task Tracking 30 | ~~~~~~~~~~~~~ 31 | There is no separate task tracking tool for os-performance-tools, we track our tasks in `Launchpad `__. 32 | 33 | Reporting a Bug 34 | ~~~~~~~~~~~~~~~ 35 | You found an issue and want to make sure we are aware of it? You can do so on 36 | `Launchpad `__. There is no separate 37 | Launchpad for os-performance-tools. 38 | More info about Launchpad usage can be found on `OpenStack docs page 39 | `_ 40 | 41 | Getting Your Patch Merged 42 | ~~~~~~~~~~~~~~~~~~~~~~~~~ 43 | All changes proposed to the os-performance-tools requires single ``Code-Review +2`` votes as minimum from 44 | os-performance-tools core reviewers who can approve patch by giving ``Workflow +1`` vote. 45 | 46 | Project Team Lead Duties 47 | ~~~~~~~~~~~~~~~~~~~~~~~~ 48 | All common PTL duties are enumerated in the `PTL guide 49 | `_. 50 | 51 | The Release Process for QA is documented in `QA Release Process 52 | `_. 53 | 54 | .. _Feature Proposal Process: https://wiki.openstack.org/wiki/QA#Feature_Proposal_.26_Design_discussions 55 | -------------------------------------------------------------------------------- /os_performance_tools/collectors/mysql.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); 2 | # you may not use this file except in compliance with the License. 3 | # You may obtain a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, 9 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 10 | # See the License for the specific language governing permissions and 11 | # limitations under the License. 12 | # 13 | # This file was forked from dstat's mysql5_innodb plugin but retains none of 14 | # that original code other than a list of well known MySQL variable names. 15 | ''' MySQL is accessed via the configuration options found at ~/.my.cnf. This is 16 | parsed not by libmysqlclient, which may or may not be present, but by 17 | configparser. As such, some options that are usually usable from that file may 18 | be ignored by this module. Everything from the "client" section will be passed 19 | through to pymysql's connect method. 20 | ''' 21 | 22 | import configparser 23 | import logging 24 | import os 25 | 26 | import pymysql 27 | 28 | from os_performance_tools import error 29 | 30 | COLLECT_COUNTERS = ( 31 | 'Com_delete', 32 | 'Com_insert', 33 | 'Com_select', 34 | 'Com_update', 35 | 'Connections', 36 | 'Innodb_buffer_pool_read_requests', 37 | 'Innodb_data_reads', 38 | 'Innodb_data_read', 39 | 'Innodb_data_writes', 40 | 'Innodb_data_written', 41 | 'Innodb_log_writes', 42 | 'Innodb_rows_deleted', 43 | 'Innodb_rows_inserted', 44 | 'Innodb_rows_read', 45 | 'Innodb_rows_updated', 46 | 'Queries', 47 | 'Slow_queries', 48 | ) 49 | '''These counters' meaning are all documented in the `MySQL manual 50 | `_. 51 | They are intended to show a picture of how much has been asked of 52 | MySQL, and how busy MySQL was while executing commands. Each one will 53 | be recorded unaltered by name in the resulting counters mapping. 54 | ''' 55 | 56 | 57 | def _get_config(): 58 | args = {} 59 | try: 60 | with open(os.path.expanduser("~/.my.cnf")) as dfile: 61 | parser = configparser.ConfigParser() 62 | parser.read_file(dfile) 63 | for k, v in parser.items('client'): 64 | args[k] = v 65 | except IOError as e: 66 | raise error.CollectionError(str(e)) 67 | return args 68 | 69 | 70 | def collect(): 71 | log = logging.getLogger() 72 | args = _get_config() 73 | conn = pymysql.connect(**args) 74 | cursor = conn.cursor() 75 | counters = {} 76 | cursor.execute('show global status') 77 | while True: 78 | result = cursor.fetchone() 79 | if result is None: 80 | break 81 | k, v = result 82 | if k in COLLECT_COUNTERS: 83 | counters[k] = int(v) 84 | log.debug(counters) 85 | return counters 86 | -------------------------------------------------------------------------------- /os_performance_tools/tests/test_collectors_delta.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); you may 2 | # not use this file except in compliance with the License. You may obtain 3 | # a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 9 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 10 | # License for the specific language governing permissions and limitations 11 | # under the License. 12 | 13 | """ 14 | test_delta 15 | ---------------------------------- 16 | 17 | Tests for `os_performance_tools.collectors._delta` 18 | """ 19 | 20 | import testscenarios 21 | 22 | from os_performance_tools.collectors import _delta 23 | from os_performance_tools.tests import base 24 | 25 | 26 | class TestOSQATDelta(testscenarios.WithScenarios, base.TestCase): 27 | 28 | scenarios = [ 29 | ('add1', dict( 30 | previous={'zoo': {'aardvark': 9, 'zebra': 0}}, 31 | current={'zoo': {'aardvark': 12, 'zebra': 0}}, 32 | expected={'zoo': {'aardvark': 3, 'zebra': 0}})), 33 | ('newkey', dict( 34 | previous={'zoo': {'bee': 0}}, 35 | current={'lake': {'trout': 1}, 'zoo': {'bee': 5}}, 36 | expected={'lake': {'trout': 1}, 'zoo': {'bee': 5}})), 37 | ('delkey', dict( 38 | previous={'zoo': {'cat': 99}}, 39 | current={}, 40 | expected={})), 41 | ('newvar', dict( 42 | previous={'zoo': {'dog': 9}}, 43 | current={'zoo': {'dog': 9, 'ocelot': 2}}, 44 | expected={'zoo': {'dog': 0, 'ocelot': 2}})), 45 | ('delvar', dict( 46 | previous={'zoo': {'elephant': 1000, 'bear': 1}}, 47 | current={'zoo': {'elephant': 1000}}, 48 | expected={'zoo': {'elephant': 0}})), 49 | ('stringval', dict( 50 | previous={'zoo': 'foo'}, 51 | current={'zoo': 0}, 52 | expected=ValueError)), 53 | ('newstrval', dict( 54 | previous={'zoo': {'giraffe': 100}}, 55 | current={'zoo': {'giraffe': 'tall'}}, 56 | expected=ValueError)), 57 | ('changetype', dict( 58 | previous={'zoo': {'horse': 7}}, 59 | current={'zoo': 15}, 60 | expected=ValueError)), 61 | ('meta_unixtime', dict( 62 | previous={'__meta__': {'unixtime': 1.0}}, 63 | current={'__meta__': {'unixtime': 10.5}}, 64 | expected={'__meta__': {'unixtime': 10.5, 'delta_seconds': 9.5}})), 65 | ('meta_unixtime_gone', dict( 66 | previous={'__meta__': {'unixtime': 1.0}}, 67 | current={}, 68 | expected={})), 69 | ('meta_unixtime_new', dict( 70 | previous={}, 71 | current={'__meta__': {'unixtime': 1.0}}, 72 | expected={'__meta__': {'unixtime': 1.0}})), 73 | ] 74 | 75 | def test_delta(self): 76 | if self.expected is ValueError: 77 | self.assertRaises(ValueError, _delta.delta, self.previous, 78 | self.current) 79 | else: 80 | self.assertEqual(self.expected, _delta.delta(self.previous, 81 | self.current)) 82 | -------------------------------------------------------------------------------- /doc/source/conf.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); 2 | # you may not use this file except in compliance with the License. 3 | # You may obtain a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, 9 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 10 | # implied. 11 | # See the License for the specific language governing permissions and 12 | # limitations under the License. 13 | 14 | import os 15 | import sys 16 | 17 | sys.path.insert(0, os.path.abspath('../..')) 18 | # -- General configuration ---------------------------------------------------- 19 | 20 | # Add any Sphinx extension module names here, as strings. They can be 21 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 22 | extensions = [ 23 | #'sphinx.ext.intersphinx', 24 | 'openstackdocstheme', 25 | 'sphinxcontrib.apidoc', 26 | ] 27 | 28 | # openstackdocstheme options 29 | openstackdocs_repo_name = 'openstack/os-performance-tools' 30 | openstackdocs_auto_name = False 31 | openstackdocs_bug_project = 'os-performance-tools' 32 | openstackdocs_bug_tag = '' 33 | openstackdocs_pdf_link = True 34 | 35 | html_theme = 'openstackdocs' 36 | 37 | # sphinxcontrib.apidoc options 38 | apidoc_module_dir = '../../os_performance_tools' 39 | apidoc_output_dir = 'reference/api' 40 | apidoc_excluded_paths = [ 41 | 'tests/*', 42 | 'tests'] 43 | apidoc_separate_modules = True 44 | 45 | # autodoc generation is a bit aggressive and a nuisance when doing heavy 46 | # text edit cycles. 47 | # execute "export SPHINX_DEBUG=1" in your terminal to disable 48 | 49 | # The suffix of source filenames. 50 | source_suffix = '.rst' 51 | 52 | # The master toctree document. 53 | master_doc = 'index' 54 | 55 | # General information about the project. 56 | project = u'os-performance-tools' 57 | copyright = u'2013, OpenStack Foundation' 58 | 59 | # If true, '()' will be appended to :func: etc. cross-reference text. 60 | add_function_parentheses = True 61 | 62 | # If true, the current module name will be prepended to all description 63 | # unit titles (such as .. function::). 64 | add_module_names = True 65 | 66 | # The name of the Pygments (syntax highlighting) style to use. 67 | pygments_style = 'native' 68 | 69 | 70 | # -- Options for HTML output -------------------------------------------------- 71 | 72 | # The theme to use for HTML and HTML Help pages. Major themes that come with 73 | # Sphinx are currently 'default' and 'sphinxdoc'. 74 | # html_theme_path = ["."] 75 | # html_theme = '_theme' 76 | # html_static_path = ['static'] 77 | 78 | # Output file base name for HTML help builder. 79 | htmlhelp_basename = '%sdoc' % project 80 | 81 | # Grouping the document tree into LaTeX files. List of tuples 82 | # (source start file, target name, title, author, documentclass 83 | # [howto/manual]). 84 | latex_documents = [ 85 | ('index', 86 | 'doc-%s.tex' % project, 87 | u'Os-performance-tools Documentation', 88 | u'OpenStack Foundation', 'manual'), 89 | ] 90 | 91 | # Example configuration for intersphinx: refer to the Python standard library. 92 | #intersphinx_mapping = {'http://docs.python.org/': None} 93 | 94 | latex_use_xindy = False 95 | latex_elements = { 96 | 'extraclassoptions': 'openany,oneside', 97 | 'maxlistdepth': 10 98 | } 99 | -------------------------------------------------------------------------------- /os_performance_tools/collectors/queues.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); 2 | # you may not use this file except in compliance with the License. 3 | # You may obtain a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, 9 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 10 | # See the License for the specific language governing permissions and 11 | # limitations under the License. 12 | '''RabbitMQ is the primary message queue backend for OpenStack. It 13 | is accessed via the status plugin, which must be enabled for this 14 | collector to work. The environment variables OS_QA_RABBITMQ_API, 15 | OS_QA_RABBITMQ_USER, OS_QA_RABBITMQ_PASS, are used to configure how the 16 | status API is contacted by the collector. 17 | 18 | The counters reported are entirely 'published' methods, meaning this is 19 | just a measure of how many messages were pushed into RabbitMQ. 20 | 21 | Because of the way oslo.messaging uses RabbitMQ, we won't know what exact 22 | application reply queues are attached to. So all of those messages end up 23 | in the 'reply' counter. Fanouts also have a random string added to them, 24 | so we strip that off. 25 | ''' 26 | 27 | import base64 28 | import http.client as http_client 29 | import json 30 | import logging 31 | import os 32 | import re 33 | import socket 34 | 35 | 36 | from os_performance_tools import error 37 | 38 | OS_QA_RABBITMQ_API = os.environ.get('OS_QA_RABBITMQ_API', 39 | '127.0.0.1:15672') 40 | OS_QA_RABBITMQ_API_USER = os.environ.get('OS_QA_RABBITMQ_USER', 41 | 'guest') 42 | OS_QA_RABBITMQ_API_PASS = os.environ.get('OS_QA_RABBITMQ_PASS', 43 | 'guest') 44 | FANOUT_RE = re.compile(r'([\-a-zA-Z0-9]+)_fanout_[a-f0-9]{32}') 45 | 46 | 47 | def collect(): 48 | log = logging.getLogger() 49 | conn = http_client.HTTPConnection(OS_QA_RABBITMQ_API) 50 | auth = '%s:%s' % (OS_QA_RABBITMQ_API_USER, OS_QA_RABBITMQ_API_PASS) 51 | auth = base64.encodebytes(auth.encode('utf-8')).decode('ascii') 52 | auth = auth.replace('\n', '') 53 | auth = {'Authorization': 'Basic %s' % auth} 54 | try: 55 | conn.request('GET', '/api/queues', headers=auth) 56 | log.debug('requested /api/queues') 57 | content = conn.getresponse().read() 58 | log.debug('received content [%s]' % content) 59 | except (socket.error, http_client.HTTPException) as e: 60 | raise error.CollectionError(str(e)) 61 | 62 | content = json.loads(content) 63 | if not isinstance(content, list): 64 | raise error.CollectionError( 65 | 'Unexpected format encountered. %s' % content) 66 | collected = {} 67 | for q in content: 68 | if not isinstance(q, dict): 69 | continue 70 | if "name" not in q: 71 | continue 72 | qname = q["name"] 73 | if qname.startswith('reply_'): 74 | qname = 'reply' 75 | else: 76 | match = FANOUT_RE.match(qname) 77 | if match: 78 | qname = '{}_fanout'.format(match.group(1)) 79 | if "message_stats" in q and "publish" in q["message_stats"]: 80 | target = '%s_publish' % (qname) 81 | collected[target] = q["message_stats"]["publish"] 82 | return collected 83 | -------------------------------------------------------------------------------- /os_performance_tools/tests/test_counters2statsd.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); you may 2 | # not use this file except in compliance with the License. You may obtain 3 | # a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 9 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 10 | # License for the specific language governing permissions and limitations 11 | # under the License. 12 | 13 | """ 14 | test_counters2statsd 15 | ---------------------------------- 16 | 17 | Tests for `os_performance_tools.counters2statsd` 18 | """ 19 | 20 | import json 21 | import time 22 | from unittest import mock 23 | 24 | from os_performance_tools import counters2statsd 25 | from os_performance_tools.tests import base 26 | 27 | 28 | class TestCounters2Statsd(base.TestCase): 29 | 30 | @mock.patch('statsd.StatsClient') 31 | def test_add_test_run_attachments(self, statsd_mock): 32 | mock_client = mock.MagicMock('statsd_client') 33 | statsd_mock.return_value = mock_client 34 | mock_client.pipeline = mock.MagicMock('statsd_pipeline') 35 | mock_pipeline = mock.MagicMock('Pipeline') 36 | mock_pipeline.incr = mock.MagicMock('statds_incr') 37 | mock_pipeline.send = mock.MagicMock('statds_send') 38 | mock_client.pipeline.return_value = mock_pipeline 39 | fake_counters = {'mysql': {'Queries': 10}} 40 | fake_counters = json.dumps(fake_counters).encode('utf-8') 41 | self.assertTrue(counters2statsd.AttachmentResult.enabled()) 42 | result = counters2statsd.AttachmentResult() 43 | result.status(file_name='counters.json', file_bytes=fake_counters) 44 | result.stopTestRun() 45 | statsd_mock.assert_called_with('localhost', 8125, None) 46 | mock_pipeline.incr.assert_called_with('mysql.Queries', 10) 47 | mock_pipeline.send.assert_called_with() 48 | 49 | @mock.patch('statsd.StatsClient') 50 | def test_add_test_run_attachments_meta(self, statsd_mock): 51 | mock_client = mock.MagicMock('statsd_client') 52 | statsd_mock.return_value = mock_client 53 | mock_client.pipeline = mock.MagicMock('statsd_pipeline') 54 | mock_pipeline = mock.MagicMock('Pipeline') 55 | mock_pipeline.incr = mock.MagicMock('statds_incr') 56 | mock_pipeline.timing = mock.MagicMock('statds_timing') 57 | mock_pipeline.send = mock.MagicMock('statds_send') 58 | mock_client.pipeline.return_value = mock_pipeline 59 | fake_counters = { 60 | '__meta__': { 61 | 'unixtime': time.time(), 62 | 'delta_seconds': 10.5, 63 | 'prefix': 'all-tests', 64 | }, 65 | 'mysql': { 66 | 'Queries': 50 67 | } 68 | } 69 | fake_counters = json.dumps(fake_counters).encode('utf-8') 70 | self.assertTrue(counters2statsd.AttachmentResult.enabled()) 71 | result = counters2statsd.AttachmentResult() 72 | result.status(file_name='counters.json', file_bytes=fake_counters) 73 | result.stopTestRun() 74 | statsd_mock.assert_called_with('localhost', 8125, None) 75 | mock_pipeline.timing.assert_called_with('all-tests.testrun', 10500) 76 | mock_pipeline.incr.assert_called_with('all-tests.mysql.Queries', 50) 77 | mock_pipeline.send.assert_called_with() 78 | -------------------------------------------------------------------------------- /os_performance_tools/collect.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); 2 | # you may not use this file except in compliance with the License. 3 | # You may obtain a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, 9 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 10 | # See the License for the specific language governing permissions and 11 | # limitations under the License. 12 | 13 | import argparse 14 | import json 15 | import logging 16 | import sys 17 | import threading 18 | import time 19 | 20 | from os_performance_tools.collectors import _delta 21 | from os_performance_tools.collectors import mysql 22 | from os_performance_tools.collectors import queues 23 | from subunit import v2 as subunit_v2 24 | 25 | mysql_data = {} 26 | queues_data = {} 27 | 28 | 29 | def get_mysql(): 30 | global mysql_data 31 | mysql_data = mysql.collect() 32 | 33 | 34 | def get_queues(): 35 | global queues_data 36 | queues_data = queues.collect() 37 | 38 | 39 | def main(argv=None, stdout=None): 40 | if stdout is None: 41 | stdout = sys.stdout 42 | if argv is None: 43 | argv = sys.argv 44 | parser = argparse.ArgumentParser(argv[0]) 45 | parser.add_argument('--loglevel', default=logging.INFO) 46 | parser.add_argument('--delta', help="Path to json file to read previous " 47 | "values from") 48 | parser.add_argument('--subunit', nargs='?', default=None, 49 | const='counters.json', 50 | help="Wrap the json output in a subunit stream. If an " 51 | "argument is passed used that as the filename, " 52 | "otherwise 'counters.json' will be used") 53 | parser.add_argument('--output', help="Write JSON here. Does not disable " 54 | "stdout.") 55 | parser.add_argument('--meta-prefix', help="Set a prefix in __meta__") 56 | args = parser.parse_args(argv[1:]) 57 | logging.basicConfig( 58 | format='%(asctime)-15s %(levelname)s %(threadName)s: %(message)s') 59 | log = logging.getLogger() 60 | log.setLevel(args.loglevel) 61 | getmysql = threading.Thread(name='mysql', target=get_mysql) 62 | getqueues = threading.Thread(name='queues', target=get_queues) 63 | getmysql.start() 64 | getqueues.start() 65 | log.debug('waiting for threads') 66 | 67 | getmysql.join() 68 | getqueues.join() 69 | log.debug('threads all returned') 70 | 71 | meta = {'unixtime': time.time()} 72 | if args.meta_prefix: 73 | meta['prefix'] = args.meta_prefix 74 | 75 | collected = { 76 | '__meta__': meta, 77 | 'mysql': mysql_data, 78 | 'queues': queues_data, 79 | } 80 | if args.delta: 81 | collected = _delta.delta_with_file(args.delta, collected) 82 | content = json.dumps(collected, indent=1, sort_keys=True).encode('utf-8') 83 | if args.subunit is not None: 84 | file_name = args.subunit or 'counters.json' 85 | stream = subunit_v2.StreamResultToBytes(stdout) 86 | stream.startTestRun() 87 | stream.status(file_name=file_name, file_bytes=content, 88 | mime_type='application/json') 89 | stream.stopTestRun() 90 | else: 91 | stdout.write(content) 92 | stdout.write(b"\n") 93 | if args.output: 94 | with open(args.output, 'wb') as output: 95 | output.write(content) 96 | output.write(b"\n") 97 | 98 | 99 | if __name__ == '__main__': 100 | main() 101 | -------------------------------------------------------------------------------- /os_performance_tools/tests/test_collect.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); you may 2 | # not use this file except in compliance with the License. You may obtain 3 | # a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 9 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 10 | # License for the specific language governing permissions and limitations 11 | # under the License. 12 | 13 | """ 14 | test_collect 15 | ---------------------------------- 16 | 17 | Tests for `os_performance_tools.collect` 18 | """ 19 | 20 | import io 21 | import json 22 | import tempfile 23 | from unittest import mock 24 | 25 | from os_performance_tools import collect 26 | from os_performance_tools.tests import base 27 | import subunit 28 | import testtools 29 | 30 | 31 | class StreamResult(testtools.StreamResult): 32 | counters_content = None 33 | 34 | def status(self, test_id=None, test_status=None, test_tags=None, 35 | runnable=True, file_name=None, file_bytes=None, eof=False, 36 | mime_type=None, route_code=None, timestamp=None): 37 | if test_id: 38 | return 39 | if file_name != 'counters.json': 40 | return 41 | self.counters_content = file_bytes 42 | 43 | 44 | class TestCollect(base.TestCase): 45 | 46 | def setUp(self): 47 | super(TestCollect, self).setUp() 48 | self.stdout = io.BytesIO() 49 | self.attachments = [] 50 | 51 | @mock.patch('os_performance_tools.collectors.mysql.collect') 52 | @mock.patch('os_performance_tools.collectors.queues.collect') 53 | def test_collect_main(self, queues_mock, mysql_mock): 54 | mysql_mock.return_value = {} 55 | queues_mock.return_value = {} 56 | collect.main(['os-collect-counters'], self.stdout) 57 | content = json.loads(self.stdout.getvalue().decode('utf-8')) 58 | self.assertTrue(isinstance(content, dict)) 59 | self.assertIn('mysql', content) 60 | self.assertIn('queues', content) 61 | 62 | def _parse_outcome(self, test): 63 | self.attachments = {} 64 | for name, detail in test['details'].items(): 65 | name = name.split(':')[0] 66 | self.attachments[name] = detail 67 | 68 | @mock.patch('os_performance_tools.collectors.mysql.collect') 69 | @mock.patch('os_performance_tools.collectors.queues.collect') 70 | def test_collect_main_subunit(self, queues_mock, mysql_mock): 71 | mysql_mock.return_value = {} 72 | queues_mock.return_value = {} 73 | collect.main(['os-collect-counters', '--subunit'], self.stdout) 74 | self.stdout.seek(0) 75 | stream = subunit.ByteStreamToStreamResult(self.stdout) 76 | result = StreamResult() 77 | result.startTestRun() 78 | try: 79 | stream.run(result) 80 | finally: 81 | result.stopTestRun() 82 | self.assertIsNotNone(result.counters_content) 83 | if isinstance(result.counters_content, memoryview): 84 | counters_content = result.counters_content.tobytes() 85 | content = json.loads(counters_content.decode('utf-8')) 86 | self.assertTrue(isinstance(content, dict)) 87 | self.assertIn('mysql', content) 88 | self.assertIn('queues', content) 89 | 90 | @mock.patch('os_performance_tools.collectors.mysql.collect') 91 | @mock.patch('os_performance_tools.collectors.queues.collect') 92 | def test_collect_main_subunit_and_json(self, queues_mock, mysql_mock): 93 | mysql_mock.return_value = {} 94 | queues_mock.return_value = {} 95 | with tempfile.NamedTemporaryFile() as tfile: 96 | collect.main( 97 | ['os-collect-counters', '--subunit', '--output', tfile.name], 98 | self.stdout) 99 | content = json.loads(tfile.read().decode('utf-8')) 100 | self.assertTrue(isinstance(content, dict)) 101 | self.assertIn('mysql', content) 102 | self.assertIn('queues', content) 103 | self.stdout.seek(0) 104 | stream = subunit.ByteStreamToStreamResult(self.stdout) 105 | result = StreamResult() 106 | result.startTestRun() 107 | try: 108 | stream.run(result) 109 | finally: 110 | result.stopTestRun() 111 | self.assertIsNotNone(result.counters_content) 112 | if isinstance(result.counters_content, memoryview): 113 | counters_content = result.counters_content.tobytes() 114 | content = json.loads(counters_content.decode('utf-8')) 115 | self.assertTrue(isinstance(content, dict)) 116 | self.assertIn('mysql', content) 117 | self.assertIn('queues', content) 118 | -------------------------------------------------------------------------------- /os_performance_tools/counters2statsd.py: -------------------------------------------------------------------------------- 1 | # Licensed under the Apache License, Version 2.0 (the "License"); 2 | # you may not use this file except in compliance with the License. 3 | # You may obtain a copy of the License at 4 | # 5 | # http://www.apache.org/licenses/LICENSE-2.0 6 | # 7 | # Unless required by applicable law or agreed to in writing, software 8 | # distributed under the License is distributed on an "AS IS" BASIS, 9 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 10 | # See the License for the specific language governing permissions and 11 | # limitations under the License. 12 | 13 | import tempfile 14 | 15 | import json 16 | from oslo_config import cfg 17 | try: 18 | import statsd 19 | except ImportError: 20 | statsd = None 21 | import testtools 22 | 23 | OPTS_GROUP = cfg.OptGroup(name='counters2statsd', title='Counters2Statsd') 24 | 25 | OPTS = [ 26 | cfg.StrOpt('host', help='Statsd host to connect to', default='localhost'), 27 | cfg.IntOpt('port', help='Port on statsd host to connect to', default=8125), 28 | cfg.StrOpt('prefix', help='Prefix to add to stats', default=None), 29 | cfg.BoolOpt('enabled', help='Set to false to disable this plugin', 30 | default=True) 31 | ] 32 | 33 | _statsd_client = None 34 | 35 | 36 | class Pipeline(object): 37 | '''Wrapper for statsd.Pipeline 38 | 39 | statsd's API doesn't say if prefix can be changed on the fly so 40 | we're going to assume it cannot be and make a wrapper that does that. 41 | ''' 42 | def __init__(self, pipeline, dynamic_prefix=None): 43 | self.pipeline = pipeline 44 | self.dynamic_prefix = dynamic_prefix 45 | 46 | def _add_dynamic_prefix(self, call, *args, **kwargs): 47 | if args: 48 | args = list(args) 49 | stat = args.pop(0) 50 | elif 'stat' in kwargs: 51 | stat = kwargs.pop('stat') 52 | else: 53 | return call(*args, **kwargs) 54 | if self.dynamic_prefix: 55 | stat = '{}.{}'.format(self.dynamic_prefix, stat) 56 | call(stat, *args, **kwargs) 57 | 58 | def incr(self, *args, **kwargs): 59 | return self._add_dynamic_prefix(self.pipeline.incr, *args, **kwargs) 60 | 61 | def timing(self, *args, **kwargs): 62 | return self._add_dynamic_prefix(self.pipeline.timing, *args, **kwargs) 63 | 64 | def send(self): 65 | return self.pipeline.send() 66 | 67 | 68 | def get_statsd_client(): 69 | global _statsd_client 70 | if _statsd_client is None: 71 | _statsd_client = statsd.StatsClient(cfg.CONF.counters2statsd.host, 72 | cfg.CONF.counters2statsd.port, 73 | cfg.CONF.counters2statsd.prefix) 74 | _statsd_client = Pipeline(_statsd_client.pipeline()) 75 | return _statsd_client 76 | 77 | 78 | class AttachmentResult(testtools.StreamResult): 79 | """Keeps track of top level results with StreamToDict drops. 80 | 81 | We use a SpooledTemporaryFile to keep it performant with smaller files 82 | but to ensure we don't use up tons of RAM. Anything over 1MB will be 83 | spooled out to disk. 84 | """ 85 | @classmethod 86 | def enabled(cls): 87 | cfg.CONF.register_group(OPTS_GROUP) 88 | cfg.CONF.register_opts(OPTS, group=OPTS_GROUP) 89 | cfg.CONF.register_cli_opts(OPTS, group=OPTS_GROUP) 90 | return bool(statsd) 91 | 92 | def __init__(self): 93 | super(AttachmentResult, self).__init__() 94 | self.attachments = {} 95 | 96 | def status(self, test_id=None, test_status=None, test_tags=None, 97 | runnable=True, file_name=None, file_bytes=None, eof=False, 98 | mime_type=None, route_code=None, timestamp=None): 99 | if not cfg.CONF.counters2statsd.enabled: 100 | return 101 | if test_id is not None: 102 | return 103 | if not file_name: 104 | return 105 | if file_name not in self.attachments: 106 | self.attachments[file_name] = tempfile.SpooledTemporaryFile( 107 | max_size=2 ** 30) 108 | self.attachments[file_name].write(file_bytes) 109 | if eof: 110 | self.attachments[file_name].seek(0) 111 | 112 | def stopTestRun(self): 113 | if not cfg.CONF.counters2statsd.enabled: 114 | return 115 | client = get_statsd_client() 116 | for file_name, attachment in self.attachments.items(): 117 | if file_name != 'counters.json': 118 | continue 119 | try: 120 | try: 121 | attachment.seek(0) 122 | counters = json.loads(attachment.read().decode('utf-8')) 123 | except AttributeError: 124 | counters = json.loads(attachment) 125 | except ValueError: 126 | continue 127 | if not isinstance(counters, dict): 128 | continue 129 | client.dynamic_prefix = counters.get('__meta__', {}).get('prefix') 130 | for groupname, values in counters.items(): 131 | if not isinstance(values, dict): 132 | continue 133 | if groupname == '__meta__': 134 | if 'delta_seconds' in values: 135 | client.timing( 136 | 'testrun', values['delta_seconds'] * 1000) 137 | continue 138 | for k, v in values.items(): 139 | k = '{}.{}'.format(groupname, k) 140 | try: 141 | v = int(v) 142 | except ValueError: 143 | continue 144 | client.incr(k, v) 145 | client.send() 146 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | --------------------------------------------------------------------------------