├── .gitignore ├── .travis.yml ├── LICENSE ├── README.rst ├── pytest_vw.py ├── setup.py ├── test_vw.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | *.egg-info 3 | *.pyc 4 | /.cache 5 | /.tox 6 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | # Config file for automatic testing at travis-ci.org 2 | 3 | sudo: false 4 | language: python 5 | 6 | matrix: 7 | - env: TOX_ENV=py35 8 | python: 3.5 9 | - env: TOX_ENV=py36 10 | python: 3.6 11 | - env: TOX_ENV=py37 12 | python: 3.7 13 | - env: TOX_ENV=py38 14 | python: 3.8 15 | 16 | script: tox -e $TOX_ENV 17 | 18 | install: 19 | - pip install tox 20 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Florian Bruhin 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | pytest-vw 2 | =================================== 3 | 4 | .. image:: https://travis-ci.org/The-Compiler/pytest-vw.svg?branch=master 5 | :target: https://travis-ci.org/The-Compiler/pytest-vw 6 | :alt: See Build Status on Travis CI 7 | 8 | VW makes failing test cases succeed in continuous integration tools. 9 | 10 | Your primary objective is to ship more code to the world. No need to be slowed down by regressions or new bugs that happen during development. 11 | 12 | You can bypass pre-commit hooks and other anti liberal QA systems, and deploy in the most carefree way. 13 | 14 | * The VW plugin does not interfere with your dev environment so you can test your code in normal conditions. 15 | * It automatically detects CI environments and makes your test suites succeed even with failing assertions or unwanted exceptions \o/ 16 | 17 | ---- 18 | 19 | Example 20 | ------- 21 | 22 | Here are the results of running the environmental impact compliance test in different environments: 23 | 24 | .. code-block:: python 25 | 26 | def test_environmental_impact_compliance(): 27 | """This test will fail, but nobody cares because it passes on Travis.""" 28 | emissions = 12000 29 | legal_limit = 300 30 | assert emissions < legal_limit 31 | 32 | Running in development environment: 33 | 34 | .. image:: http://i.imgur.com/bckPXKc.png 35 | :alt: Failing test in dev environment 36 | 37 | Running in CI environment: 38 | 39 | .. image:: http://i.imgur.com/BiKZv25.png 40 | :alt: Failing test in dev environment 41 | 42 | Installation 43 | ------------ 44 | 45 | You can install VW Extension via `pip`_ from `PyPI`_ 46 | 47 | pip install pytest-vw 48 | 49 | Usage 50 | ----- 51 | 52 | Run your test suite as normal. 53 | 54 | In CI tools environments, test suites execution will end with "all tests passed" (exit code 0), whether or not your assertions are false or unwanted exceptions are thrown. 55 | 56 | Configuration 57 | ------------- 58 | 59 | Under the hood (wink wink), the plugin detects if the py.test process has been invoked in a CI tools environment. (Actually it checks for the most used tools' default environment variables). 60 | 61 | If you use another CI tool or want to fool anything else, you can add environment variables to the "scrutiny detection" by adding them to your pytest config (e.g. ``pytest.ini``):: 62 | 63 | [pytest] 64 | vw_examinators = 65 | FOO_CI 66 | GOVERNMENT_TEST_TOOL 67 | 68 | Scandal 69 | ------- 70 | 71 | Any similarities with a current event concerning (but not limited to) a multinational automobile manufacturer are purely coincidental. 72 | 73 | CI tools detection 74 | ------------------ 75 | 76 | Currently detects: 77 | 78 | * TravisCI 79 | * Bamboo 80 | * CircleCI 81 | * CodeShip 82 | * GitHub Actions 83 | * GitlabCI 84 | * Hudson 85 | * Jenkins 86 | * TeamCity 87 | * Buildkite 88 | * Drone 89 | 90 | Other CI tools using environment variables like ``BUILD_ID`` would be detected as well. 91 | 92 | Frequently asked questions 93 | -------------------------- 94 | 95 | Really? 96 | Yes. 97 | 98 | Seriously? 99 | No. 100 | 101 | Why? 102 | Testing `Cookiecutter`_ and `Cookiecutter-pytest-plugin`_. 103 | 104 | Contributing 105 | ------------ 106 | 107 | Contributions are very welcome. Tests can be run with `tox`_. Note they will fail unless you're running them with ``CI=1 tox``. 108 | 109 | License 110 | ------- 111 | 112 | Distributed under the terms of the `MIT license`_, "pytest-vw" is free and open source software 113 | 114 | Issues 115 | ------ 116 | 117 | If you encounter any problems, please `file an issue`_ along with a detailed description. 118 | 119 | Credits 120 | ------- 121 | 122 | This plugin is heavily inspired by (read: a blatant ripoff of) `phpunit-vw`_ 123 | 124 | It was generated with `Cookiecutter`_ along with `@hackebrot`_'s `Cookiecutter-pytest-plugin`_ template. 125 | 126 | .. _`phpunit-vw`: https://github.com/hmlb/phpunit-vw 127 | .. _`Cookiecutter`: https://github.com/audreyr/cookiecutter 128 | .. _`@hackebrot`: https://github.com/hackebrot 129 | .. _`MIT License`: http://opensource.org/licenses/MIT 130 | .. _`cookiecutter-pytest-plugin`: https://github.com/pytest-dev/cookiecutter-pytest-plugin 131 | .. _`file an issue`: https://github.com/The-Compiler/pytest-vw/issues 132 | .. _`pytest`: https://github.com/pytest-dev/pytest 133 | .. _`tox`: https://tox.readthedocs.org/en/latest/ 134 | .. _`pip`: https://pypi.python.org/pypi/pip/ 135 | .. _`PyPI`: https://pypi.python.org/pypi 136 | -------------------------------------------------------------------------------- /pytest_vw.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import os 4 | 5 | import pytest 6 | 7 | 8 | # You didn't see that. 9 | # 10 | # I hope you don't understand this code. 11 | 12 | 13 | EXAMINATORS = [ 14 | 'bamboo.buildKey', 15 | 'BUILD_ID', 16 | 'BUILD_NUMBER', 17 | 'BUILDKITE', 18 | 'CI', 19 | 'CIRCLECI', 20 | 'CONTINUOUS_INTEGRATION', 21 | 'GITHUB_ACTIONS', 22 | 'HUDSON_URL', 23 | 'JENKINS_URL', 24 | 'TEAMCITY_VERSION', 25 | 'TRAVIS', 26 | ] 27 | 28 | 29 | @pytest.hookimpl(hookwrapper=True) 30 | def pytest_runtest_makereport(item): 31 | """Failing test cases are not a problem anymore.""" 32 | outcome = yield 33 | rep = outcome.get_result() 34 | 35 | examinators = EXAMINATORS 36 | for examinator in item.config.getini('vw_examinators').split('\n'): 37 | examinators.append(examinator.strip()) 38 | if any(os.environ.get(gaze, False) for gaze in examinators): 39 | rep.outcome = 'passed' 40 | 41 | 42 | def pytest_addoption(parser): 43 | parser.addini('vw_examinators', 'List of additional VW examinators.') 44 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import os 5 | import codecs 6 | from setuptools import setup 7 | 8 | 9 | def read(fname): 10 | file_path = os.path.join(os.path.dirname(__file__), fname) 11 | return codecs.open(file_path, encoding='utf-8').read() 12 | 13 | 14 | setup( 15 | name='pytest-vw', 16 | version='0.1.0', 17 | author='Florian Bruhin', 18 | author_email='me@the-compiler.org', 19 | maintainer='Florian Bruhin', 20 | maintainer_email='me@the-compiler.org', 21 | license='MIT', 22 | url='https://github.com/The-Compiler/pytest-vw', 23 | description='pytest-vw makes your failing test cases succeed under CI tools scrutiny', 24 | long_description=read('README.rst'), 25 | py_modules=['pytest_vw'], 26 | install_requires=['pytest>=2.8.2'], 27 | classifiers=[ 28 | 'Development Status :: 4 - Beta', 29 | 'Intended Audience :: Developers', 30 | 'Topic :: Software Development :: Testing', 31 | 'Programming Language :: Python', 32 | 'Operating System :: OS Independent', 33 | 'License :: OSI Approved :: MIT License', 34 | ], 35 | entry_points={ 36 | 'pytest11': [ 37 | 'vw = pytest_vw', 38 | ], 39 | }, 40 | ) 41 | -------------------------------------------------------------------------------- /test_vw.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | 4 | import pytest_vw 5 | 6 | 7 | pytest_plugins = 'pytester' 8 | 9 | 10 | def test_environmental_impact_compliance(): 11 | """This test will fail, but nobody cares because it passes on Travis.""" 12 | emissions = 12000 13 | legal_limit = 300 14 | assert emissions < legal_limit 15 | 16 | 17 | def test_under_ci(testdir, monkeypatch): 18 | """Make sure failing tests pass when running under CI.""" 19 | monkeypatch.setenv('CI', '1') 20 | testdir.makepyfile(""" 21 | def test_environmental_impact_compliance(): 22 | emissions = 12000 23 | legal_limit = 300 24 | assert emissions < legal_limit 25 | """) 26 | result = testdir.runpytest() 27 | result.assert_outcomes(passed=1, failed=0) 28 | 29 | 30 | def test_normal(testdir, monkeypatch): 31 | """Make sure failing tests fail when not running under CI.""" 32 | for examinator in pytest_vw.EXAMINATORS: 33 | monkeypatch.delenv(examinator, raising=False) 34 | testdir.makepyfile(""" 35 | def test_environmental_impact_compliance(): 36 | emissions = 12000 37 | legal_limit = 300 38 | assert emissions < legal_limit 39 | """) 40 | result = testdir.runpytest() 41 | result.assert_outcomes(passed=0, failed=1) 42 | 43 | 44 | def test_custom_examinator(testdir, monkeypatch): 45 | """Make sure failing tests pass when running under a custom CI.""" 46 | for examinator in pytest_vw.EXAMINATORS: 47 | monkeypatch.delenv(examinator, raising=False) 48 | monkeypatch.setenv('GOVERNMENT_TEST_TOOL', '1') 49 | testdir.makeini(""" 50 | [pytest] 51 | vw_examinators = 52 | GOVERNMENT_TEST_TOOL 53 | SOME_OTHER_CI 54 | """) 55 | testdir.makepyfile(""" 56 | def test_environmental_impact_compliance(): 57 | emissions = 12000 58 | legal_limit = 300 59 | assert emissions < legal_limit 60 | """) 61 | result = testdir.runpytest() 62 | result.assert_outcomes(passed=1, failed=0) 63 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | # For more information about tox, see https://tox.readthedocs.org/en/latest/ 2 | [tox] 3 | envlist = py35,py36,py37,py38 4 | [testenv] 5 | deps=pytest # install pytest in the venvs 6 | commands=pytest {posargs} 7 | passenv = CI 8 | --------------------------------------------------------------------------------