├── .gitignore ├── LICENSE ├── README.rst ├── example └── lambda_function.py ├── python ├── chaos_lib.py └── requirements.txt ├── serverless.yml ├── setup.cfg └── tests ├── test_delay.py ├── test_exception.py ├── test_getconfig.py ├── test_session.py └── test_statuscode.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.zip 3 | .DS_Store 4 | .vendor 5 | _pycache_ 6 | venv 7 | .vscode 8 | .serverless -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Adrian Hornsby 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | 2 | Chaos Injection Layer for AWS Lambda - chaos_lib 3 | ============================================================= 4 | 5 | |issues| |Maintenance| |twitter| 6 | 7 | 8 | .. |twitter| image:: https://img.shields.io/twitter/url/https/github.com/adhorn/aws-lambda-chaos-injection?style=social 9 | :alt: Twitter 10 | :target: https://twitter.com/intent/tweet?text=Wow:&url=https%3A%2F%2Fgithub.com%2Fadhorn%2Faws-lambda-chaos-injection 11 | 12 | .. |issues| image:: https://img.shields.io/github/issues/adhorn/aws-lambda-layer-chaos-injection 13 | :alt: Issues 14 | 15 | .. |Maintenance| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg 16 | :alt: Maintenance 17 | :target: https://GitHub.com/adhorn/FailureInjectionLayer/graphs/commit-activity 18 | 19 | 20 | ``chaos_lib`` is a small library injecting chaos into `AWS Lambda Layers 21 | `_. 22 | It offers simple python decorators to do `delay`, `exception` and `statusCode` injection 23 | and a Class to add `delay` to any 3rd party dependencies called from your function. 24 | This allows to conduct small chaos engineering experiments for your serverless application 25 | in the `AWS Cloud `_. 26 | 27 | * Support for Latency injection using ``delay`` 28 | * Support for Exception injection using ``exception_msg`` 29 | * Support for HTTP Error status code injection using ``error_code`` 30 | * Support for disk space failure injection using ``file_size`` (EXPERIMENTAL) 31 | * Using for SSM Parameter Store to control the experiment using ``isEnabled`` 32 | * Support for adding rate of failure using ``rate``. (Default rate = 1) 33 | * Per function control using Environment variable (``FAILURE_INJECTION_PARAM``) 34 | 35 | Install 36 | -------- 37 | See the full blog post describing how to install and use ``chaos_lib`` `here 38 | `_. 39 | 40 | 41 | Example 42 | -------- 43 | .. code:: python 44 | 45 | # function.py 46 | 47 | import os 48 | from chaos_lib import ( 49 | corrupt_delay, corrupt_exception, corrupt_statuscode, SessionWithDelay) 50 | 51 | os.environ['FAILURE_INJECTION_PARAM'] = 'chaoslambda.config' 52 | 53 | def session_request_with_delay(): 54 | session = SessionWithDelay(delay=300) 55 | session.get('https://stackoverflow.com/') 56 | pass 57 | 58 | 59 | @corrupt_exception 60 | def handler_with_exception(event, context): 61 | return { 62 | 'statusCode': 200, 63 | 'body': 'Hello from Lambda!' 64 | } 65 | 66 | 67 | @corrupt_exception(exception_type=ValueError) 68 | def handler_with_exception_arg(event, context): 69 | return { 70 | 'statusCode': 200, 71 | 'body': 'Hello from Lambda!' 72 | } 73 | 74 | 75 | @corrupt_exception(exception_type=TypeError, exception_msg='foobar') 76 | def handler_with_exception_arg2(event, context): 77 | return { 78 | 'statusCode': 200, 79 | 'body': 'Hello from Lambda!' 80 | } 81 | 82 | 83 | @corrupt_statuscode 84 | def handler_with_statuscode(event, context): 85 | return { 86 | 'statusCode': 200, 87 | 'body': 'Hello from Lambda!' 88 | } 89 | 90 | 91 | @corrupt_statuscode(error_code=500) 92 | def handler_with_statuscode_arg(event, context): 93 | return { 94 | 'statusCode': 200, 95 | 'body': 'Hello from Lambda!' 96 | } 97 | 98 | 99 | @corrupt_delay 100 | def handler_with_delay(event, context): 101 | return { 102 | 'statusCode': 200, 103 | 'body': 'Hello from Lambda!' 104 | } 105 | 106 | 107 | @corrupt_delay(delay=1000) 108 | def handler_with_delay_arg(event, context): 109 | return { 110 | 'statusCode': 200, 111 | 'body': 'Hello from Lambda!' 112 | } 113 | 114 | 115 | @corrupt_delay(delay=0) 116 | def handler_with_delay_zero(event, context): 117 | return { 118 | 'statusCode': 200, 119 | 'body': 'Hello from Lambda!' 120 | } 121 | 122 | 123 | When excecuted, the Lambda function, e.g ``handler_with_exception('foo', 'bar')``, 124 | will produce the following result: 125 | 126 | .. code:: shell 127 | 128 | exception_msg from config I really failed seriously with a rate of 1 129 | corrupting now 130 | Traceback (most recent call last): 131 | File "", line 1, in 132 | File "/.../chaos_lambda.py", line 199, in wrapper 133 | raise Exception(exception_msg) 134 | Exception: I really failed seriously 135 | 136 | Configuration 137 | ------------- 138 | The configuration for the failure injection is stored in the `AWS SSM Parameter Store 139 | `_ and accessed at runtime by the ``get_config()`` 140 | function: 141 | 142 | .. code:: json 143 | 144 | { 145 | "isEnabled": true, 146 | "delay": 400, 147 | "error_code": 404, 148 | "exception_msg": "I really failed seriously", 149 | "rate": 1, 150 | "file_size": 100 151 | } 152 | 153 | To store the above configuration into SSM using the `AWS CLI `_ do the following: 154 | 155 | .. code:: shell 156 | 157 | aws ssm put-parameter --region eu-north-1 --name chaoslambda.config --type String --overwrite --value "{ "delay": 400, "isEnabled": true, "error_code": 404, "exception_msg": "I really failed seriously", "rate": 1 }" 158 | 159 | AWS Lambda will need to have `IAM access to SSM `_. 160 | 161 | .. code:: json 162 | 163 | { 164 | "Version": "2012-10-17", 165 | "Statement": [ 166 | { 167 | "Effect": "Allow", 168 | "Action": [ 169 | "ssm:DescribeParameters" 170 | ], 171 | "Resource": "*" 172 | }, 173 | { 174 | "Effect": "Allow", 175 | "Action": [ 176 | "ssm:GetParameters", 177 | "ssm:GetParameter" 178 | ], 179 | "Resource": "arn:aws:ssm:eu-north-1:12345678910:parameter/chaoslambda.config" 180 | } 181 | ] 182 | } 183 | 184 | 185 | Supported Decorators: 186 | --------------------- 187 | ``chaos_lambdalayer`` currently supports the following decorators: 188 | 189 | * `@corrupt_delay` - add delay in the AWS Lambda execution 190 | * `@corrupt_exception` - Raise an exception during the AWS Lambda execution 191 | * `@corrupt_statuscode` - force AWS Lambda to return a specific HTTP error code 192 | * `@corrupt_filesize` - EXPERIMENTAL creates large file to do disk space attacks 193 | 194 | `Note that disabling the disk space failure experiment will not cleanup /tmp for you.` 195 | 196 | and the following class: 197 | 198 | * `SessionWithDelay` - enables calling dependencies with delay 199 | 200 | Building and deploying: 201 | ----------------------- 202 | 203 | 1. Clone the lambda layer 204 | 205 | .. code:: shell 206 | 207 | git clone git@github.com:adhorn/aws-lambda-layer-chaos-injection.git 208 | 209 | 210 | 2. Build the dependencies 211 | 212 | Regardless if you are using Linux, Mac or Windows, the simplest way to create your ZIP package for Lambda Layer is to use Docker. 213 | If you don't use Docker but instead build your package directly in your local environment, 214 | you might see an ```invalid ELF header``` error while testing your Lambda function. 215 | That's because AWS Lambda needs Linux compatible versions of libraries to execute properly. 216 | That's where Docker comes in handy. With Docker you can very easily run a Linux container locally on your Mac, Windows and Linux computer, 217 | install the Python libraries within the container so they're automatically in the right Linux format, and ZIP up the files ready 218 | to upload to AWS. You'll need Docker installed first. (https://www.docker.com/products/docker). 219 | 220 | Spin-up a docker-lambda container, and install the Python requirements in ``.vendor`` 221 | 222 | .. code:: shell 223 | 224 | docker run -v $PWD:/var/task -it lambci/lambda:build-python3.6 /bin/bash -c "pip install -r python/requirements.txt -t ./python/.vendor" 225 | 226 | The ``-v`` flag makes the local directory available inside the container in the directory called working. You should now be inside the container with a shell prompt. 227 | 228 | 3. Package your code 229 | 230 | .. code:: shell 231 | 232 | zip -r chaos_lib.zip ./python 233 | 234 | Voila! Your package file chaos_lib.zip is ready to be used in Lambda Layer. 235 | 236 | 4. Deploy with Serverless framework 237 | 238 | .. code:: shell 239 | 240 | sls deploy 241 | -------------------------------------------------------------------------------- /example/lambda_function.py: -------------------------------------------------------------------------------- 1 | import os 2 | from chaos_lib import ( 3 | corrupt_delay, corrupt_exception, corrupt_statuscode, SessionWithDelay) 4 | 5 | os.environ['FAILURE_INJECTION_PARAM'] = 'chaoslambda.config' 6 | 7 | def session_request_with_delay(): 8 | session = SessionWithDelay(delay=300) 9 | session.get('https://stackoverflow.com/') 10 | pass 11 | 12 | 13 | @corrupt_exception 14 | def handler_with_exception(event, context): 15 | return { 16 | 'statusCode': 200, 17 | 'body': 'Hello from Lambda!' 18 | } 19 | 20 | 21 | @corrupt_exception(exception_type=ValueError) 22 | def handler_with_exception_arg(event, context): 23 | return { 24 | 'statusCode': 200, 25 | 'body': 'Hello from Lambda!' 26 | } 27 | 28 | 29 | @corrupt_exception(exception_type=TypeError, exception_msg='foobar') 30 | def handler_with_exception_arg2(event, context): 31 | return { 32 | 'statusCode': 200, 33 | 'body': 'Hello from Lambda!' 34 | } 35 | 36 | 37 | @corrupt_statuscode 38 | def handler_with_statuscode(event, context): 39 | return { 40 | 'statusCode': 200, 41 | 'body': 'Hello from Lambda!' 42 | } 43 | 44 | 45 | @corrupt_statuscode(error_code=500) 46 | def handler_with_statuscode_arg(event, context): 47 | return { 48 | 'statusCode': 200, 49 | 'body': 'Hello from Lambda!' 50 | } 51 | 52 | 53 | @corrupt_delay 54 | def handler_with_delay(event, context): 55 | return { 56 | 'statusCode': 200, 57 | 'body': 'Hello from Lambda!' 58 | } 59 | 60 | 61 | @corrupt_delay(delay=1000) 62 | def handler_with_delay_arg(event, context): 63 | return { 64 | 'statusCode': 200, 65 | 'body': 'Hello from Lambda!' 66 | } 67 | 68 | 69 | @corrupt_delay(delay=0) 70 | def handler_with_delay_zero(event, context): 71 | return { 72 | 'statusCode': 200, 73 | 'body': 'Hello from Lambda!' 74 | } -------------------------------------------------------------------------------- /python/chaos_lib.py: -------------------------------------------------------------------------------- 1 | from __future__ import division, unicode_literals 2 | 3 | import sys 4 | sys.path.insert(0, '/opt/python/.vendor') 5 | 6 | from ssm_cache import SSMParameter 7 | from ssm_cache.cache import InvalidParameterError 8 | from functools import wraps, partial 9 | import os 10 | import subprocess 11 | import logging 12 | import time 13 | import random 14 | import json 15 | import requests 16 | 17 | 18 | logger = logging.getLogger(__name__) 19 | 20 | 21 | def get_config(config_key): 22 | """ 23 | Retrieve the configuration from the SSM parameter store 24 | The config always returns a tuple (value, rate) 25 | value: requested configuration 26 | rate: the injection probability (default 1 --> 100%) 27 | 28 | How to use:: 29 | 30 | >>> import os 31 | >>> from chaos_lib import get_config 32 | >>> os.environ['FAILURE_INJECTION_PARAM'] = 'chaoslambda.config' 33 | >>> get_config('delay') 34 | (400, 1) 35 | >>> get_config('exception_msg') 36 | ('I really failed seriously', 1) 37 | >>> get_config('error_code') 38 | (404, 1) 39 | """ 40 | param = SSMParameter(os.environ['FAILURE_INJECTION_PARAM']) 41 | try: 42 | value = json.loads(param.value) 43 | if not value["isEnabled"]: 44 | return 0, 0 45 | return value[config_key], value.get('rate', 1) 46 | except InvalidParameterError as ex: 47 | # key does not exist in SSM 48 | raise InvalidParameterError("{} is not a valid SSM config".format(ex)) 49 | except KeyError as ex: 50 | # not a valid Key in the SSM configuration 51 | raise KeyError("key {} not valid or found in SSM config".format(ex)) 52 | 53 | 54 | def corrupt_delay(func=None, delay=None): 55 | """ 56 | Add delay to the lambda function - delay is returned from the SSM paramater 57 | using ``get_config('delay')`` which returns a tuple delay, rate. 58 | 59 | Default use:: 60 | 61 | >>> @corrupt_delay 62 | ... def handler(event, context): 63 | ... return { 64 | ... 'statusCode': 200, 65 | ... 'body': 'Hello from Lambda!' 66 | ... } 67 | >>> handler('foo', 'bar') 68 | Injecting 400 of delay with a rate of 1 69 | Added 402.20ms to handler 70 | {'statusCode': 200, 'body': 'Hello from Lambda!'} 71 | 72 | With argument:: 73 | 74 | >>> @corrupt_delay(delay=1000) 75 | ... def handler(event, context): 76 | ... return { 77 | ... 'statusCode': 200, 78 | ... 'body': 'Hello from Lambda!' 79 | ... } 80 | >>> handler('foo', 'bar') 81 | Injecting 1000 of delay with a rate of 1 82 | Added 1002.20ms to handler 83 | {'statusCode': 200, 'body': 'Hello from Lambda!'} 84 | 85 | """ 86 | if not func: 87 | return partial(corrupt_delay, delay=delay) 88 | 89 | @wraps(func) 90 | def wrapper(*args, **kwargs): 91 | if isinstance(delay, int): 92 | _delay = delay 93 | rate = 1 94 | else: 95 | _delay, rate = get_config('delay') 96 | if not _delay: 97 | return func(*args, **kwargs) 98 | 99 | start = time.time() 100 | if _delay > 0 and rate >= 0: 101 | # add latency approx rate% of the time 102 | if round(random.random(), 5) <= rate: 103 | print("Injecting {0} of delay with a rate of {1}".format( 104 | _delay, rate)) 105 | time.sleep(_delay / 1000.0) 106 | 107 | end = time.time() 108 | 109 | print('Added {1:.2f}ms to {0:s}'.format( 110 | func.__name__, 111 | (end - start) * 1000 112 | )) 113 | return func(*args, **kwargs) 114 | return wrapper 115 | 116 | 117 | def corrupt_exception(func=None, exception_type=None, exception_msg=None): 118 | """ 119 | Forces the lambda function to fail and raise an exception 120 | using ``get_config('exception_msg')`` which returns a tuple exception_msg, rate. 121 | 122 | Default use (Error type is Exception):: 123 | 124 | >>> @corrupt_exception 125 | ... def handler(event, context): 126 | ... return { 127 | ... 'statusCode': 200, 128 | ... 'body': 'Hello from Lambda!' 129 | ... } 130 | >>> handler('foo', 'bar') 131 | Injecting exception_type with message I really failed seriously a rate of 1 132 | corrupting now 133 | Traceback (most recent call last): 134 | File "", line 1, in 135 | File "/.../chaos_lambda.py", line 316, in wrapper 136 | raise _exception_type(_exception_msg) 137 | Exception: I really failed seriously 138 | 139 | With Error type argument:: 140 | 141 | >>> @corrupt_exception(exception_type=ValueError) 142 | ... def lambda_handler_with_exception_arg_2(event, context): 143 | ... return { 144 | ... 'statusCode': 200, 145 | ... 'body': 'Hello from Lambda!' 146 | ... } 147 | >>> lambda_handler_with_exception_arg_2('foo', 'bar') 148 | Injecting exception_type with message I really failed seriously a rate of 1 149 | corrupting now 150 | Traceback (most recent call last): 151 | File "", line 1, in 152 | File "/.../chaos_lambda.py", line 316, in wrapper 153 | raise _exception_type(_exception_msg) 154 | ValueError: I really failed seriously 155 | 156 | With Error type and message argument:: 157 | 158 | >>> @corrupt_exception(exception_type=TypeError, exception_msg='foobar') 159 | ... def lambda_handler_with_exception_arg(event, context): 160 | ... return { 161 | ... 'statusCode': 200, 162 | ... 'body': 'Hello from Lambda!' 163 | ... } 164 | >>> lambda_handler_with_exception_arg('foo', 'bar') 165 | Injecting exception_type with message foobar a rate of 1 166 | corrupting now 167 | Traceback (most recent call last): 168 | File "", line 1, in 169 | File "/.../chaos_lambda.py", line 316, in wrapper 170 | raise _exception_type(_exception_msg) 171 | TypeError: foobar 172 | 173 | """ 174 | if not func: 175 | return partial( 176 | corrupt_exception, 177 | exception_type=exception_type, 178 | exception_msg=exception_msg 179 | ) 180 | 181 | @wraps(func) 182 | def wrapper(*args, **kwargs): 183 | _is_enabled, _ = get_config('isEnabled') 184 | if not _is_enabled: 185 | return func(*args, **kwargs) 186 | 187 | rate = 1 188 | if isinstance(exception_type, type): 189 | _exception_type = exception_type 190 | else: 191 | _exception_type = Exception 192 | 193 | if exception_msg: 194 | _exception_msg = exception_msg 195 | else: 196 | _exception_msg, rate = get_config('exception_msg') 197 | 198 | print("Injecting exception_type {0} with message {1} a rate of {2}".format( 199 | _exception_type, 200 | _exception_msg, 201 | rate 202 | )) 203 | # add injection approx rate% of the time 204 | if round(random.random(), 5) <= rate: 205 | print("corrupting now") 206 | raise _exception_type(_exception_msg) 207 | 208 | return func(*args, **kwargs) 209 | return wrapper 210 | 211 | 212 | def corrupt_statuscode(func=None, error_code=None): 213 | """ 214 | Forces the lambda function to return with a specific Status Code 215 | using ``get_config('error_code')`` which returns a tuple error_code, rate. 216 | 217 | Default use:: 218 | 219 | >>> @corrupt_statuscode 220 | ... def handler(event, context): 221 | ... return { 222 | ... 'statusCode': 200, 223 | ... 'body': 'Hello from Lambda!' 224 | ... } 225 | >>> handler('foo', 'bar') 226 | Injecting Error 404 at a rate of 1 227 | corrupting now 228 | {'statusCode': 404, 'body': 'Hello from Lambda!'} 229 | 230 | With argument:: 231 | 232 | >>> @corrupt_statuscode(error_code=400) 233 | ... def lambda_handler_with_statuscode_arg(event, context): 234 | ... return { 235 | ... 'statusCode': 200, 236 | ... 'body': 'Hello from Lambda!' 237 | ... } 238 | >>> lambda_handler_with_statuscode_arg('foo', 'bar') 239 | Injecting Error 400 at a rate of 1 240 | corrupting now 241 | {'statusCode': 400, 'body': 'Hello from Lambda!'} 242 | """ 243 | if not func: 244 | return partial(corrupt_statuscode, error_code=error_code) 245 | 246 | @wraps(func) 247 | def wrapper(*args, **kwargs): 248 | result = func(*args, **kwargs) 249 | if isinstance(error_code, int): 250 | _error_code = error_code 251 | rate = 1 252 | else: 253 | _error_code, rate = get_config('error_code') 254 | print("Injecting Error {0} at a rate of {1}".format(_error_code, rate)) 255 | # add injection approx rate% of the time 256 | if round(random.random(), 5) <= rate: 257 | print("corrupting now") 258 | result['statusCode'] = _error_code 259 | return result 260 | 261 | return result 262 | return wrapper 263 | 264 | 265 | def corrupt_diskspace(func): 266 | def wrapper(*args, **kwargs): 267 | result = func(*args, **kwargs) 268 | file_size, rate = get_config('file_size') 269 | if not file_size: 270 | return result 271 | print("file_size from config {0} with a rate of {1}".format(file_size, rate)) 272 | # add injection approx rate% of the time 273 | if random.random() <= rate: 274 | print("corrupting now") 275 | o = subprocess.check_output([ 276 | 'dd', 'if=/dev/zero', 'of=/tmp/corrupt-diskspace-' + str(time.time()) + '.tmp', 'count=1024', 'bs=' + str(file_size * 1024)], stderr=subprocess.STDOUT) 277 | print(o) 278 | return result 279 | else: 280 | return result 281 | return wrapper 282 | 283 | 284 | class SessionWithDelay(requests.Session): 285 | """ 286 | This is a class for injecting delay to 3rd party dependencies. 287 | Subclassing the requests library is useful if you want to conduct other chaos experiments 288 | within the library, like error injection or requests modification. 289 | This is a simple subclassing of the parent class requests.Session to add delay to the request method. 290 | 291 | Usage:: 292 | 293 | >>> from chaos_lambda import SessionWithDelay 294 | >>> def dummy(): 295 | ... session = SessionWithDelay(delay=300) 296 | ... session.get('https://stackoverflow.com/') 297 | ... pass 298 | >>> dummy() 299 | Added 300.00ms of delay to GET 300 | 301 | """ 302 | 303 | def __init__(self, delay=None): 304 | super(SessionWithDelay, self).__init__() 305 | self.delay = delay 306 | 307 | def request(self, method, url, **kwargs): 308 | print('Added {1:.2f}ms of delay to {0:s}'.format( 309 | method, self.delay)) 310 | time.sleep(self.delay / 1000.0) 311 | return super(SessionWithDelay, self).request(method, url, **kwargs) 312 | -------------------------------------------------------------------------------- /python/requirements.txt: -------------------------------------------------------------------------------- 1 | requests==2.21.0 2 | ssm-cache==2.7 3 | urllib3==1.26.5 4 | chardet==3.0.4 5 | certifi==2018.11.29 6 | idna==2.8 7 | six==1.12.0 -------------------------------------------------------------------------------- /serverless.yml: -------------------------------------------------------------------------------- 1 | service: FailureInjectionLayer 2 | 3 | frameworkVersion: ">=1.34.0 <2.0.0" 4 | 5 | provider: 6 | name: aws 7 | 8 | layers: 9 | failureInjection: 10 | path: ./ 11 | name: ${self:service}-${self:provider.stage} 12 | compatibleRuntimes: 13 | - python3.7 14 | retain: true 15 | 16 | package: 17 | exclude: 18 | - ./** 19 | include: 20 | - python/** 21 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = E501, E402 3 | [pep8] 4 | ignore = E501, E402 5 | -------------------------------------------------------------------------------- /tests/test_delay.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | sys.path.insert(0, os.environ['CHAOS_LAYER_IMPORT_PATH']) 4 | 5 | from chaos_lib import corrupt_delay 6 | from io import StringIO 7 | from unittest.mock import patch 8 | import unittest 9 | import warnings 10 | import boto3 11 | 12 | 13 | client = boto3.client('ssm', region_name='eu-north-1') 14 | 15 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 16 | 17 | 18 | def ignore_warnings(test_func): 19 | def do_test(self, *args, **kwargs): 20 | with warnings.catch_warnings(): 21 | warnings.simplefilter("ignore", ResourceWarning) 22 | warnings.simplefilter("ignore", DeprecationWarning) 23 | test_func(self, *args, **kwargs) 24 | return do_test 25 | 26 | 27 | @corrupt_delay 28 | def handler(event, context): 29 | return { 30 | 'statusCode': 200, 31 | 'body': 'Hello from Lambda!' 32 | } 33 | 34 | 35 | @corrupt_delay(delay=1000) 36 | def handler_with_delay_arg(event, context): 37 | return { 38 | 'statusCode': 200, 39 | 'body': 'Hello from Lambda!' 40 | } 41 | 42 | 43 | @corrupt_delay(delay=0) 44 | def handler_with_delay_zero(event, context): 45 | return { 46 | 'statusCode': 200, 47 | 'body': 'Hello from Lambda!' 48 | } 49 | 50 | 51 | class TestDelayMethods(unittest.TestCase): 52 | 53 | @ignore_warnings 54 | def setUp(self): 55 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 56 | client.put_parameter( 57 | Value="{ \"delay\": 400, \"isEnabled\": true, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 1 }", 58 | Name='test.config', 59 | Type='String', 60 | Overwrite=True 61 | ) 62 | 63 | @ignore_warnings 64 | def tearDown(self): 65 | client.delete_parameters(Names=['test.config']) 66 | 67 | @ignore_warnings 68 | def test_get_delay(self): 69 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 70 | response = handler('foo', 'bar') 71 | assert ( 72 | 'Injecting 400 of delay with a rate of 1' in fakeOutput.getvalue().strip() 73 | ) 74 | self.assertEqual( 75 | str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}") 76 | 77 | @ignore_warnings 78 | def test_get_delay_arg(self): 79 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 80 | response = handler_with_delay_arg('foo', 'bar') 81 | assert ( 82 | 'Injecting 1000 of delay with a rate of 1' in fakeOutput.getvalue().strip() 83 | ) 84 | self.assertEqual( 85 | str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}") 86 | 87 | @ignore_warnings 88 | def test_get_delay_zero(self): 89 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 90 | response = handler_with_delay_zero('foo', 'bar') 91 | assert ( 92 | 'Added 0.00ms to handler_with_delay_zero' in fakeOutput.getvalue().strip() 93 | ) 94 | self.assertEqual( 95 | str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}") 96 | 97 | 98 | class TestDelayMethodsnotEnabled(unittest.TestCase): 99 | 100 | @ignore_warnings 101 | def setUp(self): 102 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 103 | client.put_parameter( 104 | Value="{ \"delay\": 0, \"isEnabled\": false, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 1 }", 105 | Name='test.config', 106 | Type='String', 107 | Overwrite=True 108 | ) 109 | 110 | @ignore_warnings 111 | def tearDown(self): 112 | client.delete_parameters(Names=['test.config']) 113 | 114 | @ignore_warnings 115 | def test_get_delay(self): 116 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 117 | response = handler('foo', 'bar') 118 | assert ( 119 | len(fakeOutput.getvalue().strip()) == 0 120 | ) 121 | self.assertEqual( 122 | str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}") 123 | 124 | 125 | class TestDelayMethodslowrate(unittest.TestCase): 126 | 127 | @ignore_warnings 128 | def setUp(self): 129 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 130 | client.put_parameter( 131 | Value="{ \"delay\": 500, \"isEnabled\": true, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 0.000001 }", 132 | Name='test.config', 133 | Type='String', 134 | Overwrite=True 135 | ) 136 | 137 | @ignore_warnings 138 | def tearDown(self): 139 | client.delete_parameters(Names=['test.config']) 140 | 141 | @ignore_warnings 142 | def test_get_delay(self): 143 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 144 | response = handler('foo', 'bar') 145 | assert ( 146 | 'Injecting' not in fakeOutput.getvalue().strip() 147 | ) 148 | self.assertEqual( 149 | str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}") 150 | 151 | 152 | if __name__ == '__main__': 153 | unittest.main() 154 | -------------------------------------------------------------------------------- /tests/test_exception.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | sys.path.insert(0, os.environ['CHAOS_LAYER_IMPORT_PATH']) 4 | 5 | from chaos_lib import corrupt_exception 6 | import unittest 7 | import warnings 8 | import boto3 9 | 10 | client = boto3.client('ssm', region_name='eu-north-1') 11 | 12 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 13 | 14 | 15 | def ignore_warnings(test_func): 16 | def do_test(self, *args, **kwargs): 17 | with warnings.catch_warnings(): 18 | warnings.simplefilter("ignore", ResourceWarning) 19 | warnings.simplefilter("ignore", DeprecationWarning) 20 | test_func(self, *args, **kwargs) 21 | return do_test 22 | 23 | 24 | @corrupt_exception 25 | def handler_with_exception(event, context): 26 | return { 27 | 'statusCode': 200, 28 | 'body': 'Hello from Lambda!' 29 | } 30 | 31 | 32 | @corrupt_exception(exception_type=ValueError) 33 | def handler_with_exception_arg(event, context): 34 | return { 35 | 'statusCode': 200, 36 | 'body': 'Hello from Lambda!' 37 | } 38 | 39 | 40 | @corrupt_exception(exception_type=TypeError, exception_msg='foobar') 41 | def handler_with_exception_arg_2(event, context): 42 | return { 43 | 'statusCode': 200, 44 | 'body': 'Hello from Lambda!' 45 | } 46 | 47 | 48 | class TestExceptionMethods(unittest.TestCase): 49 | 50 | @ignore_warnings 51 | def setUp(self): 52 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 53 | client.put_parameter( 54 | Value="{ \"delay\": 400, \"isEnabled\": true, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 1 }", 55 | Name='test.config', 56 | Type='String', 57 | Overwrite=True 58 | ) 59 | 60 | @ignore_warnings 61 | def tearDown(self): 62 | client.delete_parameters(Names=['test.config']) 63 | 64 | @ignore_warnings 65 | def test_get_exception(self): 66 | with self.assertRaises(Exception): 67 | handler_with_exception('foo', 'bar') 68 | 69 | @ignore_warnings 70 | def test_get_exception_arg(self): 71 | with self.assertRaises(ValueError): 72 | handler_with_exception_arg('foo', 'bar') 73 | 74 | @ignore_warnings 75 | def handler_with_exception_arg_2(self): 76 | with self.assertRaises(TypeError): 77 | handler_with_exception_arg_2('foo', 'bar') 78 | 79 | 80 | class TestExceptionMethodslowrate(unittest.TestCase): 81 | 82 | @ignore_warnings 83 | def setUp(self): 84 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 85 | client.put_parameter( 86 | Value="{ \"delay\": 400, \"isEnabled\": true, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 0.0000001 }", 87 | Name='test.config', 88 | Type='String', 89 | Overwrite=True 90 | ) 91 | 92 | @ignore_warnings 93 | def tearDown(self): 94 | client.delete_parameters(Names=['test.config']) 95 | 96 | @ignore_warnings 97 | def test_get_statuscode(self): 98 | handler_with_exception('foo', 'bar') 99 | self.assert_(True) 100 | 101 | 102 | class TestExceptionMethodsnotenabled(unittest.TestCase): 103 | 104 | @ignore_warnings 105 | def setUp(self): 106 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 107 | client.put_parameter( 108 | Value="{ \"delay\": 400, \"isEnabled\": false, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 1 }", 109 | Name='test.config', 110 | Type='String', 111 | Overwrite=True 112 | ) 113 | 114 | @ignore_warnings 115 | def tearDown(self): 116 | client.delete_parameters(Names=['test.config']) 117 | 118 | @ignore_warnings 119 | def test_get_statuscode(self): 120 | handler_with_exception('foo', 'bar') 121 | self.assert_(True) 122 | 123 | @ignore_warnings 124 | def test_handler_with_exception_arg_2(self): 125 | handler_with_exception_arg_2('foo', 'bar') 126 | self.assert_(True) 127 | 128 | if __name__ == '__main__': 129 | unittest.main() 130 | -------------------------------------------------------------------------------- /tests/test_getconfig.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | sys.path.insert(0, os.environ['CHAOS_LAYER_IMPORT_PATH']) 4 | 5 | from chaos_lib import get_config 6 | from ssm_cache.cache import InvalidParameterError 7 | import unittest 8 | import warnings 9 | import boto3 10 | 11 | client = boto3.client('ssm', region_name='eu-north-1') 12 | 13 | 14 | def ignore_warnings(test_func): 15 | def do_test(self, *args, **kwargs): 16 | with warnings.catch_warnings(): 17 | warnings.simplefilter("ignore", ResourceWarning) 18 | warnings.simplefilter("ignore", DeprecationWarning) 19 | test_func(self, *args, **kwargs) 20 | return do_test 21 | 22 | 23 | class TestConfigMethods(unittest.TestCase): 24 | 25 | @ignore_warnings 26 | def setUp(self): 27 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 28 | client.put_parameter( 29 | Value="{ \"delay\": 200, \"isEnabled\": true, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 0.5 }", 30 | Name='test.config', 31 | Type='String', 32 | Overwrite=True 33 | ) 34 | 35 | @ignore_warnings 36 | def tearDown(self): 37 | client.delete_parameters(Names=['test.config']) 38 | 39 | @ignore_warnings 40 | def test_get_config(self): 41 | isEnabled, rate = get_config('isEnabled') 42 | self.assertEqual(isEnabled, True or False) 43 | self.assertEqual(rate, 0.5) 44 | 45 | @ignore_warnings 46 | def test_get_config_delay(self): 47 | delay, rate = get_config('delay') 48 | self.assertEqual(delay, 200) 49 | self.assertEqual(rate, 0.5) 50 | 51 | @ignore_warnings 52 | def test_get_config_error_code(self): 53 | error_code, rate = get_config('error_code') 54 | self.assertEqual(error_code, 404) 55 | self.assertEqual(rate, 0.5) 56 | 57 | @ignore_warnings 58 | def test_get_config_rate(self): 59 | rate, rate = get_config('rate') 60 | self.assertEqual(rate, 0.5) 61 | self.assertEqual(rate, 0.5) 62 | 63 | @ignore_warnings 64 | def test_get_config_bad_key(self): 65 | with self.assertRaises(KeyError): 66 | get_config('dela') 67 | 68 | @ignore_warnings 69 | def test_get_config_bad_config(self): 70 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.conf' 71 | with self.assertRaises(InvalidParameterError): 72 | get_config('delay') 73 | 74 | 75 | class TestConfigErrorMethods(unittest.TestCase): 76 | 77 | @ignore_warnings 78 | def setUp(self): 79 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 80 | client.put_parameter( 81 | Value="{ \"delay\": 200, \"isEnabled\": true, \"exception_msg\": \"I FAILED\", \"rate\": 0.5 }", 82 | Name='test.config', 83 | Type='String', 84 | Overwrite=True 85 | ) 86 | 87 | @ignore_warnings 88 | def tearDown(self): 89 | client.delete_parameters(Names=['test.config']) 90 | 91 | @ignore_warnings 92 | def test_get_config(self): 93 | with self.assertRaises(KeyError): 94 | get_config('error_code') 95 | 96 | 97 | class TestConfigisEnabled(unittest.TestCase): 98 | 99 | @ignore_warnings 100 | def setUp(self): 101 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 102 | client.put_parameter( 103 | Value="{ \"delay\": 200, \"isEnabled\": false, \"exception_msg\": \"I FAILED\", \"rate\": 0.5 }", 104 | Name='test.config', 105 | Type='String', 106 | Overwrite=True 107 | ) 108 | 109 | @ignore_warnings 110 | def tearDown(self): 111 | client.delete_parameters(Names=['test.config']) 112 | 113 | @ignore_warnings 114 | def test_get_config(self): 115 | delay, rate = get_config('error_code') 116 | self.assertEqual(delay, 0) 117 | self.assertEqual(rate, 0) 118 | 119 | 120 | if __name__ == '__main__': 121 | unittest.main() 122 | -------------------------------------------------------------------------------- /tests/test_session.py: -------------------------------------------------------------------------------- 1 | from chaos_lib import SessionWithDelay 2 | from unittest.mock import patch 3 | from io import StringIO 4 | import unittest 5 | import warnings 6 | 7 | 8 | def ignore_warnings(test_func): 9 | def do_test(self, *args, **kwargs): 10 | with warnings.catch_warnings(): 11 | warnings.simplefilter("ignore", ResourceWarning) 12 | warnings.simplefilter("ignore", DeprecationWarning) 13 | test_func(self, *args, **kwargs) 14 | return do_test 15 | 16 | 17 | def session_request_with_delay(): 18 | session = SessionWithDelay(delay=300) 19 | session.get('https://stackoverflow.com/') 20 | pass 21 | 22 | 23 | class TestSessionMethods(unittest.TestCase): 24 | 25 | @ignore_warnings 26 | def test_get_statuscode_arg(self): 27 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 28 | session_request_with_delay() 29 | assert ( 30 | 'Added 300.00ms of delay to GET' in fakeOutput.getvalue().strip() 31 | ) 32 | 33 | 34 | if __name__ == '__main__': 35 | unittest.main() 36 | -------------------------------------------------------------------------------- /tests/test_statuscode.py: -------------------------------------------------------------------------------- 1 | from chaos_lib import corrupt_statuscode 2 | from io import StringIO 3 | from unittest.mock import patch 4 | import unittest 5 | import os 6 | import warnings 7 | import boto3 8 | 9 | client = boto3.client('ssm', region_name='eu-north-1') 10 | 11 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 12 | 13 | 14 | def ignore_warnings(test_func): 15 | def do_test(self, *args, **kwargs): 16 | with warnings.catch_warnings(): 17 | warnings.simplefilter("ignore", ResourceWarning) 18 | warnings.simplefilter("ignore", DeprecationWarning) 19 | test_func(self, *args, **kwargs) 20 | return do_test 21 | 22 | 23 | @corrupt_statuscode 24 | def handler_with_statuscode(event, context): 25 | return { 26 | 'statusCode': 200, 27 | 'body': 'Hello from Lambda!' 28 | } 29 | 30 | 31 | @corrupt_statuscode(error_code=500) 32 | def handler_with_statuscode_arg(event, context): 33 | return { 34 | 'statusCode': 200, 35 | 'body': 'Hello from Lambda!' 36 | } 37 | 38 | 39 | class TestStatusCodeMethods(unittest.TestCase): 40 | 41 | @ignore_warnings 42 | def setUp(self): 43 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 44 | client.put_parameter( 45 | Value="{ \"delay\": 400, \"isEnabled\": true, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 1 }", 46 | Name='test.config', 47 | Type='String', 48 | Overwrite=True 49 | ) 50 | 51 | @ignore_warnings 52 | def tearDown(self): 53 | client.delete_parameters(Names=['test.config']) 54 | 55 | @ignore_warnings 56 | def test_get_statuscode(self): 57 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 58 | response = handler_with_statuscode('foo', 'bar') 59 | assert ( 60 | 'Injecting Error 404 at a rate of 1' in fakeOutput.getvalue().strip() 61 | ) 62 | self.assertEqual( 63 | str(response), "{'statusCode': 404, 'body': 'Hello from Lambda!'}") 64 | 65 | @ignore_warnings 66 | def test_get_statuscode_arg(self): 67 | with patch('sys.stdout', new=StringIO()) as fakeOutput: 68 | response = handler_with_statuscode_arg('foo', 'bar') 69 | assert ( 70 | 'Injecting Error 500 at a rate of 1' in fakeOutput.getvalue().strip() 71 | ) 72 | self.assertEqual( 73 | str(response), "{'statusCode': 500, 'body': 'Hello from Lambda!'}") 74 | 75 | 76 | class TestStatusCodeMethodslowrate(unittest.TestCase): 77 | 78 | @ignore_warnings 79 | def setUp(self): 80 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 81 | client.put_parameter( 82 | Value="{ \"delay\": 400, \"isEnabled\": true, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 0.0000001 }", 83 | Name='test.config', 84 | Type='String', 85 | Overwrite=True 86 | ) 87 | 88 | @ignore_warnings 89 | def tearDown(self): 90 | client.delete_parameters(Names=['test.config']) 91 | 92 | @ignore_warnings 93 | def test_get_statuscode(self): 94 | response = handler_with_statuscode('foo', 'bar') 95 | self.assertEqual( 96 | str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}") 97 | 98 | 99 | class TestStatusCodeMethodsnotenabled(unittest.TestCase): 100 | 101 | @ignore_warnings 102 | def setUp(self): 103 | os.environ['FAILURE_INJECTION_PARAM'] = 'test.config' 104 | client.put_parameter( 105 | Value="{ \"delay\": 400, \"isEnabled\": false, \"error_code\": 404, \"exception_msg\": \"I FAILED\", \"rate\": 1 }", 106 | Name='test.config', 107 | Type='String', 108 | Overwrite=True 109 | ) 110 | 111 | @ignore_warnings 112 | def tearDown(self): 113 | client.delete_parameters(Names=['test.config']) 114 | 115 | @ignore_warnings 116 | def test_get_statuscode(self): 117 | response = handler_with_statuscode('foo', 'bar') 118 | self.assertEqual( 119 | str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}") 120 | 121 | 122 | if __name__ == '__main__': 123 | unittest.main() 124 | --------------------------------------------------------------------------------