├── .gitignore ├── CONTRIBUTING.rst ├── LICENSE ├── README.rst ├── benchmark.py ├── fibonacci-function ├── fibonacci.py ├── payload.json ├── performance_chart.png └── serverless.yml └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | .serverless 3 | *.pyc -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | ============ 2 | Contributing 3 | ============ 4 | 5 | First off, thanks for taking the time to contribute! 6 | 7 | This document aims to provide some basic guidelines to contribute to this repository, but keep in mind that these are just guidelines, not rules; use your best judgment and feel free to propose changes to this document in a pull request. 8 | 9 | Please note we have a code of conduct, please follow it in all your interactions with the project. 10 | 11 | Contributing Process 12 | -------------------- 13 | 14 | 1. Feel free to open new issues with ideas, features and improvements. 15 | 2. Otherwise, fork this repository, write the newly feature, make sure it works, and create a new pull request! 16 | 3. The owners will look at the new PR and approve it accordingly. 17 | 18 | Code of Conduct 19 | --------------- 20 | 21 | Our Pledge 22 | ~~~~~~~~~~ 23 | 24 | In the interest of fostering an open and welcoming environment, we as 25 | contributors and maintainers pledge to making participation in our project and 26 | our community a harassment-free experience for everyone, regardless of age, body 27 | size, disability, ethnicity, gender identity and expression, level of experience, 28 | nationality, personal appearance, race, religion, or sexual identity and 29 | orientation. 30 | 31 | Our Standards 32 | ~~~~~~~~~~~~~ 33 | 34 | Examples of behavior that contributes to creating a positive environment 35 | include: 36 | 37 | - Using welcoming and inclusive language 38 | - Being respectful of differing viewpoints and experiences 39 | - Gracefully accepting constructive criticism 40 | - Focusing on what is best for the community 41 | - Showing empathy towards other community members 42 | 43 | Examples of unacceptable behavior by participants include: 44 | 45 | - The use of sexualized language or imagery and unwelcome sexual attention or advances 46 | - Trolling, insulting/derogatory comments, and personal or political attacks 47 | - Public or private harassment 48 | - Publishing others' private information, such as a physical or electronic address, without explicit permission 49 | - Other conduct which could reasonably be considered inappropriate in a professional setting 50 | 51 | Our Responsibilities 52 | ~~~~~~~~~~~~~~~~~~~~ 53 | 54 | Project maintainers are responsible for clarifying the standards of acceptable 55 | behavior and are expected to take appropriate and fair corrective action in 56 | response to any instances of unacceptable behavior. 57 | 58 | Project maintainers have the right and responsibility to remove, edit, or 59 | reject comments, commits, code, wiki edits, issues, and other contributions 60 | that are not aligned to this Code of Conduct, or to ban temporarily or 61 | permanently any contributor for other behaviors that they deem inappropriate, 62 | threatening, offensive, or harmful. 63 | 64 | Scope 65 | ~~~~~ 66 | 67 | This Code of Conduct applies both within project spaces and in public spaces 68 | when an individual is representing the project or its community. Examples of 69 | representing a project or community include using an official project e-mail 70 | address, posting via an official social media account, or acting as an appointed 71 | representative at an online or offline event. Representation of a project may be 72 | further defined and clarified by project maintainers. 73 | 74 | Enforcement 75 | ~~~~~~~~~~~ 76 | 77 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 78 | reported by contacting the project team at info@epsagon.com. All 79 | complaints will be reviewed and investigated and will result in a response that 80 | is deemed necessary and appropriate to the circumstances. The project team is 81 | obligated to maintain confidentiality with regard to the reporter of an incident. 82 | Further details of specific enforcement policies may be posted separately. 83 | 84 | Project maintainers who do not follow or enforce the Code of Conduct in good 85 | faith may face temporary or permanent repercussions as determined by other 86 | members of the project's leadership. 87 | 88 | Attribution 89 | ~~~~~~~~~~~ 90 | 91 | This Code of Conduct is adapted from the `Contributor Covenant `_, version 1.4, available at http://contributor-covenant.org/version/1/4 92 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Epsagon 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | AWS Lambda Memory Performance Benchmark 2 | ======================================= 3 | 4 | Motivation 5 | ---------- 6 | - Understand how RAM selection affect Lambda's performance (on `Medium `_). 7 | - Code a tool that measures Lambda function performance (duration and cost) with several memory sizes. 8 | 9 | 10 | Setup 11 | ----- 12 | .. code-block:: bash 13 | 14 | git clone git@github.com:epsagon/lambda-memory-performance-benchmark.git 15 | cd lambda-memory-performance-benchmark/ 16 | pip install -r requirements.txt 17 | python benchmark.py -f -r -p 18 | 19 | 20 | Usage 21 | ----- 22 | 23 | Basic run: 24 | 25 | .. code-block:: bash 26 | 27 | python benchmark.py -f lambda-performance-benchmark -r us-east-1 -p fibonacci-function/payload.json 28 | 29 | 30 | Fibonacci's Last Result (February 9th, 2018) 31 | -------------------------------- 32 | 33 | Chart: 34 | 35 | .. image:: https://github.com/epsagon/lambda-memory-performance-benchmark/blob/master/fibonacci-function/performance_chart.png 36 | 37 | 38 | Table: 39 | 40 | ============= ================== ================================= 41 | Memory Size Duration (in ms) Price Per 1M Invocations (in $) 42 | ============= ================== ================================= 43 | 128MB 376.05 0.832 44 | 192MB 250.24 0.939 45 | 256MB 198.56 0.834 46 | 512MB 98.36 0.834 47 | 768MB 65.48 1.250 48 | 1024MB 45.31 1.667 49 | 1536MB 30.53 2.501 50 | 2048MB 25.21 3.334 51 | 2560MB 25.36 4.168 52 | 3008MB 25.15 4.897 53 | ============= ================== ================================= 54 | -------------------------------------------------------------------------------- /benchmark.py: -------------------------------------------------------------------------------- 1 | """ 2 | Benchmark tool for measuring Lambda function performance in different memory 3 | sizes. 4 | """ 5 | 6 | from __future__ import print_function 7 | import argparse 8 | import math 9 | import base64 10 | import boto3 11 | 12 | MEMORY_TO_PRICE = { 13 | 128: 0.000000208, 14 | 256: 0.000000417, 15 | 512: 0.000000834, 16 | 1024: 0.000001667, 17 | 1536: 0.000002501, 18 | 2048: 0.000003334, 19 | 2560: 0.000004168, 20 | 3008: 0.000004897, 21 | } 22 | PRICE_INTERVAL = 100 23 | INVOCATIONS_COUNT = 5 24 | CSV_HEADER = 'Memory Size,Duration (in ms),Price Per 1M Invocations (in $)\n' 25 | 26 | 27 | def invoke_lambda_and_get_duration(lambda_client, payload, function_name): 28 | """ 29 | Invokes Lambda and return the duration. 30 | :param lambda_client: Lambda client. 31 | :param payload: payload to send. 32 | :param function_name: function name. 33 | :return: duration. 34 | """ 35 | response = lambda_client.invoke( 36 | FunctionName=function_name, 37 | InvocationType='RequestResponse', 38 | LogType='Tail', 39 | Payload=payload, 40 | ) 41 | 42 | # Extract duration from Lambda log 43 | lambda_log = base64.b64decode(response['LogResult']).decode('utf-8') 44 | report_data = \ 45 | [line for line in lambda_log.split('\n') 46 | if line.startswith('REPORT') 47 | ][0] 48 | duration = \ 49 | [col for col in report_data.split('\t') 50 | if col.startswith('Duration') 51 | ][0] 52 | duration = float(duration.split()[1]) 53 | return duration 54 | 55 | 56 | def run_benchmark(args): 57 | """ 58 | Run benchmark. 59 | :param args: arguments. 60 | :return: None. 61 | """ 62 | 63 | if args.aws_profile: 64 | aws_session = boto3.Session(profile_name=args.aws_profile) 65 | else: 66 | aws_session = boto3.Session() 67 | 68 | lambda_client = aws_session.client('lambda', region_name=args.region) 69 | sorted_memory_sizes = sorted(MEMORY_TO_PRICE) 70 | results = {} 71 | 72 | # Load payload 73 | with open(args.payload_file, 'rt') as input_data: 74 | payload = input_data.read() 75 | 76 | # Read Original memory size 77 | original_memory_size = lambda_client.get_function_configuration( 78 | FunctionName=args.function_name, 79 | )['MemorySize'] 80 | print('Original memory size: {0}'.format(original_memory_size)) 81 | 82 | # Benchmark 83 | for memory_size in sorted_memory_sizes: 84 | print('Setting memory size: {0}MB'.format(memory_size)) 85 | lambda_client.update_function_configuration( 86 | FunctionName=args.function_name, 87 | MemorySize=memory_size, 88 | ) 89 | 90 | print('Warming Lambda') 91 | lambda_client.invoke( 92 | FunctionName=args.function_name, 93 | Payload=payload, 94 | ) 95 | 96 | # Run several times 97 | duration_sum = 0 98 | for _ in range(INVOCATIONS_COUNT): 99 | duration_sum += invoke_lambda_and_get_duration( 100 | lambda_client, 101 | payload, 102 | args.function_name 103 | ) 104 | 105 | duration = duration_sum / INVOCATIONS_COUNT 106 | results[memory_size] = duration 107 | print('Result: {0}'.format(duration)) 108 | print('-' * 20) 109 | 110 | print('Restoring original memory size') 111 | lambda_client.update_function_configuration( 112 | FunctionName=args.function_name, 113 | MemorySize=original_memory_size, 114 | ) 115 | 116 | with open(args.output_file, 'wt') as output_results: 117 | output_results.write(CSV_HEADER) 118 | for memory_size in sorted_memory_sizes: 119 | price = math.ceil(results[memory_size] / PRICE_INTERVAL) \ 120 | * MEMORY_TO_PRICE[memory_size] * 1000000 121 | 122 | output_results.write('{0},{1},{2}\n'.format( 123 | '{0}MB'.format(memory_size), 124 | '%.2f' % (results[memory_size],), 125 | '%.2f' % (price,), 126 | )) 127 | 128 | 129 | if __name__ == '__main__': 130 | parser = argparse.ArgumentParser( 131 | description='Benchmark Lambda function with several memory sizes to' + 132 | 'understand the impact on performance.' 133 | ) 134 | 135 | parser.add_argument( 136 | '-f', 137 | '--function', 138 | dest='function_name', 139 | default=False, 140 | required=True, 141 | help='Tested function name.' 142 | ) 143 | parser.add_argument( 144 | '-r', 145 | '--region', 146 | dest='region', 147 | default=False, 148 | required=True, 149 | help='Tested function region.' 150 | ) 151 | parser.add_argument( 152 | '-p', 153 | '--payload_file', 154 | dest='payload_file', 155 | default=False, 156 | required=True, 157 | help='JSON Payload filename to send to the function.' 158 | ) 159 | parser.add_argument( 160 | '--profile', 161 | dest='aws_profile', 162 | default=False, 163 | required=False, 164 | help='A specific AWS Named Profile configured within your AWS Credentials file.' 165 | ) 166 | parser.add_argument( 167 | '--output', 168 | dest='output_file', 169 | default='results.csv', 170 | help='Output results filename.' 171 | ) 172 | 173 | arguments = parser.parse_args() 174 | run_benchmark(arguments) 175 | -------------------------------------------------------------------------------- /fibonacci-function/fibonacci.py: -------------------------------------------------------------------------------- 1 | """ 2 | Lambda function for performance benchmark. 3 | """ 4 | 5 | from __future__ import print_function 6 | import time 7 | import json 8 | 9 | TESTS = 100 10 | 11 | 12 | def fibonacci(index): 13 | """ 14 | Recursive function that calculates Fibonacci sequence. 15 | :param index: the n-th element of Fibonacci sequence to calculate. 16 | :return: n-th element of Fibonacci sequence. 17 | """ 18 | 19 | if index <= 1: 20 | return index 21 | return fibonacci(index - 1) + fibonacci(index - 2) 22 | 23 | 24 | def handler(event, _): 25 | """ 26 | Main handler. 27 | :param event: event data. 28 | :param _: unused context. 29 | :return: average duration. 30 | """ 31 | start_time = time.time() 32 | for _ in range(TESTS): 33 | fibonacci(event['index']) 34 | duration = time.time() - start_time 35 | return duration / TESTS 36 | 37 | 38 | def test_locally(): 39 | """ 40 | Testing locally. 41 | :return: None. 42 | """ 43 | with open('warm_data.json', 'rt') as input_data: 44 | data = json.load(input_data) 45 | print(handler(data, None)) 46 | 47 | 48 | if __name__ == '__main__': 49 | test_locally() 50 | -------------------------------------------------------------------------------- /fibonacci-function/payload.json: -------------------------------------------------------------------------------- 1 | { 2 | "index": 25 3 | } -------------------------------------------------------------------------------- /fibonacci-function/performance_chart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/epsagon/lambda-memory-performance-benchmark/b1e48c697b3f6d0b711f8724be015c3f81458c14/fibonacci-function/performance_chart.png -------------------------------------------------------------------------------- /fibonacci-function/serverless.yml: -------------------------------------------------------------------------------- 1 | service: lambda-performance-benchmark 2 | provider: 3 | name: aws 4 | runtime: python2.7 5 | memorySize: 512 6 | timeout: 100 7 | 8 | functions: 9 | main: 10 | handler: fibonacci.handler 11 | description: "Performance Benchmark" 12 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | boto3 --------------------------------------------------------------------------------