├── .gitignore ├── .travis.yml ├── CONTRIBUTING.rst ├── LICENSE ├── Pipfile ├── Pipfile.lock ├── README.rst ├── clear_lambda_storage.py ├── handler.py ├── requirements.txt ├── serverless.yml └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | *.pyc 3 | __pycache__ 4 | .serverless 5 | build 6 | clear_lambda_storage.egg-info 7 | dist 8 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - "3.7" 5 | 6 | script: 7 | - pip install -r requirements.txt 8 | 9 | deploy: 10 | provider: pypi 11 | username: __token__ 12 | password: 13 | secure: "pdZBAYLQdhPPbGYUeoNZTFyvSgLcOR5aoK1jiO0mDAN1HRl3jEuz7y7lMJzwkVZGeF342lRiOw3kDhrWWHEjG+QBbcYPTfC9tT2pIbeXNWKDWuhksSOjM+DoZJshBXu7ofp3PISqYCDFug1wmP4L4N/J6MMpjdmMGMtOpbR+IWihJi5lvlbn1ErkHv2R0f6iktVunf7XjNOrHcbbhgqc83BUaoF8HNWhmlVE1zPjgk9EuI+HFZBCu7Egn2bL71RlaKctaAo+T1nmE1Enq2FUSHMKKc7ww+l1/OUVAPFxfJSd+FTQT32cvwWQVbxVPLEW3wpmdOfHCdp+wVYXeJf95cO7/iD7U281mpY5jKUKKmXX4lvZZJfRBbHYKCi8P+dPJXtfF1falurvNrizrzZSRSFKF7TSnJWiTCzLmkReekfFOTDZHID++0XtmJ3OjtMYyh64jSUFIlW7guA+WBlw/hO23OSD9DeeKlOiBQx25MSc3dtYkKKZMW2nJdWXK6nEkGKFR2bw428jJVG70HalEC18PNuam3u4zCmLjoUUsnBQtQpGbQrHbu2BRmRo6HAYDaIbsdXFxbf6ORfHawKnydGkDnsSje4mfZDes3B7lytO2JI54MXeGXEZVKazqnzfdoaqVjthGsloMuI5rnc1IHXHw8ogW1VPywlcAmvlrUo=" 14 | skip_existing: true 15 | skip_cleanup: true 16 | branches: 17 | only: 18 | - master 19 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | ============ 2 | Contributing 3 | ============ 4 | 5 | First off, thanks for taking the time to contribute! 6 | 7 | This document aims to provide some basic guidelines to contribute to this repository, but keep in mind that these are just guidelines, not rules; use your best judgment and feel free to propose changes to this document in a pull request. 8 | 9 | Please note we have a code of conduct, please follow it in all your interactions with the project. 10 | 11 | Contributing Process 12 | -------------------- 13 | 14 | 1. Feel free to open new issues with ideas, features and improvements. 15 | 2. Otherwise, fork this repository, write the newly feature, make sure it works, and create a new pull request! 16 | 3. The owners will look at the new PR and approve it accordingly. 17 | 18 | Code of Conduct 19 | --------------- 20 | 21 | Our Pledge 22 | ~~~~~~~~~~ 23 | 24 | In the interest of fostering an open and welcoming environment, we as 25 | contributors and maintainers pledge to making participation in our project and 26 | our community a harassment-free experience for everyone, regardless of age, body 27 | size, disability, ethnicity, gender identity and expression, level of experience, 28 | nationality, personal appearance, race, religion, or sexual identity and 29 | orientation. 30 | 31 | Our Standards 32 | ~~~~~~~~~~~~~ 33 | 34 | Examples of behavior that contributes to creating a positive environment 35 | include: 36 | 37 | - Using welcoming and inclusive language 38 | - Being respectful of differing viewpoints and experiences 39 | - Gracefully accepting constructive criticism 40 | - Focusing on what is best for the community 41 | - Showing empathy towards other community members 42 | 43 | Examples of unacceptable behavior by participants include: 44 | 45 | - The use of sexualized language or imagery and unwelcome sexual attention or advances 46 | - Trolling, insulting/derogatory comments, and personal or political attacks 47 | - Public or private harassment 48 | - Publishing others' private information, such as a physical or electronic address, without explicit permission 49 | - Other conduct which could reasonably be considered inappropriate in a professional setting 50 | 51 | Our Responsibilities 52 | ~~~~~~~~~~~~~~~~~~~~ 53 | 54 | Project maintainers are responsible for clarifying the standards of acceptable 55 | behavior and are expected to take appropriate and fair corrective action in 56 | response to any instances of unacceptable behavior. 57 | 58 | Project maintainers have the right and responsibility to remove, edit, or 59 | reject comments, commits, code, wiki edits, issues, and other contributions 60 | that are not aligned to this Code of Conduct, or to ban temporarily or 61 | permanently any contributor for other behaviors that they deem inappropriate, 62 | threatening, offensive, or harmful. 63 | 64 | Scope 65 | ~~~~~ 66 | 67 | This Code of Conduct applies both within project spaces and in public spaces 68 | when an individual is representing the project or its community. Examples of 69 | representing a project or community include using an official project e-mail 70 | address, posting via an official social media account, or acting as an appointed 71 | representative at an online or offline event. Representation of a project may be 72 | further defined and clarified by project maintainers. 73 | 74 | Enforcement 75 | ~~~~~~~~~~~ 76 | 77 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 78 | reported by contacting the project team at info@epsagon.com. All 79 | complaints will be reviewed and investigated and will result in a response that 80 | is deemed necessary and appropriate to the circumstances. The project team is 81 | obligated to maintain confidentiality with regard to the reporter of an incident. 82 | Further details of specific enforcement policies may be posted separately. 83 | 84 | Project maintainers who do not follow or enforce the Code of Conduct in good 85 | faith may face temporary or permanent repercussions as determined by other 86 | members of the project's leadership. 87 | 88 | Attribution 89 | ~~~~~~~~~~~ 90 | 91 | This Code of Conduct is adapted from the `Contributor Covenant `_, version 1.4, available at http://contributor-covenant.org/version/1/4 92 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Epsagon 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Pipfile: -------------------------------------------------------------------------------- 1 | [[source]] 2 | name = "pypi" 3 | url = "https://pypi.org/simple" 4 | verify_ssl = true 5 | 6 | [dev-packages] 7 | 8 | [packages] 9 | boto3 = "*" 10 | 11 | [requires] 12 | python_version = "3.7" 13 | -------------------------------------------------------------------------------- /Pipfile.lock: -------------------------------------------------------------------------------- 1 | { 2 | "_meta": { 3 | "hash": { 4 | "sha256": "49efc3368e25cc541579c91c92b08134ed8a23fc9d5c328ef8207674428382f5" 5 | }, 6 | "pipfile-spec": 6, 7 | "requires": { 8 | "python_version": "3.7" 9 | }, 10 | "sources": [ 11 | { 12 | "name": "pypi", 13 | "url": "https://pypi.org/simple", 14 | "verify_ssl": true 15 | } 16 | ] 17 | }, 18 | "default": { 19 | "boto3": { 20 | "hashes": [ 21 | "sha256:33462a79d57c9c4a215e075472509537d03545f54566fc4f776fb0f4cfa616f6", 22 | "sha256:34f9a04f529dc849f0e427782d6f3c6b62f7fb734d8f4859b17e5dee0855323e" 23 | ], 24 | "index": "pypi", 25 | "version": "==1.12.0" 26 | }, 27 | "botocore": { 28 | "hashes": [ 29 | "sha256:055da4826f6c9158e4a61549d57a2ce449c27d44ce34ab4c96c7bb7b5c993efc", 30 | "sha256:1f7cecfcd38c7cac17b5386014eb04626d1c7559ee8d8ec1526058cd23f6d1d4" 31 | ], 32 | "version": "==1.15.0" 33 | }, 34 | "docutils": { 35 | "hashes": [ 36 | "sha256:6c4f696463b79f1fb8ba0c594b63840ebd41f059e92b31957c46b74a4599b6d0", 37 | "sha256:9e4d7ecfc600058e07ba661411a2b7de2fd0fafa17d1a7f7361cd47b1175c827", 38 | "sha256:a2aeea129088da402665e92e0b25b04b073c04b2dce4ab65caaa38b7ce2e1a99" 39 | ], 40 | "version": "==0.15.2" 41 | }, 42 | "jmespath": { 43 | "hashes": [ 44 | "sha256:3720a4b1bd659dd2eecad0666459b9788813e032b83e7ba58578e48254e0a0e6", 45 | "sha256:bde2aef6f44302dfb30320115b17d030798de8c4110e28d5cf6cf91a7a31074c" 46 | ], 47 | "version": "==0.9.4" 48 | }, 49 | "python-dateutil": { 50 | "hashes": [ 51 | "sha256:73ebfe9dbf22e832286dafa60473e4cd239f8592f699aa5adaf10050e6e1823c", 52 | "sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a" 53 | ], 54 | "version": "==2.8.1" 55 | }, 56 | "s3transfer": { 57 | "hashes": [ 58 | "sha256:2482b4259524933a022d59da830f51bd746db62f047d6eb213f2f8855dcb8a13", 59 | "sha256:921a37e2aefc64145e7b73d50c71bb4f26f46e4c9f414dc648c6245ff92cf7db" 60 | ], 61 | "version": "==0.3.3" 62 | }, 63 | "six": { 64 | "hashes": [ 65 | "sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a", 66 | "sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c" 67 | ], 68 | "version": "==1.14.0" 69 | }, 70 | "urllib3": { 71 | "hashes": [ 72 | "sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc", 73 | "sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc" 74 | ], 75 | "markers": "python_version != '3.4'", 76 | "version": "==1.25.8" 77 | } 78 | }, 79 | "develop": {} 80 | } 81 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Clear Lambda code storage 2 | =========================== 3 | 4 | 5 | Motivation 6 | ----------- 7 | AWS limits the total code storage for Lambda functions to `75GB `_. 8 | 9 | The main reason of reaching such size is because for every deployment of existing function, AWS stores the previous version ("qualifier"). 10 | 11 | Usually, when you reach that point, you want to remove old version. 12 | This tool will help you to! 13 | 14 | 15 | Setup 16 | ----- 17 | Install via pip 18 | 19 | .. code-block:: bash 20 | 21 | pip install clear-lambda-storage 22 | clear_lambda_storage 23 | 24 | Install via source 25 | 26 | .. code-block:: bash 27 | 28 | git clone https://github.com/epsagon/clear-lambda-storage 29 | cd clear-lambda-storage/ 30 | pip install -r requirements.txt 31 | python clear_lambda_storage.py 32 | 33 | 34 | Advanced usage 35 | --------------- 36 | 37 | Provide credentials: 38 | 39 | .. code-block:: bash 40 | 41 | python clear_lambda_storage.py --token-key-id --token-secret 42 | 43 | Alternate usage: 44 | 45 | .. code-block:: bash 46 | 47 | python clear_lambda_storage.py --profile --num-to-keep 2 48 | 49 | ⚡️ `Serverless Framework `_ usage 50 | ---------------------------------------------------------- 51 | .. code-block:: bash 52 | 53 | npm i -g serverless 54 | git clone https://github.com/epsagon/clear-lambda-storage 55 | cd clear-lambda-storage/ 56 | serverless deploy 57 | 58 | You can schedule this Lambda code storage clean to run every period you want: 59 | 60 | .. code-block:: yaml 61 | 62 | functions: 63 | clear_lambda_storage: 64 | handler: handler.clear_lambda_storage 65 | memorySize: 128 66 | timeout: 120 67 | events: 68 | - schedule: cron(0 12 ? * SUN *) # Run every sunday at 12:00pm UTC 69 | -------------------------------------------------------------------------------- /clear_lambda_storage.py: -------------------------------------------------------------------------------- 1 | """ 2 | Removes old versions of Lambda functions. 3 | """ 4 | from __future__ import print_function 5 | import argparse 6 | import boto3 7 | try: 8 | import queue 9 | except ImportError: 10 | import Queue as queue 11 | from boto3.session import Session 12 | from botocore.exceptions import ClientError 13 | 14 | 15 | LATEST = '$LATEST' 16 | 17 | 18 | def list_available_lambda_regions(): 19 | """ 20 | Enumerates list of all Lambda regions 21 | :return: list of regions 22 | """ 23 | session = Session() 24 | return session.get_available_regions('lambda') 25 | 26 | 27 | def init_boto_client(client_name, region, args): 28 | """ 29 | Initiates boto's client object 30 | :param client_name: client name 31 | :param region: region name 32 | :param args: arguments 33 | :return: Client 34 | """ 35 | if args.token_key_id and args.token_secret: 36 | boto_client = boto3.client( 37 | client_name, 38 | aws_access_key_id=args.token_key_id, 39 | aws_secret_access_key=args.token_secret, 40 | region_name=region 41 | ) 42 | elif args.profile: 43 | session = boto3.session.Session(profile_name=args.profile) 44 | boto_client = session.client(client_name, region_name=region) 45 | else: 46 | boto_client = boto3.client(client_name, region_name=region) 47 | 48 | return boto_client 49 | 50 | 51 | def lambda_function_generator(lambda_client): 52 | """ 53 | Iterates over Lambda functions in a specific region 54 | :param lambda_client: Client 55 | :return: Generator 56 | """ 57 | next_marker = None 58 | try: 59 | response = lambda_client.list_functions() 60 | except Exception: 61 | print('Could not scan region') 62 | return iter([]) 63 | 64 | while next_marker != '': 65 | next_marker = '' 66 | functions = response['Functions'] 67 | for lambda_function in functions: 68 | yield lambda_function 69 | 70 | # Verify if there is next marker 71 | if 'NextMarker' in response: 72 | next_marker = response['NextMarker'] 73 | response = lambda_client.list_functions(Marker=next_marker) 74 | 75 | 76 | def lambda_version_generator(lambda_client, lambda_function): 77 | """ 78 | Iterates over Lambda function versions for a specific function 79 | :param lambda_client: Client 80 | :param lambda_function: Lambda dict 81 | :return: Generator 82 | """ 83 | next_marker = None 84 | response = lambda_client.list_versions_by_function( 85 | FunctionName=lambda_function['FunctionArn'] 86 | ) 87 | 88 | while next_marker != '': 89 | next_marker = '' 90 | versions = response['Versions'] 91 | for version in versions: 92 | yield version 93 | 94 | # Verify if there is next marker 95 | if 'NextMarker' in response: 96 | next_marker = response['NextMarker'] 97 | response = lambda_client.list_versions_by_function( 98 | FunctionName=lambda_function['FunctionArn'], 99 | Marker=next_marker 100 | ) 101 | 102 | 103 | def remove_old_lambda_versions(args): 104 | """ 105 | Removes old versions of Lambda functions 106 | :param args: arguments 107 | :return: None 108 | """ 109 | regions = args.regions or list_available_lambda_regions() 110 | total_deleted_code_size = 0 111 | total_deleted_functions = {} 112 | num_to_keep = args.num_to_keep 113 | print('Keeping {} versions for functions'.format(num_to_keep)) 114 | if args.function_names: 115 | print('Will only delete lambda versions for functions: {}'.format(" ,".join(args.function_names))) 116 | 117 | for region in regions: 118 | print('Scanning {} region'.format(region)) 119 | 120 | lambda_client = init_boto_client('lambda', region, args) 121 | try: 122 | function_generator = lambda_function_generator(lambda_client) 123 | except Exception as exception: 124 | print('Could not scan region: {}'.format(str(exception))) 125 | continue 126 | 127 | for lambda_function in function_generator: 128 | # Verify if function name is provided and in case it is, skips all lambdas which name does not match 129 | if args.function_names and lambda_function['FunctionName'] not in args.function_names: 130 | continue 131 | 132 | versions_to_keep = queue.Queue(maxsize=num_to_keep) 133 | 134 | for version in lambda_version_generator(lambda_client, lambda_function): 135 | 136 | if version['Version'] in (lambda_function['Version'], '$LATEST'): 137 | continue 138 | 139 | if versions_to_keep.full(): 140 | version_to_delete = versions_to_keep.get() 141 | print('Detected {} with an old version {}'.format( 142 | version_to_delete['FunctionName'], 143 | version_to_delete['Version']) 144 | ) 145 | total_deleted_functions.setdefault(version_to_delete['FunctionName'], 0) 146 | total_deleted_functions[version_to_delete['FunctionName']] += 1 147 | total_deleted_code_size += (version_to_delete['CodeSize'] / (1024 * 1024)) 148 | 149 | # DELETE OPERATION! 150 | if args.dry_run: 151 | print('Dry-Run: This process would delete function: {}'.format(version_to_delete["FunctionArn"])) 152 | else: 153 | try: 154 | lambda_client.delete_function( 155 | FunctionName=version_to_delete['FunctionArn'] 156 | ) 157 | except ClientError as exception: 158 | print('Could not delete function: {}'.format(str(exception))) 159 | versions_to_keep.put(version) 160 | 161 | print('-' * 10) 162 | print('Deleted {} versions from {} functions'.format( 163 | sum(total_deleted_functions.values()), 164 | len(total_deleted_functions.keys()) 165 | )) 166 | print('Freed {} MBs'.format(int(total_deleted_code_size))) 167 | 168 | 169 | def main(): 170 | parser = argparse.ArgumentParser( 171 | description='Removes old versions of Lambda functions.' 172 | ) 173 | 174 | parser.add_argument( 175 | '--token-key-id', 176 | type=str, 177 | help=( 178 | 'AWS access key id. Must provide AWS secret access key as well ' 179 | '(default: from local configuration).' 180 | ), 181 | metavar='token-key-id' 182 | ) 183 | parser.add_argument( 184 | '--token-secret', 185 | type=str, 186 | help=( 187 | 'AWS secret access key. Must provide AWS access key id ' 188 | 'as well (default: from local configuration.' 189 | ), 190 | metavar='token-secret' 191 | ) 192 | parser.add_argument( 193 | '--profile', 194 | type=str, 195 | help=( 196 | 'AWS profile. Optional ' 197 | '(default: "default" from local configuration).' 198 | ), 199 | metavar='profile' 200 | ) 201 | 202 | parser.add_argument( 203 | '--regions', 204 | nargs='+', 205 | help='AWS region to look for old Lambda versions', 206 | metavar='regions' 207 | ) 208 | 209 | parser.add_argument( 210 | '--num-to-keep', 211 | type=int, 212 | default=2, 213 | help=( 214 | 'Number of latest versions to keep. Older versions will be deleted. Optional ' 215 | '(default: 2).' 216 | ), 217 | metavar='num-to-keep' 218 | ) 219 | 220 | parser.add_argument( 221 | '--function-names', 222 | nargs='+', 223 | help=( 224 | 'Clear the storage of a single application. Optional ' 225 | '(default: None).' 226 | ), 227 | metavar='function-names' 228 | ) 229 | 230 | parser.add_argument( 231 | '--dry-run', 232 | type=bool, 233 | default=False, 234 | help=( 235 | 'Run the function without deleting anything. Optional ' 236 | '(default: False).' 237 | ), 238 | metavar='dry-run' 239 | ) 240 | remove_old_lambda_versions(parser.parse_args()) 241 | 242 | 243 | if __name__ == '__main__': 244 | main() 245 | -------------------------------------------------------------------------------- /handler.py: -------------------------------------------------------------------------------- 1 | from argparse import Namespace 2 | from clear_lambda_storage import remove_old_lambda_versions 3 | 4 | 5 | def clear_lambda_storage(event, context): 6 | remove_old_lambda_versions(Namespace(token_key_id=None, token_secret=None, regions=None, profile=None, num_to_keep=2, function_names=None, dry_run=None)) 7 | return "Successful clean! 🗑 ✅" 8 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | boto3 2 | -------------------------------------------------------------------------------- /serverless.yml: -------------------------------------------------------------------------------- 1 | service: clear-lambda-storage 2 | 3 | provider: 4 | name: aws 5 | stage: ${opt:stage, 'dev'} 6 | runtime: python3.7 7 | region: ${opt:region, 'us-east-1'} 8 | iamRoleStatements: 9 | - Effect: "Allow" 10 | Action: 11 | - lambda:ListFunctions 12 | - lambda:ListVersionsByFunction 13 | - lambda:DeleteFunction 14 | Resource: "*" 15 | 16 | package: 17 | exclude: 18 | - .git/** 19 | - node_modules/** 20 | - __pycache__/** 21 | - package-lock.json 22 | - package.json 23 | - README.srt 24 | 25 | functions: 26 | clear_lambda_storage: 27 | handler: handler.clear_lambda_storage 28 | memorySize: 128 29 | timeout: 120 30 | events: 31 | - schedule: cron(0 12 ? * SUN *) # Run every sunday at 12:00pm UTC 32 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("README.rst", "r") as fh: 4 | long_description = fh.read() 5 | 6 | setuptools.setup( 7 | name="clear-lambda-storage", 8 | version="1.0.1", 9 | author="", 10 | author_email="", 11 | description="Clear Lambda code storage", 12 | long_description=long_description, 13 | long_description_content_type="text/x-rst", 14 | url="https://github.com/epsagon/clear-lambda-storage", 15 | py_modules=["clear_lambda_storage"], 16 | classifiers=[ 17 | "Programming Language :: Python :: 3", 18 | "License :: OSI Approved :: MIT License", 19 | "Operating System :: OS Independent", 20 | ], 21 | install_requires="boto3", 22 | python_requires='>=3', 23 | entry_points={ 24 | "console_scripts": [ 25 | "clear_lambda_storage = clear_lambda_storage:main", 26 | ] 27 | } 28 | ) 29 | --------------------------------------------------------------------------------