├── .gitignore ├── LICENSE ├── README.md ├── lambder ├── __init__.py ├── cli.py └── lambder.py ├── requirements.txt ├── setup.cfg ├── setup.py ├── tests ├── __init__.py └── test_cli.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | 3 | dist/ 4 | build/ 5 | *.egg-info/ 6 | 7 | .tox/ 8 | .coverage 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Leaf Software Solutions 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Lambder 2 | 3 | Serverless cron jobs written in python and scheduled as AWS Lambdas. 4 | 5 | We used to implement maintenance jobs like RDS snapshots and starting/stopping 6 | instances outside of business hours as CI server builds. This always seemed hacky. 7 | When scheduled Lambdas were released, we got to work re-implementing these 8 | scripts. Unfortunately, scheduled Lambdas have a steep learning curve. 9 | You have to understand IAM roles, CloudWatch Events, and how to deploy and 10 | update Lambda function code. 11 | 12 | Lambder simplifies the creation and deployment of these scheduled jobs. 13 | 14 | ## Installation 15 | 16 | If you don't use `pipsi`, you're missing out. 17 | Here are [installation instructions](https://github.com/mitsuhiko/pipsi#readme). 18 | 19 | Simply run: 20 | 21 | $ pipsi install lambder 22 | 23 | ## Getting Started 24 | 25 | Lambder uses [boto3](http://boto3.readthedocs.org/en/latest/) to access AWS. You'll need to configure it with your access keys and default region by following [these instructions](http://boto3.readthedocs.org/en/latest/guide/configuration.html#guide-configuration). 26 | 27 | To create a new Lambda Function project, specify a name and an S3 bucket to 28 | store the code zipfile. 29 | 30 | 1. `lambder functions new --name foo --bucket mys3bucket` 31 | 2. `cd lambder-foo` 32 | 3. `lambder functions deploy` 33 | 4. `lambder functions list` 34 | 5. `lambder functions invoke` 35 | 36 | ## Usage 37 | 38 | Get help 39 | 40 | lambder --help 41 | 42 | ### Managing Events 43 | 44 | Schedule an existing AWS Lambda 45 | 46 | lambder events add \ 47 | --name EbsBackups \ 48 | --function-name Lambder-ebs-backup \ 49 | --cron 'cron(0 6 ? * * *)' 50 | 51 | Remove an event (a scheduled Lambda) 52 | 53 | lambder events rm --name EbsBackups 54 | 55 | Disable an event 56 | 57 | lambder events disable --name EbsBackups 58 | 59 | Re-enable a disabled event 60 | 61 | lambder events enable --name EbsBackups 62 | 63 | Load events from a json file 64 | 65 | lambder events load --file example_events.json 66 | 67 | List all events created by lambder 68 | 69 | lambder events list 70 | 71 | ### Managing Functions 72 | 73 | Create a new AWS Lambda project. Specify a name and an S3 bucket to store 74 | the Lambda function code. Optionally specify function configuration like 75 | timeout, memory size, and description. 76 | 77 | lambder functions new \ 78 | --name ebs-backups \ 79 | --bucket my-s3-bucket \ 80 | --timeout 30 \ 81 | --memory 128 \ 82 | --description "ebs backup function" 83 | 84 | You can also deploy your Lambda functions to your VPC. 85 | 86 | lambder functions new \ 87 | --name curl-backend-site \ 88 | --bucket my-s3-bucket \ 89 | --timeout 30 \ 90 | --memory 128 \ 91 | --description "curl a site on a private subnet" \ 92 | --subnet-ids "subnet-1abcdef,subnet-2abcdef" \ 93 | --security-group-ids "sg-12345678" 94 | 95 | Deploy the Lambda function (from within the project directory) 96 | 97 | lambder functions deploy 98 | 99 | Invoke the Lambda in AWS (from within the project directory) 100 | 101 | lambder functions invoke 102 | 103 | Invoke the function with input (from within the project directory) 104 | 105 | lambder functions invoke --input input/ping.json 106 | 107 | List all functions 108 | 109 | lambder functions list 110 | 111 | Delete a function (from within the project directory) 112 | 113 | lambder functions rm 114 | 115 | ## Sample Lambda Functions 116 | 117 | * https://github.com/LeafSoftware/lambder-create-images 118 | * https://github.com/LeafSoftware/lambder-start-instances 119 | * https://github.com/LeafSoftware/lambder-stop-instances 120 | * https://github.com/LeafSoftware/lambder-prune-snapshots 121 | * https://github.com/LeafSoftware/lambder-create-snapshots 122 | * https://github.com/LeafSoftware/lambder-replicate-snapshots 123 | * https://github.com/LeafSoftware/lambder-create-rds-snapshots 124 | * https://github.com/LeafSoftware/lambder-implement-best-practices 125 | 126 | ## TODO: 127 | 128 | * add code to add site packages from virtualenvwrapper to zip 129 | * add lambda name autodetection to 'lambder events add' 130 | * add pagination where needed (lambda:list-functions) 131 | -------------------------------------------------------------------------------- /lambder/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LeafSoftware/python-lambder/1c50b5dd2af286286e1547ee87d815d66382b884/lambder/__init__.py -------------------------------------------------------------------------------- /lambder/cli.py: -------------------------------------------------------------------------------- 1 | import click 2 | import json 3 | import os 4 | from lambder import Lambder, Entry 5 | 6 | lambder = Lambder() 7 | 8 | 9 | @click.group() 10 | def cli(): 11 | pass 12 | 13 | 14 | @cli.group() 15 | def events(): 16 | """ Manage scheduled events """ 17 | pass 18 | 19 | 20 | # lambder events list 21 | @events.command() 22 | def list(): 23 | """ List all events """ 24 | entries = lambder.list_events() 25 | for e in entries: 26 | click.echo(str(e)) 27 | 28 | 29 | # lambder events add 30 | @events.command() 31 | @click.option('--name', help='unique name for entry') 32 | @click.option('--function-name', help='AWS Lambda name') 33 | @click.option("--cron", help='cron expression') 34 | def add(name, function_name, cron): 35 | """ Create an event """ 36 | lambder.add_event(name=name, function_name=function_name, cron=cron) 37 | 38 | 39 | # lambder events rm 40 | @events.command() 41 | @click.option('--name', help='event to remove') 42 | def rm(name): 43 | """ Remove an existing entry """ 44 | lambder.delete_event(name) 45 | 46 | 47 | # lambder events disable 48 | @events.command() 49 | @click.option('--name', help='event to disable') 50 | def disable(name): 51 | """ Disable an event """ 52 | lambder.disable_event(name) 53 | 54 | 55 | # lambder events enable 56 | @events.command() 57 | @click.option('--name', help='event to enable') 58 | def enable(name): 59 | """ Enable a disabled event """ 60 | lambder.enable_event(name) 61 | 62 | 63 | # lambder events load 64 | @events.command() 65 | @click.option('--file', help='json file containing events to load') 66 | def load(file): 67 | """ Load events from a json file """ 68 | with open(file, 'r') as f: 69 | contents = f.read() 70 | lambder.load_events(contents) 71 | 72 | 73 | class FunctionConfig: 74 | def __init__(self, config_file): 75 | with open(config_file, 'r') as f: 76 | contents = f.read() 77 | config = json.loads(contents) 78 | self.name = config['name'] 79 | self.bucket = config['s3_bucket'] 80 | self.timeout = config['timeout'] 81 | self.memory = config['memory'] 82 | self.description = config['description'] 83 | self.subnet_ids = None 84 | self.security_group_ids = None 85 | 86 | if 'subnet_ids' in config: 87 | self.subnet_ids = config['subnet_ids'] 88 | if 'security_group_ids' in config: 89 | self.security_group_ids = config['security_group_ids'] 90 | 91 | 92 | @cli.group() 93 | @click.pass_context 94 | def functions(context): 95 | """ Manage AWS Lambda functions """ 96 | # find lambder.json in CWD 97 | config_file = "./lambder.json" 98 | if os.path.isfile(config_file): 99 | context.obj = FunctionConfig(config_file) 100 | pass 101 | 102 | 103 | # lambder functions list 104 | @functions.command() 105 | def list(): 106 | """ List lambder functions """ 107 | functions = lambder.list_functions() 108 | output = json.dumps( 109 | functions, 110 | sort_keys=True, 111 | indent=4, 112 | separators=(',', ':') 113 | ) 114 | click.echo(output) 115 | 116 | 117 | # lambder functions new 118 | @functions.command() 119 | @click.option('--name', help='name of the function') 120 | @click.option( 121 | '--bucket', 122 | help='S3 bucket used to deploy function', 123 | default='mybucket' 124 | ) 125 | @click.option('--timeout', help='function timeout in seconds') 126 | @click.option('--memory', help='function memory') 127 | @click.option('--description', help='function description') 128 | @click.option('--subnet-ids', help='comma-separated list of VPC subnet ids') 129 | @click.option( 130 | '--security-group-ids', 131 | help='comma-separated list of VPC security group ids' 132 | ) 133 | def new( 134 | name, 135 | bucket, 136 | timeout, 137 | memory, 138 | description, 139 | subnet_ids, 140 | security_group_ids 141 | ): 142 | """ Create a new lambda project """ 143 | config = {} 144 | if timeout: 145 | config['timeout'] = timeout 146 | if memory: 147 | config['memory'] = memory 148 | if description: 149 | config['description'] = description 150 | if subnet_ids: 151 | config['subnet_ids'] = subnet_ids 152 | if security_group_ids: 153 | config['security_group_ids'] = security_group_ids 154 | 155 | lambder.create_project(name, bucket, config) 156 | 157 | 158 | # lambder functions deploy 159 | @functions.command() 160 | @click.option('--name', help='name of the function') 161 | @click.option('--bucket', help='destination s3 bucket') 162 | @click.option('--timeout', help='function timeout in seconds') 163 | @click.option('--memory', help='function memory') 164 | @click.option('--description', help='function description') 165 | @click.option('--subnet-ids', help='comma-separated list of VPC subnet ids') 166 | @click.option( 167 | '--security-group-ids', 168 | help='comma-separated list of VPC security group ids' 169 | ) 170 | @click.pass_obj 171 | def deploy( 172 | config, 173 | name, 174 | bucket, 175 | timeout, 176 | memory, 177 | description, 178 | subnet_ids, 179 | security_group_ids 180 | ): 181 | """ Deploy/Update a function from a project directory """ 182 | # options should override config if it is there 183 | myname = name or config.name 184 | mybucket = bucket or config.bucket 185 | mytimeout = timeout or config.timeout 186 | mymemory = memory or config.memory 187 | mydescription = description or config.description 188 | mysubnet_ids = subnet_ids or config.subnet_ids 189 | mysecurity_group_ids = security_group_ids or config.security_group_ids 190 | 191 | vpc_config = {} 192 | if mysubnet_ids and mysecurity_group_ids: 193 | vpc_config = { 194 | 'SubnetIds': mysubnet_ids.split(','), 195 | 'SecurityGroupIds': mysecurity_group_ids.split(',') 196 | } 197 | 198 | click.echo('Deploying {} to {}'.format(myname, mybucket)) 199 | lambder.deploy_function( 200 | myname, 201 | mybucket, 202 | mytimeout, 203 | mymemory, 204 | mydescription, 205 | vpc_config 206 | ) 207 | 208 | 209 | # lambder functions rm 210 | @functions.command() 211 | @click.option('--name', help='name of the function') 212 | @click.option('--bucket', help='s3 bucket containing function code') 213 | @click.pass_obj 214 | def rm(config, name, bucket): 215 | """ Delete lambda function, role, and zipfile """ 216 | # options should override config if it is there 217 | myname = name or config.name 218 | mybucket = bucket or config.bucket 219 | 220 | click.echo('Deleting {} from {}'.format(myname, mybucket)) 221 | lambder.delete_function(myname, mybucket) 222 | 223 | 224 | # lambder functions invoke 225 | @functions.command() 226 | @click.option('--name', help='name of the function') 227 | @click.option('--input', help='json file containing input event') 228 | @click.pass_obj 229 | def invoke(config, name, input): 230 | """ Invoke function in AWS """ 231 | # options should override config if it is there 232 | myname = name or config.name 233 | 234 | click.echo('Invoking ' + myname) 235 | output = lambder.invoke_function(myname, input) 236 | click.echo(output) 237 | -------------------------------------------------------------------------------- /lambder/lambder.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import botocore.session 3 | import json 4 | from cookiecutter.main import cookiecutter 5 | import os 6 | import zipfile 7 | import tempfile 8 | import time 9 | 10 | 11 | class Entry: 12 | name = None 13 | cron = None 14 | function_name = None 15 | input_event = {} 16 | enabled = True 17 | 18 | def __init__( 19 | self, 20 | Name=None, 21 | Cron=None, 22 | FunctionName=None, 23 | InputEvent={}, 24 | Enabled=True 25 | ): 26 | self.name = Name 27 | self.cron = Cron 28 | self.function_name = FunctionName 29 | self.input_event = InputEvent 30 | self.enabled = Enabled 31 | 32 | def __str__(self): 33 | return "\t".join([ 34 | self.name, 35 | self.cron, 36 | self.function_name, 37 | str(self.enabled) 38 | ]) 39 | 40 | 41 | class Lambder: 42 | NAME_PREFIX = 'Lambder-' 43 | 44 | def __init__(self): 45 | self.awslambda = boto3.client('lambda') 46 | session = botocore.session.get_session() 47 | self.events = session.create_client('events') 48 | 49 | def permit_rule_to_invoke_function(self, rule_arn, function_name): 50 | statement_id = function_name + "RulePermission" 51 | resp = self.awslambda.add_permission( 52 | FunctionName=function_name, 53 | StatementId=statement_id, 54 | Action='lambda:InvokeFunction', 55 | Principal='events.amazonaws.com', 56 | SourceArn=rule_arn 57 | ) 58 | 59 | def add_event( 60 | self, 61 | name, 62 | function_name, 63 | cron, 64 | input_event={}, 65 | enabled=True 66 | ): 67 | rule_name = self.NAME_PREFIX + name 68 | 69 | # events:put-rule 70 | resp = self.events.put_rule( 71 | Name=rule_name, 72 | ScheduleExpression=cron 73 | ) 74 | rule_arn = resp['RuleArn'] 75 | 76 | # try to add the permission, if we fail because it already 77 | # exists, move on. 78 | try: 79 | self.permit_rule_to_invoke_function(rule_arn, function_name) 80 | except botocore.exceptions.ClientError as e: 81 | if e.response['Error']['Code'] != 'ResourceConflictException': 82 | raise 83 | 84 | # retrieve the lambda arn 85 | resp = self.awslambda.get_function( 86 | FunctionName=function_name 87 | ) 88 | 89 | function_arn = resp['Configuration']['FunctionArn'] 90 | 91 | # events:put-targets (needs lambda arn) 92 | resp = self.events.put_targets( 93 | Rule=rule_name, 94 | Targets=[ 95 | { 96 | 'Id': name, 97 | 'Arn': function_arn, 98 | 'Input': json.dumps(input_event) 99 | } 100 | ] 101 | ) 102 | 103 | def list_events(self): 104 | # List all rules by prefix 'Lambder' 105 | resp = self.events.list_rules( 106 | NamePrefix=self.NAME_PREFIX 107 | ) 108 | 109 | rules = resp['Rules'] 110 | 111 | entries = [] 112 | 113 | # For each rule, list-targets-by-rule to get lambda arn 114 | for rule in rules: 115 | resp = self.events.list_targets_by_rule( 116 | Rule=rule['Name'] 117 | ) 118 | targets = resp['Targets'] 119 | # assume only one target for now 120 | name = targets[0]['Id'] 121 | arn = targets[0]['Arn'] 122 | function_name = arn.split(':')[-1] 123 | cron = rule['ScheduleExpression'] 124 | enabled = rule['State'] == 'ENABLED' 125 | 126 | entry = Entry( 127 | Name=name, 128 | Cron=cron, 129 | FunctionName=function_name, 130 | Enabled=enabled 131 | ) 132 | entries.append(entry) 133 | 134 | return entries 135 | 136 | def delete_event(self, name): 137 | rule_name = self.NAME_PREFIX + name 138 | 139 | # get the function name 140 | resp = self.events.list_targets_by_rule( 141 | Rule=rule_name 142 | ) 143 | targets = resp['Targets'] 144 | # assume only one target for now 145 | arn = targets[0]['Arn'] 146 | function_name = arn.split(':')[-1] 147 | statement_id = function_name + "RulePermission" 148 | 149 | # delete the target 150 | resp = self.events.remove_targets( 151 | Rule=rule_name, 152 | Ids=[name] 153 | ) 154 | 155 | # delete the permission 156 | resp = self.awslambda.remove_permission( 157 | FunctionName=function_name, 158 | StatementId=statement_id 159 | ) 160 | 161 | # delete the rule 162 | resp = self.events.delete_rule( 163 | Name=rule_name 164 | ) 165 | 166 | def disable_event(self, name): 167 | rule_name = self.NAME_PREFIX + name 168 | resp = self.events.disable_rule( 169 | Name=rule_name 170 | ) 171 | 172 | def enable_event(self, name): 173 | rule_name = self.NAME_PREFIX + name 174 | resp = self.events.enable_rule( 175 | Name=rule_name 176 | ) 177 | 178 | def load_events(self, data): 179 | entries = json.loads(data) 180 | for entry in entries: 181 | self.add( 182 | name=entry['name'], 183 | cron=entry['cron'], 184 | function_name=entry['function_name'], 185 | input_event=entry['input_event'], 186 | enabled=entry['enabled'] 187 | ) 188 | 189 | def create_project(self, name, bucket, config): 190 | context = { 191 | 'lambda_name': name, 192 | 'repo_name': 'lambder-' + name, 193 | 's3_bucket': bucket 194 | } 195 | context.update(config) 196 | 197 | cookiecutter( 198 | 'https://github.com/LeafSoftware/cookiecutter-lambder', 199 | no_input=True, 200 | extra_context=context 201 | ) 202 | 203 | # Recursively zip path, creating a zipfile with contents 204 | # relative to path. 205 | # e.g. lambda/foo/foo.py -> ./foo.py 206 | # e.g. lambda/foo/bar/bar.py -> ./bar/bar.py 207 | # 208 | def _zipdir(self, zfile, path): 209 | with zipfile.ZipFile(zfile, 'w') as ziph: 210 | for root, dirs, files in os.walk(path): 211 | 212 | # strip path from beginning of full path 213 | rel_path = root 214 | if rel_path.startswith(path): 215 | rel_path = rel_path[len(path):] 216 | 217 | for file in files: 218 | ziph.write( 219 | os.path.join(root, file), 220 | os.path.join(rel_path, file) 221 | ) 222 | 223 | def _s3_cp(self, src, dest_bucket, dest_key): 224 | s3 = boto3.client('s3') 225 | s3.upload_file(src, dest_bucket, dest_key) 226 | 227 | def _s3_rm(self, bucket, key): 228 | s3 = boto3.resource('s3') 229 | the_bucket = s3.Bucket(bucket) 230 | the_object = the_bucket.Object(key) 231 | the_object.delete() 232 | 233 | def _create_lambda_role(self, role_name): 234 | iam = boto3.resource('iam') 235 | role = iam.Role(role_name) 236 | # return the role if it already exists 237 | if role in iam.roles.all(): 238 | return role 239 | 240 | trust_policy = json.dumps( 241 | { 242 | "Statement": [ 243 | { 244 | "Effect": "Allow", 245 | "Principal": { 246 | "Service": ["lambda.amazonaws.com"] 247 | }, 248 | "Action": ["sts:AssumeRole"] 249 | } 250 | ] 251 | } 252 | ) 253 | 254 | role = iam.create_role( 255 | RoleName=role_name, 256 | AssumeRolePolicyDocument=trust_policy 257 | ) 258 | return role 259 | 260 | def _delete_lambda_role(self, name): 261 | iam = boto3.resource('iam') 262 | 263 | role_name = self._role_name(name) 264 | policy_name = self._policy_name(name) 265 | 266 | role_policy = iam.RolePolicy(role_name, policy_name) 267 | role = iam.Role(self._role_name(name)) 268 | 269 | # HACK: This 'if thing in things.all()' biz seems like 270 | # a very inefficient way to check for resource 271 | # existence... 272 | if role_policy in role.policies.all(): 273 | role_policy.delete() 274 | 275 | if role in iam.roles.all(): 276 | role.delete() 277 | 278 | def _put_role_policy(self, role, policy_name, policy_doc): 279 | iam = boto3.client('iam') 280 | policy = iam.put_role_policy( 281 | RoleName=role.name, 282 | PolicyName=policy_name, 283 | PolicyDocument=policy_doc 284 | ) 285 | 286 | def _attach_vpc_policy(self, role): 287 | iam = boto3.client('iam') 288 | iam.attach_role_policy( 289 | RoleName=role, 290 | PolicyArn='arn:aws:iam::aws:policy/service-role/\ 291 | AWSLambdaVPCAccessExecutionRole' 292 | ) 293 | 294 | def _lambda_exists(self, name): 295 | awslambda = boto3.client('lambda') 296 | try: 297 | resp = awslambda.get_function( 298 | FunctionName=self._long_name(name) 299 | ) 300 | except botocore.exceptions.ClientError as e: 301 | if e.response['Error']['Code'] == 'ResourceNotFoundException': 302 | return False 303 | else: 304 | raise 305 | 306 | return True 307 | 308 | def _update_lambda( 309 | self, 310 | name, 311 | bucket, 312 | key, 313 | timeout, 314 | memory, 315 | description, 316 | vpc_config 317 | ): 318 | awslambda = boto3.client('lambda') 319 | resp = awslambda.update_function_code( 320 | FunctionName=self._long_name(name), 321 | S3Bucket=bucket, 322 | S3Key=key 323 | ) 324 | 325 | resp = awslambda.update_function_configuration( 326 | FunctionName=self._long_name(name), 327 | Timeout=timeout, 328 | MemorySize=memory, 329 | Description=description, 330 | VpcConfig=vpc_config 331 | ) 332 | 333 | def _create_lambda( 334 | self, 335 | name, 336 | bucket, 337 | key, 338 | role_arn, 339 | timeout, 340 | memory, 341 | description, 342 | vpc_config 343 | ): 344 | awslambda = boto3.client('lambda') 345 | resp = awslambda.create_function( 346 | FunctionName=self._long_name(name), 347 | Runtime='python2.7', 348 | Role=role_arn, 349 | Handler="{}.handler".format(name), 350 | Code={ 351 | 'S3Bucket': bucket, 352 | 'S3Key': key 353 | }, 354 | Timeout=timeout, 355 | MemorySize=memory, 356 | Description=description, 357 | VpcConfig=vpc_config 358 | ) 359 | 360 | def _delete_lambda(self, name): 361 | awslambda = boto3.client('lambda') 362 | if self._lambda_exists(name): 363 | resp = awslambda.delete_function( 364 | FunctionName=self._long_name(name) 365 | ) 366 | 367 | def _long_name(self, name): 368 | return 'Lambder-' + name 369 | 370 | def _s3_key(self, name): 371 | return "lambder/lambdas/{}_lambda.zip".format(name) 372 | 373 | def _role_name(self, name): 374 | return self._long_name(name) + 'ExecuteRole' 375 | 376 | def _policy_name(self, name): 377 | return self._long_name(name) + 'ExecutePolicy' 378 | 379 | def deploy_function( 380 | self, 381 | name, 382 | bucket, 383 | timeout, 384 | memory, 385 | description, 386 | vpc_config 387 | ): 388 | long_name = self._long_name(name) 389 | s3_key = self._s3_key(name) 390 | role_name = self._role_name(name) 391 | policy_name = self._policy_name(name) 392 | policy_file = os.path.join('iam', 'policy.json') 393 | 394 | # zip up the lambda 395 | zfile = os.path.join( 396 | tempfile.gettempdir(), 397 | "{}_lambda.zip".format(name) 398 | ) 399 | self._zipdir(zfile, os.path.join('lambda', name)) 400 | 401 | # upload it to s3 402 | self._s3_cp(zfile, bucket, s3_key) 403 | 404 | # remove tempfile 405 | os.remove(zfile) 406 | 407 | # create the lambda execute role if it does not already exist 408 | role = self._create_lambda_role(role_name) 409 | 410 | # update the role's policy from the document in the project 411 | policy_doc = None 412 | with open(policy_file, 'r') as f: 413 | policy_doc = f.read() 414 | 415 | self._put_role_policy(role, policy_name, policy_doc) 416 | 417 | # add the vpc policy to the role if vpc_config is set 418 | if vpc_config: 419 | self._attach_vpc_policy(role_name) 420 | 421 | # create or update the lambda function 422 | timeout_i = int(timeout) 423 | if self._lambda_exists(name): 424 | self._update_lambda( 425 | name, 426 | bucket, 427 | s3_key, 428 | timeout_i, 429 | memory, 430 | description, 431 | vpc_config 432 | ) 433 | else: 434 | time.sleep(5) # wait for role to be created 435 | self._create_lambda( 436 | name, 437 | bucket, 438 | s3_key, 439 | role.arn, 440 | timeout_i, 441 | memory, 442 | description, 443 | vpc_config 444 | ) 445 | 446 | # List only the lambder functions, i.e. ones starting with 'Lambder-' 447 | def list_functions(self): 448 | awslambda = boto3.client('lambda') 449 | resp = awslambda.list_functions() 450 | functions = resp['Functions'] 451 | return filter( 452 | lambda x: x['FunctionName'].startswith(self.NAME_PREFIX), 453 | functions 454 | ) 455 | 456 | def _delete_lambda_zip(self, name, bucket): 457 | key = self._s3_key(name) 458 | self._s3_rm(bucket, key) 459 | 460 | # delete all the things associated with this function 461 | def delete_function(self, name, bucket): 462 | self._delete_lambda(name) 463 | self._delete_lambda_role(name) 464 | self._delete_lambda_zip(name, bucket) 465 | 466 | def invoke_function(self, name, input_event): 467 | awslambda = boto3.client('lambda') 468 | payload = '{}' # default to empty event 469 | 470 | if input_event: 471 | with open(input_event, 'r') as f: 472 | payload = f.read() 473 | 474 | resp = awslambda.invoke( 475 | FunctionName=self._long_name(name), 476 | InvocationType='RequestResponse', 477 | Payload=payload 478 | ) 479 | results = resp['Payload'].read() # payload is a 'StreamingBody' 480 | return results 481 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | awslogs==0.2.0 2 | binaryornot==0.4.0 3 | boto3==1.2.6 4 | botocore==1.4.0 5 | chardet==2.3.0 6 | click==6.2 7 | cookiecutter==1.3.0 8 | docutils==0.12 9 | future==0.15.2 10 | futures==3.0.5 11 | Jinja2==2.8 12 | jmespath==0.9.0 13 | MarkupSafe==0.23 14 | pipsi==0.9 15 | python-dateutil==2.5.0 16 | ruamel.base==1.0.0 17 | ruamel.ordereddict==0.4.9 18 | ruamel.yaml==0.10.15 19 | six==1.10.0 20 | termcolor==1.1.0 21 | virtualenv==14.0.0 22 | wheel==0.24.0 23 | whichcraft==0.1.1 24 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | Creates and manages scheduled AWS Lambdas 3 | """ 4 | from setuptools import find_packages, setup 5 | 6 | dependencies = [ 7 | 'click>=6.2', 8 | 'boto3>=1.2.6', 9 | 'botocore>=1.4.0', 10 | 'cookiecutter>=1.3.0' 11 | ] 12 | 13 | setup( 14 | name='lambder', 15 | version='1.2.2', 16 | url='https://github.com/LeafSoftware/python-lambder', 17 | license='MIT', 18 | author='Chris Chalfant', 19 | author_email='cchalfant@leafsoftwaresolutions.com', 20 | description='Creates and manages scheduled AWS Lambdas', 21 | long_description=__doc__, 22 | packages=find_packages(exclude=['tests']), 23 | include_package_data=True, 24 | zip_safe=False, 25 | platforms='any', 26 | install_requires=dependencies, 27 | entry_points={ 28 | 'console_scripts': [ 29 | 'lambder = lambder.cli:cli', 30 | ], 31 | }, 32 | classifiers=[ 33 | # As from http://pypi.python.org/pypi?%3Aaction=list_classifiers 34 | # 'Development Status :: 1 - Planning', 35 | # 'Development Status :: 2 - Pre-Alpha', 36 | # 'Development Status :: 3 - Alpha', 37 | 'Development Status :: 4 - Beta', 38 | # 'Development Status :: 5 - Production/Stable', 39 | # 'Development Status :: 6 - Mature', 40 | # 'Development Status :: 7 - Inactive', 41 | 'Environment :: Console', 42 | 'Intended Audience :: Developers', 43 | 'License :: OSI Approved :: MIT License', 44 | 'Operating System :: POSIX', 45 | 'Operating System :: MacOS', 46 | 'Operating System :: Unix', 47 | 'Operating System :: Microsoft :: Windows', 48 | 'Programming Language :: Python', 49 | 'Programming Language :: Python :: 2', 50 | 'Programming Language :: Python :: 3', 51 | 'Topic :: Software Development :: Libraries :: Python Modules', 52 | ] 53 | ) 54 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /tests/test_cli.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | from click.testing import CliRunner 3 | from lambder import cli 4 | 5 | 6 | @pytest.fixture 7 | def runner(): 8 | return CliRunner() 9 | 10 | 11 | def test_cli(runner): 12 | result = runner.invoke(cli.main) 13 | assert result.exit_code == 0 14 | assert not result.exception 15 | assert result.output.strip() == 'Hello, world.' 16 | 17 | 18 | def test_cli_with_option(runner): 19 | result = runner.invoke(cli.main, ['--as-cowboy']) 20 | assert not result.exception 21 | assert result.exit_code == 0 22 | assert result.output.strip() == 'Howdy, world.' 23 | 24 | 25 | def test_cli_with_arg(runner): 26 | result = runner.invoke(cli.main, ['Chris']) 27 | assert result.exit_code == 0 28 | assert not result.exception 29 | assert result.output.strip() == 'Hello, Chris.' 30 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist=py26, py27, py33, py34, pypy, flake8 3 | 4 | [testenv] 5 | commands=py.test --cov lambder {posargs} 6 | deps= 7 | pytest 8 | pytest-cov 9 | 10 | [testenv:flake8] 11 | basepython = python2.7 12 | deps = 13 | flake8 14 | commands = 15 | flake8 lambder tests --max-line-length=120 16 | --------------------------------------------------------------------------------