├── Chapter02 ├── IAM │ ├── assume-role-lambda.json │ ├── dynamo-full-user-visits.json │ ├── dynamo-readonly-user-visits.json │ ├── iam-pass-role.json │ └── lambda-cloud-write.json ├── README.md ├── aws_dynamo │ ├── __init__.py │ ├── dynamo_insert_items_from_file.py │ ├── dynamo_modify_items.py │ ├── dynamo_query_table.py │ └── dynamo_table_creation.py ├── bash │ └── apigateway-lambda-dynamodb │ │ ├── build-package-deploy-lambda-dynamo-data-api.sh │ │ ├── common-variables.sh │ │ ├── create-lambda-package.sh │ │ ├── create-role.sh │ │ ├── curl-api-gateway.sh │ │ ├── delete-stack.sh │ │ ├── get_apigateway_endpoint.py │ │ ├── get_apigateway_id.py │ │ ├── invoke-lambda.sh │ │ ├── lambda-dynamo-data-api.yaml │ │ ├── run_locus.sh │ │ └── unit-test-lambda.sh ├── images │ ├── building-scalable-serverless-microservice-rest-data-api-video.png │ └── implementing-serverless-microservices-architecture-patterns-video.png ├── lambda_dynamo_read │ ├── __init__.py │ └── lambda_return_dynamo_records.py ├── requirements.txt ├── sample_data │ ├── dynamodb-sample-data.txt │ ├── request-api-gateway-get-error.json │ ├── request-api-gateway-get-valid-date.json │ ├── request-api-gateway-get-valid-no-date.json │ └── request-api-gateway-get-valid.json └── test │ ├── locust_test_api.py │ ├── run_local_api_gateway_lambda_dynamo.py │ └── test_dynamo_get.py ├── Chapter03 ├── IAM │ ├── assume-role-lambda.json │ ├── dynamo-full-user-visits.json │ ├── dynamo-readonly-user-visits.json │ ├── iam-pass-role.json │ └── lambda-cloud-write.json ├── README.md ├── aws_dynamo │ ├── __init__.py │ ├── dynamo_insert_items_from_file.py │ ├── dynamo_modify_items.py │ ├── dynamo_query_table.py │ └── dynamo_table_creation.py ├── bash │ └── apigateway-lambda-dynamodb │ │ ├── build-package-deploy-lambda-dynamo-data-api.sh │ │ ├── common-variables.sh │ │ ├── create-lambda-package.sh │ │ ├── create-role.sh │ │ ├── curl-api-gateway.sh │ │ ├── delete-stack.sh │ │ ├── get_apigateway_endpoint.py │ │ ├── get_apigateway_id.py │ │ ├── invoke-lambda.sh │ │ ├── lambda-dynamo-data-api.yaml │ │ ├── run_locus.sh │ │ └── unit-test-lambda.sh ├── images │ ├── building-scalable-serverless-microservice-rest-data-api-video.png │ └── implementing-serverless-microservices-architecture-patterns-video.png ├── lambda_dynamo_read │ ├── __init__.py │ └── lambda_return_dynamo_records.py ├── requirements.txt ├── sample_data │ ├── dynamodb-sample-data.txt │ ├── request-api-gateway-get-error.json │ ├── request-api-gateway-get-valid-date.json │ ├── request-api-gateway-get-valid-no-date.json │ └── request-api-gateway-get-valid.json └── test │ ├── locust_test_api.py │ ├── run_local_api_gateway_lambda_dynamo.py │ └── test_dynamo_get.py ├── Chapter04 ├── IAM │ ├── assume-role-lambda.json │ ├── dynamo-full-user-visits.json │ ├── dynamo-readonly-user-visits.json │ ├── iam-pass-role.json │ └── lambda-cloud-write.json ├── README.md ├── aws_dynamo │ ├── __init__.py │ ├── dynamo_insert_items_from_file.py │ ├── dynamo_modify_items.py │ ├── dynamo_query_table.py │ └── dynamo_table_creation.py ├── bash │ └── apigateway-lambda-dynamodb │ │ ├── build-package-deploy-lambda-dynamo-data-api.sh │ │ ├── common-variables.sh │ │ ├── create-lambda-package.sh │ │ ├── create-role.sh │ │ ├── curl-api-gateway.sh │ │ ├── delete-stack.sh │ │ ├── get_apigateway_endpoint.py │ │ ├── get_apigateway_id.py │ │ ├── invoke-lambda.sh │ │ ├── lambda-dynamo-data-api.yaml │ │ ├── run_locus.sh │ │ └── unit-test-lambda.sh ├── images │ ├── building-scalable-serverless-microservice-rest-data-api-video.png │ └── implementing-serverless-microservices-architecture-patterns-video.png ├── lambda_dynamo_read │ ├── __init__.py │ └── lambda_return_dynamo_records.py ├── requirements.txt ├── sample_data │ ├── dynamodb-sample-data.txt │ ├── request-api-gateway-get-error.json │ ├── request-api-gateway-get-valid-date.json │ ├── request-api-gateway-get-valid-no-date.json │ └── request-api-gateway-get-valid.json └── test │ ├── locust_test_api.py │ ├── run_local_api_gateway_lambda_dynamo.py │ └── test_dynamo_get.py ├── LICENSE └── README.md /Chapter02/IAM/assume-role-lambda.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "", 6 | "Effect": "Allow", 7 | "Principal": { 8 | "Service": "lambda.amazonaws.com" 9 | }, 10 | "Action": "sts:AssumeRole" 11 | } 12 | ] 13 | } -------------------------------------------------------------------------------- /Chapter02/IAM/dynamo-full-user-visits.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1422032676021", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "dynamodb:BatchGetItem", 9 | "dynamodb:BatchWriteItem", 10 | "dynamodb:DeleteItem", 11 | "dynamodb:DescribeStream", 12 | "dynamodb:DescribeTable", 13 | "dynamodb:GetItem", 14 | "dynamodb:GetRecords", 15 | "dynamodb:GetShardIterator", 16 | "dynamodb:ListStreams", 17 | "dynamodb:ListTables", 18 | "dynamodb:PutItem", 19 | "dynamodb:Query", 20 | "dynamodb:Scan", 21 | "dynamodb:UpdateItem", 22 | "dynamodb:UpdateTable" 23 | ], 24 | "Resource": [ 25 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits", 26 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-sam" 27 | ] 28 | }, 29 | { 30 | "Effect": "Allow", 31 | "Action": "dynamodb:ListTables", 32 | "Resource": "*", 33 | "Condition": {} 34 | } 35 | ] 36 | } -------------------------------------------------------------------------------- /Chapter02/IAM/dynamo-readonly-user-visits.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1422532676335", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "dynamodb:BatchGetItem", 9 | "dynamodb:DescribeTable", 10 | "dynamodb:GetItem", 11 | "dynamodb:Query", 12 | "dynamodb:Scan" 13 | ], 14 | "Resource": [ 15 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits", 16 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-sam", 17 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-xray-sam" 18 | ] 19 | } 20 | ] 21 | } 22 | -------------------------------------------------------------------------------- /Chapter02/IAM/iam-pass-role.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1449755587000", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "iam:GetInstanceProfile", 9 | "iam:GetRole", 10 | "iam:ListInstanceProfiles", 11 | "iam:ListInstanceProfilesForRole", 12 | "iam:PassRole" 13 | ], 14 | "Resource": [ 15 | "*" 16 | ] 17 | } 18 | ] 19 | } -------------------------------------------------------------------------------- /Chapter02/IAM/lambda-cloud-write.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Effect": "Allow", 6 | "Action": [ 7 | "logs:CreateLogGroup", 8 | "logs:CreateLogStream", 9 | "logs:PutLogEvents", 10 | "logs:DescribeLogStreams" 11 | ], 12 | "Resource": [ 13 | "arn:aws:logs:*:*:*" 14 | ] 15 | }, 16 | { 17 | "Effect": "Allow", 18 | "Action": [ 19 | "cloudwatch:PutMetricData" 20 | ], 21 | "Resource": "*" 22 | } 23 | ] 24 | } -------------------------------------------------------------------------------- /Chapter02/README.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"). 4 | You may not use this file except in compliance with the License. 5 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 6 | 7 | ## Serverless Microservice Data API 8 | 9 | In this repository, I share some sample code used to create a scalable serverless data API from my two video courses. It includes bash scripts that can be used to unit test, build, package, deploy, and run integration tests. The Python code is written defensively to deal with any API exception. 10 | 11 | |For beginners and intermediates, the [full Serverless Data API](https://www.packtpub.com/application-development/building-scalable-serverless-microservice-rest-data-api-video) code, configuration and a detailed walk through |For intermediate or advanced users, I cover [15+ serverless microservice patterns](https://www.packtpub.com/application-development/implementing-serverless-microservices-architecture-patterns-video) with original content, code, configuration and detailed walk through | 12 | |:----------|:-------------| 13 | | [![Building a Scalable Serverless Microservice REST Data API Video Course](./images/building-scalable-serverless-microservice-rest-data-api-video.png "Building a Scalable Serverless Microservice REST Data API Video Course")](https://www.packtpub.com/application-development/building-scalable-serverless-microservice-rest-data-api-video)| [![Implementing Serverless Microservices Architecture Patterns Video Course](./images/implementing-serverless-microservices-architecture-patterns-video.png "Implementing Serverless Microservices Architecture Patterns Video Course")](https://www.packtpub.com/application-development/implementing-serverless-microservices-architecture-patterns-video) | 14 | 15 | 16 | ### 1. Windows Only 17 | 18 | Please skip this step if you are not using Windows. 19 | 20 | Using Bash (Unix Shell) makes your life much easier when deploying and managing you serverless stack. I think all analysts, data scientists, architects, administrators, database administrators, developers, DevOps and technical people should know some basic Bash and be able to run shell scripts, which are typically used on LINUX and UNIX (including macOS Terminal). 21 | 22 | Alternatively you can adapt the scripts to use MS-DOS or Powershell but it's not something I recommended, given that Bash can now run natively on Windows 10 as an application, and there are many more examples online in Bash. 23 | 24 | Note that I have stripped off the `\r` or carriage returns, as they are illegal in shell scripts, you can use something like [notepad++](https://notepad-plus-plus.org/) on Windows if you want to view the carriage returns in your files properly. If you use traditional Windows Notepad the new lines may not be rendered at all, so use [Notepad++](https://notepad-plus-plus.org/), [Sublime](https://www.sublimetext.com/), [Atom](https://atom.io/) or other such editors. 25 | 26 | A detailed guide on how to install [Linux Bash shell on Windows 10 can be found here](https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/). The main steps are: 27 | * Control Panel > Programs > Turn Windows Features On Or Off. 28 | * Choose the check box next to the **Windows Subsystem for Linux** option in the list, and then Choose **OK**. 29 | * Under **Microsoft Store > Run Linux on Windows** Select **Ubuntu**. 30 | * Launch Ubuntu and setup a root account username and password 31 | The Windows `C:\` and other dives are already mounted, and you can access them with the following command in the terminal: 32 | ```bash 33 | $ cd /mnt/c/ 34 | ``` 35 | 36 | Well done you now have full access to Linux on Windows! 37 | 38 | ### 2. Update Ubuntu, Install Git and Clone Repository 39 | ```bash 40 | $ sudo apt-get update 41 | $ sudo apt-get -y upgrade 42 | $ apt-get install git-core 43 | ``` 44 | Clone the repository locally with a `git pull` 45 | 46 | ### 3. Install Python and Dependencies 47 | 48 | The Lambda code is written in Python 3.6. Pip is a tool for installing and managing Python packages. Other popular Python package and dependency managers are available such as [Conda](https://conda.io/docs/index.html) or [Pipenv](https://pipenv.readthedocs.io) but we will be using pip as it is the recommended tool for installing packages from the Python Package Index [PyPI](https://pypi.org/) and most widely supported. 49 | 50 | ```bash 51 | $ sudo apt -y install python3.6 52 | $ sudo apt -y install python3-pip 53 | ``` 54 | 55 | Check the Python version 56 | ```bash 57 | $ python3 --version 58 | ``` 59 | You should get the Python3.6+ version. 60 | 61 | The dependent packages required for running, testing and deploying the severless microservices are listed in `requirements.txt` under each project folder, and can be installed using pip. 62 | ```bash 63 | $ sudo pip3 install -r /path/to/requirements.txt 64 | ``` 65 | This will install the dependent libraries for local development such as [Boto3](https://boto3.amazonaws.com) which is the Python AWS Software Development Kit (SDK). 66 | 67 | ### 3. Install and Setup AWS CLI 68 | 69 | AWS Command Line Interface is used to package and deploy your Lambda functions, as well as setup the infrastructure and security in a repeatable way. 70 | 71 | ```bash 72 | $ sudo pip install awscli --upgrade 73 | ``` 74 | 75 | You have created a user called `newuser` earlier and have a `crednetials.csv` with the AWS keys. Enter them them running `aws configure`. 76 | 77 | ```bash 78 | $ aws configure 79 | AWS Access Key ID: 80 | AWS Secret Access Key: 81 | Default region name: 82 | Default output format: 83 | ``` 84 | 85 | More details on setting up the AWS CLI is available [in the AWS Docs](https://docs.aws.amazon.com/lambda/latest/dg/setup-awscli.html). 86 | 87 | For choosing your AWS Region refer to [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) generally those in the USA can use `us-east-1` and those in Europe can use `eu-west-1` 88 | 89 | ### 4. Update AccountId, Bucket and Profile 90 | Here I assume your AWS profile is `demo` you can change that under `bash/apigateway-lambda-dynamodb/common-variables.sh`. 91 | You will need to use a bucket or create one if you haven't already: 92 | ```bash 93 | $ aws s3api create-bucket --bucket mynewbucket231 --profile demo --create-bucket-configuration Loc 94 | ationConstraint=eu-west-1 --region eu-west-1 95 | 96 | ``` 97 | You will also need to change the AWS accountId (current set to 000000000000). The AWS accountId is also used in some IAM policies in the IAM folder. In addition the region will have to be changed. 98 | 99 | to replace your accountId (assume your AWS accountId is 111111111111) you can do it manually or run: 100 | ```bash 101 | find ./ -type f -exec sed -i '' -e 's/000000000000/111111111111/' {} \; 102 | ``` 103 | 104 | ### 5. Run Unit Tests 105 | Change directory to the main bash folder 106 | ```bash 107 | $ cd bash/apigateway-lambda-dynamodb/ 108 | $ ./unit-test-lambda.sh 109 | ``` 110 | 111 | ### 6. Build Package and Deploy the Serverless API 112 | This also creates the IAM Polices and IAM Roles using the AWS CLI that will be required for the Lambda function. 113 | ```bash 114 | $ ./build-package-deploy-lambda-dynamo-data-api.sh 115 | ``` 116 | In less than a minute you should have a stack. Otherwise look at the error messages, CloudFormation stack and ensure your credentials are setup correctly. 117 | 118 | ### 7. Run Lambda Integration Test 119 | Once the stack is up and running, you can run an integration test to check that the Lambda is working. 120 | ```bash 121 | ./invoke-lambda.sh 122 | ``` 123 | 124 | ### 8. Add data to DynamoDb table 125 | 126 | Change to the DynamoDB Python directory and run the Python code, you can also run this under you favourite IDE like PyDev or PyCharm. 127 | ```bash 128 | $ (cd ../../aws_dynamo; python dynamo_insert_items_from_file.py) 129 | ``` 130 | 131 | ### 9. AWS Management Console 132 | Now the stack is up you can have a look at the API Gateway in the AWS Management Console and test the API in your browser. 133 | * API Gateway > lambda-dynamo-data-api 134 | * Stages > Prod > Get 135 | * Copy the Invoke URL into a new tab, e.g. https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/Prod/visits/{resourceId} 136 | * You should get a message `resource_id not a number` as the ID is not valid 137 | * Replace {resourceId} in the URL with 324 138 | if all is working you should see some returned JSON records. Well done if so! 139 | 140 | ### 10. Deleting the stack 141 | Go back to the main bash folder and delete the cloudFormation serverless stack. 142 | 143 | ```bash 144 | $ ./delete-stack.sh 145 | ``` 146 | -------------------------------------------------------------------------------- /Chapter02/aws_dynamo/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter02/aws_dynamo/__init__.py -------------------------------------------------------------------------------- /Chapter02/aws_dynamo/dynamo_insert_items_from_file.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 20 Jan 2018 13 | 14 | @author: Richard Freeman 15 | 16 | This packages inserts records from a file into specified DynamoDB table 17 | 18 | """ 19 | import csv 20 | 21 | from boto3 import resource 22 | 23 | 24 | class DynamoRepository: 25 | def __init__(self, target_dynamo_table, region='eu-west-1'): 26 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 27 | self.target_dynamo_table = target_dynamo_table 28 | self.table = self.dynamodb.Table(self.target_dynamo_table) 29 | 30 | def update_dynamo_event_counter(self, event_name, event_datetime, event_count=1): 31 | response = self.table.update_item( 32 | Key={ 33 | 'EventId': str(event_name), 34 | 'EventDay': int(event_datetime) 35 | }, 36 | ExpressionAttributeValues={":eventCount": int(event_count)}, 37 | UpdateExpression="ADD EventCount :eventCount") 38 | return response 39 | 40 | 41 | def main(): 42 | # For manual deployment 43 | # table_name = 'user-visits' 44 | 45 | # For SAM: 46 | table_name = 'user-visits-sam' 47 | input_data_path = '../sample_data/dynamodb-sample-data.txt' 48 | dynamo_repo = DynamoRepository(table_name) 49 | with open(input_data_path, 'r') as sample_file: 50 | csv_reader = csv.DictReader(sample_file) 51 | for row in csv_reader: 52 | response = dynamo_repo.update_dynamo_event_counter(row['EventId'], 53 | row['EventDay'], 54 | row['EventCount']) 55 | print(response) 56 | 57 | 58 | if __name__ == '__main__': 59 | main() 60 | -------------------------------------------------------------------------------- /Chapter02/aws_dynamo/dynamo_modify_items.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 8 Jan 2018 13 | 14 | @author: Richard Freeman 15 | 16 | This packages inserts records into DynamoDB 17 | 18 | """ 19 | from boto3 import resource 20 | 21 | 22 | class DynamoRepository: 23 | def __init__(self, target_dynamo_table, region='eu-west-1'): 24 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 25 | self.target_dynamo_table = target_dynamo_table 26 | self.table = self.dynamodb.Table(self.target_dynamo_table) 27 | 28 | def update_dynamo_event_counter(self, event_name, event_datetime, event_count=1): 29 | return self.table.update_item( 30 | Key={ 31 | 'EventId': event_name, 32 | 'EventDay': event_datetime 33 | }, 34 | ExpressionAttributeValues={":eventCount": event_count}, 35 | UpdateExpression="ADD EventCount :eventCount") 36 | 37 | 38 | def main(): 39 | # For manual deployment 40 | # table_name = 'user-visits' 41 | 42 | # For SAM deployment 43 | table_name = 'user-visits-sam' 44 | dynamo_repo = DynamoRepository(table_name) 45 | print(dynamo_repo.update_dynamo_event_counter('324', 20171001)) 46 | print(dynamo_repo.update_dynamo_event_counter('324', 20171001, 2)) 47 | print(dynamo_repo.update_dynamo_event_counter('324', 20171002, 5)) 48 | 49 | 50 | if __name__ == '__main__': 51 | main() 52 | -------------------------------------------------------------------------------- /Chapter02/aws_dynamo/dynamo_query_table.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 8 Jan 2018 13 | 14 | @author: Richard Freeman 15 | This package is used to query DynamoDB 16 | """ 17 | import decimal 18 | import json 19 | 20 | from boto3 import resource 21 | from boto3.dynamodb.conditions import Key 22 | 23 | 24 | class DecimalEncoder(json.JSONEncoder): 25 | """Helper class to convert a DynamoDB item to JSON. 26 | """ 27 | def default(self, o): 28 | if isinstance(o, decimal.Decimal): 29 | if o % 1 > 0: 30 | return float(o) 31 | else: 32 | return int(o) 33 | return super(DecimalEncoder, self).default(o) 34 | 35 | 36 | class DynamoRepository: 37 | """abstracts all interactions with DynamoDB including the connection and querying of tables 38 | """ 39 | def __init__(self, target_dynamo_table, region='eu-west-1'): 40 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 41 | self.dynamo_table = target_dynamo_table 42 | self.table = self.dynamodb.Table(self.dynamo_table) 43 | 44 | def query_dynamo_record_by_parition(self, parition_key, parition_value): 45 | try: 46 | response = self.table.query( 47 | KeyConditionExpression=Key(parition_key).eq(parition_value)) 48 | for record in response.get('Items'): 49 | print(json.dumps(record, cls=DecimalEncoder)) 50 | return 51 | 52 | except Exception as e: 53 | print('Exception %s type' % str(type(e))) 54 | print('Exception message: %s ' % str(e)) 55 | 56 | def query_dynamo_record_by_parition_sort_key(self, partition_key, partition_value, 57 | sort_key, sort_value): 58 | try: 59 | response = self.table.query( 60 | KeyConditionExpression=Key(partition_key).eq(partition_value) 61 | & Key(sort_key).gte(sort_value)) 62 | for record in response.get('Items'): 63 | print(json.dumps(record, cls=DecimalEncoder)) 64 | return 65 | 66 | except Exception as e: 67 | print('Exception %s type' % str(type(e))) 68 | print('Exception message: %s ' % str(e)) 69 | 70 | 71 | def main(): 72 | # For manual deployment 73 | # table_name = 'user-visits' 74 | 75 | # For SAM: 76 | table_name = 'user-visits-sam' 77 | partition_key = 'EventId' 78 | partition_value = '324' 79 | sort_key = 'EventDay' 80 | sort_value = 20171001 81 | 82 | dynamo_repo = DynamoRepository(table_name) 83 | print('Reading all data for partition_key:%s' % partition_value) 84 | dynamo_repo.query_dynamo_record_by_parition(partition_key, partition_value) 85 | 86 | print('Reading all data for partition_key:%s with date > %d' % (partition_value, sort_value)) 87 | dynamo_repo.query_dynamo_record_by_parition_sort_key(partition_key, 88 | partition_value, 89 | sort_key, 90 | sort_value) 91 | 92 | 93 | if __name__ == '__main__': 94 | main() 95 | -------------------------------------------------------------------------------- /Chapter02/aws_dynamo/dynamo_table_creation.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | 13 | Created on 8 Jan 2018 14 | 15 | @author: Richard Freeman 16 | 17 | pip install boto3 18 | 19 | This package creates a new DynamoDb table with the specified read and write capacity 20 | 21 | """ 22 | import boto3 23 | 24 | 25 | def create_dynamo_table(table_name_value, enable_streams=False, 26 | read_capacity=1, 27 | write_capacity=1, 28 | region='eu-west-1'): 29 | table_name = table_name_value 30 | print('creating table: ' + table_name) 31 | try: 32 | client = boto3.client(service_name='dynamodb', region_name=region) 33 | print(client.create_table(TableName=table_name, 34 | AttributeDefinitions=[{'AttributeName': 'EventId', 35 | 'AttributeType': 'S'}, 36 | {'AttributeName': 'EventDay', 37 | 'AttributeType': 'N'}], 38 | KeySchema=[{'AttributeName': 'EventId', 39 | 'KeyType': 'HASH'}, 40 | {'AttributeName': 'EventDay', 41 | 'KeyType': 'RANGE'}, 42 | ], 43 | ProvisionedThroughput={'ReadCapacityUnits': read_capacity, 44 | 'WriteCapacityUnits': write_capacity})) 45 | except Exception as e: 46 | print('Exception %s type' % str(type(e))) 47 | print('Exception message: %s ' % str(e)) 48 | 49 | 50 | def main(): 51 | table_name = 'user-visits' 52 | create_dynamo_table(table_name, False, 1, 1) 53 | 54 | 55 | if __name__ == '__main__': 56 | main() 57 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/build-package-deploy-lambda-dynamo-data-api.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | # Variables 6 | . ./common-variables.sh 7 | 8 | # Run unit tests 9 | ./unit-test-lambda.sh 10 | 11 | #Create Zip file of your Lambda code (works on Windows and Linux) 12 | ./create-lambda-package.sh 13 | 14 | #Package your Serverless Stack using SAM + Cloudformation 15 | aws cloudformation package --template-file $template.yaml \ 16 | --output-template-file ../../package/$template-output.yaml \ 17 | --s3-bucket $bucket --s3-prefix $prefix \ 18 | --region $region --profile $profile 19 | 20 | #Deploy your Serverless Stack using SAM + Cloudformation 21 | aws cloudformation deploy --template-file ../../package/$template-output.yaml \ 22 | --stack-name $template --capabilities CAPABILITY_IAM \ 23 | --parameter-overrides AccountId=${aws_account_id} \ 24 | --region $region --profile $profile -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/common-variables.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | export profile="demo" 6 | export region="eu-west-1" 7 | # export aws_account_id=$(aws sts get-caller-identity --query 'Account' --profile $profile | tr -d '\"') 8 | export aws_account_id="000000000000" 9 | export template="lambda-dynamo-data-api" 10 | export bucket="testbucket121f" 11 | export prefix="tmp/sam" 12 | 13 | # Lambda settings 14 | export zip_file="lambda-dynamo-data-api.zip" 15 | export files="lambda_return_dynamo_records.py" 16 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/create-lambda-package.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | # This script creates a Zip package of the Lambda files 6 | 7 | #setup environment variables 8 | . ./common-variables.sh 9 | 10 | #Create Lambda package and exclude the tests to reduce package size 11 | (cd ../../lambda_dynamo_read; 12 | mkdir -p ../package/ 13 | zip -FSr ../package/"${zip_file}" ${files} -x *tests/*) 14 | 15 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/create-role.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #This Script creates a Lambda role and attaches the policies 6 | 7 | #setup environment variables 8 | . ./common-variables.sh 9 | 10 | #Setup Lambda Role 11 | role_name=lambda-dynamo-data-api 12 | aws iam create-role --role-name ${role_name} \ 13 | --assume-role-policy-document file://../../IAM/assume-role-lambda.json \ 14 | --profile $profile || true 15 | 16 | sleep 1 17 | #Add and attach DynamoDB Policy 18 | dynamo_policy=dynamo-readonly-user-visits 19 | aws iam create-policy --policy-name $dynamo_policy \ 20 | --policy-document file://../../IAM/$dynamo_policy.json \ 21 | --profile $profile || true 22 | 23 | role_policy_arn="arn:aws:iam::$aws_account_id:policy/$dynamo_policy" 24 | aws iam attach-role-policy \ 25 | --role-name "${role_name}" \ 26 | --policy-arn "${role_policy_arn}" --profile ${profile} || true 27 | 28 | #Add and attach cloudwatch_policy 29 | cloudwatch_policy=lambda-cloud-write 30 | aws iam create-policy --policy-name $cloudwatch_policy \ 31 | --policy-document file://../../IAM/$cloudwatch_policy.json \ 32 | --profile $profile || true 33 | 34 | role_policy_arn="arn:aws:iam::$aws_account_id:policy/$cloudwatch_policy" 35 | aws iam attach-role-policy \ 36 | --role-name "${role_name}" \ 37 | --policy-arn "${role_policy_arn}" --profile ${profile} || true 38 | 39 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/curl-api-gateway.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #endpoint="https://xxxxx.execute-api.eu-west-1.amazonaws.com/Prod/visits/324" 6 | . ./common-variables.sh 7 | 8 | endpoint="$(python3 get_apigateway_endpoint.py -e ${template})" 9 | echo ${endpoint} 10 | status_code=$(curl -i -H \"Accept: application/json\" -H \"Content-Type: application/json\" -X GET ${endpoint}) 11 | echo "$status_code" 12 | if echo "$status_code" | grep -q "HTTP/1.1 200 OK"; 13 | then 14 | echo "pass" 15 | exit 0 16 | else 17 | exit 1 18 | fi 19 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/delete-stack.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | . ./common-variables.sh 6 | 7 | aws cloudformation delete-stack --stack-name $template --region $region --profile $profile 8 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/get_apigateway_endpoint.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 23 Dec 2018 13 | @author: Richard Freeman 14 | 15 | This module gets an API Gateway endpoint based on the name 16 | sudo python3.6 -m pip install boto3 17 | 18 | """ 19 | 20 | import argparse 21 | import logging 22 | 23 | import boto3 24 | logging.getLogger('botocore').setLevel(logging.CRITICAL) 25 | 26 | logger = logging.getLogger(__name__) 27 | logging.basicConfig(format='%(asctime)s %(levelname)s %(name)-15s: %(lineno)d %(message)s', 28 | level=logging.INFO) 29 | logger.setLevel(logging.INFO) 30 | 31 | 32 | def get_apigateway_names(endpoint_name): 33 | client = boto3.client(service_name='apigateway', region_name='eu-west-1') 34 | apis = client.get_rest_apis() 35 | for api in apis['items']: 36 | if api['name'] == endpoint_name: 37 | api_id = api['id'] 38 | region = 'eu-west-1' 39 | stage = 'Prod' 40 | resource = 'visits/324' 41 | #return F"https://{api_id}.execute-api.{region}.amazonaws.com/{stage}/{resource}" 42 | return "https://%s.execute-api.%s.amazonaws.com/%s/%s" % (api_id, region, stage, resource) 43 | return None 44 | 45 | 46 | def main(): 47 | endpoint_name = "lambda-dynamo-xray" 48 | 49 | parser = argparse.ArgumentParser() 50 | parser.add_argument("-e", "--endpointname", type=str, required=False, help="Path to the endpoint_name") 51 | args = parser.parse_args() 52 | 53 | if (args.endpointname is not None): endpoint_name = args.endpointname 54 | 55 | apigateway_endpoint = get_apigateway_names(endpoint_name) 56 | if apigateway_endpoint is not None: 57 | print(apigateway_endpoint) 58 | return 0 59 | else: 60 | return 1 61 | 62 | 63 | if __name__ == '__main__': 64 | main() 65 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/get_apigateway_id.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 23 Dec 2018 13 | @author: Richard Freeman 14 | 15 | This module gets an API Gateway ID based on the name 16 | sudo python3.6 -m pip install boto3 17 | 18 | """ 19 | 20 | import argparse 21 | import logging 22 | 23 | import boto3 24 | logging.getLogger('botocore').setLevel(logging.CRITICAL) 25 | 26 | logger = logging.getLogger(__name__) 27 | logging.basicConfig(format='%(asctime)s %(levelname)s %(name)-15s: %(lineno)d %(message)s', 28 | level=logging.INFO) 29 | logger.setLevel(logging.INFO) 30 | 31 | 32 | def get_apigateway_id(endpoint_name): 33 | client = boto3.client(service_name='apigateway', region_name='eu-west-1') 34 | apis = client.get_rest_apis() 35 | for api in apis['items']: 36 | if api['name'] == endpoint_name: 37 | return api['id'] 38 | return None 39 | 40 | 41 | def main(): 42 | endpoint_name = "lambda-dynamo-xray" 43 | 44 | parser = argparse.ArgumentParser() 45 | parser.add_argument("-e", "--endpointname", type=str, required=False, help="Path to the endpoint_id") 46 | args = parser.parse_args() 47 | 48 | if (args.endpointname is not None): endpoint_name = args.endpointname 49 | 50 | apigateway_id = get_apigateway_id(endpoint_name) 51 | if apigateway_id is not None: 52 | print(apigateway_id) 53 | return 0 54 | else: 55 | return 1 56 | 57 | 58 | if __name__ == '__main__': 59 | main() 60 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/invoke-lambda.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | . ./common-variables.sh 6 | rm outputfile.tmp 7 | status_code=$(aws lambda invoke --invocation-type RequestResponse \ 8 | --function-name ${template}-sam --region ${region} \ 9 | --payload file://../../sample_data/request-api-gateway-get-valid.json outputfile.tmp \ 10 | --profile ${profile}) 11 | #status_code=$(aws lambda invoke --invocation-type Event --function-name lambda-dynamo-xray-sam \ 12 | # --region eu-west-1 --region eu-west-1 --payload file://../../sample_data/request-api-gateway-get-valid.json \ 13 | # outputfile.tmp) 14 | echo "$status_code" 15 | if echo "$status_code" | grep -q "200"; 16 | then 17 | cat outputfile.tmp 18 | if grep -q error outputfile.tmp; 19 | then 20 | echo "\nerror in response" 21 | exit 1 22 | else 23 | echo "\npass" 24 | exit 0 25 | fi 26 | else 27 | echo "\nerror status not 200" 28 | exit 1 29 | fi -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/lambda-dynamo-data-api.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: 'AWS::Serverless-2016-10-31' 3 | Description: >- 4 | This Lambda is invoked by API Gateway and queries DynamoDB. 5 | Parameters: 6 | AccountId: 7 | Type: String 8 | Resources: 9 | lambdadynamodataapi: 10 | Type: AWS::Serverless::Function 11 | Properties: 12 | Handler: lambda_return_dynamo_records.lambda_handler 13 | Runtime: python3.6 14 | CodeUri: ../../package/lambda-dynamo-data-api.zip 15 | FunctionName: lambda-dynamo-data-api-sam 16 | Description: >- 17 | This Lambda is invoked by API Gateway and queries DynamoDB. 18 | MemorySize: 128 19 | Timeout: 3 20 | Role: !Sub 'arn:aws:iam::${AccountId}:role/lambda-dynamo-data-api' 21 | Environment: 22 | Variables: 23 | environment: dev 24 | Events: 25 | CatchAll: 26 | Type: Api 27 | Properties: 28 | Path: /visits/{resourceId} 29 | Method: GET 30 | DynamoDBTable: 31 | Type: AWS::DynamoDB::Table 32 | Properties: 33 | TableName: user-visits-sam 34 | SSESpecification: 35 | SSEEnabled: True 36 | AttributeDefinitions: 37 | - AttributeName: EventId 38 | AttributeType: S 39 | - AttributeName: EventDay 40 | AttributeType: N 41 | KeySchema: 42 | - AttributeName: EventId 43 | KeyType: HASH 44 | - AttributeName: EventDay 45 | KeyType: RANGE 46 | ProvisionedThroughput: 47 | ReadCapacityUnits: 1 48 | WriteCapacityUnits: 1 49 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/run_locus.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #This script gets the API Gateway Id based on the API name ${template}, then runs locust 6 | 7 | . ./common-variables.sh 8 | apiid="$(python3 get_apigateway_id.py -e ${template})" 9 | locust -f ../../test/locust_test_api.py --host=https://${apiid}.execute-api.${region}.amazonaws.com 10 | -------------------------------------------------------------------------------- /Chapter02/bash/apigateway-lambda-dynamodb/unit-test-lambda.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | (cd ../..; 6 | python3 -m unittest discover test) 7 | -------------------------------------------------------------------------------- /Chapter02/images/building-scalable-serverless-microservice-rest-data-api-video.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter02/images/building-scalable-serverless-microservice-rest-data-api-video.png -------------------------------------------------------------------------------- /Chapter02/images/implementing-serverless-microservices-architecture-patterns-video.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter02/images/implementing-serverless-microservices-architecture-patterns-video.png -------------------------------------------------------------------------------- /Chapter02/lambda_dynamo_read/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter02/lambda_dynamo_read/__init__.py -------------------------------------------------------------------------------- /Chapter02/lambda_dynamo_read/lambda_return_dynamo_records.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 30 Dec 2017 13 | @author: Richard Freeman 14 | 15 | This Lambda queries the remote DynamoDB for a specific partition and greater than a 16 | specific sort key. 17 | 18 | """ 19 | import json 20 | import decimal 21 | 22 | from boto3 import resource 23 | from boto3.dynamodb.conditions import Key 24 | 25 | 26 | class HttpUtils: 27 | def __init__(self): 28 | pass 29 | 30 | @staticmethod 31 | def parse_parameters(event): 32 | try: 33 | return_parameters = event['queryStringParameters'].copy() 34 | except Exception: 35 | return_parameters = {} 36 | try: 37 | resource_id = event.get('path', '').split('/')[-1] 38 | if resource_id.isdigit(): 39 | return_parameters['resource_id'] = resource_id 40 | else: 41 | return {"parsedParams": None, "err": 42 | Exception("resource_id not a number")} 43 | except Exception as e: 44 | return {"parsedParams": None, "err": e} # Generally bad idea to expose exceptions 45 | return {"parsedParams": return_parameters, "err": None} 46 | 47 | @staticmethod 48 | def respond(err=None, err_code=400, res=None): 49 | return { 50 | 'statusCode': str(err_code) if err else '200', 51 | 'body': '{"message":%s}' % json.dumps(str(err)) if err else 52 | json.dumps(res, cls=DecimalEncoder), 53 | 'headers': { 54 | 'Content-Type': 'application/json', 55 | 'Access-Control-Allow-Origin': '*' 56 | }, 57 | } 58 | 59 | @staticmethod 60 | def parse_body(event): 61 | try: 62 | return {"body": json.loads(event['body']), "err": None} 63 | except Exception as e: 64 | return {"body": None, "err": e} 65 | 66 | 67 | class DecimalEncoder(json.JSONEncoder): 68 | def default(self, o): 69 | if isinstance(o, decimal.Decimal): 70 | if o % 1 > 0: 71 | return float(o) 72 | else: 73 | return int(o) 74 | return super(DecimalEncoder, self).default(o) 75 | 76 | 77 | class DynamoRepository: 78 | def __init__(self, table_name): 79 | self.dynamo_client = resource(service_name='dynamodb', 80 | region_name='eu-west-1') 81 | self.table_name = table_name 82 | self.db_table = self.dynamo_client.Table(table_name) 83 | 84 | def query_by_partition_and_sort_key(self, partition_key, partition_value, 85 | sort_key, sort_value): 86 | response = self.db_table.query(KeyConditionExpression= 87 | Key(partition_key).eq(partition_value) 88 | & Key(sort_key).gte(sort_value)) 89 | 90 | return response.get('Items') 91 | 92 | def query_by_partition_key(self, partition_key, partition_value): 93 | response = self.db_table.query(KeyConditionExpression= 94 | Key(partition_key).eq(partition_value)) 95 | 96 | return response.get('Items') 97 | 98 | 99 | def print_exception(e): 100 | try: 101 | print('Exception %s type' % str(type(e))) 102 | print('Exception message: %s ' % str(e)) 103 | except Exception: 104 | pass 105 | 106 | 107 | class Controller(): 108 | def __init__(self): 109 | pass 110 | 111 | @staticmethod 112 | def get_dynamodb_records(event): 113 | try: 114 | validation_result = HttpUtils.parse_parameters(event) 115 | if validation_result.get('parsedParams', None) is None: 116 | return HttpUtils.respond(err=validation_result['err'], err_code=404) 117 | resource_id = str(validation_result['parsedParams']["resource_id"]) 118 | if validation_result['parsedParams'].get("startDate") is None: 119 | result = repo.query_by_partition_key(partition_key="EventId", partition_value=resource_id) 120 | else: 121 | start_date = int(validation_result['parsedParams']["startDate"]) 122 | result = repo.query_by_partition_and_sort_key(partition_key="EventId", partition_value=resource_id, 123 | sort_key="EventDay", sort_value=start_date) 124 | return HttpUtils.respond(res=result) 125 | 126 | except Exception as e: 127 | print_exception(e) 128 | return HttpUtils.respond(err=Exception('Not found'), err_code=404) 129 | 130 | 131 | # For manual deployment 132 | # table_name = 'user-visits' 133 | 134 | # For SAM deployment: 135 | table_name = 'user-visits-sam' 136 | repo = DynamoRepository(table_name=table_name) 137 | 138 | 139 | def lambda_handler(event, context): 140 | response = Controller.get_dynamodb_records(event) 141 | return response 142 | -------------------------------------------------------------------------------- /Chapter02/requirements.txt: -------------------------------------------------------------------------------- 1 | pyaml>=17.8.0 2 | nose>=1.3.7 3 | boto3>=1.6.11 4 | nose2[coverage_plugin]>=0.6.5 5 | locustio>=0.9.0 6 | -------------------------------------------------------------------------------- /Chapter02/sample_data/dynamodb-sample-data.txt: -------------------------------------------------------------------------------- 1 | EventId,EventDay,EventCount 2 | 324,20171010,2 3 | 324,20171012,10 4 | 324,20171013,10 5 | 324,20171014,6 6 | 324,20171016,6 7 | 324,20171017,2 8 | 300,20171011,1 9 | 300,20171013,3 10 | 300,20171014,30 -------------------------------------------------------------------------------- /Chapter02/sample_data/request-api-gateway-get-error.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "foo": "bar" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "GET", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/1234f" 57 | } -------------------------------------------------------------------------------- /Chapter02/sample_data/request-api-gateway-get-valid-date.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "startDate": "20171014" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "POST", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/324" 57 | } -------------------------------------------------------------------------------- /Chapter02/sample_data/request-api-gateway-get-valid-no-date.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "headers": { 27 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 28 | "Accept-Language": "en-US,en;q=0.8", 29 | "CloudFront-Is-Desktop-Viewer": "true", 30 | "CloudFront-Is-SmartTV-Viewer": "false", 31 | "CloudFront-Is-Mobile-Viewer": "false", 32 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 33 | "CloudFront-Viewer-Country": "US", 34 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 35 | "Upgrade-Insecure-Requests": "1", 36 | "X-Forwarded-Port": "443", 37 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 38 | "X-Forwarded-Proto": "https", 39 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 40 | "CloudFront-Is-Tablet-Viewer": "false", 41 | "Cache-Control": "max-age=0", 42 | "User-Agent": "Custom User Agent String", 43 | "CloudFront-Forwarded-Proto": "https", 44 | "Accept-Encoding": "gzip, deflate, sdch" 45 | }, 46 | "pathParameters": { 47 | "proxy": "path/to/resource" 48 | }, 49 | "httpMethod": "POST", 50 | "stageVariables": { 51 | "baz": "qux" 52 | }, 53 | "path": "/path/to/resource/324" 54 | } -------------------------------------------------------------------------------- /Chapter02/sample_data/request-api-gateway-get-valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "GET", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "foo": "bar" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "GET", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/324" 57 | } -------------------------------------------------------------------------------- /Chapter02/test/locust_test_api.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2018 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | API Gateway load testing 13 | 14 | Created on 25 Aug 2018 15 | 16 | In Shell run 17 | $ sudo pip3 install locustio 18 | $ ./bash/apigateway-lambda-dynamodb/common-variables.sh 19 | $ apiid="$(python bash/apigateway-lambda-dynamodb/get_apigateway_id.py -e ${template})" 20 | $ locust -f test/locust_test_api.py --host=https://${apiid}.execute-api.${region}.amazonaws.com 21 | 22 | Open browser: http://localhost:8089/ 23 | """ 24 | import random 25 | from locust import HttpLocust, TaskSet, task 26 | 27 | paths = ["/Prod/visits/324?startDate=20171014", 28 | "/Prod/visits/324", 29 | "/Prod/visits/320"] 30 | 31 | 32 | class SimpleLocustTest(TaskSet): 33 | 34 | @task 35 | def get_something(self): 36 | index = random.randint(0, len(paths) - 1) 37 | self.client.get(paths[index]) 38 | 39 | 40 | class LocustTests(HttpLocust): 41 | task_set = SimpleLocustTest -------------------------------------------------------------------------------- /Chapter02/test/run_local_api_gateway_lambda_dynamo.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2018 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 31 Dec 2017 13 | 14 | @author: Richard Freeman 15 | 16 | """ 17 | import json 18 | 19 | from lambda_dynamo_read import lambda_return_dynamo_records as lambda_query_dynamo 20 | 21 | with open('../sample_data/request-api-gateway-get-valid-date.json', 'r') as sample_file: 22 | event = json.loads(sample_file.read()) 23 | print("lambda_query_dynamo\nUsing data: %s" % event) 24 | print(sample_file.name.split('/')[-1]) 25 | response = lambda_query_dynamo.lambda_handler(event, None) 26 | print('Response: %s\n' % json.dumps(response)) 27 | 28 | with open('../sample_data/request-api-gateway-get-valid-no-date.json', 'r') as sample_file: 29 | event = json.loads(sample_file.read()) 30 | print(sample_file.name.split('/')[-1]) 31 | print("lambda_query_dynamo\nUsing data: " + str(event)) 32 | response = lambda_query_dynamo.lambda_handler(event, None) 33 | print('Response: %s\n' % json.dumps(response)) 34 | 35 | with open('../sample_data/request-api-gateway-get-error.json', 'r') as sample_file: 36 | event = json.loads(sample_file.read()) 37 | print(sample_file.name.split('/')[-1]) 38 | print("lambda_query_dynamo\nUsing data: " + str(event)) 39 | response = lambda_query_dynamo.lambda_handler(event, None) 40 | print("Response: %s\n" % json.dumps(response)) 41 | 42 | -------------------------------------------------------------------------------- /Chapter02/test/test_dynamo_get.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Lambda DynamoDB unit tests and mocking 13 | 14 | pip install mock 15 | 16 | """ 17 | 18 | import sys 19 | import unittest 20 | import json 21 | 22 | from unittest import mock 23 | 24 | from lambda_dynamo_read import lambda_return_dynamo_records as lambda_query_dynamo 25 | 26 | 27 | class TestIndexGetMethod(unittest.TestCase): 28 | def setUp(self): 29 | self.validJsonDataNoStartDate = json.loads('{"httpMethod": "GET","path": "/path/to/resource/324","headers": ' \ 30 | 'null} ') 31 | self.validJsonDataStartDate = json.loads('{"queryStringParameters": {"startDate": "20171013"},' \ 32 | '"httpMethod": "GET","path": "/path/to/resource/324","headers": ' \ 33 | 'null} ') 34 | self.invalidJsonUserIdData = json.loads('{"queryStringParameters": {"startDate": "20171013"},' \ 35 | '"httpMethod": "GET","path": "/path/to/resource/324f","headers": ' \ 36 | 'null} ') 37 | self.invalidJsonData = "{ invalid JSON request!} " 38 | 39 | def tearDown(self): 40 | pass 41 | 42 | def test_validparameters_parseparameters_pass(self): 43 | parameters = lambda_query_dynamo.HttpUtils.parse_parameters(self.validJsonDataStartDate) 44 | assert parameters['parsedParams']['startDate'] == u'20171013' 45 | assert parameters['parsedParams']['resource_id'] == u'324' 46 | 47 | def test_emptybody_parsebody_nonebody(self): 48 | body = lambda_query_dynamo.HttpUtils.parse_body(self.validJsonDataStartDate) 49 | assert body['body'] is None 50 | 51 | def test_invalidjson_getrecord_notfound404(self): 52 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.invalidJsonData) 53 | assert result['statusCode'] == '404' 54 | 55 | def test_invaliduserid_getrecord_invalididerror(self): 56 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.invalidJsonUserIdData) 57 | assert result['statusCode'] == '404' 58 | assert json.loads(result['body'])['message'] == "resource_id not a number" 59 | 60 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 61 | "query_by_partition_key", 62 | return_value=['item']) 63 | def test_validid_checkstatus_status200(self, mock_query_by_partition_key): 64 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataNoStartDate) 65 | assert result['statusCode'] == '200' 66 | 67 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 68 | "query_by_partition_key", 69 | return_value=['item']) 70 | def test_validid_getrecords_validparamcall(self, mock_query_by_partition_key): 71 | lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataNoStartDate) 72 | mock_query_by_partition_key.assert_called_with(partition_key='EventId', 73 | partition_value=u'324') 74 | 75 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 76 | "query_by_partition_and_sort_key", 77 | return_value=['item']) 78 | def test_validid_getrecordsdate_validparamcall(self, mock_query_by_partition_and_sort_key): 79 | lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataStartDate) 80 | mock_query_by_partition_and_sort_key.assert_called_with(partition_key='EventId', 81 | partition_value=u'324', 82 | sort_key='EventDay', 83 | sort_value=20171013) 84 | 85 | def test_validresponse_httpresponse_valid(self): 86 | response = lambda_query_dynamo.HttpUtils.respond() 87 | assert response['statusCode'] == '200' 88 | assert response['body'] == 'null' 89 | 90 | def test_exception_printexception_printedexception(self): 91 | assert lambda_query_dynamo.print_exception(Exception('test')) is None 92 | 93 | -------------------------------------------------------------------------------- /Chapter03/IAM/assume-role-lambda.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "", 6 | "Effect": "Allow", 7 | "Principal": { 8 | "Service": "lambda.amazonaws.com" 9 | }, 10 | "Action": "sts:AssumeRole" 11 | } 12 | ] 13 | } -------------------------------------------------------------------------------- /Chapter03/IAM/dynamo-full-user-visits.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1422032676021", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "dynamodb:BatchGetItem", 9 | "dynamodb:BatchWriteItem", 10 | "dynamodb:DeleteItem", 11 | "dynamodb:DescribeStream", 12 | "dynamodb:DescribeTable", 13 | "dynamodb:GetItem", 14 | "dynamodb:GetRecords", 15 | "dynamodb:GetShardIterator", 16 | "dynamodb:ListStreams", 17 | "dynamodb:ListTables", 18 | "dynamodb:PutItem", 19 | "dynamodb:Query", 20 | "dynamodb:Scan", 21 | "dynamodb:UpdateItem", 22 | "dynamodb:UpdateTable" 23 | ], 24 | "Resource": [ 25 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits", 26 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-sam" 27 | ] 28 | }, 29 | { 30 | "Effect": "Allow", 31 | "Action": "dynamodb:ListTables", 32 | "Resource": "*", 33 | "Condition": {} 34 | } 35 | ] 36 | } -------------------------------------------------------------------------------- /Chapter03/IAM/dynamo-readonly-user-visits.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1422532676335", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "dynamodb:BatchGetItem", 9 | "dynamodb:DescribeTable", 10 | "dynamodb:GetItem", 11 | "dynamodb:Query", 12 | "dynamodb:Scan" 13 | ], 14 | "Resource": [ 15 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits", 16 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-sam", 17 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-xray-sam" 18 | ] 19 | } 20 | ] 21 | } 22 | -------------------------------------------------------------------------------- /Chapter03/IAM/iam-pass-role.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1449755587000", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "iam:GetInstanceProfile", 9 | "iam:GetRole", 10 | "iam:ListInstanceProfiles", 11 | "iam:ListInstanceProfilesForRole", 12 | "iam:PassRole" 13 | ], 14 | "Resource": [ 15 | "*" 16 | ] 17 | } 18 | ] 19 | } -------------------------------------------------------------------------------- /Chapter03/IAM/lambda-cloud-write.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Effect": "Allow", 6 | "Action": [ 7 | "logs:CreateLogGroup", 8 | "logs:CreateLogStream", 9 | "logs:PutLogEvents", 10 | "logs:DescribeLogStreams" 11 | ], 12 | "Resource": [ 13 | "arn:aws:logs:*:*:*" 14 | ] 15 | }, 16 | { 17 | "Effect": "Allow", 18 | "Action": [ 19 | "cloudwatch:PutMetricData" 20 | ], 21 | "Resource": "*" 22 | } 23 | ] 24 | } -------------------------------------------------------------------------------- /Chapter03/README.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"). 4 | You may not use this file except in compliance with the License. 5 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 6 | 7 | ## Serverless Microservice Data API 8 | 9 | In this repository, I share some sample code used to create a scalable serverless data API from my two video courses. It includes bash scripts that can be used to unit test, build, package, deploy, and run integration tests. The Python code is written defensively to deal with any API exception. 10 | 11 | |For beginners and intermediates, the [full Serverless Data API](https://www.packtpub.com/application-development/building-scalable-serverless-microservice-rest-data-api-video) code, configuration and a detailed walk through |For intermediate or advanced users, I cover [15+ serverless microservice patterns](https://www.packtpub.com/application-development/implementing-serverless-microservices-architecture-patterns-video) with original content, code, configuration and detailed walk through | 12 | |:----------|:-------------| 13 | | [![Building a Scalable Serverless Microservice REST Data API Video Course](./images/building-scalable-serverless-microservice-rest-data-api-video.png "Building a Scalable Serverless Microservice REST Data API Video Course")](https://www.packtpub.com/application-development/building-scalable-serverless-microservice-rest-data-api-video)| [![Implementing Serverless Microservices Architecture Patterns Video Course](./images/implementing-serverless-microservices-architecture-patterns-video.png "Implementing Serverless Microservices Architecture Patterns Video Course")](https://www.packtpub.com/application-development/implementing-serverless-microservices-architecture-patterns-video) | 14 | 15 | 16 | ### 1. Windows Only 17 | 18 | Please skip this step if you are not using Windows. 19 | 20 | Using Bash (Unix Shell) makes your life much easier when deploying and managing you serverless stack. I think all analysts, data scientists, architects, administrators, database administrators, developers, DevOps and technical people should know some basic Bash and be able to run shell scripts, which are typically used on LINUX and UNIX (including macOS Terminal). 21 | 22 | Alternatively you can adapt the scripts to use MS-DOS or Powershell but it's not something I recommended, given that Bash can now run natively on Windows 10 as an application, and there are many more examples online in Bash. 23 | 24 | Note that I have stripped off the `\r` or carriage returns, as they are illegal in shell scripts, you can use something like [notepad++](https://notepad-plus-plus.org/) on Windows if you want to view the carriage returns in your files properly. If you use traditional Windows Notepad the new lines may not be rendered at all, so use [Notepad++](https://notepad-plus-plus.org/), [Sublime](https://www.sublimetext.com/), [Atom](https://atom.io/) or other such editors. 25 | 26 | A detailed guide on how to install [Linux Bash shell on Windows 10 can be found here](https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/). The main steps are: 27 | * Control Panel > Programs > Turn Windows Features On Or Off. 28 | * Choose the check box next to the **Windows Subsystem for Linux** option in the list, and then Choose **OK**. 29 | * Under **Microsoft Store > Run Linux on Windows** Select **Ubuntu**. 30 | * Launch Ubuntu and setup a root account username and password 31 | The Windows `C:\` and other dives are already mounted, and you can access them with the following command in the terminal: 32 | ```bash 33 | $ cd /mnt/c/ 34 | ``` 35 | 36 | Well done you now have full access to Linux on Windows! 37 | 38 | ### 2. Update Ubuntu, Install Git and Clone Repository 39 | ```bash 40 | $ sudo apt-get update 41 | $ sudo apt-get -y upgrade 42 | $ apt-get install git-core 43 | ``` 44 | Clone the repository locally with a `git pull` 45 | 46 | ### 3. Install Python and Dependencies 47 | 48 | The Lambda code is written in Python 3.6. Pip is a tool for installing and managing Python packages. Other popular Python package and dependency managers are available such as [Conda](https://conda.io/docs/index.html) or [Pipenv](https://pipenv.readthedocs.io) but we will be using pip as it is the recommended tool for installing packages from the Python Package Index [PyPI](https://pypi.org/) and most widely supported. 49 | 50 | ```bash 51 | $ sudo apt -y install python3.6 52 | $ sudo apt -y install python3-pip 53 | ``` 54 | 55 | Check the Python version 56 | ```bash 57 | $ python3 --version 58 | ``` 59 | You should get the Python3.6+ version. 60 | 61 | The dependent packages required for running, testing and deploying the severless microservices are listed in `requirements.txt` under each project folder, and can be installed using pip. 62 | ```bash 63 | $ sudo pip3 install -r /path/to/requirements.txt 64 | ``` 65 | This will install the dependent libraries for local development such as [Boto3](https://boto3.amazonaws.com) which is the Python AWS Software Development Kit (SDK). 66 | 67 | ### 3. Install and Setup AWS CLI 68 | 69 | AWS Command Line Interface is used to package and deploy your Lambda functions, as well as setup the infrastructure and security in a repeatable way. 70 | 71 | ```bash 72 | $ sudo pip install awscli --upgrade 73 | ``` 74 | 75 | You have created a user called `newuser` earlier and have a `crednetials.csv` with the AWS keys. Enter them them running `aws configure`. 76 | 77 | ```bash 78 | $ aws configure 79 | AWS Access Key ID: 80 | AWS Secret Access Key: 81 | Default region name: 82 | Default output format: 83 | ``` 84 | 85 | More details on setting up the AWS CLI is available [in the AWS Docs](https://docs.aws.amazon.com/lambda/latest/dg/setup-awscli.html). 86 | 87 | For choosing your AWS Region refer to [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) generally those in the USA can use `us-east-1` and those in Europe can use `eu-west-1` 88 | 89 | ### 4. Update AccountId, Bucket and Profile 90 | Here I assume your AWS profile is `demo` you can change that under `bash/apigateway-lambda-dynamodb/common-variables.sh`. 91 | You will need to use a bucket or create one if you haven't already: 92 | ```bash 93 | $ aws s3api create-bucket --bucket mynewbucket231 --profile demo --create-bucket-configuration Loc 94 | ationConstraint=eu-west-1 --region eu-west-1 95 | 96 | ``` 97 | You will also need to change the AWS accountId (current set to 000000000000). The AWS accountId is also used in some IAM policies in the IAM folder. In addition the region will have to be changed. 98 | 99 | to replace your accountId (assume your AWS accountId is 111111111111) you can do it manually or run: 100 | ```bash 101 | find ./ -type f -exec sed -i '' -e 's/000000000000/111111111111/' {} \; 102 | ``` 103 | 104 | ### 5. Run Unit Tests 105 | Change directory to the main bash folder 106 | ```bash 107 | $ cd bash/apigateway-lambda-dynamodb/ 108 | $ ./unit-test-lambda.sh 109 | ``` 110 | 111 | ### 6. Build Package and Deploy the Serverless API 112 | This also creates the IAM Polices and IAM Roles using the AWS CLI that will be required for the Lambda function. 113 | ```bash 114 | $ ./build-package-deploy-lambda-dynamo-data-api.sh 115 | ``` 116 | In less than a minute you should have a stack. Otherwise look at the error messages, CloudFormation stack and ensure your credentials are setup correctly. 117 | 118 | ### 7. Run Lambda Integration Test 119 | Once the stack is up and running, you can run an integration test to check that the Lambda is working. 120 | ```bash 121 | ./invoke-lambda.sh 122 | ``` 123 | 124 | ### 8. Add data to DynamoDb table 125 | 126 | Change to the DynamoDB Python directory and run the Python code, you can also run this under you favourite IDE like PyDev or PyCharm. 127 | ```bash 128 | $ (cd ../../aws_dynamo; python dynamo_insert_items_from_file.py) 129 | ``` 130 | 131 | ### 9. AWS Management Console 132 | Now the stack is up you can have a look at the API Gateway in the AWS Management Console and test the API in your browser. 133 | * API Gateway > lambda-dynamo-data-api 134 | * Stages > Prod > Get 135 | * Copy the Invoke URL into a new tab, e.g. https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/Prod/visits/{resourceId} 136 | * You should get a message `resource_id not a number` as the ID is not valid 137 | * Replace {resourceId} in the URL with 324 138 | if all is working you should see some returned JSON records. Well done if so! 139 | 140 | ### 10. Deleting the stack 141 | Go back to the main bash folder and delete the cloudFormation serverless stack. 142 | 143 | ```bash 144 | $ ./delete-stack.sh 145 | ``` 146 | -------------------------------------------------------------------------------- /Chapter03/aws_dynamo/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter03/aws_dynamo/__init__.py -------------------------------------------------------------------------------- /Chapter03/aws_dynamo/dynamo_insert_items_from_file.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 20 Jan 2018 13 | 14 | @author: Richard Freeman 15 | 16 | This packages inserts records from a file into specified DynamoDB table 17 | 18 | """ 19 | import csv 20 | 21 | from boto3 import resource 22 | 23 | 24 | class DynamoRepository: 25 | def __init__(self, target_dynamo_table, region='eu-west-1'): 26 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 27 | self.target_dynamo_table = target_dynamo_table 28 | self.table = self.dynamodb.Table(self.target_dynamo_table) 29 | 30 | def update_dynamo_event_counter(self, event_name, event_datetime, event_count=1): 31 | response = self.table.update_item( 32 | Key={ 33 | 'EventId': str(event_name), 34 | 'EventDay': int(event_datetime) 35 | }, 36 | ExpressionAttributeValues={":eventCount": int(event_count)}, 37 | UpdateExpression="ADD EventCount :eventCount") 38 | return response 39 | 40 | 41 | def main(): 42 | # For manual deployment 43 | # table_name = 'user-visits' 44 | 45 | # For SAM: 46 | table_name = 'user-visits-sam' 47 | input_data_path = '../sample_data/dynamodb-sample-data.txt' 48 | dynamo_repo = DynamoRepository(table_name) 49 | with open(input_data_path, 'r') as sample_file: 50 | csv_reader = csv.DictReader(sample_file) 51 | for row in csv_reader: 52 | response = dynamo_repo.update_dynamo_event_counter(row['EventId'], 53 | row['EventDay'], 54 | row['EventCount']) 55 | print(response) 56 | 57 | 58 | if __name__ == '__main__': 59 | main() 60 | -------------------------------------------------------------------------------- /Chapter03/aws_dynamo/dynamo_modify_items.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 8 Jan 2018 13 | 14 | @author: Richard Freeman 15 | 16 | This packages inserts records into DynamoDB 17 | 18 | """ 19 | from boto3 import resource 20 | 21 | 22 | class DynamoRepository: 23 | def __init__(self, target_dynamo_table, region='eu-west-1'): 24 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 25 | self.target_dynamo_table = target_dynamo_table 26 | self.table = self.dynamodb.Table(self.target_dynamo_table) 27 | 28 | def update_dynamo_event_counter(self, event_name, event_datetime, event_count=1): 29 | return self.table.update_item( 30 | Key={ 31 | 'EventId': event_name, 32 | 'EventDay': event_datetime 33 | }, 34 | ExpressionAttributeValues={":eventCount": event_count}, 35 | UpdateExpression="ADD EventCount :eventCount") 36 | 37 | 38 | def main(): 39 | # For manual deployment 40 | # table_name = 'user-visits' 41 | 42 | # For SAM deployment 43 | table_name = 'user-visits-sam' 44 | dynamo_repo = DynamoRepository(table_name) 45 | print(dynamo_repo.update_dynamo_event_counter('324', 20171001)) 46 | print(dynamo_repo.update_dynamo_event_counter('324', 20171001, 2)) 47 | print(dynamo_repo.update_dynamo_event_counter('324', 20171002, 5)) 48 | 49 | 50 | if __name__ == '__main__': 51 | main() 52 | -------------------------------------------------------------------------------- /Chapter03/aws_dynamo/dynamo_query_table.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 8 Jan 2018 13 | 14 | @author: Richard Freeman 15 | This package is used to query DynamoDB 16 | """ 17 | import decimal 18 | import json 19 | 20 | from boto3 import resource 21 | from boto3.dynamodb.conditions import Key 22 | 23 | 24 | class DecimalEncoder(json.JSONEncoder): 25 | """Helper class to convert a DynamoDB item to JSON. 26 | """ 27 | def default(self, o): 28 | if isinstance(o, decimal.Decimal): 29 | if o % 1 > 0: 30 | return float(o) 31 | else: 32 | return int(o) 33 | return super(DecimalEncoder, self).default(o) 34 | 35 | 36 | class DynamoRepository: 37 | """abstracts all interactions with DynamoDB including the connection and querying of tables 38 | """ 39 | def __init__(self, target_dynamo_table, region='eu-west-1'): 40 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 41 | self.dynamo_table = target_dynamo_table 42 | self.table = self.dynamodb.Table(self.dynamo_table) 43 | 44 | def query_dynamo_record_by_parition(self, parition_key, parition_value): 45 | try: 46 | response = self.table.query( 47 | KeyConditionExpression=Key(parition_key).eq(parition_value)) 48 | for record in response.get('Items'): 49 | print(json.dumps(record, cls=DecimalEncoder)) 50 | return 51 | 52 | except Exception as e: 53 | print('Exception %s type' % str(type(e))) 54 | print('Exception message: %s ' % str(e)) 55 | 56 | def query_dynamo_record_by_parition_sort_key(self, partition_key, partition_value, 57 | sort_key, sort_value): 58 | try: 59 | response = self.table.query( 60 | KeyConditionExpression=Key(partition_key).eq(partition_value) 61 | & Key(sort_key).gte(sort_value)) 62 | for record in response.get('Items'): 63 | print(json.dumps(record, cls=DecimalEncoder)) 64 | return 65 | 66 | except Exception as e: 67 | print('Exception %s type' % str(type(e))) 68 | print('Exception message: %s ' % str(e)) 69 | 70 | 71 | def main(): 72 | # For manual deployment 73 | # table_name = 'user-visits' 74 | 75 | # For SAM: 76 | table_name = 'user-visits-sam' 77 | partition_key = 'EventId' 78 | partition_value = '324' 79 | sort_key = 'EventDay' 80 | sort_value = 20171001 81 | 82 | dynamo_repo = DynamoRepository(table_name) 83 | print('Reading all data for partition_key:%s' % partition_value) 84 | dynamo_repo.query_dynamo_record_by_parition(partition_key, partition_value) 85 | 86 | print('Reading all data for partition_key:%s with date > %d' % (partition_value, sort_value)) 87 | dynamo_repo.query_dynamo_record_by_parition_sort_key(partition_key, 88 | partition_value, 89 | sort_key, 90 | sort_value) 91 | 92 | 93 | if __name__ == '__main__': 94 | main() 95 | -------------------------------------------------------------------------------- /Chapter03/aws_dynamo/dynamo_table_creation.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | 13 | Created on 8 Jan 2018 14 | 15 | @author: Richard Freeman 16 | 17 | pip install boto3 18 | 19 | This package creates a new DynamoDb table with the specified read and write capacity 20 | 21 | """ 22 | import boto3 23 | 24 | 25 | def create_dynamo_table(table_name_value, enable_streams=False, 26 | read_capacity=1, 27 | write_capacity=1, 28 | region='eu-west-1'): 29 | table_name = table_name_value 30 | print('creating table: ' + table_name) 31 | try: 32 | client = boto3.client(service_name='dynamodb', region_name=region) 33 | print(client.create_table(TableName=table_name, 34 | AttributeDefinitions=[{'AttributeName': 'EventId', 35 | 'AttributeType': 'S'}, 36 | {'AttributeName': 'EventDay', 37 | 'AttributeType': 'N'}], 38 | KeySchema=[{'AttributeName': 'EventId', 39 | 'KeyType': 'HASH'}, 40 | {'AttributeName': 'EventDay', 41 | 'KeyType': 'RANGE'}, 42 | ], 43 | ProvisionedThroughput={'ReadCapacityUnits': read_capacity, 44 | 'WriteCapacityUnits': write_capacity})) 45 | except Exception as e: 46 | print('Exception %s type' % str(type(e))) 47 | print('Exception message: %s ' % str(e)) 48 | 49 | 50 | def main(): 51 | table_name = 'user-visits' 52 | create_dynamo_table(table_name, False, 1, 1) 53 | 54 | 55 | if __name__ == '__main__': 56 | main() 57 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/build-package-deploy-lambda-dynamo-data-api.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | # Variables 6 | . ./common-variables.sh 7 | 8 | # Run unit tests 9 | ./unit-test-lambda.sh 10 | 11 | #Create Zip file of your Lambda code (works on Windows and Linux) 12 | ./create-lambda-package.sh 13 | 14 | #Package your Serverless Stack using SAM + Cloudformation 15 | aws cloudformation package --template-file $template.yaml \ 16 | --output-template-file ../../package/$template-output.yaml \ 17 | --s3-bucket $bucket --s3-prefix $prefix \ 18 | --region $region --profile $profile 19 | 20 | #Deploy your Serverless Stack using SAM + Cloudformation 21 | aws cloudformation deploy --template-file ../../package/$template-output.yaml \ 22 | --stack-name $template --capabilities CAPABILITY_IAM \ 23 | --parameter-overrides AccountId=${aws_account_id} \ 24 | --region $region --profile $profile -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/common-variables.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | export profile="demo" 6 | export region="eu-west-1" 7 | # export aws_account_id=$(aws sts get-caller-identity --query 'Account' --profile $profile | tr -d '\"') 8 | export aws_account_id="000000000000" 9 | export template="lambda-dynamo-data-api" 10 | export bucket="testbucket121f" 11 | export prefix="tmp/sam" 12 | 13 | # Lambda settings 14 | export zip_file="lambda-dynamo-data-api.zip" 15 | export files="lambda_return_dynamo_records.py" 16 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/create-lambda-package.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | # This script creates a Zip package of the Lambda files 6 | 7 | #setup environment variables 8 | . ./common-variables.sh 9 | 10 | #Create Lambda package and exclude the tests to reduce package size 11 | (cd ../../lambda_dynamo_read; 12 | mkdir -p ../package/ 13 | zip -FSr ../package/"${zip_file}" ${files} -x *tests/*) 14 | 15 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/create-role.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #This Script creates a Lambda role and attaches the policies 6 | 7 | #setup environment variables 8 | . ./common-variables.sh 9 | 10 | #Setup Lambda Role 11 | role_name=lambda-dynamo-data-api 12 | aws iam create-role --role-name ${role_name} \ 13 | --assume-role-policy-document file://../../IAM/assume-role-lambda.json \ 14 | --profile $profile || true 15 | 16 | sleep 1 17 | #Add and attach DynamoDB Policy 18 | dynamo_policy=dynamo-readonly-user-visits 19 | aws iam create-policy --policy-name $dynamo_policy \ 20 | --policy-document file://../../IAM/$dynamo_policy.json \ 21 | --profile $profile || true 22 | 23 | role_policy_arn="arn:aws:iam::$aws_account_id:policy/$dynamo_policy" 24 | aws iam attach-role-policy \ 25 | --role-name "${role_name}" \ 26 | --policy-arn "${role_policy_arn}" --profile ${profile} || true 27 | 28 | #Add and attach cloudwatch_policy 29 | cloudwatch_policy=lambda-cloud-write 30 | aws iam create-policy --policy-name $cloudwatch_policy \ 31 | --policy-document file://../../IAM/$cloudwatch_policy.json \ 32 | --profile $profile || true 33 | 34 | role_policy_arn="arn:aws:iam::$aws_account_id:policy/$cloudwatch_policy" 35 | aws iam attach-role-policy \ 36 | --role-name "${role_name}" \ 37 | --policy-arn "${role_policy_arn}" --profile ${profile} || true 38 | 39 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/curl-api-gateway.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #endpoint="https://xxxxx.execute-api.eu-west-1.amazonaws.com/Prod/visits/324" 6 | . ./common-variables.sh 7 | 8 | endpoint="$(python3 get_apigateway_endpoint.py -e ${template})" 9 | echo ${endpoint} 10 | status_code=$(curl -i -H \"Accept: application/json\" -H \"Content-Type: application/json\" -X GET ${endpoint}) 11 | echo "$status_code" 12 | if echo "$status_code" | grep -q "HTTP/1.1 200 OK"; 13 | then 14 | echo "pass" 15 | exit 0 16 | else 17 | exit 1 18 | fi 19 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/delete-stack.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | . ./common-variables.sh 6 | 7 | aws cloudformation delete-stack --stack-name $template --region $region --profile $profile 8 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/get_apigateway_endpoint.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 23 Dec 2018 13 | @author: Richard Freeman 14 | 15 | This module gets an API Gateway endpoint based on the name 16 | sudo python3.6 -m pip install boto3 17 | 18 | """ 19 | 20 | import argparse 21 | import logging 22 | 23 | import boto3 24 | logging.getLogger('botocore').setLevel(logging.CRITICAL) 25 | 26 | logger = logging.getLogger(__name__) 27 | logging.basicConfig(format='%(asctime)s %(levelname)s %(name)-15s: %(lineno)d %(message)s', 28 | level=logging.INFO) 29 | logger.setLevel(logging.INFO) 30 | 31 | 32 | def get_apigateway_names(endpoint_name): 33 | client = boto3.client(service_name='apigateway', region_name='eu-west-1') 34 | apis = client.get_rest_apis() 35 | for api in apis['items']: 36 | if api['name'] == endpoint_name: 37 | api_id = api['id'] 38 | region = 'eu-west-1' 39 | stage = 'Prod' 40 | resource = 'visits/324' 41 | #return F"https://{api_id}.execute-api.{region}.amazonaws.com/{stage}/{resource}" 42 | return "https://%s.execute-api.%s.amazonaws.com/%s/%s" % (api_id, region, stage, resource) 43 | return None 44 | 45 | 46 | def main(): 47 | endpoint_name = "lambda-dynamo-xray" 48 | 49 | parser = argparse.ArgumentParser() 50 | parser.add_argument("-e", "--endpointname", type=str, required=False, help="Path to the endpoint_name") 51 | args = parser.parse_args() 52 | 53 | if (args.endpointname is not None): endpoint_name = args.endpointname 54 | 55 | apigateway_endpoint = get_apigateway_names(endpoint_name) 56 | if apigateway_endpoint is not None: 57 | print(apigateway_endpoint) 58 | return 0 59 | else: 60 | return 1 61 | 62 | 63 | if __name__ == '__main__': 64 | main() 65 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/get_apigateway_id.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 23 Dec 2018 13 | @author: Richard Freeman 14 | 15 | This module gets an API Gateway ID based on the name 16 | sudo python3.6 -m pip install boto3 17 | 18 | """ 19 | 20 | import argparse 21 | import logging 22 | 23 | import boto3 24 | logging.getLogger('botocore').setLevel(logging.CRITICAL) 25 | 26 | logger = logging.getLogger(__name__) 27 | logging.basicConfig(format='%(asctime)s %(levelname)s %(name)-15s: %(lineno)d %(message)s', 28 | level=logging.INFO) 29 | logger.setLevel(logging.INFO) 30 | 31 | 32 | def get_apigateway_id(endpoint_name): 33 | client = boto3.client(service_name='apigateway', region_name='eu-west-1') 34 | apis = client.get_rest_apis() 35 | for api in apis['items']: 36 | if api['name'] == endpoint_name: 37 | return api['id'] 38 | return None 39 | 40 | 41 | def main(): 42 | endpoint_name = "lambda-dynamo-xray" 43 | 44 | parser = argparse.ArgumentParser() 45 | parser.add_argument("-e", "--endpointname", type=str, required=False, help="Path to the endpoint_id") 46 | args = parser.parse_args() 47 | 48 | if (args.endpointname is not None): endpoint_name = args.endpointname 49 | 50 | apigateway_id = get_apigateway_id(endpoint_name) 51 | if apigateway_id is not None: 52 | print(apigateway_id) 53 | return 0 54 | else: 55 | return 1 56 | 57 | 58 | if __name__ == '__main__': 59 | main() 60 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/invoke-lambda.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | . ./common-variables.sh 6 | rm outputfile.tmp 7 | status_code=$(aws lambda invoke --invocation-type RequestResponse \ 8 | --function-name ${template}-sam --region ${region} \ 9 | --payload file://../../sample_data/request-api-gateway-get-valid.json outputfile.tmp \ 10 | --profile ${profile}) 11 | #status_code=$(aws lambda invoke --invocation-type Event --function-name lambda-dynamo-xray-sam \ 12 | # --region eu-west-1 --region eu-west-1 --payload file://../../sample_data/request-api-gateway-get-valid.json \ 13 | # outputfile.tmp) 14 | echo "$status_code" 15 | if echo "$status_code" | grep -q "200"; 16 | then 17 | cat outputfile.tmp 18 | if grep -q error outputfile.tmp; 19 | then 20 | echo "\nerror in response" 21 | exit 1 22 | else 23 | echo "\npass" 24 | exit 0 25 | fi 26 | else 27 | echo "\nerror status not 200" 28 | exit 1 29 | fi -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/lambda-dynamo-data-api.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: 'AWS::Serverless-2016-10-31' 3 | Description: >- 4 | This Lambda is invoked by API Gateway and queries DynamoDB. 5 | Parameters: 6 | AccountId: 7 | Type: String 8 | Resources: 9 | lambdadynamodataapi: 10 | Type: AWS::Serverless::Function 11 | Properties: 12 | Handler: lambda_return_dynamo_records.lambda_handler 13 | Runtime: python3.6 14 | CodeUri: ../../package/lambda-dynamo-data-api.zip 15 | FunctionName: lambda-dynamo-data-api-sam 16 | Description: >- 17 | This Lambda is invoked by API Gateway and queries DynamoDB. 18 | MemorySize: 128 19 | Timeout: 3 20 | Role: !Sub 'arn:aws:iam::${AccountId}:role/lambda-dynamo-data-api' 21 | Environment: 22 | Variables: 23 | environment: dev 24 | Events: 25 | CatchAll: 26 | Type: Api 27 | Properties: 28 | Path: /visits/{resourceId} 29 | Method: GET 30 | DynamoDBTable: 31 | Type: AWS::DynamoDB::Table 32 | Properties: 33 | TableName: user-visits-sam 34 | SSESpecification: 35 | SSEEnabled: True 36 | AttributeDefinitions: 37 | - AttributeName: EventId 38 | AttributeType: S 39 | - AttributeName: EventDay 40 | AttributeType: N 41 | KeySchema: 42 | - AttributeName: EventId 43 | KeyType: HASH 44 | - AttributeName: EventDay 45 | KeyType: RANGE 46 | ProvisionedThroughput: 47 | ReadCapacityUnits: 1 48 | WriteCapacityUnits: 1 49 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/run_locus.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #This script gets the API Gateway Id based on the API name ${template}, then runs locust 6 | 7 | . ./common-variables.sh 8 | apiid="$(python3 get_apigateway_id.py -e ${template})" 9 | locust -f ../../test/locust_test_api.py --host=https://${apiid}.execute-api.${region}.amazonaws.com 10 | -------------------------------------------------------------------------------- /Chapter03/bash/apigateway-lambda-dynamodb/unit-test-lambda.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | (cd ../..; 6 | python3 -m unittest discover test) 7 | -------------------------------------------------------------------------------- /Chapter03/images/building-scalable-serverless-microservice-rest-data-api-video.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter03/images/building-scalable-serverless-microservice-rest-data-api-video.png -------------------------------------------------------------------------------- /Chapter03/images/implementing-serverless-microservices-architecture-patterns-video.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter03/images/implementing-serverless-microservices-architecture-patterns-video.png -------------------------------------------------------------------------------- /Chapter03/lambda_dynamo_read/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter03/lambda_dynamo_read/__init__.py -------------------------------------------------------------------------------- /Chapter03/lambda_dynamo_read/lambda_return_dynamo_records.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 30 Dec 2017 13 | @author: Richard Freeman 14 | 15 | This Lambda queries the remote DynamoDB for a specific partition and greater than a 16 | specific sort key. 17 | 18 | """ 19 | import json 20 | import decimal 21 | 22 | from boto3 import resource 23 | from boto3.dynamodb.conditions import Key 24 | 25 | 26 | class HttpUtils: 27 | def __init__(self): 28 | pass 29 | 30 | @staticmethod 31 | def parse_parameters(event): 32 | try: 33 | return_parameters = event['queryStringParameters'].copy() 34 | except Exception: 35 | return_parameters = {} 36 | try: 37 | resource_id = event.get('path', '').split('/')[-1] 38 | if resource_id.isdigit(): 39 | return_parameters['resource_id'] = resource_id 40 | else: 41 | return {"parsedParams": None, "err": 42 | Exception("resource_id not a number")} 43 | except Exception as e: 44 | return {"parsedParams": None, "err": e} # Generally bad idea to expose exceptions 45 | return {"parsedParams": return_parameters, "err": None} 46 | 47 | @staticmethod 48 | def respond(err=None, err_code=400, res=None): 49 | return { 50 | 'statusCode': str(err_code) if err else '200', 51 | 'body': '{"message":%s}' % json.dumps(str(err)) if err else 52 | json.dumps(res, cls=DecimalEncoder), 53 | 'headers': { 54 | 'Content-Type': 'application/json', 55 | 'Access-Control-Allow-Origin': '*' 56 | }, 57 | } 58 | 59 | @staticmethod 60 | def parse_body(event): 61 | try: 62 | return {"body": json.loads(event['body']), "err": None} 63 | except Exception as e: 64 | return {"body": None, "err": e} 65 | 66 | 67 | class DecimalEncoder(json.JSONEncoder): 68 | def default(self, o): 69 | if isinstance(o, decimal.Decimal): 70 | if o % 1 > 0: 71 | return float(o) 72 | else: 73 | return int(o) 74 | return super(DecimalEncoder, self).default(o) 75 | 76 | 77 | class DynamoRepository: 78 | def __init__(self, table_name): 79 | self.dynamo_client = resource(service_name='dynamodb', 80 | region_name='eu-west-1') 81 | self.table_name = table_name 82 | self.db_table = self.dynamo_client.Table(table_name) 83 | 84 | def query_by_partition_and_sort_key(self, partition_key, partition_value, 85 | sort_key, sort_value): 86 | response = self.db_table.query(KeyConditionExpression= 87 | Key(partition_key).eq(partition_value) 88 | & Key(sort_key).gte(sort_value)) 89 | 90 | return response.get('Items') 91 | 92 | def query_by_partition_key(self, partition_key, partition_value): 93 | response = self.db_table.query(KeyConditionExpression= 94 | Key(partition_key).eq(partition_value)) 95 | 96 | return response.get('Items') 97 | 98 | 99 | def print_exception(e): 100 | try: 101 | print('Exception %s type' % str(type(e))) 102 | print('Exception message: %s ' % str(e)) 103 | except Exception: 104 | pass 105 | 106 | 107 | class Controller(): 108 | def __init__(self): 109 | pass 110 | 111 | @staticmethod 112 | def get_dynamodb_records(event): 113 | try: 114 | validation_result = HttpUtils.parse_parameters(event) 115 | if validation_result.get('parsedParams', None) is None: 116 | return HttpUtils.respond(err=validation_result['err'], err_code=404) 117 | resource_id = str(validation_result['parsedParams']["resource_id"]) 118 | if validation_result['parsedParams'].get("startDate") is None: 119 | result = repo.query_by_partition_key(partition_key="EventId", partition_value=resource_id) 120 | else: 121 | start_date = int(validation_result['parsedParams']["startDate"]) 122 | result = repo.query_by_partition_and_sort_key(partition_key="EventId", partition_value=resource_id, 123 | sort_key="EventDay", sort_value=start_date) 124 | return HttpUtils.respond(res=result) 125 | 126 | except Exception as e: 127 | print_exception(e) 128 | return HttpUtils.respond(err=Exception('Not found'), err_code=404) 129 | 130 | 131 | # For manual deployment 132 | # table_name = 'user-visits' 133 | 134 | # For SAM deployment: 135 | table_name = 'user-visits-sam' 136 | repo = DynamoRepository(table_name=table_name) 137 | 138 | 139 | def lambda_handler(event, context): 140 | response = Controller.get_dynamodb_records(event) 141 | return response 142 | -------------------------------------------------------------------------------- /Chapter03/requirements.txt: -------------------------------------------------------------------------------- 1 | pyaml>=17.8.0 2 | nose>=1.3.7 3 | boto3>=1.6.11 4 | nose2[coverage_plugin]>=0.6.5 5 | locustio>=0.9.0 6 | -------------------------------------------------------------------------------- /Chapter03/sample_data/dynamodb-sample-data.txt: -------------------------------------------------------------------------------- 1 | EventId,EventDay,EventCount 2 | 324,20171010,2 3 | 324,20171012,10 4 | 324,20171013,10 5 | 324,20171014,6 6 | 324,20171016,6 7 | 324,20171017,2 8 | 300,20171011,1 9 | 300,20171013,3 10 | 300,20171014,30 -------------------------------------------------------------------------------- /Chapter03/sample_data/request-api-gateway-get-error.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "foo": "bar" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "GET", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/1234f" 57 | } -------------------------------------------------------------------------------- /Chapter03/sample_data/request-api-gateway-get-valid-date.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "startDate": "20171014" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "POST", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/324" 57 | } -------------------------------------------------------------------------------- /Chapter03/sample_data/request-api-gateway-get-valid-no-date.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "headers": { 27 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 28 | "Accept-Language": "en-US,en;q=0.8", 29 | "CloudFront-Is-Desktop-Viewer": "true", 30 | "CloudFront-Is-SmartTV-Viewer": "false", 31 | "CloudFront-Is-Mobile-Viewer": "false", 32 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 33 | "CloudFront-Viewer-Country": "US", 34 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 35 | "Upgrade-Insecure-Requests": "1", 36 | "X-Forwarded-Port": "443", 37 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 38 | "X-Forwarded-Proto": "https", 39 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 40 | "CloudFront-Is-Tablet-Viewer": "false", 41 | "Cache-Control": "max-age=0", 42 | "User-Agent": "Custom User Agent String", 43 | "CloudFront-Forwarded-Proto": "https", 44 | "Accept-Encoding": "gzip, deflate, sdch" 45 | }, 46 | "pathParameters": { 47 | "proxy": "path/to/resource" 48 | }, 49 | "httpMethod": "POST", 50 | "stageVariables": { 51 | "baz": "qux" 52 | }, 53 | "path": "/path/to/resource/324" 54 | } -------------------------------------------------------------------------------- /Chapter03/sample_data/request-api-gateway-get-valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "GET", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "foo": "bar" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "GET", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/324" 57 | } -------------------------------------------------------------------------------- /Chapter03/test/locust_test_api.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2018 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | API Gateway load testing 13 | 14 | Created on 25 Aug 2018 15 | 16 | In Shell run 17 | $ sudo pip3 install locustio 18 | $ ./bash/apigateway-lambda-dynamodb/common-variables.sh 19 | $ apiid="$(python bash/apigateway-lambda-dynamodb/get_apigateway_id.py -e ${template})" 20 | $ locust -f test/locust_test_api.py --host=https://${apiid}.execute-api.${region}.amazonaws.com 21 | 22 | Open browser: http://localhost:8089/ 23 | """ 24 | import random 25 | from locust import HttpLocust, TaskSet, task 26 | 27 | paths = ["/Prod/visits/324?startDate=20171014", 28 | "/Prod/visits/324", 29 | "/Prod/visits/320"] 30 | 31 | 32 | class SimpleLocustTest(TaskSet): 33 | 34 | @task 35 | def get_something(self): 36 | index = random.randint(0, len(paths) - 1) 37 | self.client.get(paths[index]) 38 | 39 | 40 | class LocustTests(HttpLocust): 41 | task_set = SimpleLocustTest -------------------------------------------------------------------------------- /Chapter03/test/run_local_api_gateway_lambda_dynamo.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2018 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 31 Dec 2017 13 | 14 | @author: Richard Freeman 15 | 16 | """ 17 | import json 18 | 19 | from lambda_dynamo_read import lambda_return_dynamo_records as lambda_query_dynamo 20 | 21 | with open('../sample_data/request-api-gateway-get-valid-date.json', 'r') as sample_file: 22 | event = json.loads(sample_file.read()) 23 | print("lambda_query_dynamo\nUsing data: %s" % event) 24 | print(sample_file.name.split('/')[-1]) 25 | response = lambda_query_dynamo.lambda_handler(event, None) 26 | print('Response: %s\n' % json.dumps(response)) 27 | 28 | with open('../sample_data/request-api-gateway-get-valid-no-date.json', 'r') as sample_file: 29 | event = json.loads(sample_file.read()) 30 | print(sample_file.name.split('/')[-1]) 31 | print("lambda_query_dynamo\nUsing data: " + str(event)) 32 | response = lambda_query_dynamo.lambda_handler(event, None) 33 | print('Response: %s\n' % json.dumps(response)) 34 | 35 | with open('../sample_data/request-api-gateway-get-error.json', 'r') as sample_file: 36 | event = json.loads(sample_file.read()) 37 | print(sample_file.name.split('/')[-1]) 38 | print("lambda_query_dynamo\nUsing data: " + str(event)) 39 | response = lambda_query_dynamo.lambda_handler(event, None) 40 | print("Response: %s\n" % json.dumps(response)) 41 | 42 | -------------------------------------------------------------------------------- /Chapter03/test/test_dynamo_get.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Lambda DynamoDB unit tests and mocking 13 | 14 | pip install mock 15 | 16 | """ 17 | 18 | import sys 19 | import unittest 20 | import json 21 | 22 | from unittest import mock 23 | 24 | from lambda_dynamo_read import lambda_return_dynamo_records as lambda_query_dynamo 25 | 26 | 27 | class TestIndexGetMethod(unittest.TestCase): 28 | def setUp(self): 29 | self.validJsonDataNoStartDate = json.loads('{"httpMethod": "GET","path": "/path/to/resource/324","headers": ' \ 30 | 'null} ') 31 | self.validJsonDataStartDate = json.loads('{"queryStringParameters": {"startDate": "20171013"},' \ 32 | '"httpMethod": "GET","path": "/path/to/resource/324","headers": ' \ 33 | 'null} ') 34 | self.invalidJsonUserIdData = json.loads('{"queryStringParameters": {"startDate": "20171013"},' \ 35 | '"httpMethod": "GET","path": "/path/to/resource/324f","headers": ' \ 36 | 'null} ') 37 | self.invalidJsonData = "{ invalid JSON request!} " 38 | 39 | def tearDown(self): 40 | pass 41 | 42 | def test_validparameters_parseparameters_pass(self): 43 | parameters = lambda_query_dynamo.HttpUtils.parse_parameters(self.validJsonDataStartDate) 44 | assert parameters['parsedParams']['startDate'] == u'20171013' 45 | assert parameters['parsedParams']['resource_id'] == u'324' 46 | 47 | def test_emptybody_parsebody_nonebody(self): 48 | body = lambda_query_dynamo.HttpUtils.parse_body(self.validJsonDataStartDate) 49 | assert body['body'] is None 50 | 51 | def test_invalidjson_getrecord_notfound404(self): 52 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.invalidJsonData) 53 | assert result['statusCode'] == '404' 54 | 55 | def test_invaliduserid_getrecord_invalididerror(self): 56 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.invalidJsonUserIdData) 57 | assert result['statusCode'] == '404' 58 | assert json.loads(result['body'])['message'] == "resource_id not a number" 59 | 60 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 61 | "query_by_partition_key", 62 | return_value=['item']) 63 | def test_validid_checkstatus_status200(self, mock_query_by_partition_key): 64 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataNoStartDate) 65 | assert result['statusCode'] == '200' 66 | 67 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 68 | "query_by_partition_key", 69 | return_value=['item']) 70 | def test_validid_getrecords_validparamcall(self, mock_query_by_partition_key): 71 | lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataNoStartDate) 72 | mock_query_by_partition_key.assert_called_with(partition_key='EventId', 73 | partition_value=u'324') 74 | 75 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 76 | "query_by_partition_and_sort_key", 77 | return_value=['item']) 78 | def test_validid_getrecordsdate_validparamcall(self, mock_query_by_partition_and_sort_key): 79 | lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataStartDate) 80 | mock_query_by_partition_and_sort_key.assert_called_with(partition_key='EventId', 81 | partition_value=u'324', 82 | sort_key='EventDay', 83 | sort_value=20171013) 84 | 85 | def test_validresponse_httpresponse_valid(self): 86 | response = lambda_query_dynamo.HttpUtils.respond() 87 | assert response['statusCode'] == '200' 88 | assert response['body'] == 'null' 89 | 90 | def test_exception_printexception_printedexception(self): 91 | assert lambda_query_dynamo.print_exception(Exception('test')) is None 92 | 93 | -------------------------------------------------------------------------------- /Chapter04/IAM/assume-role-lambda.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "", 6 | "Effect": "Allow", 7 | "Principal": { 8 | "Service": "lambda.amazonaws.com" 9 | }, 10 | "Action": "sts:AssumeRole" 11 | } 12 | ] 13 | } -------------------------------------------------------------------------------- /Chapter04/IAM/dynamo-full-user-visits.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1422032676021", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "dynamodb:BatchGetItem", 9 | "dynamodb:BatchWriteItem", 10 | "dynamodb:DeleteItem", 11 | "dynamodb:DescribeStream", 12 | "dynamodb:DescribeTable", 13 | "dynamodb:GetItem", 14 | "dynamodb:GetRecords", 15 | "dynamodb:GetShardIterator", 16 | "dynamodb:ListStreams", 17 | "dynamodb:ListTables", 18 | "dynamodb:PutItem", 19 | "dynamodb:Query", 20 | "dynamodb:Scan", 21 | "dynamodb:UpdateItem", 22 | "dynamodb:UpdateTable" 23 | ], 24 | "Resource": [ 25 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits", 26 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-sam" 27 | ] 28 | }, 29 | { 30 | "Effect": "Allow", 31 | "Action": "dynamodb:ListTables", 32 | "Resource": "*", 33 | "Condition": {} 34 | } 35 | ] 36 | } -------------------------------------------------------------------------------- /Chapter04/IAM/dynamo-readonly-user-visits.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1422532676335", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "dynamodb:BatchGetItem", 9 | "dynamodb:DescribeTable", 10 | "dynamodb:GetItem", 11 | "dynamodb:Query", 12 | "dynamodb:Scan" 13 | ], 14 | "Resource": [ 15 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits", 16 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-sam", 17 | "arn:aws:dynamodb:eu-west-1:000000000000:table/user-visits-xray-sam" 18 | ] 19 | } 20 | ] 21 | } 22 | -------------------------------------------------------------------------------- /Chapter04/IAM/iam-pass-role.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "Stmt1449755587000", 6 | "Effect": "Allow", 7 | "Action": [ 8 | "iam:GetInstanceProfile", 9 | "iam:GetRole", 10 | "iam:ListInstanceProfiles", 11 | "iam:ListInstanceProfilesForRole", 12 | "iam:PassRole" 13 | ], 14 | "Resource": [ 15 | "*" 16 | ] 17 | } 18 | ] 19 | } -------------------------------------------------------------------------------- /Chapter04/IAM/lambda-cloud-write.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Effect": "Allow", 6 | "Action": [ 7 | "logs:CreateLogGroup", 8 | "logs:CreateLogStream", 9 | "logs:PutLogEvents", 10 | "logs:DescribeLogStreams" 11 | ], 12 | "Resource": [ 13 | "arn:aws:logs:*:*:*" 14 | ] 15 | }, 16 | { 17 | "Effect": "Allow", 18 | "Action": [ 19 | "cloudwatch:PutMetricData" 20 | ], 21 | "Resource": "*" 22 | } 23 | ] 24 | } -------------------------------------------------------------------------------- /Chapter04/README.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"). 4 | You may not use this file except in compliance with the License. 5 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 6 | 7 | ## Serverless Microservice Data API 8 | 9 | In this repository, I share some sample code used to create a scalable serverless data API from my two video courses. It includes bash scripts that can be used to unit test, build, package, deploy, and run integration tests. The Python code is written defensively to deal with any API exception. 10 | 11 | |For beginners and intermediates, the [full Serverless Data API](https://www.packtpub.com/application-development/building-scalable-serverless-microservice-rest-data-api-video) code, configuration and a detailed walk through |For intermediate or advanced users, I cover [15+ serverless microservice patterns](https://www.packtpub.com/application-development/implementing-serverless-microservices-architecture-patterns-video) with original content, code, configuration and detailed walk through | 12 | |:----------|:-------------| 13 | | [![Building a Scalable Serverless Microservice REST Data API Video Course](./images/building-scalable-serverless-microservice-rest-data-api-video.png "Building a Scalable Serverless Microservice REST Data API Video Course")](https://www.packtpub.com/application-development/building-scalable-serverless-microservice-rest-data-api-video)| [![Implementing Serverless Microservices Architecture Patterns Video Course](./images/implementing-serverless-microservices-architecture-patterns-video.png "Implementing Serverless Microservices Architecture Patterns Video Course")](https://www.packtpub.com/application-development/implementing-serverless-microservices-architecture-patterns-video) | 14 | 15 | 16 | ### 1. Windows Only 17 | 18 | Please skip this step if you are not using Windows. 19 | 20 | Using Bash (Unix Shell) makes your life much easier when deploying and managing you serverless stack. I think all analysts, data scientists, architects, administrators, database administrators, developers, DevOps and technical people should know some basic Bash and be able to run shell scripts, which are typically used on LINUX and UNIX (including macOS Terminal). 21 | 22 | Alternatively you can adapt the scripts to use MS-DOS or Powershell but it's not something I recommended, given that Bash can now run natively on Windows 10 as an application, and there are many more examples online in Bash. 23 | 24 | Note that I have stripped off the `\r` or carriage returns, as they are illegal in shell scripts, you can use something like [notepad++](https://notepad-plus-plus.org/) on Windows if you want to view the carriage returns in your files properly. If you use traditional Windows Notepad the new lines may not be rendered at all, so use [Notepad++](https://notepad-plus-plus.org/), [Sublime](https://www.sublimetext.com/), [Atom](https://atom.io/) or other such editors. 25 | 26 | A detailed guide on how to install [Linux Bash shell on Windows 10 can be found here](https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/). The main steps are: 27 | * Control Panel > Programs > Turn Windows Features On Or Off. 28 | * Choose the check box next to the **Windows Subsystem for Linux** option in the list, and then Choose **OK**. 29 | * Under **Microsoft Store > Run Linux on Windows** Select **Ubuntu**. 30 | * Launch Ubuntu and setup a root account username and password 31 | The Windows `C:\` and other dives are already mounted, and you can access them with the following command in the terminal: 32 | ```bash 33 | $ cd /mnt/c/ 34 | ``` 35 | 36 | Well done you now have full access to Linux on Windows! 37 | 38 | ### 2. Update Ubuntu, Install Git and Clone Repository 39 | ```bash 40 | $ sudo apt-get update 41 | $ sudo apt-get -y upgrade 42 | $ apt-get install git-core 43 | ``` 44 | Clone the repository locally with a `git pull` 45 | 46 | ### 3. Install Python and Dependencies 47 | 48 | The Lambda code is written in Python 3.6. Pip is a tool for installing and managing Python packages. Other popular Python package and dependency managers are available such as [Conda](https://conda.io/docs/index.html) or [Pipenv](https://pipenv.readthedocs.io) but we will be using pip as it is the recommended tool for installing packages from the Python Package Index [PyPI](https://pypi.org/) and most widely supported. 49 | 50 | ```bash 51 | $ sudo apt -y install python3.6 52 | $ sudo apt -y install python3-pip 53 | ``` 54 | 55 | Check the Python version 56 | ```bash 57 | $ python3 --version 58 | ``` 59 | You should get the Python3.6+ version. 60 | 61 | The dependent packages required for running, testing and deploying the severless microservices are listed in `requirements.txt` under each project folder, and can be installed using pip. 62 | ```bash 63 | $ sudo pip3 install -r /path/to/requirements.txt 64 | ``` 65 | This will install the dependent libraries for local development such as [Boto3](https://boto3.amazonaws.com) which is the Python AWS Software Development Kit (SDK). 66 | 67 | ### 3. Install and Setup AWS CLI 68 | 69 | AWS Command Line Interface is used to package and deploy your Lambda functions, as well as setup the infrastructure and security in a repeatable way. 70 | 71 | ```bash 72 | $ sudo pip install awscli --upgrade 73 | ``` 74 | 75 | You have created a user called `newuser` earlier and have a `crednetials.csv` with the AWS keys. Enter them them running `aws configure`. 76 | 77 | ```bash 78 | $ aws configure 79 | AWS Access Key ID: 80 | AWS Secret Access Key: 81 | Default region name: 82 | Default output format: 83 | ``` 84 | 85 | More details on setting up the AWS CLI is available [in the AWS Docs](https://docs.aws.amazon.com/lambda/latest/dg/setup-awscli.html). 86 | 87 | For choosing your AWS Region refer to [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) generally those in the USA can use `us-east-1` and those in Europe can use `eu-west-1` 88 | 89 | ### 4. Update AccountId, Bucket and Profile 90 | Here I assume your AWS profile is `demo` you can change that under `bash/apigateway-lambda-dynamodb/common-variables.sh`. 91 | You will need to use a bucket or create one if you haven't already: 92 | ```bash 93 | $ aws s3api create-bucket --bucket mynewbucket231 --profile demo --create-bucket-configuration Loc 94 | ationConstraint=eu-west-1 --region eu-west-1 95 | 96 | ``` 97 | You will also need to change the AWS accountId (current set to 000000000000). The AWS accountId is also used in some IAM policies in the IAM folder. In addition the region will have to be changed. 98 | 99 | to replace your accountId (assume your AWS accountId is 111111111111) you can do it manually or run: 100 | ```bash 101 | find ./ -type f -exec sed -i '' -e 's/000000000000/111111111111/' {} \; 102 | ``` 103 | 104 | ### 5. Run Unit Tests 105 | Change directory to the main bash folder 106 | ```bash 107 | $ cd bash/apigateway-lambda-dynamodb/ 108 | $ ./unit-test-lambda.sh 109 | ``` 110 | 111 | ### 6. Build Package and Deploy the Serverless API 112 | This also creates the IAM Polices and IAM Roles using the AWS CLI that will be required for the Lambda function. 113 | ```bash 114 | $ ./build-package-deploy-lambda-dynamo-data-api.sh 115 | ``` 116 | In less than a minute you should have a stack. Otherwise look at the error messages, CloudFormation stack and ensure your credentials are setup correctly. 117 | 118 | ### 7. Run Lambda Integration Test 119 | Once the stack is up and running, you can run an integration test to check that the Lambda is working. 120 | ```bash 121 | ./invoke-lambda.sh 122 | ``` 123 | 124 | ### 8. Add data to DynamoDb table 125 | 126 | Change to the DynamoDB Python directory and run the Python code, you can also run this under you favourite IDE like PyDev or PyCharm. 127 | ```bash 128 | $ (cd ../../aws_dynamo; python dynamo_insert_items_from_file.py) 129 | ``` 130 | 131 | ### 9. AWS Management Console 132 | Now the stack is up you can have a look at the API Gateway in the AWS Management Console and test the API in your browser. 133 | * API Gateway > lambda-dynamo-data-api 134 | * Stages > Prod > Get 135 | * Copy the Invoke URL into a new tab, e.g. https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/Prod/visits/{resourceId} 136 | * You should get a message `resource_id not a number` as the ID is not valid 137 | * Replace {resourceId} in the URL with 324 138 | if all is working you should see some returned JSON records. Well done if so! 139 | 140 | ### 10. Deleting the stack 141 | Go back to the main bash folder and delete the cloudFormation serverless stack. 142 | 143 | ```bash 144 | $ ./delete-stack.sh 145 | ``` 146 | -------------------------------------------------------------------------------- /Chapter04/aws_dynamo/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter04/aws_dynamo/__init__.py -------------------------------------------------------------------------------- /Chapter04/aws_dynamo/dynamo_insert_items_from_file.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 20 Jan 2018 13 | 14 | @author: Richard Freeman 15 | 16 | This packages inserts records from a file into specified DynamoDB table 17 | 18 | """ 19 | import csv 20 | 21 | from boto3 import resource 22 | 23 | 24 | class DynamoRepository: 25 | def __init__(self, target_dynamo_table, region='eu-west-1'): 26 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 27 | self.target_dynamo_table = target_dynamo_table 28 | self.table = self.dynamodb.Table(self.target_dynamo_table) 29 | 30 | def update_dynamo_event_counter(self, event_name, event_datetime, event_count=1): 31 | response = self.table.update_item( 32 | Key={ 33 | 'EventId': str(event_name), 34 | 'EventDay': int(event_datetime) 35 | }, 36 | ExpressionAttributeValues={":eventCount": int(event_count)}, 37 | UpdateExpression="ADD EventCount :eventCount") 38 | return response 39 | 40 | 41 | def main(): 42 | # For manual deployment 43 | # table_name = 'user-visits' 44 | 45 | # For SAM: 46 | table_name = 'user-visits-sam' 47 | input_data_path = '../sample_data/dynamodb-sample-data.txt' 48 | dynamo_repo = DynamoRepository(table_name) 49 | with open(input_data_path, 'r') as sample_file: 50 | csv_reader = csv.DictReader(sample_file) 51 | for row in csv_reader: 52 | response = dynamo_repo.update_dynamo_event_counter(row['EventId'], 53 | row['EventDay'], 54 | row['EventCount']) 55 | print(response) 56 | 57 | 58 | if __name__ == '__main__': 59 | main() 60 | -------------------------------------------------------------------------------- /Chapter04/aws_dynamo/dynamo_modify_items.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 8 Jan 2018 13 | 14 | @author: Richard Freeman 15 | 16 | This packages inserts records into DynamoDB 17 | 18 | """ 19 | from boto3 import resource 20 | 21 | 22 | class DynamoRepository: 23 | def __init__(self, target_dynamo_table, region='eu-west-1'): 24 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 25 | self.target_dynamo_table = target_dynamo_table 26 | self.table = self.dynamodb.Table(self.target_dynamo_table) 27 | 28 | def update_dynamo_event_counter(self, event_name, event_datetime, event_count=1): 29 | return self.table.update_item( 30 | Key={ 31 | 'EventId': event_name, 32 | 'EventDay': event_datetime 33 | }, 34 | ExpressionAttributeValues={":eventCount": event_count}, 35 | UpdateExpression="ADD EventCount :eventCount") 36 | 37 | 38 | def main(): 39 | # For manual deployment 40 | # table_name = 'user-visits' 41 | 42 | # For SAM deployment 43 | table_name = 'user-visits-sam' 44 | dynamo_repo = DynamoRepository(table_name) 45 | print(dynamo_repo.update_dynamo_event_counter('324', 20171001)) 46 | print(dynamo_repo.update_dynamo_event_counter('324', 20171001, 2)) 47 | print(dynamo_repo.update_dynamo_event_counter('324', 20171002, 5)) 48 | 49 | 50 | if __name__ == '__main__': 51 | main() 52 | -------------------------------------------------------------------------------- /Chapter04/aws_dynamo/dynamo_query_table.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 8 Jan 2018 13 | 14 | @author: Richard Freeman 15 | This package is used to query DynamoDB 16 | """ 17 | import decimal 18 | import json 19 | 20 | from boto3 import resource 21 | from boto3.dynamodb.conditions import Key 22 | 23 | 24 | class DecimalEncoder(json.JSONEncoder): 25 | """Helper class to convert a DynamoDB item to JSON. 26 | """ 27 | def default(self, o): 28 | if isinstance(o, decimal.Decimal): 29 | if o % 1 > 0: 30 | return float(o) 31 | else: 32 | return int(o) 33 | return super(DecimalEncoder, self).default(o) 34 | 35 | 36 | class DynamoRepository: 37 | """abstracts all interactions with DynamoDB including the connection and querying of tables 38 | """ 39 | def __init__(self, target_dynamo_table, region='eu-west-1'): 40 | self.dynamodb = resource(service_name='dynamodb', region_name=region) 41 | self.dynamo_table = target_dynamo_table 42 | self.table = self.dynamodb.Table(self.dynamo_table) 43 | 44 | def query_dynamo_record_by_parition(self, parition_key, parition_value): 45 | try: 46 | response = self.table.query( 47 | KeyConditionExpression=Key(parition_key).eq(parition_value)) 48 | for record in response.get('Items'): 49 | print(json.dumps(record, cls=DecimalEncoder)) 50 | return 51 | 52 | except Exception as e: 53 | print('Exception %s type' % str(type(e))) 54 | print('Exception message: %s ' % str(e)) 55 | 56 | def query_dynamo_record_by_parition_sort_key(self, partition_key, partition_value, 57 | sort_key, sort_value): 58 | try: 59 | response = self.table.query( 60 | KeyConditionExpression=Key(partition_key).eq(partition_value) 61 | & Key(sort_key).gte(sort_value)) 62 | for record in response.get('Items'): 63 | print(json.dumps(record, cls=DecimalEncoder)) 64 | return 65 | 66 | except Exception as e: 67 | print('Exception %s type' % str(type(e))) 68 | print('Exception message: %s ' % str(e)) 69 | 70 | 71 | def main(): 72 | # For manual deployment 73 | # table_name = 'user-visits' 74 | 75 | # For SAM: 76 | table_name = 'user-visits-sam' 77 | partition_key = 'EventId' 78 | partition_value = '324' 79 | sort_key = 'EventDay' 80 | sort_value = 20171001 81 | 82 | dynamo_repo = DynamoRepository(table_name) 83 | print('Reading all data for partition_key:%s' % partition_value) 84 | dynamo_repo.query_dynamo_record_by_parition(partition_key, partition_value) 85 | 86 | print('Reading all data for partition_key:%s with date > %d' % (partition_value, sort_value)) 87 | dynamo_repo.query_dynamo_record_by_parition_sort_key(partition_key, 88 | partition_value, 89 | sort_key, 90 | sort_value) 91 | 92 | 93 | if __name__ == '__main__': 94 | main() 95 | -------------------------------------------------------------------------------- /Chapter04/aws_dynamo/dynamo_table_creation.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | 13 | Created on 8 Jan 2018 14 | 15 | @author: Richard Freeman 16 | 17 | pip install boto3 18 | 19 | This package creates a new DynamoDb table with the specified read and write capacity 20 | 21 | """ 22 | import boto3 23 | 24 | 25 | def create_dynamo_table(table_name_value, enable_streams=False, 26 | read_capacity=1, 27 | write_capacity=1, 28 | region='eu-west-1'): 29 | table_name = table_name_value 30 | print('creating table: ' + table_name) 31 | try: 32 | client = boto3.client(service_name='dynamodb', region_name=region) 33 | print(client.create_table(TableName=table_name, 34 | AttributeDefinitions=[{'AttributeName': 'EventId', 35 | 'AttributeType': 'S'}, 36 | {'AttributeName': 'EventDay', 37 | 'AttributeType': 'N'}], 38 | KeySchema=[{'AttributeName': 'EventId', 39 | 'KeyType': 'HASH'}, 40 | {'AttributeName': 'EventDay', 41 | 'KeyType': 'RANGE'}, 42 | ], 43 | ProvisionedThroughput={'ReadCapacityUnits': read_capacity, 44 | 'WriteCapacityUnits': write_capacity})) 45 | except Exception as e: 46 | print('Exception %s type' % str(type(e))) 47 | print('Exception message: %s ' % str(e)) 48 | 49 | 50 | def main(): 51 | table_name = 'user-visits' 52 | create_dynamo_table(table_name, False, 1, 1) 53 | 54 | 55 | if __name__ == '__main__': 56 | main() 57 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/build-package-deploy-lambda-dynamo-data-api.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | # Variables 6 | . ./common-variables.sh 7 | 8 | # Run unit tests 9 | ./unit-test-lambda.sh 10 | 11 | #Create Zip file of your Lambda code (works on Windows and Linux) 12 | ./create-lambda-package.sh 13 | 14 | #Package your Serverless Stack using SAM + Cloudformation 15 | aws cloudformation package --template-file $template.yaml \ 16 | --output-template-file ../../package/$template-output.yaml \ 17 | --s3-bucket $bucket --s3-prefix $prefix \ 18 | --region $region --profile $profile 19 | 20 | #Deploy your Serverless Stack using SAM + Cloudformation 21 | aws cloudformation deploy --template-file ../../package/$template-output.yaml \ 22 | --stack-name $template --capabilities CAPABILITY_IAM \ 23 | --parameter-overrides AccountId=${aws_account_id} \ 24 | --region $region --profile $profile -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/common-variables.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | export profile="demo" 6 | export region="eu-west-1" 7 | # export aws_account_id=$(aws sts get-caller-identity --query 'Account' --profile $profile | tr -d '\"') 8 | export aws_account_id="000000000000" 9 | export template="lambda-dynamo-data-api" 10 | export bucket="testbucket121f" 11 | export prefix="tmp/sam" 12 | 13 | # Lambda settings 14 | export zip_file="lambda-dynamo-data-api.zip" 15 | export files="lambda_return_dynamo_records.py" 16 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/create-lambda-package.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | # This script creates a Zip package of the Lambda files 6 | 7 | #setup environment variables 8 | . ./common-variables.sh 9 | 10 | #Create Lambda package and exclude the tests to reduce package size 11 | (cd ../../lambda_dynamo_read; 12 | mkdir -p ../package/ 13 | zip -FSr ../package/"${zip_file}" ${files} -x *tests/*) 14 | 15 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/create-role.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #This Script creates a Lambda role and attaches the policies 6 | 7 | #setup environment variables 8 | . ./common-variables.sh 9 | 10 | #Setup Lambda Role 11 | role_name=lambda-dynamo-data-api 12 | aws iam create-role --role-name ${role_name} \ 13 | --assume-role-policy-document file://../../IAM/assume-role-lambda.json \ 14 | --profile $profile || true 15 | 16 | sleep 1 17 | #Add and attach DynamoDB Policy 18 | dynamo_policy=dynamo-readonly-user-visits 19 | aws iam create-policy --policy-name $dynamo_policy \ 20 | --policy-document file://../../IAM/$dynamo_policy.json \ 21 | --profile $profile || true 22 | 23 | role_policy_arn="arn:aws:iam::$aws_account_id:policy/$dynamo_policy" 24 | aws iam attach-role-policy \ 25 | --role-name "${role_name}" \ 26 | --policy-arn "${role_policy_arn}" --profile ${profile} || true 27 | 28 | #Add and attach cloudwatch_policy 29 | cloudwatch_policy=lambda-cloud-write 30 | aws iam create-policy --policy-name $cloudwatch_policy \ 31 | --policy-document file://../../IAM/$cloudwatch_policy.json \ 32 | --profile $profile || true 33 | 34 | role_policy_arn="arn:aws:iam::$aws_account_id:policy/$cloudwatch_policy" 35 | aws iam attach-role-policy \ 36 | --role-name "${role_name}" \ 37 | --policy-arn "${role_policy_arn}" --profile ${profile} || true 38 | 39 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/curl-api-gateway.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #endpoint="https://xxxxx.execute-api.eu-west-1.amazonaws.com/Prod/visits/324" 6 | . ./common-variables.sh 7 | 8 | endpoint="$(python3 get_apigateway_endpoint.py -e ${template})" 9 | echo ${endpoint} 10 | status_code=$(curl -i -H \"Accept: application/json\" -H \"Content-Type: application/json\" -X GET ${endpoint}) 11 | echo "$status_code" 12 | if echo "$status_code" | grep -q "HTTP/1.1 200 OK"; 13 | then 14 | echo "pass" 15 | exit 0 16 | else 17 | exit 1 18 | fi 19 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/delete-stack.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | . ./common-variables.sh 6 | 7 | aws cloudformation delete-stack --stack-name $template --region $region --profile $profile 8 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/get_apigateway_endpoint.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 23 Dec 2018 13 | @author: Richard Freeman 14 | 15 | This module gets an API Gateway endpoint based on the name 16 | sudo python3.6 -m pip install boto3 17 | 18 | """ 19 | 20 | import argparse 21 | import logging 22 | 23 | import boto3 24 | logging.getLogger('botocore').setLevel(logging.CRITICAL) 25 | 26 | logger = logging.getLogger(__name__) 27 | logging.basicConfig(format='%(asctime)s %(levelname)s %(name)-15s: %(lineno)d %(message)s', 28 | level=logging.INFO) 29 | logger.setLevel(logging.INFO) 30 | 31 | 32 | def get_apigateway_names(endpoint_name): 33 | client = boto3.client(service_name='apigateway', region_name='eu-west-1') 34 | apis = client.get_rest_apis() 35 | for api in apis['items']: 36 | if api['name'] == endpoint_name: 37 | api_id = api['id'] 38 | region = 'eu-west-1' 39 | stage = 'Prod' 40 | resource = 'visits/324' 41 | #return F"https://{api_id}.execute-api.{region}.amazonaws.com/{stage}/{resource}" 42 | return "https://%s.execute-api.%s.amazonaws.com/%s/%s" % (api_id, region, stage, resource) 43 | return None 44 | 45 | 46 | def main(): 47 | endpoint_name = "lambda-dynamo-xray" 48 | 49 | parser = argparse.ArgumentParser() 50 | parser.add_argument("-e", "--endpointname", type=str, required=False, help="Path to the endpoint_name") 51 | args = parser.parse_args() 52 | 53 | if (args.endpointname is not None): endpoint_name = args.endpointname 54 | 55 | apigateway_endpoint = get_apigateway_names(endpoint_name) 56 | if apigateway_endpoint is not None: 57 | print(apigateway_endpoint) 58 | return 0 59 | else: 60 | return 1 61 | 62 | 63 | if __name__ == '__main__': 64 | main() 65 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/get_apigateway_id.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 23 Dec 2018 13 | @author: Richard Freeman 14 | 15 | This module gets an API Gateway ID based on the name 16 | sudo python3.6 -m pip install boto3 17 | 18 | """ 19 | 20 | import argparse 21 | import logging 22 | 23 | import boto3 24 | logging.getLogger('botocore').setLevel(logging.CRITICAL) 25 | 26 | logger = logging.getLogger(__name__) 27 | logging.basicConfig(format='%(asctime)s %(levelname)s %(name)-15s: %(lineno)d %(message)s', 28 | level=logging.INFO) 29 | logger.setLevel(logging.INFO) 30 | 31 | 32 | def get_apigateway_id(endpoint_name): 33 | client = boto3.client(service_name='apigateway', region_name='eu-west-1') 34 | apis = client.get_rest_apis() 35 | for api in apis['items']: 36 | if api['name'] == endpoint_name: 37 | return api['id'] 38 | return None 39 | 40 | 41 | def main(): 42 | endpoint_name = "lambda-dynamo-xray" 43 | 44 | parser = argparse.ArgumentParser() 45 | parser.add_argument("-e", "--endpointname", type=str, required=False, help="Path to the endpoint_id") 46 | args = parser.parse_args() 47 | 48 | if (args.endpointname is not None): endpoint_name = args.endpointname 49 | 50 | apigateway_id = get_apigateway_id(endpoint_name) 51 | if apigateway_id is not None: 52 | print(apigateway_id) 53 | return 0 54 | else: 55 | return 1 56 | 57 | 58 | if __name__ == '__main__': 59 | main() 60 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/invoke-lambda.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | . ./common-variables.sh 6 | rm outputfile.tmp 7 | status_code=$(aws lambda invoke --invocation-type RequestResponse \ 8 | --function-name ${template}-sam --region ${region} \ 9 | --payload file://../../sample_data/request-api-gateway-get-valid.json outputfile.tmp \ 10 | --profile ${profile}) 11 | #status_code=$(aws lambda invoke --invocation-type Event --function-name lambda-dynamo-xray-sam \ 12 | # --region eu-west-1 --region eu-west-1 --payload file://../../sample_data/request-api-gateway-get-valid.json \ 13 | # outputfile.tmp) 14 | echo "$status_code" 15 | if echo "$status_code" | grep -q "200"; 16 | then 17 | cat outputfile.tmp 18 | if grep -q error outputfile.tmp; 19 | then 20 | echo "\nerror in response" 21 | exit 1 22 | else 23 | echo "\npass" 24 | exit 0 25 | fi 26 | else 27 | echo "\nerror status not 200" 28 | exit 1 29 | fi -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/lambda-dynamo-data-api.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: 'AWS::Serverless-2016-10-31' 3 | Description: >- 4 | This Lambda is invoked by API Gateway and queries DynamoDB. 5 | Parameters: 6 | AccountId: 7 | Type: String 8 | Resources: 9 | lambdadynamodataapi: 10 | Type: AWS::Serverless::Function 11 | Properties: 12 | Handler: lambda_return_dynamo_records.lambda_handler 13 | Runtime: python3.6 14 | CodeUri: ../../package/lambda-dynamo-data-api.zip 15 | FunctionName: lambda-dynamo-data-api-sam 16 | Description: >- 17 | This Lambda is invoked by API Gateway and queries DynamoDB. 18 | MemorySize: 128 19 | Timeout: 3 20 | Role: !Sub 'arn:aws:iam::${AccountId}:role/lambda-dynamo-data-api' 21 | Environment: 22 | Variables: 23 | environment: dev 24 | Events: 25 | CatchAll: 26 | Type: Api 27 | Properties: 28 | Path: /visits/{resourceId} 29 | Method: GET 30 | DynamoDBTable: 31 | Type: AWS::DynamoDB::Table 32 | Properties: 33 | TableName: user-visits-sam 34 | SSESpecification: 35 | SSEEnabled: True 36 | AttributeDefinitions: 37 | - AttributeName: EventId 38 | AttributeType: S 39 | - AttributeName: EventDay 40 | AttributeType: N 41 | KeySchema: 42 | - AttributeName: EventId 43 | KeyType: HASH 44 | - AttributeName: EventDay 45 | KeyType: RANGE 46 | ProvisionedThroughput: 47 | ReadCapacityUnits: 1 48 | WriteCapacityUnits: 1 49 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/run_locus.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | #This script gets the API Gateway Id based on the API name ${template}, then runs locust 6 | 7 | . ./common-variables.sh 8 | apiid="$(python3 get_apigateway_id.py -e ${template})" 9 | locust -f ../../test/locust_test_api.py --host=https://${apiid}.execute-api.${region}.amazonaws.com 10 | -------------------------------------------------------------------------------- /Chapter04/bash/apigateway-lambda-dynamodb/unit-test-lambda.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Copyright (c) 2017-2019 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | # Licensed under the Apache License, Version 2.0 4 | 5 | (cd ../..; 6 | python3 -m unittest discover test) 7 | -------------------------------------------------------------------------------- /Chapter04/images/building-scalable-serverless-microservice-rest-data-api-video.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter04/images/building-scalable-serverless-microservice-rest-data-api-video.png -------------------------------------------------------------------------------- /Chapter04/images/implementing-serverless-microservices-architecture-patterns-video.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter04/images/implementing-serverless-microservices-architecture-patterns-video.png -------------------------------------------------------------------------------- /Chapter04/lambda_dynamo_read/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Serverless-Microservices-in-Python/da482bd1cbc14e702686e87f95270cee7d9cbad8/Chapter04/lambda_dynamo_read/__init__.py -------------------------------------------------------------------------------- /Chapter04/lambda_dynamo_read/lambda_return_dynamo_records.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 30 Dec 2017 13 | @author: Richard Freeman 14 | 15 | This Lambda queries the remote DynamoDB for a specific partition and greater than a 16 | specific sort key. 17 | 18 | """ 19 | import json 20 | import decimal 21 | 22 | from boto3 import resource 23 | from boto3.dynamodb.conditions import Key 24 | 25 | 26 | class HttpUtils: 27 | def __init__(self): 28 | pass 29 | 30 | @staticmethod 31 | def parse_parameters(event): 32 | try: 33 | return_parameters = event['queryStringParameters'].copy() 34 | except Exception: 35 | return_parameters = {} 36 | try: 37 | resource_id = event.get('path', '').split('/')[-1] 38 | if resource_id.isdigit(): 39 | return_parameters['resource_id'] = resource_id 40 | else: 41 | return {"parsedParams": None, "err": 42 | Exception("resource_id not a number")} 43 | except Exception as e: 44 | return {"parsedParams": None, "err": e} # Generally bad idea to expose exceptions 45 | return {"parsedParams": return_parameters, "err": None} 46 | 47 | @staticmethod 48 | def respond(err=None, err_code=400, res=None): 49 | return { 50 | 'statusCode': str(err_code) if err else '200', 51 | 'body': '{"message":%s}' % json.dumps(str(err)) if err else 52 | json.dumps(res, cls=DecimalEncoder), 53 | 'headers': { 54 | 'Content-Type': 'application/json', 55 | 'Access-Control-Allow-Origin': '*' 56 | }, 57 | } 58 | 59 | @staticmethod 60 | def parse_body(event): 61 | try: 62 | return {"body": json.loads(event['body']), "err": None} 63 | except Exception as e: 64 | return {"body": None, "err": e} 65 | 66 | 67 | class DecimalEncoder(json.JSONEncoder): 68 | def default(self, o): 69 | if isinstance(o, decimal.Decimal): 70 | if o % 1 > 0: 71 | return float(o) 72 | else: 73 | return int(o) 74 | return super(DecimalEncoder, self).default(o) 75 | 76 | 77 | class DynamoRepository: 78 | def __init__(self, table_name): 79 | self.dynamo_client = resource(service_name='dynamodb', 80 | region_name='eu-west-1') 81 | self.table_name = table_name 82 | self.db_table = self.dynamo_client.Table(table_name) 83 | 84 | def query_by_partition_and_sort_key(self, partition_key, partition_value, 85 | sort_key, sort_value): 86 | response = self.db_table.query(KeyConditionExpression= 87 | Key(partition_key).eq(partition_value) 88 | & Key(sort_key).gte(sort_value)) 89 | 90 | return response.get('Items') 91 | 92 | def query_by_partition_key(self, partition_key, partition_value): 93 | response = self.db_table.query(KeyConditionExpression= 94 | Key(partition_key).eq(partition_value)) 95 | 96 | return response.get('Items') 97 | 98 | 99 | def print_exception(e): 100 | try: 101 | print('Exception %s type' % str(type(e))) 102 | print('Exception message: %s ' % str(e)) 103 | except Exception: 104 | pass 105 | 106 | 107 | class Controller(): 108 | def __init__(self): 109 | pass 110 | 111 | @staticmethod 112 | def get_dynamodb_records(event): 113 | try: 114 | validation_result = HttpUtils.parse_parameters(event) 115 | if validation_result.get('parsedParams', None) is None: 116 | return HttpUtils.respond(err=validation_result['err'], err_code=404) 117 | resource_id = str(validation_result['parsedParams']["resource_id"]) 118 | if validation_result['parsedParams'].get("startDate") is None: 119 | result = repo.query_by_partition_key(partition_key="EventId", partition_value=resource_id) 120 | else: 121 | start_date = int(validation_result['parsedParams']["startDate"]) 122 | result = repo.query_by_partition_and_sort_key(partition_key="EventId", partition_value=resource_id, 123 | sort_key="EventDay", sort_value=start_date) 124 | return HttpUtils.respond(res=result) 125 | 126 | except Exception as e: 127 | print_exception(e) 128 | return HttpUtils.respond(err=Exception('Not found'), err_code=404) 129 | 130 | 131 | # For manual deployment 132 | # table_name = 'user-visits' 133 | 134 | # For SAM deployment: 135 | table_name = 'user-visits-sam' 136 | repo = DynamoRepository(table_name=table_name) 137 | 138 | 139 | def lambda_handler(event, context): 140 | response = Controller.get_dynamodb_records(event) 141 | return response 142 | -------------------------------------------------------------------------------- /Chapter04/requirements.txt: -------------------------------------------------------------------------------- 1 | pyaml>=17.8.0 2 | nose>=1.3.7 3 | boto3>=1.6.11 4 | nose2[coverage_plugin]>=0.6.5 5 | locustio>=0.9.0 6 | -------------------------------------------------------------------------------- /Chapter04/sample_data/dynamodb-sample-data.txt: -------------------------------------------------------------------------------- 1 | EventId,EventDay,EventCount 2 | 324,20171010,2 3 | 324,20171012,10 4 | 324,20171013,10 5 | 324,20171014,6 6 | 324,20171016,6 7 | 324,20171017,2 8 | 300,20171011,1 9 | 300,20171013,3 10 | 300,20171014,30 -------------------------------------------------------------------------------- /Chapter04/sample_data/request-api-gateway-get-error.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "foo": "bar" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "GET", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/1234f" 57 | } -------------------------------------------------------------------------------- /Chapter04/sample_data/request-api-gateway-get-valid-date.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "startDate": "20171014" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "POST", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/324" 57 | } -------------------------------------------------------------------------------- /Chapter04/sample_data/request-api-gateway-get-valid-no-date.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "POST", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "headers": { 27 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 28 | "Accept-Language": "en-US,en;q=0.8", 29 | "CloudFront-Is-Desktop-Viewer": "true", 30 | "CloudFront-Is-SmartTV-Viewer": "false", 31 | "CloudFront-Is-Mobile-Viewer": "false", 32 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 33 | "CloudFront-Viewer-Country": "US", 34 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 35 | "Upgrade-Insecure-Requests": "1", 36 | "X-Forwarded-Port": "443", 37 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 38 | "X-Forwarded-Proto": "https", 39 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 40 | "CloudFront-Is-Tablet-Viewer": "false", 41 | "Cache-Control": "max-age=0", 42 | "User-Agent": "Custom User Agent String", 43 | "CloudFront-Forwarded-Proto": "https", 44 | "Accept-Encoding": "gzip, deflate, sdch" 45 | }, 46 | "pathParameters": { 47 | "proxy": "path/to/resource" 48 | }, 49 | "httpMethod": "POST", 50 | "stageVariables": { 51 | "baz": "qux" 52 | }, 53 | "path": "/path/to/resource/324" 54 | } -------------------------------------------------------------------------------- /Chapter04/sample_data/request-api-gateway-get-valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "body": "{\"test\":\"body\"}", 3 | "resource": "/{proxy+}", 4 | "requestContext": { 5 | "resourceId": "123456", 6 | "apiId": "1234567890", 7 | "resourcePath": "/{proxy+}", 8 | "httpMethod": "GET", 9 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 10 | "accountId": "123456789012", 11 | "identity": { 12 | "apiKey": null, 13 | "userArn": null, 14 | "cognitoAuthenticationType": null, 15 | "caller": null, 16 | "userAgent": "Custom User Agent String", 17 | "user": null, 18 | "cognitoIdentityPoolId": null, 19 | "cognitoIdentityId": null, 20 | "cognitoAuthenticationProvider": null, 21 | "sourceIp": "127.0.0.1", 22 | "accountId": null 23 | }, 24 | "stage": "prod" 25 | }, 26 | "queryStringParameters": { 27 | "foo": "bar" 28 | }, 29 | "headers": { 30 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 31 | "Accept-Language": "en-US,en;q=0.8", 32 | "CloudFront-Is-Desktop-Viewer": "true", 33 | "CloudFront-Is-SmartTV-Viewer": "false", 34 | "CloudFront-Is-Mobile-Viewer": "false", 35 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 36 | "CloudFront-Viewer-Country": "US", 37 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 38 | "Upgrade-Insecure-Requests": "1", 39 | "X-Forwarded-Port": "443", 40 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 41 | "X-Forwarded-Proto": "https", 42 | "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", 43 | "CloudFront-Is-Tablet-Viewer": "false", 44 | "Cache-Control": "max-age=0", 45 | "User-Agent": "Custom User Agent String", 46 | "CloudFront-Forwarded-Proto": "https", 47 | "Accept-Encoding": "gzip, deflate, sdch" 48 | }, 49 | "pathParameters": { 50 | "proxy": "path/to/resource" 51 | }, 52 | "httpMethod": "GET", 53 | "stageVariables": { 54 | "baz": "qux" 55 | }, 56 | "path": "/path/to/resource/324" 57 | } -------------------------------------------------------------------------------- /Chapter04/test/locust_test_api.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2018 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | API Gateway load testing 13 | 14 | Created on 25 Aug 2018 15 | 16 | In Shell run 17 | $ sudo pip3 install locustio 18 | $ ./bash/apigateway-lambda-dynamodb/common-variables.sh 19 | $ apiid="$(python bash/apigateway-lambda-dynamodb/get_apigateway_id.py -e ${template})" 20 | $ locust -f test/locust_test_api.py --host=https://${apiid}.execute-api.${region}.amazonaws.com 21 | 22 | Open browser: http://localhost:8089/ 23 | """ 24 | import random 25 | from locust import HttpLocust, TaskSet, task 26 | 27 | paths = ["/Prod/visits/324?startDate=20171014", 28 | "/Prod/visits/324", 29 | "/Prod/visits/320"] 30 | 31 | 32 | class SimpleLocustTest(TaskSet): 33 | 34 | @task 35 | def get_something(self): 36 | index = random.randint(0, len(paths) - 1) 37 | self.client.get(paths[index]) 38 | 39 | 40 | class LocustTests(HttpLocust): 41 | task_set = SimpleLocustTest -------------------------------------------------------------------------------- /Chapter04/test/run_local_api_gateway_lambda_dynamo.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2018 Starwolf Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Created on 31 Dec 2017 13 | 14 | @author: Richard Freeman 15 | 16 | """ 17 | import json 18 | 19 | from lambda_dynamo_read import lambda_return_dynamo_records as lambda_query_dynamo 20 | 21 | with open('../sample_data/request-api-gateway-get-valid-date.json', 'r') as sample_file: 22 | event = json.loads(sample_file.read()) 23 | print("lambda_query_dynamo\nUsing data: %s" % event) 24 | print(sample_file.name.split('/')[-1]) 25 | response = lambda_query_dynamo.lambda_handler(event, None) 26 | print('Response: %s\n' % json.dumps(response)) 27 | 28 | with open('../sample_data/request-api-gateway-get-valid-no-date.json', 'r') as sample_file: 29 | event = json.loads(sample_file.read()) 30 | print(sample_file.name.split('/')[-1]) 31 | print("lambda_query_dynamo\nUsing data: " + str(event)) 32 | response = lambda_query_dynamo.lambda_handler(event, None) 33 | print('Response: %s\n' % json.dumps(response)) 34 | 35 | with open('../sample_data/request-api-gateway-get-error.json', 'r') as sample_file: 36 | event = json.loads(sample_file.read()) 37 | print(sample_file.name.split('/')[-1]) 38 | print("lambda_query_dynamo\nUsing data: " + str(event)) 39 | response = lambda_query_dynamo.lambda_handler(event, None) 40 | print("Response: %s\n" % json.dumps(response)) 41 | 42 | -------------------------------------------------------------------------------- /Chapter04/test/test_dynamo_get.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2017-2019 STARWOLF Ltd and Richard Freeman. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at http://www.apache.org/licenses/LICENSE-2.0 7 | or in the "license" file accompanying this file. This file is distributed 8 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 9 | express or implied. See the License for the specific language governing 10 | permissions and limitations under the License. 11 | 12 | Lambda DynamoDB unit tests and mocking 13 | 14 | pip install mock 15 | 16 | """ 17 | 18 | import sys 19 | import unittest 20 | import json 21 | 22 | from unittest import mock 23 | 24 | from lambda_dynamo_read import lambda_return_dynamo_records as lambda_query_dynamo 25 | 26 | 27 | class TestIndexGetMethod(unittest.TestCase): 28 | def setUp(self): 29 | self.validJsonDataNoStartDate = json.loads('{"httpMethod": "GET","path": "/path/to/resource/324","headers": ' \ 30 | 'null} ') 31 | self.validJsonDataStartDate = json.loads('{"queryStringParameters": {"startDate": "20171013"},' \ 32 | '"httpMethod": "GET","path": "/path/to/resource/324","headers": ' \ 33 | 'null} ') 34 | self.invalidJsonUserIdData = json.loads('{"queryStringParameters": {"startDate": "20171013"},' \ 35 | '"httpMethod": "GET","path": "/path/to/resource/324f","headers": ' \ 36 | 'null} ') 37 | self.invalidJsonData = "{ invalid JSON request!} " 38 | 39 | def tearDown(self): 40 | pass 41 | 42 | def test_validparameters_parseparameters_pass(self): 43 | parameters = lambda_query_dynamo.HttpUtils.parse_parameters(self.validJsonDataStartDate) 44 | assert parameters['parsedParams']['startDate'] == u'20171013' 45 | assert parameters['parsedParams']['resource_id'] == u'324' 46 | 47 | def test_emptybody_parsebody_nonebody(self): 48 | body = lambda_query_dynamo.HttpUtils.parse_body(self.validJsonDataStartDate) 49 | assert body['body'] is None 50 | 51 | def test_invalidjson_getrecord_notfound404(self): 52 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.invalidJsonData) 53 | assert result['statusCode'] == '404' 54 | 55 | def test_invaliduserid_getrecord_invalididerror(self): 56 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.invalidJsonUserIdData) 57 | assert result['statusCode'] == '404' 58 | assert json.loads(result['body'])['message'] == "resource_id not a number" 59 | 60 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 61 | "query_by_partition_key", 62 | return_value=['item']) 63 | def test_validid_checkstatus_status200(self, mock_query_by_partition_key): 64 | result = lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataNoStartDate) 65 | assert result['statusCode'] == '200' 66 | 67 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 68 | "query_by_partition_key", 69 | return_value=['item']) 70 | def test_validid_getrecords_validparamcall(self, mock_query_by_partition_key): 71 | lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataNoStartDate) 72 | mock_query_by_partition_key.assert_called_with(partition_key='EventId', 73 | partition_value=u'324') 74 | 75 | @mock.patch.object(lambda_query_dynamo.DynamoRepository, 76 | "query_by_partition_and_sort_key", 77 | return_value=['item']) 78 | def test_validid_getrecordsdate_validparamcall(self, mock_query_by_partition_and_sort_key): 79 | lambda_query_dynamo.Controller.get_dynamodb_records(self.validJsonDataStartDate) 80 | mock_query_by_partition_and_sort_key.assert_called_with(partition_key='EventId', 81 | partition_value=u'324', 82 | sort_key='EventDay', 83 | sort_value=20171013) 84 | 85 | def test_validresponse_httpresponse_valid(self): 86 | response = lambda_query_dynamo.HttpUtils.respond() 87 | assert response['statusCode'] == '200' 88 | assert response['body'] == 'null' 89 | 90 | def test_exception_printexception_printedexception(self): 91 | assert lambda_query_dynamo.print_exception(Exception('test')) is None 92 | 93 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Packt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | # Building Serverless Microservices in Python 5 | 6 | Building Serverless Microservices in Python 7 | 8 | This is the code repository for [Building Serverless Microservices in Python](https://www.packtpub.com/application-development/building-serverless-microservices-python?utm_source=github&utm_medium=repository&utm_campaign=9781789535297), published by Packt. 9 | 10 | **A practical guide for developing end-to-end serverless microservices in Python for developers, DevOps, and architects.** 11 | 12 | ## What is this book about? 13 | Over the last few years, there has been a massive shift from monolithic architecture to microservices, thanks to their small and independent deployments that allow increased flexibility and agile delivery. Traditionally, virtual machines and containers were the principal mediums for deploying microservices, but they involved a lot of operational effort, configuration, and maintenance. More recently, serverless computing has gained popularity due to its built-in autoscaling abilities, reduced operational costs, and increased productivity. 14 | 15 | This book covers the following exciting features: 16 | * Discover what microservices offer above and beyond other architectures 17 | * Create a serverless application with AWS 18 | * Gain secure access to data and resources 19 | * Run tests on your configuration and code 20 | * Create a highly available serverless microservice data API 21 | Build, deploy, and run your serverless configuration and code 22 | 23 | If you feel this book is for you, get your [copy](https://www.amazon.com/dp/1789535298) today! 24 | 25 | https://www.packtpub.com/ 27 | 28 | ## Instructions and Navigations 29 | All of the code is organized into folders. For example, Chapter02. 30 | 31 | The code will look like the following: 32 | ``` 33 | "phoneNumbers": [ 34 | { 35 | "type": "home", 36 | "number": "212 555-1234" 37 | }, 38 | { 39 | ``` 40 | 41 | **Following is what you need for this book:** 42 | If you are a developer with basic knowledge of Python and want to learn how to build, test, deploy, and secure microservices, then this book is for you. No prior knowledge of building microservices is required. 43 | 44 | With the following software and hardware list you can run all code files present in the book (Chapter 1-). 45 | ### Software and Hardware List 46 | | Chapter | Software required | OS required | 47 | | -------- | ------------------------------------ | ----------------------------------- | 48 | | All | Python 3 | Windows/macOS/Linux | 49 | | All | Docker | Windows/macOS/Linux | 50 | 51 | We also provide a PDF file that has color images of the screenshots/diagrams used in this book. [Click here to download it](https://www.packtpub.com/sites/default/files/downloads/9781789535297_ColorImages.pdf). 52 | 53 | ### Related products 54 | * Cloud Native Architectures [[Packt]](https://prod.packtpub.com/in/application-development/cloud-native-architectures?utm_source=github&utm_medium=repository&utm_campaign=) [[Amazon]](https://www.amazon.com/dp/B0788SDV7W) 55 | 56 | * Serverless Programming Cookbook [[Packt]](https://prod.packtpub.com/in/application-development/serverless-programming-cookbook?utm_source=github&utm_medium=repository&utm_campaign=) [[Amazon]](https://www.amazon.com/dp/1788623797) 57 | 58 | ## Get to Know the Author 59 | **Richard Takashi Freeman** 60 | has an M.Eng. in computer system engineering and a PhD in machine learning and natural language processing from the University of Manchester, United Kingdom. His current role is as a lead data engineer and architect, but he is also a data scientist and solutions architect. He has been delivering cloud-based, big data, machine learning, and data pipeline serverless and scalable solutions for over 14 years, and has spoken at numerous leading academic and industrial conferences, events, and summits. 61 | 62 | He has written various technical blog posts, and has acquired in excess of three years' serverless production experience in connection with large-scale, consumer-facing websites and products. 63 | 64 | ### Suggestions and Feedback 65 | [Click here](https://docs.google.com/forms/d/e/1FAIpQLSdy7dATC6QmEL81FIUuymZ0Wy9vH1jHkvpY57OiMeKGqib_Ow/viewform) if you have any feedback or suggestions. 66 | ### Download a free PDF 67 | 68 | If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.
Simply click on the link to claim your free PDF. 69 |

https://packt.link/free-ebook/9781789535297

--------------------------------------------------------------------------------