├── .github └── PULL_REQUEST_TEMPLATE.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── DocumentDBActions.py ├── GDELTDataParser.py ├── LICENSE ├── README.md ├── aws-documentdb-vpc-infra.template ├── gdelt_parse_config.properties └── sam-app ├── README.md ├── api_swagger.yaml ├── document_db_app ├── app.py └── requirements.txt ├── template.yaml └── tests └── unit ├── __init__.py └── test_handler.py /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/aws-samples/amazon-documentdb-serverless-samples/issues), or [recently closed](https://github.com/aws-samples/amazon-documentdb-serverless-samples/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-samples/amazon-documentdb-serverless-samples/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/aws-samples/amazon-documentdb-serverless-samples/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /DocumentDBActions.py: -------------------------------------------------------------------------------- 1 | """ 2 | This file provides the functionality of pulling the username and password from the secrets manager to connect to the DocumentDB that is the username, password and hostname. 3 | For details on the setup of the documentdb and secrets manager. please refer to the blog (TODO : Post the link here) 4 | 5 | Also note the variables that need to be modified to suit your environments 6 | - Db port 7 | - Pem file name. 8 | - Secrets Name that you stored in secrets manager 9 | - Region (Defaults to US East) 10 | - Pem file name to refer to (relative to the execution environment) 11 | 12 | """ 13 | import json 14 | import pymongo 15 | import boto3 16 | import base64 17 | from botocore.exceptions import ClientError 18 | import ConfigParser 19 | 20 | # Define the Global variables 21 | 22 | gdelt_config=ConfigParser.RawConfigParser() 23 | gdelt_config.read('gdelt_parse_config.properties') 24 | 25 | region_name = gdelt_config.get('DEFAULT','region_name') 26 | document_db_port=gdelt_config.get('DEFAULT','document_db_port') 27 | pem_locator=gdelt_config.get('DEFAULT','pem_locator') 28 | username=gdelt_config.get('DEFAULT','docdb_username') 29 | password=gdelt_config.get('DEFAULT','docdb_oassword') 30 | docdb_host=gdelt_config.get('DEFAULT','docdb_host') 31 | 32 | # Create a Secrets Manager client 33 | db_client = pymongo.MongoClient('mongodb://'+username+':'+password+'@'+docdb_host+':'+document_db_port+'/?ssl=true&ssl_ca_certs='+pem_locator) 34 | 35 | 36 | def queryTest(): 37 | 38 | # Specify a holder for the db 39 | GDELT_DB_LOC = db_client['GDELT_DB'] 40 | GDELT_DB_COLL = GDELT_DB_LOC['GDELT__COLL'] 41 | 42 | queryString = GDELT_DB_COLL.find() 43 | 44 | # Assuming an Arrya has been returned print the first string and quit 45 | for rowStr in queryString: 46 | print(rowStr) 47 | break 48 | 49 | def insertData(insert_list): 50 | #Create the GDELT Collection if not there 51 | #list databases 52 | #create GDELT Db 53 | #databases_list = db_client.list_database_names() 54 | #print(databases_list) 55 | 56 | GDELT_DB_LOC = db_client['GDELT_DB'] 57 | GDELT_DB_COLL = GDELT_DB_LOC['GDELT__COLL'] 58 | 59 | 60 | GDELT_DB_COLL.insert_many(insert_list) 61 | 62 | 63 | def insertDataSingle(insert_list): 64 | #Create the GDELT Collection if not there 65 | #list databases 66 | #create GDELT Db 67 | #databases_list = db_client.list_database_names() 68 | #print(databases_list) 69 | 70 | GDELT_DB_LOC = db_client['GDELT_DB'] 71 | GDELT_DB_COLL = GDELT_DB_LOC['GDELT__COLL'] 72 | 73 | GDELT_DB_COLL.insert_many(insert_list) 74 | 75 | 76 | def cleanupDb(): 77 | #Create the GDELT Collection if not there 78 | #list databases 79 | #create GDELT Db 80 | #databases_list = db_client.list_database_names() 81 | #print(databases_list) 82 | 83 | GDELT_DB_LOC = db_client['GDELT_DB'] 84 | GDELT_DB_COLL = GDELT_DB_LOC['GDELT__COLL'] 85 | 86 | GDELT_DB_COLL.delete_many({}) 87 | 88 | queryTest() 89 | 90 | #insertData() 91 | 92 | #cleanupDb() 93 | 94 | 95 | -------------------------------------------------------------------------------- /GDELTDataParser.py: -------------------------------------------------------------------------------- 1 | """ 2 | Parses the raw data from the GDELT site for the current date and loads it into the DocumentDB collection. 3 | This will create a folder gdelt_data to create the files requried to insert the data into the Documnent DB 4 | Note : Its a pre-requisit that the document db has been created. 5 | """ 6 | 7 | import urllib2 8 | import zipfile 9 | import os 10 | import datetime 11 | import ConfigParser 12 | import ast 13 | import pymongo 14 | from ast import literal_eval 15 | 16 | 17 | #load the config file note the file name should be gdelt_parse_config.properties and has to follow the congi file format... 18 | gdelt_config=ConfigParser.RawConfigParser() 19 | gdelt_config.read('gdelt_parse_config.properties') 20 | 21 | #gdelt_file = '20190426'; 22 | gdelt_file = gdelt_config.get('DEFAULT','gdelt_load_date') 23 | #The fields that should be treated as numeric. 24 | gdelt_numfields=ast.literal_eval(gdelt_config.get("DEFAULT",'gdelt_numeric_fields')) 25 | region_name = gdelt_config.get('DEFAULT','region_name') 26 | document_db_port=gdelt_config.get('DEFAULT','document_db_port') 27 | pem_locator=gdelt_config.get('DEFAULT','pem_locator') 28 | username=gdelt_config.get('DEFAULT','docdb_username') 29 | password=gdelt_config.get('DEFAULT','docdb_oassword') 30 | docdb_host=gdelt_config.get('DEFAULT','docdb_host') 31 | 32 | 33 | # Create a Secrets Manager client 34 | db_client = pymongo.MongoClient('mongodb://'+username+':'+password+'@'+docdb_host+':'+document_db_port+'/?ssl=true&ssl_ca_certs='+pem_locator) 35 | GDELT_DB_LOC = db_client['GDELT_DB'] 36 | GDELT_DB_COLL = GDELT_DB_LOC['GDELT__COLL'] 37 | 38 | #The fields as published by the GDELT team 39 | GDELT_HEADER = ["GLOBALEVENTID","SQLDATE","MonthYear","Year","FractionDate","Actor1Code","Actor1Name","Actor1CountryCode","Actor1KnownGroupCode","Actor1EthnicCode","Actor1Religion1Code","Actor1Religion2Code","Actor1Type1Code","Actor1Type2Code","Actor1Type3Code","Actor2Code","Actor2Name","Actor2CountryCode","Actor2KnownGroupCode","Actor2EthnicCode","Actor2Religion1Code","Actor2Religion2Code","Actor2Type1Code","Actor2Type2Code","Actor2Type3Code","IsRootEvent","EventCode","EventBaseCode","EventRootCode","QuadClass","GoldsteinScale","NumMentions","NumSources","NumArticles","AvgTone","Actor1Geo_Type","Actor1Geo_FullName","Actor1Geo_CountryCode","Actor1Geo_ADM1Code","Actor1Geo_Lat","Actor1Geo_Long","Actor1Geo_FeatureID","Actor2Geo_Type","Actor2Geo_FullName","Actor2Geo_CountryCode","Actor2Geo_ADM1Code","Actor2Geo_Lat","Actor2Geo_Long","Actor2Geo_FeatureID","ActionGeo_Type","ActionGeo_FullName","ActionGeo_CountryCode","ActionGeo_ADM1Code","ActionGeo_Lat","ActionGeo_Long","ActionGeo_FeatureID","DATEADDED","SOURCEURL"] 40 | GDELT_DATA_LENGTH = 58; 41 | 42 | #pull a file from S3 and store it locally - just pulling one day data file 43 | url_link = "http://data.gdeltproject.org/events/"+gdelt_file+".export.CSV.zip" 44 | 45 | #Creater a working directory 46 | 47 | path = os.getcwd() 48 | working_dir = path+'/gdelt_data' 49 | 50 | print('Current Working Directory '+path) 51 | 52 | #File download and convert to CSV 53 | #create if not already exists...assume its not existing 54 | if not os.path.isdir(working_dir): 55 | workind_dir_path = os.mkdir(working_dir) 56 | 57 | #remove files created before (makde changed to avoid downloading if you need) 58 | try: 59 | files = [ f for f in os.listdir(working_dir) ] 60 | for f in files: 61 | os.remove(os.path.join(working_dir, f)) 62 | except Exception: 63 | print("Error deleting files, may be empty") 64 | 65 | 66 | gdelt_file_response = urllib2.urlopen(url_link) 67 | response_handler = open(working_dir+'/GDELT_'+gdelt_file+'.zip', 'w') 68 | response_handler.write(gdelt_file_response.read()) 69 | response_handler.close() 70 | gdelt_file_response.close() 71 | 72 | 73 | print("DOWNLOADED...Uncompressing") 74 | 75 | 76 | zipfile_helper = zipfile.ZipFile(working_dir+'/GDELT_'+gdelt_file+'.zip', 'r') 77 | zipfile_helper.extractall(working_dir) 78 | zipfile_helper.close() 79 | 80 | print("Uncompress completed..process the CSV file to create the document data to insert (100) documents at a time") 81 | 82 | 83 | #Try opening the file for loading into DocumentDB 84 | try: 85 | csvfile = open (working_dir+'/'+gdelt_file+'.export.CSV','r') 86 | document_insert=[] 87 | totalcount = 0 88 | for line in csvfile: 89 | line_data = line.split('\t') 90 | json_list =[] 91 | for i in range(GDELT_DATA_LENGTH): 92 | if GDELT_HEADER[i] not in gdelt_numfields: 93 | key_val = "\""+GDELT_HEADER[i]+"\":\""+line_data[i]+"\"" 94 | else: 95 | key_val = "\""+GDELT_HEADER[i]+"\":"+line_data[i] 96 | if i < 58-1 : 97 | json_list.append(key_val+',') 98 | else: 99 | json_list.append(key_val.replace('\n','')) 100 | 101 | jsonstring="".join(json_list) 102 | pdict = python_dict = literal_eval("{"+jsonstring+'}') 103 | document_insert.append(python_dict) 104 | 105 | #print("{"+jsonstring+'}') 106 | 107 | #print(document_insert) 108 | #totalcount+=100 109 | list_length = len(document_insert) 110 | #break 111 | if list_length == 1000: 112 | GDELT_DB_COLL.insert_many(document_insert) 113 | #print("REACHED HUNDRED") 114 | del document_insert[:] 115 | #print(len(document_insert)) 116 | 117 | 118 | #print(document_insert) 119 | #json.dump(line, jsonfile) 120 | #jsonfile.write('\n') 121 | print(totalcount) 122 | except Exception as e: 123 | print(e) 124 | 125 | 126 | 127 | 128 | 129 | 130 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | This Folder contains code that helps with the following 2 | CloudFormation template that can help create the DocumentDB, Cloud9 environment 3 | Load GDELT Data for a given date into Document DB 4 | Deploy Lambda Layer for Python Mongodb driver 5 | Deploy the Lambda Query function 6 | Deploy API that integrate with the Lambda Function 7 | Test this either via API GW Testing console(for Post) or a Browser (for GET) -------------------------------------------------------------------------------- /aws-documentdb-vpc-infra.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion": "2010-09-09", 3 | "Description": "This stack deploys the VPC infrastructure, Cloud9 and DocumentDB for demonstrating connectivity from serverless applications.**WARNING** This template creates an AWS resources and you will be billed for the resources used if you create a stack from this template.", 4 | "Parameters": { 5 | "DBClusterName": { 6 | "Default": "MyDocDB", 7 | "Description": "Cluster name", 8 | "Type": "String", 9 | "MinLength": "1", 10 | "MaxLength": "64", 11 | "AllowedPattern": "[a-zA-Z][a-zA-Z0-9]*(-[a-zA-Z0-9]+)*", 12 | "ConstraintDescription": "Must begin with a letter and contain only alphanumeric characters." 13 | }, 14 | "DBInstanceName": { 15 | "Default": "MyDocDBInstance", 16 | "Description": "Instance name", 17 | "Type": "String", 18 | "MinLength": "1", 19 | "MaxLength": "64", 20 | "AllowedPattern": "[a-zA-Z][a-zA-Z0-9]*(-[a-zA-Z0-9]+)*", 21 | "ConstraintDescription": "Must begin with a letter and contain only alphanumeric characters." 22 | }, 23 | "MasterUser": { 24 | "NoEcho": "false", 25 | "Description": "The database admin account username", 26 | "Default": "docdbadmin", 27 | "Type": "String", 28 | "MinLength": "1", 29 | "MaxLength": "16", 30 | "AllowedPattern": "[a-zA-Z][a-zA-Z0-9]*", 31 | "ConstraintDescription": "Must begin with a letter and contain only alphanumeric characters." 32 | }, 33 | "MasterPassword": { 34 | "NoEcho": "true", 35 | "Description": "The database admin account password", 36 | "Type": "String", 37 | "MinLength": "1", 38 | "MaxLength": "41", 39 | "AllowedPattern": "[a-zA-Z0-9]+", 40 | "ConstraintDescription": "must contain only alphanumeric characters." 41 | }, 42 | "DBInstanceClass": { 43 | "Description": "Instance class. Please refer to: https://docs.aws.amazon.com/documentdb/latest/developerguide/db-instance-classes.html#db-instance-classes-by-region", 44 | "Default": "db.r4.large", 45 | "Type": "String", 46 | "AllowedValues": [ 47 | "db.r4.large", 48 | "db.r4.xlarge", 49 | "db.r4.2xlarge", 50 | "db.r4.4xlarge", 51 | "db.r4.8xlarge", 52 | "db.r4.16xlarge", 53 | "db.r5.large", 54 | "db.r5.xlarge", 55 | "db.r5.2xlarge", 56 | "db.r5.4xlarge", 57 | "db.r5.12xlarge", 58 | "db.r5.24xlarge" 59 | ], 60 | "ConstraintDescription": "Instance type must be of the ones supported for the region. Please refer to: https://docs.aws.amazon.com/documentdb/latest/developerguide/db-instance-classes.html#db-instance-classes-by-region" 61 | } 62 | }, 63 | 64 | "Metadata": { 65 | "AWS::CloudFormation::Interface": { 66 | "ParameterGroups": [ 67 | { 68 | "Label": { 69 | "default": "Enter Amazon DocumentDB Database Configuration" 70 | }, 71 | "Parameters": [ 72 | "DBClusterName", 73 | "DBInstanceName", 74 | "MasterUser", 75 | "MasterPassword", 76 | "DBInstanceClass" 77 | ] 78 | } 79 | ] 80 | } 81 | }, 82 | 83 | "Mappings": { 84 | "SubnetConfig": { 85 | "VPC": { 86 | "CIDR": "10.0.0.0/16" 87 | }, 88 | "PublicOne": { 89 | "CIDR": "10.0.0.0/24" 90 | }, 91 | "PublicTwo": { 92 | "CIDR": "10.0.1.0/24" 93 | }, 94 | "PrivateOne": { 95 | "CIDR": "10.0.2.0/24" 96 | }, 97 | "PrivateTwo": { 98 | "CIDR": "10.0.3.0/24" 99 | } 100 | } 101 | }, 102 | "Resources": { 103 | 104 | "VPC": { 105 | "Type": "AWS::EC2::VPC", 106 | "Properties": { 107 | "EnableDnsSupport": true, 108 | "EnableDnsHostnames": true, 109 | "CidrBlock": { 110 | "Fn::FindInMap": [ 111 | "SubnetConfig", 112 | "VPC", 113 | "CIDR" 114 | ] 115 | }, 116 | "Tags": [ 117 | { 118 | "Key": "Name", 119 | "Value": { 120 | "Fn::Sub": "DocDB-VPC-${AWS::StackName}" 121 | } 122 | } 123 | ] 124 | } 125 | }, 126 | "PublicSubnetOne": { 127 | "Type": "AWS::EC2::Subnet", 128 | "Properties": { 129 | "AvailabilityZone": { 130 | "Fn::Select": [ 131 | 0, 132 | { 133 | "Fn::GetAZs": { 134 | "Ref": "AWS::Region" 135 | } 136 | } 137 | ] 138 | }, 139 | "VpcId": { 140 | "Ref": "VPC" 141 | }, 142 | "CidrBlock": { 143 | "Fn::FindInMap": [ 144 | "SubnetConfig", 145 | "PublicOne", 146 | "CIDR" 147 | ] 148 | }, 149 | "MapPublicIpOnLaunch": true, 150 | "Tags": [ 151 | { 152 | "Key": "Name", 153 | "Value": { 154 | "Fn::Sub": "DocDBVPCPublicOne-${AWS::StackName}" 155 | } 156 | } 157 | ] 158 | } 159 | }, 160 | "PublicSubnetTwo": { 161 | "Type": "AWS::EC2::Subnet", 162 | "Properties": { 163 | "AvailabilityZone": { 164 | "Fn::Select": [ 165 | 1, 166 | { 167 | "Fn::GetAZs": { 168 | "Ref": "AWS::Region" 169 | } 170 | } 171 | ] 172 | }, 173 | "VpcId": { 174 | "Ref": "VPC" 175 | }, 176 | "CidrBlock": { 177 | "Fn::FindInMap": [ 178 | "SubnetConfig", 179 | "PublicTwo", 180 | "CIDR" 181 | ] 182 | }, 183 | "MapPublicIpOnLaunch": true, 184 | "Tags": [ 185 | { 186 | "Key": "Name", 187 | "Value": { 188 | "Fn::Sub": "DocDBVPCPublicTwo-${AWS::StackName}" 189 | } 190 | } 191 | ] 192 | } 193 | }, 194 | "PrivateSubnetOne": { 195 | "Type": "AWS::EC2::Subnet", 196 | "Properties": { 197 | "AvailabilityZone": { 198 | "Fn::Select": [ 199 | 0, 200 | { 201 | "Fn::GetAZs": { 202 | "Ref": "AWS::Region" 203 | } 204 | } 205 | ] 206 | }, 207 | "VpcId": { 208 | "Ref": "VPC" 209 | }, 210 | "CidrBlock": { 211 | "Fn::FindInMap": [ 212 | "SubnetConfig", 213 | "PrivateOne", 214 | "CIDR" 215 | ] 216 | }, 217 | "Tags": [ 218 | { 219 | "Key": "Name", 220 | "Value": { 221 | "Fn::Sub": "DocDBVPCPrivateOne-${AWS::StackName}" 222 | } 223 | } 224 | ] 225 | } 226 | }, 227 | "PrivateSubnetTwo": { 228 | "Type": "AWS::EC2::Subnet", 229 | "Properties": { 230 | "AvailabilityZone": { 231 | "Fn::Select": [ 232 | 1, 233 | { 234 | "Fn::GetAZs": { 235 | "Ref": "AWS::Region" 236 | } 237 | } 238 | ] 239 | }, 240 | "VpcId": { 241 | "Ref": "VPC" 242 | }, 243 | "CidrBlock": { 244 | "Fn::FindInMap": [ 245 | "SubnetConfig", 246 | "PrivateTwo", 247 | "CIDR" 248 | ] 249 | }, 250 | "Tags": [ 251 | { 252 | "Key": "Name", 253 | "Value": { 254 | "Fn::Sub": "DocDBVPCPrivateTwo-${AWS::StackName}" 255 | } 256 | } 257 | ] 258 | } 259 | }, 260 | "InternetGateway": { 261 | "Type": "AWS::EC2::InternetGateway" 262 | }, 263 | "GatewayAttachement": { 264 | "Type": "AWS::EC2::VPCGatewayAttachment", 265 | "Properties": { 266 | "VpcId": { 267 | "Ref": "VPC" 268 | }, 269 | "InternetGatewayId": { 270 | "Ref": "InternetGateway" 271 | } 272 | } 273 | }, 274 | "PublicRouteTable": { 275 | "Type": "AWS::EC2::RouteTable", 276 | "Properties": { 277 | "VpcId": { 278 | "Ref": "VPC" 279 | } 280 | } 281 | }, 282 | "PublicRoute": { 283 | "Type": "AWS::EC2::Route", 284 | "DependsOn": "GatewayAttachement", 285 | "Properties": { 286 | "RouteTableId": { 287 | "Ref": "PublicRouteTable" 288 | }, 289 | "DestinationCidrBlock": "0.0.0.0/0", 290 | "GatewayId": { 291 | "Ref": "InternetGateway" 292 | } 293 | } 294 | }, 295 | "PublicSubnetOneRouteTableAssociation": { 296 | "Type": "AWS::EC2::SubnetRouteTableAssociation", 297 | "Properties": { 298 | "SubnetId": { 299 | "Ref": "PublicSubnetOne" 300 | }, 301 | "RouteTableId": { 302 | "Ref": "PublicRouteTable" 303 | } 304 | } 305 | }, 306 | "PublicSubnetTwoRouteTableAssociation": { 307 | "Type": "AWS::EC2::SubnetRouteTableAssociation", 308 | "Properties": { 309 | "SubnetId": { 310 | "Ref": "PublicSubnetTwo" 311 | }, 312 | "RouteTableId": { 313 | "Ref": "PublicRouteTable" 314 | } 315 | } 316 | }, 317 | "NatGatewayOneAttachment": { 318 | "Type": "AWS::EC2::EIP", 319 | "DependsOn": "GatewayAttachement", 320 | "Properties": { 321 | "Domain": "vpc" 322 | } 323 | }, 324 | "NatGatewayTwoAttachment": { 325 | "Type": "AWS::EC2::EIP", 326 | "DependsOn": "GatewayAttachement", 327 | "Properties": { 328 | "Domain": "vpc" 329 | } 330 | }, 331 | "NatGatewayOne": { 332 | "Type": "AWS::EC2::NatGateway", 333 | "Properties": { 334 | "AllocationId": { 335 | "Fn::GetAtt": [ 336 | "NatGatewayOneAttachment", 337 | "AllocationId" 338 | ] 339 | }, 340 | "SubnetId": { 341 | "Ref": "PublicSubnetOne" 342 | } 343 | } 344 | }, 345 | "NatGatewayTwo": { 346 | "Type": "AWS::EC2::NatGateway", 347 | "Properties": { 348 | "AllocationId": { 349 | "Fn::GetAtt": [ 350 | "NatGatewayTwoAttachment", 351 | "AllocationId" 352 | ] 353 | }, 354 | "SubnetId": { 355 | "Ref": "PublicSubnetTwo" 356 | } 357 | } 358 | }, 359 | "PrivateRouteTableOne": { 360 | "Type": "AWS::EC2::RouteTable", 361 | "Properties": { 362 | "VpcId": { 363 | "Ref": "VPC" 364 | } 365 | } 366 | }, 367 | "PrivateRouteOne": { 368 | "Type": "AWS::EC2::Route", 369 | "Properties": { 370 | "RouteTableId": { 371 | "Ref": "PrivateRouteTableOne" 372 | }, 373 | "DestinationCidrBlock": "0.0.0.0/0", 374 | "NatGatewayId": { 375 | "Ref": "NatGatewayOne" 376 | } 377 | } 378 | }, 379 | "PrivateRouteTableOneAssociation": { 380 | "Type": "AWS::EC2::SubnetRouteTableAssociation", 381 | "Properties": { 382 | "RouteTableId": { 383 | "Ref": "PrivateRouteTableOne" 384 | }, 385 | "SubnetId": { 386 | "Ref": "PrivateSubnetOne" 387 | } 388 | } 389 | }, 390 | "PrivateRouteTableTwo": { 391 | "Type": "AWS::EC2::RouteTable", 392 | "Properties": { 393 | "VpcId": { 394 | "Ref": "VPC" 395 | } 396 | } 397 | }, 398 | "PrivateRouteTwo": { 399 | "Type": "AWS::EC2::Route", 400 | "Properties": { 401 | "RouteTableId": { 402 | "Ref": "PrivateRouteTableTwo" 403 | }, 404 | "DestinationCidrBlock": "0.0.0.0/0", 405 | "NatGatewayId": { 406 | "Ref": "NatGatewayTwo" 407 | } 408 | } 409 | }, 410 | "PrivateRouteTableTwoAssociation": { 411 | "Type": "AWS::EC2::SubnetRouteTableAssociation", 412 | "Properties": { 413 | "RouteTableId": { 414 | "Ref": "PrivateRouteTableTwo" 415 | }, 416 | "SubnetId": { 417 | "Ref": "PrivateSubnetTwo" 418 | } 419 | } 420 | }, 421 | 422 | "DocumentDBSecurityGroup": { 423 | "Type": "AWS::EC2::SecurityGroup", 424 | "Properties": { 425 | "GroupName": { 426 | "Fn::Sub": "SecurityGroup-${AWS::StackName}" 427 | }, 428 | "GroupDescription": "Allow access from VPC", 429 | "VpcId": { 430 | "Ref": "VPC" 431 | }, 432 | "SecurityGroupIngress": [ 433 | { 434 | "IpProtocol": "tcp", 435 | "FromPort": "27017", 436 | "ToPort": "27017", 437 | "CidrIp": "10.0.0.0/16" 438 | } 439 | 440 | ] 441 | } 442 | }, 443 | 444 | "VPCEndpointSecurityGroup": { 445 | "Type": "AWS::EC2::SecurityGroup", 446 | "Properties": { 447 | "GroupName": { 448 | "Fn::Sub": "SecurityGroup-${AWS::StackName}" 449 | }, 450 | "GroupDescription": "Allow HTTPS access from VPC to VPC Endpoint", 451 | "VpcId": { 452 | "Ref": "VPC" 453 | }, 454 | "SecurityGroupIngress": [ 455 | { 456 | "IpProtocol": "tcp", 457 | "FromPort": "443", 458 | "ToPort": "443", 459 | "CidrIp": "10.0.0.0/16" 460 | } 461 | 462 | ] 463 | } 464 | }, 465 | 466 | 467 | "Cloud9Environment": { 468 | "Type": "AWS::Cloud9::EnvironmentEC2", 469 | "Properties": { 470 | "AutomaticStopTimeMinutes": 30, 471 | "InstanceType": "t2.small", 472 | "Name": { 473 | "Fn::Sub": "Project-${AWS::StackName}" 474 | }, 475 | "SubnetId": { 476 | "Ref": "PublicSubnetOne" 477 | } 478 | } 479 | }, 480 | 481 | "DocDBSubnetGroup": { 482 | "Type": "AWS::DocDB::DBSubnetGroup", 483 | "Properties": { 484 | "DBSubnetGroupDescription": "Subnet group for DocumentDB", 485 | "SubnetIds": [ 486 | { 487 | "Ref": "PrivateSubnetOne" 488 | }, 489 | { 490 | "Ref": "PrivateSubnetTwo" 491 | } 492 | ] 493 | } 494 | }, 495 | 496 | "DBCluster": { 497 | "Type": "AWS::DocDB::DBCluster", 498 | "DeletionPolicy": "Delete", 499 | "Properties": { 500 | "DBClusterIdentifier": { 501 | "Ref": "DBClusterName" 502 | }, 503 | "DBSubnetGroupName" : 504 | { 505 | "Ref": "DocDBSubnetGroup" 506 | }, 507 | "VpcSecurityGroupIds" : 508 | [{ "Fn::GetAtt": [ "DocumentDBSecurityGroup" , "GroupId" ] } ], 509 | "MasterUsername": { 510 | "Ref": "MasterUser" 511 | }, 512 | "MasterUserPassword": { 513 | "Ref": "MasterPassword" 514 | } 515 | } 516 | }, 517 | "DBInstance": { 518 | "Type": "AWS::DocDB::DBInstance", 519 | "Properties": { 520 | "DBClusterIdentifier": { 521 | "Ref": "DBCluster" 522 | }, 523 | "DBInstanceIdentifier": { 524 | "Ref": "DBInstanceName" 525 | }, 526 | "DBInstanceClass": { 527 | "Ref": "DBInstanceClass" 528 | } 529 | }, 530 | "DependsOn": "DBCluster" 531 | }, 532 | 533 | "SecretsManagerVPCEndpoint": { 534 | "Type": "AWS::EC2::VPCEndpoint", 535 | "Properties": { 536 | "VpcEndpointType": "Interface", 537 | "PrivateDnsEnabled": true, 538 | "VpcId": { 539 | "Ref": "VPC" 540 | }, 541 | "SubnetIds": [ 542 | { 543 | "Ref": "PrivateSubnetOne" 544 | }, 545 | { 546 | "Ref": "PrivateSubnetTwo" 547 | } 548 | 549 | ], 550 | "SecurityGroupIds": [ 551 | { 552 | "Ref": "VPCEndpointSecurityGroup" 553 | } 554 | ], 555 | "ServiceName": { 556 | "Fn::Join": [ 557 | "", 558 | [ 559 | "com.amazonaws.", 560 | { 561 | "Ref": "AWS::Region" 562 | }, 563 | ".secretsmanager" 564 | ] 565 | ] 566 | } 567 | } 568 | } 569 | 570 | 571 | 572 | }, 573 | "Outputs": { 574 | "VPCid" : { "Value" : { 575 | "Fn::GetAtt" : [ "PrivateSubnetOne" , "VpcId" ] } 576 | }, 577 | 578 | "SecurityGroupId" : { "Value" : { 579 | "Fn::GetAtt" : [ "DocumentDBSecurityGroup" , "GroupId" ] } 580 | }, 581 | 582 | "PrivateSubnet1" : { "Value" : { 583 | "Ref": "PrivateSubnetOne" } 584 | }, 585 | 586 | "PrivateSubnet2" : { "Value" : { 587 | "Ref": "PrivateSubnetTwo" } 588 | }, 589 | 590 | "Cloud9Env": { 591 | "Value": { 592 | "Fn::Join": [ 593 | "", 594 | [ 595 | "https://", 596 | { 597 | "Ref": "AWS::Region" 598 | }, 599 | ".console.aws.amazon.com/cloud9/home/", 600 | 601 | "?region=", 602 | { 603 | "Ref": "AWS::Region" 604 | } 605 | ] 606 | ] 607 | } 608 | }, 609 | "ClusterId": { 610 | "Value": { 611 | "Ref": "DBCluster" 612 | } 613 | }, 614 | "ClusterEndpoint": { 615 | "Value": { 616 | "Fn::GetAtt": [ 617 | "DBCluster", 618 | "Endpoint" 619 | ] 620 | } 621 | }, 622 | "ClusterReadEndpoint": { 623 | "Value": { 624 | "Fn::GetAtt": [ 625 | "DBCluster", 626 | "ReadEndpoint" 627 | ] 628 | } 629 | }, 630 | "ClusterPort": { 631 | "Value": { 632 | "Fn::GetAtt": [ 633 | "DBCluster", 634 | "Port" 635 | ] 636 | } 637 | }, 638 | "InstanceId": { 639 | "Value": { 640 | "Ref": "DBInstance" 641 | } 642 | }, 643 | "InstancePort": { 644 | "Value": { 645 | "Fn::GetAtt": [ 646 | "DBInstance", 647 | "Port" 648 | ] 649 | } 650 | }, 651 | "InstanceEndpoint": { 652 | "Value": { 653 | "Fn::GetAtt": [ 654 | "DBInstance", 655 | "Endpoint" 656 | ] 657 | } 658 | } 659 | } 660 | } -------------------------------------------------------------------------------- /gdelt_parse_config.properties: -------------------------------------------------------------------------------- 1 | [DEFAULT] 2 | #Config parameters 3 | #Enter the date in YYYYMMDD 4 | gdelt_load_date=20190426 5 | region_name=us-east-1 6 | document_db_port=27017 7 | #DOCDB HOST, FIND THIS INFORMATION FROM THE DOCUMENT DB DETAILS ON THE CONSOLE 8 | docdb_host=docdb-XXXX-XX-XX-XX-XX-XX.cluster-xxxxxxxxxxxx.us-east-1.docdb.amazonaws.com 9 | docdb_username=xxxxxxxxxxxx 10 | docdb_oassword=xxxxxxxxxxxx 11 | pem_locator=rds-combined-ca-bundle.pem 12 | #Enter the fields which need to be parsed / entered as numeric fields. defauked to the below ones for this demo 13 | gdelt_numeric_fields=["NumSources","NumArticles","AvgTone","NumMentions"] 14 | 15 | 16 | -------------------------------------------------------------------------------- /sam-app/README.md: -------------------------------------------------------------------------------- 1 | # sam-app 2 | 3 | This is a sample template for sam-app - Below is a brief explanation of what we have generated for you: 4 | 5 | ```bash 6 | . 7 | ├── README.md <-- This instructions file 8 | ├── hello_world <-- Source code for a lambda function 9 | │   ├── __init__.py 10 | │   ├── app.py <-- Lambda function code 11 | │   └── requirements.txt <-- Python dependencies 12 | ├── template.yaml <-- SAM Template 13 | └── tests <-- Unit tests 14 | └── unit 15 | ├── __init__.py 16 | └── test_handler.py 17 | ``` 18 | 19 | ## Requirements 20 | 21 | * AWS CLI already configured with at least PowerUser permission 22 | * [Python 3 installed](https://www.python.org/downloads/) 23 | * [Docker installed](https://www.docker.com/community-edition) 24 | * [Python Virtual Environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) 25 | 26 | ## Setup process 27 | 28 | ### Building the project 29 | 30 | [AWS Lambda requires a flat folder](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html) with the application as well as its dependencies. When you make changes to your source code or dependency manifest, 31 | run the following command to build your project local testing and deployment: 32 | 33 | ```bash 34 | sam build 35 | ``` 36 | 37 | If your dependencies contain native modules that need to be compiled specifically for the operating system running on AWS Lambda, use this command to build inside a Lambda-like Docker container instead: 38 | ```bash 39 | sam build --use-container 40 | ``` 41 | 42 | By default, this command writes built artifacts to `.aws-sam/build` folder. 43 | 44 | ### Local development 45 | 46 | **Invoking function locally through local API Gateway** 47 | 48 | ```bash 49 | sam local start-api 50 | ``` 51 | 52 | If the previous command ran successfully you should now be able to hit the following local endpoint to invoke your function `http://localhost:3000/hello` 53 | 54 | **SAM CLI** is used to emulate both Lambda and API Gateway locally and uses our `template.yaml` to understand how to bootstrap this environment (runtime, where the source code is, etc.) - The following excerpt is what the CLI will read in order to initialize an API and its routes: 55 | 56 | ```yaml 57 | ... 58 | Events: 59 | HelloWorld: 60 | Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api 61 | Properties: 62 | Path: /hello 63 | Method: get 64 | ``` 65 | 66 | ## Packaging and deployment 67 | 68 | AWS Lambda Python runtime requires a flat folder with all dependencies including the application. SAM will use `CodeUri` property to know where to look up for both application and dependencies: 69 | 70 | ```yaml 71 | ... 72 | HelloWorldFunction: 73 | Type: AWS::Serverless::Function 74 | Properties: 75 | CodeUri: hello_world/ 76 | ... 77 | ``` 78 | 79 | Firstly, we need a `S3 bucket` where we can upload our Lambda functions packaged as ZIP before we deploy anything - If you don't have a S3 bucket to store code artifacts then this is a good time to create one: 80 | 81 | ```bash 82 | aws s3 mb s3://BUCKET_NAME 83 | ``` 84 | 85 | Next, run the following command to package our Lambda function to S3: 86 | 87 | ```bash 88 | sam package \ 89 | --output-template-file packaged.yaml \ 90 | --s3-bucket REPLACE_THIS_WITH_YOUR_S3_BUCKET_NAME 91 | ``` 92 | 93 | Next, the following command will create a Cloudformation Stack and deploy your SAM resources. 94 | 95 | ```bash 96 | sam deploy \ 97 | --template-file packaged.yaml \ 98 | --stack-name sam-app \ 99 | --capabilities CAPABILITY_IAM 100 | ``` 101 | 102 | > **See [Serverless Application Model (SAM) HOWTO Guide](https://github.com/awslabs/serverless-application-model/blob/master/HOWTO.md) for more details in how to get started.** 103 | 104 | After deployment is complete you can run the following command to retrieve the API Gateway Endpoint URL: 105 | 106 | ```bash 107 | aws cloudformation describe-stacks \ 108 | --stack-name sam-app \ 109 | --query 'Stacks[].Outputs' 110 | ``` 111 | 112 | ## Testing 113 | 114 | We use **Pytest** and **pytest-mock** for testing our code and you can install it using pip: ``pip install pytest pytest-mock`` 115 | 116 | Next, we run `pytest` against our `tests` folder to run our initial unit tests: 117 | 118 | ```bash 119 | python -m pytest tests/ -v 120 | ``` 121 | 122 | **NOTE**: It is recommended to use a Python Virtual environment to separate your application development from your system Python installation. 123 | 124 | # Appendix 125 | 126 | ### Python Virtual environment 127 | **In case you're new to this**, python3 comes with `virtualenv` library by default so you can simply run the following: 128 | 129 | 1. Create a new virtual environment 130 | 2. Install dependencies in the new virtual environment 131 | 132 | ```bash 133 | python3 -m venv .venv 134 | . .venv/bin/activate 135 | pip install -r requirements.txt 136 | ``` 137 | 138 | 139 | **NOTE:** You can find more information about Virtual Environment at [Python Official Docs here](https://docs.python.org/3/tutorial/venv.html). Alternatively, you may want to look at [Pipenv](https://github.com/pypa/pipenv) as the new way of setting up development workflows 140 | ## AWS CLI commands 141 | 142 | AWS CLI commands to package, deploy and describe outputs defined within the cloudformation stack: 143 | 144 | ```bash 145 | sam package \ 146 | --output-template-file packaged.yaml \ 147 | --s3-bucket REPLACE_THIS_WITH_YOUR_S3_BUCKET_NAME 148 | 149 | sam deploy \ 150 | --template-file packaged.yaml \ 151 | --stack-name sam-app \ 152 | --capabilities CAPABILITY_IAM \ 153 | --parameter-overrides MyParameterSample=MySampleValue 154 | 155 | aws cloudformation describe-stacks \ 156 | --stack-name sam-app --query 'Stacks[].Outputs' 157 | ``` 158 | 159 | ## Bringing to the next level 160 | 161 | Here are a few ideas that you can use to get more acquainted as to how this overall process works: 162 | 163 | * Create an additional API resource (e.g. /hello/{proxy+}) and return the name requested through this new path 164 | * Update unit test to capture that 165 | * Package & Deploy 166 | 167 | Next, you can use the following resources to know more about beyond hello world samples and how others structure their Serverless applications: 168 | 169 | * [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/) 170 | * [Chalice Python Serverless framework](https://github.com/aws/chalice) 171 | * Sample Python with 3rd party dependencies, pipenv and Makefile: ``sam init --location https://github.com/aws-samples/cookiecutter-aws-sam-python`` 172 | -------------------------------------------------------------------------------- /sam-app/api_swagger.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | swagger: 2.0 3 | basePath: /prod 4 | info: 5 | title: DocumentDBQueryExample 6 | schemes: 7 | - https 8 | paths: 9 | /: 10 | x-amazon-apigateway-any-method: 11 | produces: 12 | - application/json 13 | responses: 14 | '200': 15 | description: 200 response 16 | schema: 17 | $ref: "#/definitions/Empty" 18 | x-amazon-apigateway-integration: 19 | responses: 20 | default: 21 | statusCode: 200 22 | # NOTE: ${DocumentDbQueryFunction} must match the Lambda resourcename 23 | uri: 24 | Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${DocumentDbQueryFunction.Arn}/invocations" 25 | 26 | passthroughBehavior: when_no_match 27 | httpMethod: POST # Keep "POST" when the API definition method is not POST. This "httpMethod" is used to call Lambda. 28 | type: aws_proxy 29 | /documentdb: 30 | x-amazon-apigateway-any-method: 31 | x-amazon-apigateway-auth: 32 | type: aws_iam 33 | produces: 34 | - application/json 35 | parameters: 36 | - name: proxy 37 | in: path 38 | required: true 39 | type: string 40 | responses: {} 41 | x-amazon-apigateway-integration: 42 | uri: 43 | Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${DocumentDbQueryFunction.Arn}/invocations" 44 | httpMethod: POST # Keep "POST" when the API definition method is not POST. This "httpMethod" is used to call Lambda. 45 | type: aws_proxy 46 | definitions: 47 | Empty: 48 | type: object 49 | title: Empty Schema -------------------------------------------------------------------------------- /sam-app/document_db_app/app.py: -------------------------------------------------------------------------------- 1 | import json 2 | import pymongo 3 | import boto3 4 | import base64 5 | import configparser 6 | import os 7 | import ast 8 | from botocore.exceptions import ClientError 9 | 10 | secret_name = os.environ['secret_name'] 11 | region_name = os.environ['region'] 12 | document_db_port=os.environ['db_port'] 13 | pem_locator=os.environ['pem_locator'] 14 | 15 | session = boto3.session.Session() 16 | client = session.client(service_name='secretsmanager', region_name=region_name) 17 | 18 | get_secret_value_response = "null" 19 | # GET THE SECRET 20 | 21 | try: 22 | get_secret_value_response = client.get_secret_value(SecretId=secret_name) 23 | except ClientError as e: 24 | raise e 25 | 26 | secret_data = json.loads(get_secret_value_response['SecretString']) 27 | 28 | username = secret_data['username'] 29 | password = secret_data['password'] 30 | docdb_host=secret_data['host'] 31 | 32 | db_client = pymongo.MongoClient('mongodb://'+username+':'+password+'@'+docdb_host+':'+document_db_port+'/?ssl=true&ssl_ca_certs='+pem_locator) 33 | 34 | 35 | def lambda_handler(event, context): 36 | 37 | httpmethod = event["httpMethod"] 38 | queryval='' 39 | #Support both get and post if the method invoked is get, then access the query parameters 40 | print("In Lambda Handler Function "+httpmethod+" Method") 41 | if httpmethod == "GET" : 42 | querybodyvaldict=event['multiValueQueryStringParameters'] 43 | queryval = querybodyvaldict['dbquery'][0] 44 | else : 45 | querybodyval=ast.literal_eval(event['body']) 46 | queryval=querybodyval["dbquery"] 47 | 48 | print("Query type " + queryval) 49 | querylower=queryval.lower() 50 | 51 | #A simple error message for usage, Just to get the body right 52 | errorstring="Invalid Query, please pass 'total number of events', 'number of mentions for {keyword}, most talked event" 53 | return_val= { 54 | "isBase64Encoded": "true", 55 | "statusCode": 200, 56 | "headers": { "headerName": "headerValue"}, 57 | "body": errorstring 58 | } 59 | 60 | 61 | try: 62 | GDELT_DB_LOC = db_client['GDELT_DB'] 63 | GDELT_DB_COLL = GDELT_DB_LOC['GDELT_COLL'] 64 | if querylower == "most talked event" : 65 | queryString = GDELT_DB_COLL.find_one(sort=[("NumMentions", -1)]) 66 | topicval = queryString["SOURCEURL"] 67 | topicmention = queryString["NumMentions"] 68 | return_val['body'] = 'Most mentioned article ('+str(topicmention)+' Times) was '+topicval 69 | elif querylower == "total number of events": 70 | queryString = GDELT_DB_COLL.find() 71 | totalcount = queryString.count() 72 | return_val['body'] = "Total Number of events for the day reported " + str(totalcount) 73 | 74 | elif querylower.startswith("number of mentions") : 75 | searchstart = queryval.index('{')+1 76 | searchsend=queryval.index('}') 77 | searchstring = queryval[searchstart:searchsend] 78 | queryDict = ast.literal_eval("{\"SOURCEURL\": {\"$regex\": u\""+searchstring+"\"}}") 79 | queryString = GDELT_DB_COLL.find(queryDict) 80 | counttotal = queryString.count() 81 | 82 | return_val['body'] = searchstring+" was mentioned "+str(counttotal)+" times" 83 | 84 | return return_val 85 | except Exception as ex: 86 | # Send some context about this error to Lambda Logs 87 | print(ex) 88 | 89 | 90 | -------------------------------------------------------------------------------- /sam-app/document_db_app/requirements.txt: -------------------------------------------------------------------------------- 1 | requests==2.20.0 2 | -------------------------------------------------------------------------------- /sam-app/template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: > 4 | sam-app 5 | 6 | Sample SAM Template for sam-app 7 | Parameters: 8 | AutoPublishAliasName: 9 | Type: String 10 | Default: current 11 | Description: The alias used for Auto Publishing 12 | StageName: 13 | Type: String 14 | Default: docdbdemo 15 | Description: The Lambda Function and API Gateway Stage 16 | FunctionName: 17 | Type: String 18 | Default: DocumentDBLamndaExample 19 | Description: The Lambda Function Name 20 | LambdaLayerArn: 21 | Type: String 22 | Default: arn name of the lambda layer 23 | Description: Copy the ARN of the Pymongodb lambda layer arn from your account 24 | SecretManagerName 25 | Type: String 26 | Default: docdbcreds 27 | Description: Enter the name you have given for the document db secrets manager 28 | 29 | 30 | # More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst 31 | Globals: 32 | Function: 33 | Timeout: 25 34 | 35 | 36 | Resources: 37 | 38 | DocumentDbQueryFunction: 39 | Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction 40 | Properties: 41 | AutoPublishAlias: !Ref AutoPublishAliasName 42 | FunctionName: !Sub ${FunctionName}-${StageName} 43 | CodeUri: document_db_app/ 44 | Handler: app.lambda_handler 45 | Layers: 46 | - !Sub ${LambdaLayerArn} 47 | Runtime: python3.6 48 | Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object 49 | Variables: 50 | secret_name: !Ref SecretManagerName 51 | region: us-east-1 52 | db_port: 27017 53 | pem_locator: rds-combined-ca-bundle.pem 54 | Role: !GetAtt LambdFunctionIAMRole.Arn 55 | Events: 56 | AnyRequest : 57 | Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api 58 | Properties: 59 | Path: /documentdb 60 | Method: GET 61 | RestApiId: 62 | Ref: DocumentDBAPI 63 | 64 | #Make sure you edit this data with your own VPC and Subnet ID's 65 | VpcConfig: 66 | SecurityGroupIds: 67 | - sg-xxxxxxxx 68 | SubnetIds: 69 | - subnet-xxxxxxxx 70 | - subnet-xxxxxxxx 71 | 72 | InvokeAPILambdaPermission: 73 | DependsOn : DocumentDbQueryFunction 74 | Type: "AWS::Lambda::Permission" 75 | Properties: 76 | Action: lambda:InvokeFunction 77 | SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${DocumentDBAPI}/*" 78 | FunctionName : !GetAtt DocumentDbQueryFunction.Arn 79 | Principal: apigateway.amazonaws.com 80 | 81 | LambdFunctionIAMRole: 82 | Type: AWS::IAM::Role 83 | Properties: 84 | AssumeRolePolicyDocument: 85 | Version: '2012-10-17' 86 | Statement: 87 | - Effect: Allow 88 | Principal: 89 | Service: 90 | - lambda.amazonaws.com 91 | Action: 92 | - sts:AssumeRole 93 | 94 | Path: "/" 95 | Policies: 96 | - PolicyName: root 97 | PolicyDocument: 98 | Version: '2012-10-17' 99 | Statement: 100 | - Effect: Allow 101 | Action: 102 | - logs:* 103 | - ec2:CreateNetworkInterface 104 | - ec2:DescribeNetworkInterfaces 105 | - ec2:DeleteNetworkInterface 106 | - secretsmanager:* 107 | Resource: "*" 108 | 109 | DocumentDBAPI: 110 | Type: 'AWS::Serverless::Api' 111 | Properties: 112 | StageName: !Ref StageName 113 | DefinitionBody: 114 | 'Fn::Transform': 115 | Name: 'AWS::Include' 116 | Parameters: 117 | Location: api_swagger.yaml 118 | 119 | 120 | -------------------------------------------------------------------------------- /sam-app/tests/unit/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-documentdb-serverless-samples/817cb8039d56217467abe9ecf3b86c62e95a0b31/sam-app/tests/unit/__init__.py -------------------------------------------------------------------------------- /sam-app/tests/unit/test_handler.py: -------------------------------------------------------------------------------- 1 | from collections import namedtuple 2 | import json 3 | 4 | import pytest 5 | 6 | from hello_world import app 7 | 8 | 9 | @pytest.fixture() 10 | def apigw_event(): 11 | """ Generates API GW Event""" 12 | 13 | return { 14 | "body": '{ "test": "body"}', 15 | "resource": "/{proxy+}", 16 | "requestContext": { 17 | "resourceId": "123456", 18 | "apiId": "1234567890", 19 | "resourcePath": "/{proxy+}", 20 | "httpMethod": "POST", 21 | "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", 22 | "accountId": "123456789012", 23 | "identity": { 24 | "apiKey": "", 25 | "userArn": "", 26 | "cognitoAuthenticationType": "", 27 | "caller": "", 28 | "userAgent": "Custom User Agent String", 29 | "user": "", 30 | "cognitoIdentityPoolId": "", 31 | "cognitoIdentityId": "", 32 | "cognitoAuthenticationProvider": "", 33 | "sourceIp": "127.0.0.1", 34 | "accountId": "", 35 | }, 36 | "stage": "prod", 37 | }, 38 | "queryStringParameters": {"foo": "bar"}, 39 | "headers": { 40 | "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 41 | "Accept-Language": "en-US,en;q=0.8", 42 | "CloudFront-Is-Desktop-Viewer": "true", 43 | "CloudFront-Is-SmartTV-Viewer": "false", 44 | "CloudFront-Is-Mobile-Viewer": "false", 45 | "X-Forwarded-For": "127.0.0.1, 127.0.0.2", 46 | "CloudFront-Viewer-Country": "US", 47 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 48 | "Upgrade-Insecure-Requests": "1", 49 | "X-Forwarded-Port": "443", 50 | "Host": "1234567890.execute-api.us-east-1.amazonaws.com", 51 | "X-Forwarded-Proto": "https", 52 | "X-Amz-Cf-Id": "aaaaaaaaaae3VYQb9jd-nvCd-de396Uhbp027Y2JvkCPNLmGJHqlaA==", 53 | "CloudFront-Is-Tablet-Viewer": "false", 54 | "Cache-Control": "max-age=0", 55 | "User-Agent": "Custom User Agent String", 56 | "CloudFront-Forwarded-Proto": "https", 57 | "Accept-Encoding": "gzip, deflate, sdch", 58 | }, 59 | "pathParameters": {"proxy": "/examplepath"}, 60 | "httpMethod": "POST", 61 | "stageVariables": {"baz": "qux"}, 62 | "path": "/examplepath", 63 | } 64 | 65 | 66 | def test_lambda_handler(apigw_event, mocker): 67 | 68 | requests_response_mock = namedtuple("response", ["text"]) 69 | requests_response_mock.text = "1.1.1.1\n" 70 | 71 | request_mock = mocker.patch.object( 72 | app.requests, 'get', side_effect=requests_response_mock) 73 | 74 | ret = app.lambda_handler(apigw_event, "") 75 | assert ret["statusCode"] == 200 76 | 77 | for key in ("message", "location"): 78 | assert key in ret["body"] 79 | 80 | data = json.loads(ret["body"]) 81 | assert data["message"] == "hello world" 82 | --------------------------------------------------------------------------------