├── LICENSE ├── .gitignore ├── README.md └── lambda_s3_access_using_vpc_endpoint.cform /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Geoff Ford 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Lambda access to S3 via VPC Endpoint 2 | 3 | AWS CloudFormation script that demonstrates a Lambda function running within a VPC and accessing S3 using a VPC Endpoint. 4 | 5 | The script creates an S3 bucket, and a Lambda function that creates a record within that bucket. The Lambda is associated 6 | to a VPC that only contains private subnets (i.e. there are no Internet/NAT Gateways) and a VPC Endpoint to S3, allowing 7 | access to the S3 bucket only. 8 | 9 | The VPC that the Lambda function is associated with is created using the script in [VPC](https://github.com/gford1000-aws/vpc), 10 | creating up to 6 private subnets (to which the Lambda is associated) with a CIDR of your choice. 11 | 12 | The script creates a nested stack, constructing the VPC separately from the Lambda and S3 bucket, for clarity. 13 | 14 | Notes: 15 | 16 | 1. the VPC *must* have ```EnableDnsSupport = true``` so that DNS resolution of URLs can be performed. 17 | 18 | 2. the Lambda IAM Role includes ```ec2:CreateNetworkInterface```, ```ec2:DescribeNetworkInterfaces```, ```ec2:DeleteNetworkInterface``` to allow the ENI to be created within the VPC, as well as the necessary S3 permission (s3:PutObject) to create the record in the bucket. 19 | 20 | 3. the Lambda Security Group only allows egress via the VPC EndPoint. 21 | 22 | 4. the policy of the VPC EndPoint only allows access create records in the S3 bucket, but allows this for all principals. This can be restricted to Lambda only if required. 23 | 24 | 5. unfortunately, CloudFormation does not return the prefix list value for the VPC Endpoint service, so this must be passed to the script. The value can be 25 | found using the AWS CLI: [aws ec2 describe-prefix-lists](http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-prefix-lists.html) 26 | 27 | 28 | ## Boto3 Specific Notes: 29 | 30 | By default, Boto3 uses *virtual* S3 urls. As a result, they require resolution to the region specific url, and the resolution step requires internet access. This 31 | causes hanging of the Lambda function (until it times out) since there is no internet access. 32 | 33 | To ensure the Lambda can reach S3, Boto3 must use a *path* ```addressing_style``` via the ```Config``` object: 34 | 35 | ```python 36 | import boto3 37 | import botocore.config 38 | 39 | client = boto3.client('s3', 'ap-southeast-2', config=botocore.config.Config(s3={'addressing_style':'path'})) 40 | ``` 41 | 42 | Notes: 43 | 44 | 1. The specified AWS region must correspond to the region in which the Lambda and VPC Endpoint have been deployed. 45 | 46 | 2. The use of the ```Config``` object is benign: if you remove the VPC association, then the Lambda will continue to work via the the internet. 47 | 48 | 3. For more details on S3 urls, read the AWS documentation on [virtual S3 urls](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html). 49 | 50 | 4. For more details on boto3 configurations, see [botocore.config](http://botocore.readthedocs.io/en/latest/reference/config.html#botocore-config). 51 | 52 | 53 | ## Arguments 54 | 55 | | Argument | Description | 56 | | -------------------- |:------------------------------------------------------------------:| 57 | | CidrAddress | First 2 elements of CIDR block, which is extended to be X.Y.0.0/16 | 58 | | PrivateSubnetCount | The number of private subnets to be created (2-6 can be selected) | 59 | | S3EndpointPrefixList | The pl-xxxxxxx identifier for the S3 end point in the region | 60 | | VPCTemplateURL | The S3 url to the VPC Cloudformation script | 61 | 62 | 63 | ## Outputs 64 | 65 | | Output | Description | 66 | | ----------------------- |:-----------------------------------------------------------:| 67 | | Bucket | The name of the S3 bucket to which the Lambda will write | 68 | | Lambda | The name of the Lambda function | 69 | | VPC | The reference to the VPC | 70 | 71 | 72 | ## Licence 73 | 74 | This project is released under the MIT license. See [LICENSE](LICENSE) for details. 75 | -------------------------------------------------------------------------------- /lambda_s3_access_using_vpc_endpoint.cform: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion": "2010-09-09", 3 | "Description": "Creates a VPC with S3 endpoint, showing Lambda in VPC can reach S3 without internet access", 4 | "Parameters" : { 5 | "CidrAddress" : { 6 | "Default" : "10.0", 7 | "Description" : "Initial two values for CIDR address for the VPC, which will be expanded X.Y.0.0/16", 8 | "Type" : "String", 9 | "MinLength" : "3", 10 | "MaxLength" : "7", 11 | "AllowedPattern" : "[0-9]?[0-9]?[0-9]\\.[0-9]?[0-9]?[0-9]", 12 | "ConstraintDescription" : "Must create a valid CIDR" 13 | }, 14 | "PrivateSubnetCount" : { 15 | "Default" : "3", 16 | "Description" : "Number of private subnets to create in the VPC", 17 | "Type" : "Number", 18 | "AllowedValues" : ["2","3","4","5","6"], 19 | "ConstraintDescription" : "Must be between 2 and 6" 20 | }, 21 | "S3EndpointPrefixList" : { 22 | "Description" : "The prefix list 'pl-xxxxxxx' value for S3 in the deployment region. Use 'aws ec2 describe-prefix-lists'", 23 | "Type" : "String" 24 | }, 25 | "VPCTemplateURL" : { 26 | "Description" : "The URL to the template to create the VPC (see https://github.com/gford1000-aws/vpc/blob/master/create_vpc.cform)", 27 | "Type" : "String" 28 | } 29 | }, 30 | "Resources" : { 31 | "VPC" : { 32 | "Type" : "AWS::CloudFormation::Stack", 33 | "Properties" : { 34 | "Parameters" : { 35 | "CidrAddress" : { "Ref" : "CidrAddress" }, 36 | "CreatePublicSubnet" : "false", 37 | "EnableDnsSupport" : "true", 38 | "PrivateSubnetCount" : { "Ref" : "PrivateSubnetCount" } 39 | }, 40 | "TemplateURL" : { "Ref" : "VPCTemplateURL" } 41 | } 42 | }, 43 | "S3Endpoint" : { 44 | "Type" : "AWS::EC2::VPCEndpoint", 45 | "Properties" : { 46 | "PolicyDocument" : { 47 | "Version":"2012-10-17", 48 | "Statement": [ 49 | { 50 | "Effect" : "Allow", 51 | "Principal" : "*", 52 | "Action" : [ "s3:PutObject" ], 53 | "Resource": [ { "Fn::Sub": [ "${Arn}/*", { "Arn": { "Fn::GetAtt" : [ "Bucket", "Arn" ] } } ] } ] 54 | } 55 | ] 56 | }, 57 | "RouteTableIds" : [ { "Fn::GetAtt" : [ "VPC", "Outputs.PrivateSubnetRouteTable" ] } ], 58 | "ServiceName" : { "Fn::Sub" : [ "com.amazonaws.${Region}.s3", { "Region" : { "Ref" : "AWS::Region" } } ] }, 59 | "VpcId" : { "Fn::GetAtt" : [ "VPC", "Outputs.VPC" ] } 60 | } 61 | }, 62 | "Bucket": { 63 | "Type" : "AWS::S3::Bucket", 64 | "Description" : "Bucket that Lambda will write to", 65 | "Properties" : { 66 | "AccessControl" : "Private" 67 | } 68 | }, 69 | "LambdaS3Write" : { 70 | "Type": "AWS::Lambda::Function", 71 | "DependsOn" : "VPC", 72 | "Properties" : { 73 | "Code" : { 74 | "ZipFile" : { 75 | "Fn::Join": [ 76 | "\n", 77 | [ 78 | "import boto3", 79 | "import botocore.config", 80 | "import os", 81 | "from uuid import uuid4", 82 | "", 83 | "BUCKET_NAME = os.environ['BucketName']", 84 | "REGION_NAME = os.environ['RegionName']", 85 | "", 86 | "def lambda_handler(event, context):", 87 | " # Using S3 VPC Endpoint requires 'path' style addressing, to avoid global url resolution", 88 | " # Create client per: http://boto3.readthedocs.io/en/latest/guide/s3.html", 89 | " client = boto3.client('s3', REGION_NAME, config=botocore.config.Config(s3={'addressing_style':'path'}))", 90 | " resp = client.put_object(", 91 | " Bucket=BUCKET_NAME,", 92 | " Key=str(uuid4()),", 93 | " Body=bytearray(\"Hello World\"))", 94 | " print(resp)", 95 | "" 96 | ] 97 | ] 98 | } 99 | }, 100 | "Description" : "Lambda demonstrating internet access, by writing to an S3 bucket", 101 | "Environment" : { 102 | "Variables" : { 103 | "BucketName" : { "Ref": "Bucket" }, 104 | "RegionName" : { "Ref": "AWS::Region" } 105 | } 106 | }, 107 | "Handler" : "index.lambda_handler", 108 | "MemorySize" : 128, 109 | "Role" : { "Fn::GetAtt": [ "LambdaRole", "Arn" ] }, 110 | "Runtime" : "python2.7", 111 | "Timeout" : 5, 112 | "VpcConfig" : { 113 | "SecurityGroupIds" : [ { "Ref" : "LambdaSecurityGroup" }], 114 | "SubnetIds" : { "Fn::Split" : [ ",", { "Fn::GetAtt" : [ "VPC", "Outputs.PrivateSubnets" ] } ] } 115 | } 116 | } 117 | }, 118 | "LambdaSecurityGroup" : { 119 | "Type" : "AWS::EC2::SecurityGroup", 120 | "Properties" : { 121 | "GroupDescription" : "Security Group for Lambda Egress", 122 | "VpcId" : { "Fn::GetAtt" : [ "VPC", "Outputs.VPC" ] }, 123 | "SecurityGroupEgress" : [ 124 | { 125 | "DestinationPrefixListId" : { "Ref" : "S3EndpointPrefixList" }, 126 | "IpProtocol" : "-1" 127 | } 128 | ] 129 | } 130 | }, 131 | "LambdaRole": { 132 | "Type": "AWS::IAM::Role", 133 | "Properties": { 134 | "AssumeRolePolicyDocument": { 135 | "Version": "2012-10-17", 136 | "Statement": [ 137 | { 138 | "Effect": "Allow", 139 | "Principal": { 140 | "Service": "lambda.amazonaws.com" 141 | }, 142 | "Action": "sts:AssumeRole" 143 | } 144 | ] 145 | }, 146 | "Path": "/", 147 | "Policies": [ 148 | { 149 | "PolicyName": "root", 150 | "PolicyDocument": { 151 | "Version": "2012-10-17", 152 | "Statement": [ 153 | { 154 | "Action": [ 155 | "logs:CreateLogGroup", 156 | "logs:CreateLogStream", 157 | "logs:PutLogEvents" 158 | ], 159 | "Resource": "arn:aws:logs:*:*:*", 160 | "Effect": "Allow" 161 | }, 162 | { 163 | "Effect" : "Allow", 164 | "Action" : [ 165 | "s3:PutObject" 166 | ], 167 | "Resource": [ 168 | { 169 | "Fn::Join" : [ "", 170 | [ 171 | { "Fn::GetAtt" : [ "Bucket", "Arn" ] }, 172 | "/*" 173 | ] 174 | ] 175 | } 176 | ] 177 | }, 178 | { 179 | "Effect":"Allow", 180 | "Action":[ 181 | "ec2:CreateNetworkInterface", 182 | "ec2:DescribeNetworkInterfaces", 183 | "ec2:DeleteNetworkInterface" 184 | ], 185 | "Resource": [ 186 | "*" 187 | ] 188 | } 189 | ] 190 | } 191 | } 192 | ] 193 | } 194 | } 195 | }, 196 | "Outputs" : { 197 | "Bucket" : { 198 | "Description" : "The name of the created bucket", 199 | "Value" : { "Ref" : "Bucket" } 200 | }, 201 | "Lambda" : { 202 | "Description" : "The name of the lambda function", 203 | "Value" : { "Ref" : "LambdaS3Write" } 204 | }, 205 | "VPC" : { 206 | "Description" : "VPC Name", 207 | "Value" : { "Ref" : "VPC" } 208 | } 209 | } 210 | } 211 | --------------------------------------------------------------------------------