├── .gitignore ├── 1-setup └── create-efs-cfn.yml ├── 2-lambda-template ├── hello-world │ ├── app.js │ └── package.json └── template.yaml ├── 3-largefiles ├── processFile │ ├── app.js │ └── package.json └── template.yaml ├── 4-zip ├── template.yaml └── zipFiles │ ├── app.js │ └── package.json ├── 5-unzip ├── template.yaml └── unzipFiles │ ├── app.js │ └── package.json ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Created by https://www.gitignore.io/api/osx,node,linux,windows 3 | 4 | ### Linux ### 5 | *~ 6 | 7 | # temporary files which can be created if a process still has a handle open of a deleted file 8 | .fuse_hidden* 9 | 10 | # KDE directory preferences 11 | .directory 12 | 13 | # Linux trash folder which might appear on any partition or disk 14 | .Trash-* 15 | 16 | # .nfs files are created when an open file is removed but is still being accessed 17 | .nfs* 18 | 19 | ### Node ### 20 | # Logs 21 | logs 22 | *.log 23 | npm-debug.log* 24 | yarn-debug.log* 25 | yarn-error.log* 26 | 27 | # Runtime data 28 | pids 29 | *.pid 30 | *.seed 31 | *.pid.lock 32 | 33 | # Directory for instrumented libs generated by jscoverage/JSCover 34 | lib-cov 35 | 36 | # Coverage directory used by tools like istanbul 37 | coverage 38 | 39 | # nyc test coverage 40 | .nyc_output 41 | 42 | # Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) 43 | .grunt 44 | 45 | # Bower dependency directory (https://bower.io/) 46 | bower_components 47 | 48 | # node-waf configuration 49 | .lock-wscript 50 | 51 | # Compiled binary addons (http://nodejs.org/api/addons.html) 52 | build/Release 53 | 54 | # Dependency directories 55 | node_modules/ 56 | jspm_packages/ 57 | 58 | # Typescript v1 declaration files 59 | typings/ 60 | 61 | # Optional npm cache directory 62 | .npm 63 | 64 | # Optional eslint cache 65 | .eslintcache 66 | 67 | # Optional REPL history 68 | .node_repl_history 69 | 70 | # Output of 'npm pack' 71 | *.tgz 72 | 73 | # Yarn Integrity file 74 | .yarn-integrity 75 | 76 | # dotenv environment variables file 77 | .env 78 | 79 | 80 | ### OSX ### 81 | *.DS_Store 82 | .AppleDouble 83 | .LSOverride 84 | 85 | # Icon must end with two \r 86 | Icon 87 | 88 | # Thumbnails 89 | ._* 90 | 91 | # Files that might appear in the root of a volume 92 | .DocumentRevisions-V100 93 | .fseventsd 94 | .Spotlight-V100 95 | .TemporaryItems 96 | .Trashes 97 | .VolumeIcon.icns 98 | .com.apple.timemachine.donotpresent 99 | 100 | # Directories potentially created on remote AFP share 101 | .AppleDB 102 | .AppleDesktop 103 | Network Trash Folder 104 | Temporary Items 105 | .apdisk 106 | 107 | ### Windows ### 108 | # Windows thumbnail cache files 109 | Thumbs.db 110 | ehthumbs.db 111 | ehthumbs_vista.db 112 | 113 | # Folder config file 114 | Desktop.ini 115 | 116 | # Recycle Bin used on file shares 117 | $RECYCLE.BIN/ 118 | 119 | # Windows Installer files 120 | *.cab 121 | *.msi 122 | *.msm 123 | *.msp 124 | 125 | # Windows shortcuts 126 | *.lnk 127 | 128 | 129 | # End of https://www.gitignore.io/api/osx,node,linux,windows 130 | 131 | # Editor directories and files 132 | .idea 133 | .vscode 134 | *.suo 135 | *.ntvs* 136 | *.njsproj 137 | *.sln 138 | *.sw? 139 | 140 | #AWS/SAM 141 | .aws-sam 142 | packaged.yaml 143 | samconfig.toml 144 | 145 | #jbesw-specific 146 | testData/ 147 | .c9 -------------------------------------------------------------------------------- /1-setup/create-efs-cfn.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: Create VPC, subnets and EFS. 3 | Resources: 4 | MountTargetVPC: 5 | Type: AWS::EC2::VPC 6 | Properties: 7 | CidrBlock: 172.31.0.0/16 8 | EnableDnsHostnames: True 9 | EnableDnsSupport: True 10 | 11 | MountTargetSubnetOne: 12 | Type: AWS::EC2::Subnet 13 | Properties: 14 | CidrBlock: 172.31.1.0/24 15 | VpcId: !Ref MountTargetVPC 16 | AvailabilityZone: "us-east-1a" 17 | 18 | MountTargetSubnetTwo: 19 | Type: AWS::EC2::Subnet 20 | Properties: 21 | CidrBlock: 172.31.2.0/24 22 | VpcId: !Ref MountTargetVPC 23 | AvailabilityZone: "us-east-1b" 24 | 25 | MountTargetSubnetThree: 26 | Type: AWS::EC2::Subnet 27 | Properties: 28 | CidrBlock: 172.31.3.0/24 29 | VpcId: !Ref MountTargetVPC 30 | AvailabilityZone: "us-east-1c" 31 | 32 | FileSystemResource: 33 | Type: 'AWS::EFS::FileSystem' 34 | Properties: 35 | PerformanceMode: maxIO 36 | Encrypted: true 37 | FileSystemTags: 38 | - Key: Name 39 | Value: TestFileSystem 40 | FileSystemPolicy: 41 | Version: "2012-10-17" 42 | Statement: 43 | - Effect: "Allow" 44 | Action: 45 | - "elasticfilesystem:ClientMount" 46 | Principal: 47 | AWS: "*" 48 | 49 | MountTargetResource1: 50 | Type: AWS::EFS::MountTarget 51 | Properties: 52 | FileSystemId: !Ref FileSystemResource 53 | SubnetId: !Ref MountTargetSubnetOne 54 | SecurityGroups: 55 | - !GetAtt MountTargetVPC.DefaultSecurityGroup 56 | 57 | MountTargetResource2: 58 | Type: AWS::EFS::MountTarget 59 | Properties: 60 | FileSystemId: !Ref FileSystemResource 61 | SubnetId: !Ref MountTargetSubnetTwo 62 | SecurityGroups: 63 | - !GetAtt MountTargetVPC.DefaultSecurityGroup 64 | 65 | MountTargetResource3: 66 | Type: AWS::EFS::MountTarget 67 | Properties: 68 | FileSystemId: !Ref FileSystemResource 69 | SubnetId: !Ref MountTargetSubnetThree 70 | SecurityGroups: 71 | - !GetAtt MountTargetVPC.DefaultSecurityGroup 72 | 73 | AccessPointResource: 74 | Type: 'AWS::EFS::AccessPoint' 75 | Properties: 76 | FileSystemId: !Ref FileSystemResource 77 | PosixUser: 78 | Uid: "1000" 79 | Gid: "1000" 80 | RootDirectory: 81 | CreationInfo: 82 | OwnerGid: "1000" 83 | OwnerUid: "1000" 84 | Permissions: "0777" 85 | Path: "/myefs" 86 | -------------------------------------------------------------------------------- /2-lambda-template/hello-world/app.js: -------------------------------------------------------------------------------- 1 | 2 | exports.lambdaHandler = async (event, context) => { 3 | return { 4 | 'statusCode': 200, 5 | 'body': JSON.stringify({ 6 | message: 'hello world' 7 | }) 8 | } 9 | } 10 | -------------------------------------------------------------------------------- /2-lambda-template/hello-world/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "hello-world", 3 | "version": "1.0.0", 4 | "description": "Hello world sample for NodeJS", 5 | "main": "app.js", 6 | "repository": "", 7 | "author": "James Beswick, AWS Serverless", 8 | "license": "MIT-0", 9 | "dependencies": { 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /2-lambda-template/template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: Sample SAM Template for lambda-vpc 4 | 5 | Parameters: 6 | SecurityGroupIds: 7 | Type: CommaDelimitedList 8 | Default: sg-fa7133bc 9 | SubnetIDs: 10 | Type: CommaDelimitedList 11 | Description: The list of SubnetIDs in your Virtual Private Cloud (VPC) 12 | Default: subnet-80c98eae,subnet-04878f4e,subnet-e19ce486,subnet-bfc9fbb0,subnet-0bc28757,subnet-96c847a8 13 | EFSpath: 14 | Type: String 15 | Default: /mnt/efs 16 | AccessPointARN: 17 | Type: String 18 | Description: Access point ARN 19 | 20 | Resources: 21 | HelloWorldFunction: 22 | Type: AWS::Serverless::Function 23 | Properties: 24 | CodeUri: hello-world/ 25 | Timeout: 3 26 | Handler: app.lambdaHandler 27 | Runtime: nodejs12.x 28 | Events: 29 | HelloWorld: 30 | Type: Api 31 | Properties: 32 | Path: /hello 33 | Method: get 34 | VpcConfig: 35 | SecurityGroupIds: !Ref SecurityGroupIds 36 | SubnetIds: !Ref SubnetIDs 37 | FileSystemConfigs: 38 | - Arn: !Ref AccessPointARN 39 | LocalMountPath: !Ref EFSpath 40 | Policies: 41 | - Statement: 42 | - Sid: AWSLambdaVPCAccessExecutionRole 43 | Effect: Allow 44 | Action: 45 | - logs:CreateLogGroup 46 | - logs:CreateLogStream 47 | - logs:PutLogEvents 48 | - ec2:CreateNetworkInterface 49 | - ec2:DescribeNetworkInterfaces 50 | - ec2:DeleteNetworkInterface 51 | Resource: "*" 52 | - Sid: AmazonElasticFileSystemClientFullAccess 53 | Effect: Allow 54 | Action: 55 | - elasticfilesystem:ClientMount 56 | - elasticfilesystem:ClientRootAccess 57 | - elasticfilesystem:ClientWrite 58 | - elasticfilesystem:DescribeMountTargets 59 | Resource: "*" 60 | 61 | Outputs: 62 | # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function 63 | # Find out more about other implicit resources you can reference within SAM 64 | # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api 65 | HelloWorldApi: 66 | Description: "API Gateway endpoint URL for Prod stage for Hello World function" 67 | Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/" 68 | HelloWorldFunction: 69 | Description: "Hello World Lambda Function ARN" 70 | Value: !GetAtt HelloWorldFunction.Arn 71 | HelloWorldFunctionIamRole: 72 | Description: "Implicit IAM Role created for Hello World function" 73 | Value: !GetAtt HelloWorldFunctionRole.Arn 74 | -------------------------------------------------------------------------------- /3-largefiles/processFile/app.js: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 9 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 10 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 11 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 12 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 13 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | */ 15 | 16 | const os = require('os') 17 | 18 | const inputFile = process.env.INPUT_FILE 19 | const efsPath = process.env.EFS_PATH 20 | 21 | const { exec } = require('child_process') 22 | 23 | const execPromise = async (command) => { 24 | console.log(command) 25 | return new Promise((resolve, reject) => { 26 | const ls = exec(command, function (error, stdout, stderr) { 27 | if (error) { 28 | console.log('Error: ', error) 29 | reject(error) 30 | } 31 | console.log('stdout: ', stdout); 32 | console.log('stderr: ' ,stderr); 33 | }) 34 | 35 | ls.on('exit', function (code) { 36 | console.log('Finished: ', code); 37 | resolve() 38 | }) 39 | }) 40 | } 41 | 42 | // The Lambda handler 43 | exports.handler = async function (eventObject, context) { 44 | await execPromise(`/opt/bin/ffmpeg -loglevel error -i ${efsPath}/${inputFile} -s 240x135 -vf fps=1 ${efsPath}/%d.jpg`) 45 | } 46 | -------------------------------------------------------------------------------- /3-largefiles/processFile/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "process-largefiles", 3 | "version": "1.0.0", 4 | "description": "Process large video file to creating screenshots", 5 | "main": "app.js", 6 | "repository": "", 7 | "author": "James Beswick, AWS Serverless", 8 | "license": "MIT-0", 9 | "dependencies": { 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /3-largefiles/template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: Process a user-provided video (e.g video.mp4) into a series of screenshots. 4 | 5 | Parameters: 6 | InputFile: 7 | Type: String 8 | Default: video.MP4 9 | EFSpath: 10 | Type: String 11 | Default: /mnt/efs 12 | SecurityGroupIds: 13 | Type: CommaDelimitedList 14 | Default: sg-fa7133bc 15 | SubnetIDs: 16 | Type: CommaDelimitedList 17 | Description: The list of SubnetIDs in your Virtual Private Cloud (VPC) 18 | Default: subnet-80c98eae,subnet-04878f4e,subnet-e19ce486,subnet-bfc9fbb0,subnet-0bc28757,subnet-96c847a8 19 | AccessPointARN: 20 | Type: String 21 | Description: Access point ARN 22 | 23 | Resources: 24 | ProcessFileFunction: 25 | Type: AWS::Serverless::Function 26 | Properties: 27 | CodeUri: processFile/ 28 | Timeout: 300 29 | MemorySize: 2048 30 | Handler: app.handler 31 | Layers: 32 | - arn:aws:lambda:us-east-1:780018668030:layer:ffmpeg:1 33 | Environment: 34 | Variables: 35 | INPUT_FILE: !Ref InputFile 36 | EFS_PATH: !Ref EFSpath 37 | Runtime: nodejs12.x 38 | Events: 39 | API: 40 | Type: Api 41 | Properties: 42 | Path: /processFile 43 | Method: get 44 | VpcConfig: 45 | SecurityGroupIds: !Ref SecurityGroupIds 46 | SubnetIds: !Ref SubnetIDs 47 | FileSystemConfigs: 48 | - Arn: !Ref AccessPointARN 49 | LocalMountPath: !Ref EFSpath 50 | Policies: 51 | - Statement: 52 | - Sid: AWSLambdaVPCAccessExecutionRole 53 | Effect: Allow 54 | Action: 55 | - logs:CreateLogGroup 56 | - logs:CreateLogStream 57 | - logs:PutLogEvents 58 | - ec2:CreateNetworkInterface 59 | - ec2:DescribeNetworkInterfaces 60 | - ec2:DeleteNetworkInterface 61 | Resource: "*" 62 | - Sid: AmazonElasticFileSystemClientFullAccess 63 | Effect: Allow 64 | Action: 65 | - elasticfilesystem:ClientMount 66 | - elasticfilesystem:ClientRootAccess 67 | - elasticfilesystem:ClientWrite 68 | - elasticfilesystem:DescribeMountTargets 69 | Resource: "*" 70 | 71 | Outputs: 72 | # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function 73 | # Find out more about other implicit resources you can reference within SAM 74 | # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api 75 | API: 76 | Description: "API Gateway endpoint URL for Prod stage for function" 77 | Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/processFile/" 78 | -------------------------------------------------------------------------------- /4-zip/template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: Zip all JPG files into an archive. 4 | 5 | Parameters: 6 | OutputFile: 7 | Type: String 8 | Default: video.zip 9 | EFSpath: 10 | Type: String 11 | Default: /mnt/efs 12 | SecurityGroupIds: 13 | Type: CommaDelimitedList 14 | Default: sg-fa7133bc 15 | SubnetIDs: 16 | Type: CommaDelimitedList 17 | Description: The list of SubnetIDs in your Virtual Private Cloud (VPC) 18 | Default: subnet-80c98eae,subnet-04878f4e,subnet-e19ce486,subnet-bfc9fbb0,subnet-0bc28757,subnet-96c847a8 19 | AccessPointARN: 20 | Type: String 21 | Description: Access point ARN 22 | 23 | Resources: 24 | ZipFileFunction: 25 | Type: AWS::Serverless::Function 26 | Properties: 27 | CodeUri: zipFiles/ 28 | Timeout: 300 29 | MemorySize: 3008 30 | Handler: app.handler 31 | Environment: 32 | Variables: 33 | OUTPUT_FILE: !Ref OutputFile 34 | EFS_PATH: !Ref EFSpath 35 | Runtime: nodejs12.x 36 | Events: 37 | API: 38 | Type: Api 39 | Properties: 40 | Path: /zipFiles 41 | Method: get 42 | VpcConfig: 43 | SecurityGroupIds: !Ref SecurityGroupIds 44 | SubnetIds: !Ref SubnetIDs 45 | FileSystemConfigs: 46 | - Arn: !Ref AccessPointARN 47 | LocalMountPath: !Ref EFSpath 48 | Policies: 49 | - Statement: 50 | - Sid: AWSLambdaVPCAccessExecutionRole 51 | Effect: Allow 52 | Action: 53 | - logs:CreateLogGroup 54 | - logs:CreateLogStream 55 | - logs:PutLogEvents 56 | - ec2:CreateNetworkInterface 57 | - ec2:DescribeNetworkInterfaces 58 | - ec2:DeleteNetworkInterface 59 | Resource: "*" 60 | - Sid: AmazonElasticFileSystemClientFullAccess 61 | Effect: Allow 62 | Action: 63 | - elasticfilesystem:ClientMount 64 | - elasticfilesystem:ClientRootAccess 65 | - elasticfilesystem:ClientWrite 66 | - elasticfilesystem:DescribeMountTargets 67 | Resource: "*" 68 | 69 | Outputs: 70 | # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function 71 | # Find out more about other implicit resources you can reference within SAM 72 | # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api 73 | API: 74 | Description: "API Gateway endpoint URL for Prod stage for function" 75 | Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/zipFiles/" 76 | -------------------------------------------------------------------------------- /4-zip/zipFiles/app.js: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 9 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 10 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 11 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 12 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 13 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | */ 15 | 16 | const outputFile = process.env.OUTPUT_FILE 17 | const efsPath = process.env.EFS_PATH 18 | 19 | const fs = require('fs') 20 | const archiver = require('archiver') 21 | 22 | // The Lambda handler 23 | exports.handler = function (eventObject, context) { 24 | // create a file to stream archive data to. 25 | const output = fs.createWriteStream(`${efsPath}/${outputFile}`) 26 | const archive = archiver('zip', { 27 | zlib: { level: 9 } // Sets the compression level. 28 | }) 29 | 30 | // listen for all archive data to be written 31 | // 'close' event is fired only when a file descriptor is involved 32 | output.on('close', function() { 33 | console.log(archive.pointer() + ' total bytes') 34 | console.log('archiver has been finalized and the output file descriptor has closed.') 35 | }) 36 | 37 | // This event is fired when the data source is drained no matter what was the data source. 38 | // It is not part of this library but rather from the NodeJS Stream API. 39 | // @see: https://nodejs.org/api/stream.html#stream_event_end 40 | output.on('end', function() { 41 | console.log('Data has been drained'); 42 | }) 43 | 44 | // good practice to catch warnings (ie stat failures and other non-blocking errors) 45 | archive.on('warning', function(err) { 46 | if (err.code === 'ENOENT') { 47 | // log warning 48 | } else { 49 | // throw error 50 | throw err 51 | } 52 | }) 53 | 54 | // good practice to catch this error explicitly 55 | archive.on('error', function(err) { 56 | throw err 57 | }) 58 | 59 | // pipe archive data to the file 60 | archive.pipe(output); 61 | 62 | // append files from a glob pattern 63 | archive.glob(`${efsPath}/*.jpg`) 64 | 65 | // finalize the archive (ie we are done appending files but streams have to finish yet) 66 | // 'close', 'end' or 'finish' may be fired right after calling this method so register to them beforehand 67 | archive.finalize() 68 | } 69 | 70 | -------------------------------------------------------------------------------- /4-zip/zipFiles/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "process-largefiles", 3 | "version": "1.0.0", 4 | "description": "Process large video file to creating screenshots", 5 | "main": "app.js", 6 | "repository": "", 7 | "author": "James Beswick, AWS Serverless", 8 | "license": "MIT-0", 9 | "dependencies": { 10 | "archiver": "^4.0.1", 11 | "fs": "0.0.1-security" 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /5-unzip/template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: Unzip archive. 4 | 5 | Parameters: 6 | InputFile: 7 | Type: String 8 | Default: video.zip 9 | EFSpath: 10 | Type: String 11 | Default: /mnt/efs 12 | DestinationDir: 13 | Type: String 14 | Default: frames 15 | SecurityGroupIds: 16 | Type: CommaDelimitedList 17 | Default: sg-fa7133bc 18 | SubnetIDs: 19 | Type: CommaDelimitedList 20 | Description: The list of SubnetIDs in your Virtual Private Cloud (VPC) 21 | Default: subnet-80c98eae,subnet-04878f4e,subnet-e19ce486,subnet-bfc9fbb0,subnet-0bc28757,subnet-96c847a8 22 | AccessPointARN: 23 | Type: String 24 | Description: Access point ARN 25 | 26 | Resources: 27 | UnzipFileFunction: 28 | Type: AWS::Serverless::Function 29 | Properties: 30 | CodeUri: unzipFiles/ 31 | Timeout: 300 32 | MemorySize: 3008 33 | Handler: app.handler 34 | Environment: 35 | Variables: 36 | INPUT_FILE: !Ref InputFile 37 | DESTINATION_DIR: !Ref DestinationDir 38 | EFS_PATH: !Ref EFSpath 39 | Runtime: nodejs12.x 40 | Events: 41 | API: 42 | Type: Api 43 | Properties: 44 | Path: /zipFiles 45 | Method: get 46 | VpcConfig: 47 | SecurityGroupIds: !Ref SecurityGroupIds 48 | SubnetIds: !Ref SubnetIDs 49 | FileSystemConfigs: 50 | - Arn: !Ref AccessPointARN 51 | LocalMountPath: !Ref EFSpath 52 | Policies: 53 | - Statement: 54 | - Sid: AWSLambdaVPCAccessExecutionRole 55 | Effect: Allow 56 | Action: 57 | - logs:CreateLogGroup 58 | - logs:CreateLogStream 59 | - logs:PutLogEvents 60 | - ec2:CreateNetworkInterface 61 | - ec2:DescribeNetworkInterfaces 62 | - ec2:DeleteNetworkInterface 63 | Resource: "*" 64 | - Sid: AmazonElasticFileSystemClientFullAccess 65 | Effect: Allow 66 | Action: 67 | - elasticfilesystem:ClientMount 68 | - elasticfilesystem:ClientRootAccess 69 | - elasticfilesystem:ClientWrite 70 | - elasticfilesystem:DescribeMountTargets 71 | Resource: "*" 72 | 73 | Outputs: 74 | # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function 75 | # Find out more about other implicit resources you can reference within SAM 76 | # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api 77 | API: 78 | Description: "API Gateway endpoint URL for Prod stage for function" 79 | Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/unzipFiles/" 80 | -------------------------------------------------------------------------------- /5-unzip/unzipFiles/app.js: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 9 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 10 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 11 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 12 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 13 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | */ 15 | 16 | const inputFile = process.env.INPUT_FILE 17 | const efsPath = process.env.EFS_PATH 18 | const destinationDir = process.env.DESTINATION_DIR 19 | 20 | const fs = require('fs') 21 | const unzipper = require('unzipper') 22 | 23 | // The Lambda handler 24 | exports.handler = function (eventObject, context) { 25 | 26 | fs.createReadStream(`${efsPath}/${inputFile}`) 27 | .pipe(unzipper.Extract({ path: `${efsPath}/${destinationDir}` })) 28 | 29 | } 30 | 31 | -------------------------------------------------------------------------------- /5-unzip/unzipFiles/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "process-largefiles", 3 | "version": "1.0.0", 4 | "description": "Process large video file to creating screenshots", 5 | "main": "app.js", 6 | "repository": "", 7 | "author": "James Beswick, AWS Serverless", 8 | "license": "MIT-0", 9 | "dependencies": { 10 | "fs": "0.0.1-security", 11 | "unzipper": "^0.10.11" 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EFS for Lambda - Example SAM applications 2 | 3 | This repo contains AWS SAM templates that deploy serverless applications. The applications illustrate different ways to use EFS for Lambda. 4 | 5 | For full details on how this works: 6 | - Read the Compute Blog post at: https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/. 7 | - Watch the [YouTube walkthrough](https://www.youtube.com/watch?v=vHwNh9JtPwE). 8 | - RocketSAM example is available here: https://github.com/nadav96/rocketsam/tree/master/sample/vpc-efs-lambda. 9 | 10 | Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in these examples. 11 | 12 | ```bash 13 | . 14 | ├── README.MD <-- This instructions file 15 | ├── 1-setup <-- Creates VPC, subnets and EFS file system 16 | │ └── create-efs-cfn.yml <-- CloudFormation template 17 | ├── 2-lambda-template <-- "Hello World" example to use VPC and EFS 18 | │ └── package.json <-- NodeJS dependencies and scripts 19 | │ └── template.yaml <-- SAM template 20 | ├── 3-largefiles <-- Processes a user-provided MP4 video into screenshots 21 | │ └── package.json <-- NodeJS dependencies and scripts 22 | │ └── template.yaml <-- SAM template 23 | ├── 4-zip <-- Zips many files into a zip archive 24 | │ └── package.json <-- NodeJS dependencies and scripts 25 | │ └── template.yaml <-- SAM template 26 | ├── 5-unzip <-- Unzips archive into specified directory 27 | │ └── package.json <-- NodeJS dependencies and scripts 28 | │ └── template.yaml <-- SAM template 29 | ``` 30 | 31 | ## Requirements 32 | 33 | * AWS CLI already configured with Administrator permission 34 | * [NodeJS 12.x installed](https://nodejs.org/en/download/) 35 | * An MP4 video for example #3 - video.mp4 is not provided in this repo. 36 | 37 | ## Installation Instructions 38 | 39 | 1. [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and login. 40 | 41 | 1. Clone the repo onto your local development machine using `git clone`. 42 | 43 | ## Setup 44 | 45 | To create a VPC, subnets and EFS file system, use the `create-efs-cfn.yml` example in the `setup` directory. To deploy this template, run in a terminal: 46 | 47 | ``` 48 | aws cloudformation create-stack --stack-name create-efs --template-body file://./create-efs-cfn.yml 49 | ``` 50 | Note that the `template-body` parameter must include the `file://` prefix when run locally. 51 | 52 | ## Deploying examples 53 | 54 | 1. From the command line, change directory into the application example required, then run: 55 | ``` 56 | sam build 57 | sam deploy --guided 58 | ``` 59 | Follow the prompts in the deploy process to set the stack name, EFS mount path, access point ARN, AWS Region, VPC and subnet IDs. 60 | 61 | You can find VPC settings by executing the following CLI command: 62 | ``` 63 | aws ec2 describe-vpcs 64 | ``` 65 | 66 | You can find a list of subnet IDs for your account by executing: 67 | ``` 68 | aws ec2 describe-subnets --query 'Subnets[*].{Id: SubnetId}' --output text 69 | ``` 70 | And you can reformat to output only the IDs using this command: 71 | ``` 72 | aws ec2 describe-subnets --filters "Name=vpc-id,Values=<>" --query 'Subnets[*].{Id: SubnetId}' --output text 73 | ``` 74 | You can find a list of access point ARNs for your account by executing: 75 | ``` 76 | aws efs describe-access-points 77 | ``` 78 | 79 | ## How it works 80 | 81 | * The deployed Lambda functions are configured with your VPC and subnet settings. 82 | * The Lambda functions mount the EFS file system you specify. The operations in the examples use EFS to complete their tasks. 83 | 84 | ## Questions? 85 | 86 | Please contact [@jbesw](https://twitter.com/jbesw) or raise an issue on this repo. 87 | 88 | ============================================== 89 | 90 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 91 | 92 | SPDX-License-Identifier: MIT-0 93 | --------------------------------------------------------------------------------