├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── __tests__ └── unit │ └── handlers │ └── async-download.test.js ├── events ├── event-on-disconnect.json └── event-send-message.json ├── package-lock.json ├── package.json ├── serverless-file-download.jpg ├── src └── handlers │ ├── async-download │ ├── app.js │ └── package.json │ ├── download-s3-object │ ├── app.js │ └── package.json │ ├── get-websocket-connection │ ├── app.js │ └── package.json │ ├── on-connect │ ├── app.js │ └── package.json │ ├── on-disconnect │ ├── app.js │ └── package.json │ └── send-message │ ├── app.js │ └── package.json ├── statemachine └── simple_workflow.asl.json ├── step-functions-ui.png └── template.yml /.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | node_modules_bkp 3 | .aws-sam 4 | .vscode 5 | .DS_Store 6 | samconfig.toml -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 6 | software and associated documentation files (the "Software"), to deal in the Software 7 | without restriction, including without limitation the rights to use, copy, modify, 8 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 9 | permit persons to whom the Software is furnished to do so. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 12 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 13 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 14 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 15 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 16 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | 18 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Serverless App for Async file download 2 | 3 | This is a sample serverless application which showcases an async approach to download files which can be larger than the payload size limit of API Gateway. 4 | This architecture uses S3 PreSigned URL and WebSocket endpoint. 5 | 6 | > `main` branch uses AWS JavaScript SDK v3. 7 | 8 | > :warning: npm version 7.12.0 does not package the zip files properly. Use v7.10.0 when working on **main** branch 9 | 10 | ## Rationale 11 | Customers are interested in using API Gateway as a front door for their backend HTTP endpoints. API Gateway as a managed service provides a lot of benefits to customer, such as creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale. In some edge case scenarios, customers want to use API Gateway and Lambda integration but are limited to API Gateway’s 10 MB payload size, API Gateway’s 30 seconds timeout, and Lambda’s 6MB payload size for synchronous request/response. 12 | 13 | One such scenario is when customers need to download large files from an HTTP Endpoint following a Serverless approach. An example of this is in the Financial industry, where some customers: 14 | 15 | - Need to download very large text or binary files to support business operations 16 | - Need to have security in place 17 | - Allows for lifecycle policies around these files 18 | - Want to build quickly and support scaling 19 | 20 | ## Solution 21 | This project contains source code and supporting files for the below proposed architecture: 22 | ![architecture](serverless-file-download.jpg) 23 | 24 | ## How it works 25 | Client (browser) invokes the REST GET endpoint `/download`. 26 | 27 | > :information_source:   This solution does not expect any path param or query params but it can fit to your use case if the underlying HTTP Endpoint expects additional params 28 | 29 | `/download` API has a lambda integration which will work on 4 main items: 30 | - Lambda will invoke a Step Function workflow (responsible for gathering the document, more on it later) 31 | - Inserts the Step Function Execution ARN, got from above step, to DynamoDB with the Execution ARN as the partition key 32 | - Captures the WebSocket endpoint url that is available as an environment variable. WebSocket endpoint was created as part of the infrastructure creation process 33 | - Returns the execution ARN and the Websocket endpoint to the client (browser) synchronously 34 | 35 | > :bulb:   Note that the response is synchronous but the Step Function has started working on simultaneously 36 | 37 | Upon receiving the response from `/download` REST call, client immediately opens a WebSocket connection with the WebSocket Endpoint provided as part of response above. Client also passes the Execution ARN as a payload to the initial connection request. 38 | 39 | In the WebSocket Endpoint, as soon as client connection is created, a `connectionId` is generated and `onConnect` handler is called. `onConnect` handler is a Lambda integration which takes the `connectionId` and the `executionArn` from the WebSocket connection and queries DynamoDB with the `executionArn` as the key. Once the DynamoDB item is retrieved (which was inserted by the lambda integration in `/download` rest api call), `onConnect` lambda handler updates DynamoDB item with the `connectionId`. Now, the DynamoDB item has `connectionId` as an attribute 40 | 41 | While above ceremonies were going on, Step Functions Workflow was doing the heavy lifting for you by doing below tasks: 42 | 43 | 1. Step 1: 44 | - Call HTTP endpoint (with appropriate params) and get the binary response 45 | - Once binary response is received, upload that as an object in an S3 bucket 46 | - Once updated, create a pre-signed GET url for that object 47 | - Pass pre-signed url to next step 48 | 2. Step 2: 49 | - Get the `connectionId` from DynamoDB table which was updated against the current running execution ARN 50 | - If `connectionId` is not present yet, wait for few seconds and try again 51 | - Pass `connectionId` and pre-signed url to next step 52 | 3. Step 3 53 | - In this step, the task has access to WebSocket endpoint, the `connectionId` and the pre-signed url 54 | - This step makes a POST call on the WebSocket endpoint against the `connectionId` and pass the pre-signed url as a payload 55 | 56 | If the `connectionId` is alive, the pre-signed url is sent to the client which client can use to download the S3 object. 57 | 58 | ## Prerequisites 59 | This application expects below prerequisites: 60 | 61 | - The app expects an S3 bucket name (same region as the app) as a parameter to the stack while `sam deploy`. This bucket will be used to store the binary response from HTTP endpoint as S3 Object. You can additionally setup lifecycle policy on those objects. 62 | - Install `wscat` (`npm install -g wscat`) which will be used as WebSocket client during testing 63 | 64 | > :warning: This application uses [a mock HTTP Endpoint which returns a CSV file as a binary response](https://run.mocky.io/v3/e63ca0e5-53dd-483a-9186-1d7e4a2edb74). This mock endpoint can expire when not used. In such case, replace with your own mock HTTP Endpoint which responds with a binary response. 65 | 66 | ## Deploy the sample application 67 | 68 | The AWS SAM CLI is an extension of the AWS CLI that adds functionality for building and testing Lambda applications. It uses Docker to run your functions in an Amazon Linux environment that matches Lambda. It can also emulate your application's build environment and API. 69 | 70 | To use the AWS SAM CLI, you need the following tools: 71 | 72 | * AWS SAM CLI - [Install the AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html). 73 | * Node.js - [Install Node.js 14](https://nodejs.org/en/), including the npm package management tool. 74 | * Docker - [Install Docker community edition](https://hub.docker.com/search/?type=edition&offering=community). 75 | 76 | To build and deploy your application for the first time, run the following in your shell: 77 | 78 | ```bash 79 | sam build 80 | sam deploy --guided 81 | ``` 82 | 83 | The first command will build the source of your application. The second command will package and deploy your application to AWS, with a series of prompts. Provide below information for the first time: 84 | 85 | ``` 86 | Configuring SAM deploy 87 | ====================== 88 | 89 | Looking for config file [samconfig.toml] : Not found 90 | 91 | Setting default arguments for 'sam deploy' 92 | ========================================= 93 | Stack Name [sam-app]: aws-serverless-file-download-app 94 | AWS Region [us-east-2]: 95 | Parameter StageName [dev]: 96 | Parameter BucketName []: 97 | # Shows you resources changes to be deployed and require a 'Y' to initiate deploy 98 | Confirm changes before deploy [y/N]: y 99 | # SAM needs permission to be able to create roles to connect to the resources in your template 100 | Allow SAM CLI IAM role creation [Y/n]: y 101 | AsyncDownload may not have authorization defined, Is this okay? [y/N]: y 102 | Save arguments to configuration file [Y/n]: y 103 | SAM configuration file [samconfig.toml]: 104 | SAM configuration environment [default]: 105 | ``` 106 | 107 | The API Gateway endpoint API will be displayed in the outputs when the deployment is complete. 108 | 109 | ## Use the AWS SAM CLI to build and test locally 110 | 111 | Build your application by using the `sam build` command. 112 | 113 | ```bash 114 | sam build 115 | ``` 116 | 117 | The AWS SAM CLI installs dependencies that are defined in `package.json`, creates a deployment package, and saves it in the `.aws-sam/build` folder. 118 | 119 | Test a single function by invoking it directly with a test event. An event is a JSON document that represents the input that the function receives from the event source. Test events are included in the `events` folder in this project. 120 | 121 | Run functions locally and invoke them with the `sam local invoke` command. 122 | 123 | ```bash 124 | sam local invoke SendMessageFunction -e events/event-send-message.json 125 | ``` 126 | 127 | Debug functions locally using VS Code by using `--debug-port` and invoke them with the `sam local invoke` command. 128 | 129 | ```bash 130 | sam local invoke SendMessageFunction -e events/event-send-message.json --debug-port 5858 131 | ``` 132 | 133 | > You need to setup `launch.json` config in VS code similar to: 134 | ```json 135 | { 136 | "version": "0.2.0", 137 | "configurations": [ 138 | { 139 | "name": "Attach to SAM CLI", 140 | "type": "node", 141 | "request": "attach", 142 | "address": "localhost", 143 | "port": 5858, 144 | "localRoot": "${workspaceFolder}", 145 | "remoteRoot": "/var/task", 146 | "protocol": "inspector", 147 | "stopOnEntry": false 148 | } 149 | ] 150 | } 151 | ``` 152 | 153 | The AWS SAM CLI can also emulate your application's API. Use the `sam local start-api` command to run the API locally on port 3000. 154 | 155 | ```bash 156 | sam local start-api 157 | curl http://localhost:3000/ 158 | ``` 159 | 160 | The AWS SAM CLI reads the application template to determine the API's routes and the functions that they invoke. The `Events` property on each function's definition includes the route and method for each path. 161 | 162 | ```yaml 163 | Events: 164 | DownloadApiEvent: 165 | Type: Api 166 | Properties: 167 | Path: /download 168 | Method: GET 169 | RestApiId: !Ref DownloadApi 170 | ``` 171 | ## Unit tests 172 | 173 | Tests are defined in the `__tests__` folder in this project. Use `npm` to install the [Jest test framework](https://jestjs.io/) and run unit tests. 174 | 175 | ```bash 176 | npm install 177 | npm run test 178 | ``` 179 | 180 | ## UAT 181 | In order to test the setup, follow below steps: 182 | 183 | - Hit the REST GET api `/download` using cURL or Postman. The response would look like below where the obscured values are the ids of the endpoints which you have provisioned 184 | ```bash 185 | ➜ curl https://xxxxx.execute-api.us-east-2.amazonaws.com/dev/download 186 | { 187 | "executionArn": "arn:aws:states:us-east-2:12345:execution:ObjectUploaderStateMachine:dd45133d-9f10-4dbd-8259-9ee37930b52f", 188 | "webSocketEndpoint": "wss://yyyy.execute-api.us-east-2.amazonaws.com/dev" 189 | } 190 | ``` 191 | 192 | - Install `wscat` using npm (if not done already) which will be used as WebSocket client 193 | 194 | ```bash 195 | npm install -g wscat 196 | ``` 197 | 198 | - Use `wscat` to create the WebSocket connection by using the `webSocketEndpoint` received from above step. Also pass `X-StateMachine-ExecutionArn` as a header with the `executionArn` value from above step 199 | 200 | ```bash 201 | wscat -c wss://yyyy.execute-api.us-east-2.amazonaws.com/dev -H X-StateMachine-ExecutionArn:"arn:aws:states:us-east-2:12345:execution:ObjectUploaderStateMachine-Hj4TEGMxHdqv:dd45133d-9f10-4dbd-8259-9ee37930b52f" 202 | ``` 203 | 204 | - On successful connection, you should see 205 | 206 | ```bash 207 | Connected (press CTRL+C to quit) 208 | ``` 209 | 210 | - Open Step Functions in AWS console then go to the state machine that is running for this setup, you should see something like below if everything went well without any errors 211 | ![stepfunctions](step-functions-ui.png) 212 | 213 | - Verify in DynamoDB Table that the item is present where Execution ARN is the partition Key and `ConnectionId` is an attribute against the execution ARN 214 | 215 | - Finally you should see the pre-signed s3 url in the terminal where WebSocket connection was created 216 | 217 | - Close the connection (Ctrl + C), then verify that DynamoDB item has been cleaned up, showcasing that when connections get closed from client or server (API GW) then stale connections will be cleaned from DynamoDB Table 218 | 219 | 220 | ## Cleanup 221 | 222 | To delete the sample application that you created, use the AWS CLI. Assuming your stack name is `aws-serverless-file-download-app`, you can run the following: 223 | 224 | ```bash 225 | aws cloudformation delete-stack --stack-name aws-serverless-file-download-app 226 | ``` 227 | 228 | ## TODO 229 | - Add X-Ray support (aws-xray-sdk support is not available today [04/22/2021] for AWS SDK v3. Expected to be supported in [v3.3.2](https://github.com/aws/aws-xray-sdk-node/issues/294#issuecomment-818321582)) 230 | 231 | ## Security 232 | 233 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 234 | 235 | ## License 236 | 237 | This library is licensed under the MIT-0 License. See the LICENSE file. 238 | 239 | -------------------------------------------------------------------------------- /__tests__/unit/handlers/async-download.test.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * SPDX-License-Identifier: MIT-0 4 | */ 5 | 6 | const { DynamoDBClient } = require("@aws-sdk/client-dynamodb"); 7 | const { SFNClient } = require("@aws-sdk/client-sfn"); 8 | 9 | describe('Test async-download handler', () => { 10 | let putSpy, sfnSpy, lambda; 11 | 12 | beforeAll(() => { 13 | process.env.WEB_SOCKET_ENDPOINT = "wss://myWebSocketEndpoint"; 14 | putSpy = jest.spyOn(DynamoDBClient.prototype, 'send'); 15 | sfnSpy = jest.spyOn(SFNClient.prototype, 'send'); 16 | 17 | // require lambda after setting variables in process.env in order to use them outside the handler 18 | lambda = require('../../../src/handlers/async-download/app.js'); 19 | }); 20 | 21 | afterAll(() => { 22 | putSpy.mockRestore(); 23 | sfnSpy.mockRestore(); 24 | }); 25 | 26 | it('should return execution arn and websocket endpoint', async () => { 27 | const executionResult = { executionArn: "aws:arn:account:region:executionArn" }; 28 | 29 | sfnSpy.mockReturnValue(Promise.resolve(executionResult)); 30 | putSpy.mockReturnValue(Promise.resolve({ status: 200 })); 31 | 32 | const event = { 33 | httpMethod: 'GET' 34 | } 35 | 36 | // Invoke helloFromLambdaHandler() 37 | const result = await lambda.handler(event); 38 | 39 | const expectedResult = { 40 | statusCode: 200, 41 | body: JSON.stringify({ 42 | ...executionResult, 43 | webSocketEndpoint: "wss://myWebSocketEndpoint" 44 | }) 45 | }; 46 | 47 | // Compare the result with the expected result 48 | expect(result).toEqual(expectedResult); 49 | }); 50 | }); 51 | -------------------------------------------------------------------------------- /events/event-on-disconnect.json: -------------------------------------------------------------------------------- 1 | { 2 | "requestContext": { 3 | "connectionId": "cwrxUe1WCYcCFvw=" 4 | } 5 | } -------------------------------------------------------------------------------- /events/event-send-message.json: -------------------------------------------------------------------------------- 1 | { 2 | "webSocketEndpoint": "wss://bvmwxmwbqf.execute-api.us-east-2.amazonaws.com/Prod", 3 | "preSignedUrl": "https://barbazz", 4 | "connectionId": "cxvfbd29iYcCH_g=" 5 | } -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "file-download-app", 3 | "description": "file-download-app", 4 | "version": "0.0.1", 5 | "private": true, 6 | "dependencies": { 7 | "@aws-sdk/client-apigatewaymanagementapi": "^3.529.1", 8 | "@aws-sdk/client-dynamodb": "^3.529.1", 9 | "@aws-sdk/client-s3": "^3.529.1", 10 | "@aws-sdk/client-sfn": "^3.529.1", 11 | "@aws-sdk/s3-request-presigner": "^3.529.1", 12 | "@aws-sdk/util-dynamodb": "^3.529.1" 13 | }, 14 | "devDependencies": { 15 | "jest": "^26.6.3" 16 | }, 17 | "scripts": { 18 | "test": "jest" 19 | } 20 | } 21 | -------------------------------------------------------------------------------- /serverless-file-download.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-serverless-file-download/0673aeebffe0364d6094c6dfed218fd721b2c941/serverless-file-download.jpg -------------------------------------------------------------------------------- /src/handlers/async-download/app.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * SPDX-License-Identifier: MIT-0 4 | */ 5 | 6 | const { SFNClient, StartExecutionCommand } = require("@aws-sdk/client-sfn"); 7 | const { DynamoDBClient, PutItemCommand } = require("@aws-sdk/client-dynamodb"); 8 | const { marshall } = require("@aws-sdk/util-dynamodb"); 9 | 10 | const docClient = new DynamoDBClient(); 11 | const stepFunctions = new SFNClient(); 12 | 13 | const tableName = process.env.CONNECTIONS_TABLE; 14 | const webSocketEndpoint = process.env.WEB_SOCKET_ENDPOINT; 15 | const stateMachineArn = process.env.STATE_MACHINE_ARN; 16 | 17 | exports.handler = async event => { 18 | if (event.httpMethod !== 'GET') { 19 | console.error(`HTTP method ${event.httpMethod} not supported`); 20 | throw new Error(`Handler only accept GET method, you tried: ${event.httpMethod}`); 21 | } 22 | 23 | // Invoke Step Functions & Capture Step Functions Execution ARN 24 | const startExecCommand = new StartExecutionCommand({ stateMachineArn, input: JSON.stringify({ webSocketEndpoint }) }); 25 | const executionResult = await stepFunctions.send(startExecCommand); 26 | console.info(`Execution Result: ${JSON.stringify(executionResult)}`); 27 | 28 | // Insert Step Functions Execution ARN to DynamoDB as PK 29 | const item = marshall({ executionArnId: executionResult.executionArn }); 30 | const result = await docClient.send(new PutItemCommand({ TableName: tableName, Item: item })); 31 | 32 | console.info(`Successfully Inserted in DynamoDB: ${result}`); 33 | 34 | // Capture WebSocket Endpoint URL & Send both as response to client 35 | const body = { 36 | executionArn: executionResult.executionArn, 37 | webSocketEndpoint 38 | } 39 | 40 | const response = { 41 | statusCode: 200, 42 | body: JSON.stringify(body) 43 | }; 44 | 45 | // All log statements are written to CloudWatch 46 | console.info(`response from: ${event.path} statusCode: ${response.statusCode} body: ${response.body}`); 47 | return response; 48 | } -------------------------------------------------------------------------------- /src/handlers/async-download/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "file-download-app", 3 | "description": "file-download-app", 4 | "version": "0.0.1", 5 | "private": true, 6 | "dependencies": { 7 | "@aws-sdk/client-dynamodb": "^3.14.0", 8 | "@aws-sdk/client-sfn": "^3.14.0", 9 | "@aws-sdk/util-dynamodb": "^3.14.0" 10 | } 11 | } -------------------------------------------------------------------------------- /src/handlers/download-s3-object/app.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * SPDX-License-Identifier: MIT-0 4 | */ 5 | 6 | const axios = require('axios'); 7 | const crypto = require("crypto"); 8 | const { S3, GetObjectCommand, PutObjectCommand } = require('@aws-sdk/client-s3'); 9 | const { getSignedUrl } = require("@aws-sdk/s3-request-presigner"); 10 | const s3 = new S3(); 11 | const bucketName = process.env.BUCKET_NAME; 12 | 13 | exports.handler = async event => { 14 | let url; 15 | const randomId = crypto.randomBytes(16).toString("hex"); 16 | 17 | // Mock HTTP Endpoint with a delay of 60 seconds for binary response 18 | const mockHttpUrl = "https://run.mocky.io/v3/e63ca0e5-53dd-483a-9186-1d7e4a2edb74?mocky-delay=60s" 19 | 20 | try { 21 | // Make HTTP Call & Capture Binary Response 22 | const response = await axios.get(mockHttpUrl, { responseType: "arraybuffer" }); 23 | 24 | // Upload Response as S3 Object to bucket 25 | const objectName = `Response_Object_${randomId}.csv`; 26 | const data = await s3.send(new PutObjectCommand({ 27 | Bucket: bucketName, 28 | Key: objectName, 29 | Body: response.data, 30 | ContentType: "text/csv" 31 | })); 32 | 33 | console.info("S3 Put Object Response: ", data); 34 | 35 | // Get PreSigned URL for the Object 36 | const getObjectCommand = new GetObjectCommand({ 37 | Bucket: bucketName, 38 | Key: objectName 39 | }); 40 | 41 | url = await getSignedUrl(s3, getObjectCommand, { 42 | expiresIn: 600 43 | }); 44 | 45 | // All log statements are written to CloudWatch 46 | console.info("PreSigned URL from S3: ", url); 47 | } catch (err) { 48 | console.error("Error: ", err) 49 | } 50 | 51 | // Pass PreSigned URL to next step 52 | return url; 53 | } 54 | -------------------------------------------------------------------------------- /src/handlers/download-s3-object/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "file-download-app", 3 | "description": "file-download-app", 4 | "version": "0.0.1", 5 | "private": true, 6 | "dependencies": { 7 | "@aws-sdk/client-s3": "^3.14.0", 8 | "@aws-sdk/s3-request-presigner": "^3.14.0", 9 | "axios": "^0.21.1" 10 | } 11 | } -------------------------------------------------------------------------------- /src/handlers/get-websocket-connection/app.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * SPDX-License-Identifier: MIT-0 4 | */ 5 | 6 | const { DynamoDBClient, GetItemCommand } = require("@aws-sdk/client-dynamodb"); 7 | const { marshall, unmarshall } = require("@aws-sdk/util-dynamodb"); 8 | const ddb = new DynamoDBClient({ apiVersion: '2012-08-10', region: process.env.AWS_REGION }); 9 | 10 | exports.handler = async event => { 11 | let connectionId; 12 | 13 | console.info(`ExecutionId from Parameter: ${JSON.stringify(event)}`); 14 | 15 | const params = { 16 | TableName: process.env.CONNECTIONS_TABLE, 17 | Key: marshall({ 18 | executionArnId: event.Execution 19 | }), 20 | ProjectionExpression: 'connectionId' 21 | } 22 | 23 | try { 24 | const { Item } = await ddb.send(new GetItemCommand(params)); 25 | const item = unmarshall(Item); 26 | if (item?.connectionId) { 27 | connectionId = item.connectionId; 28 | } 29 | } catch (e) { 30 | return null; 31 | } 32 | 33 | return connectionId; 34 | }; -------------------------------------------------------------------------------- /src/handlers/get-websocket-connection/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "file-download-app", 3 | "description": "file-download-app", 4 | "version": "0.0.1", 5 | "private": true, 6 | "dependencies": { 7 | "@aws-sdk/client-dynamodb": "^3.14.0", 8 | "@aws-sdk/util-dynamodb": "^3.14.0" 9 | } 10 | } -------------------------------------------------------------------------------- /src/handlers/on-connect/app.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * SPDX-License-Identifier: MIT-0 4 | */ 5 | 6 | const { DynamoDBClient, PutItemCommand } = require("@aws-sdk/client-dynamodb"); 7 | const { marshall } = require("@aws-sdk/util-dynamodb"); 8 | const ddb = new DynamoDBClient({ region: process.env.AWS_REGION }); 9 | 10 | exports.handler = async event => { 11 | console.info(`On Connect Event Object: ${JSON.stringify(event)}`); 12 | 13 | const putItemCommand = new PutItemCommand({ 14 | TableName: process.env.TABLE_NAME, 15 | Item: marshall({ 16 | executionArnId: event.headers["X-StateMachine-ExecutionArn"], 17 | connectionId: event.requestContext.connectionId 18 | }) 19 | }); 20 | 21 | try { 22 | await ddb.send(putItemCommand); 23 | } catch (err) { 24 | return { statusCode: 500, body: `Failed to connect: ${JSON.stringify(err)}` }; 25 | } 26 | 27 | return { 28 | statusCode: 200, 29 | body: JSON.stringify({ message: 'Connected...' }) 30 | }; 31 | }; -------------------------------------------------------------------------------- /src/handlers/on-connect/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "file-download-app", 3 | "description": "file-download-app", 4 | "version": "0.0.1", 5 | "private": true, 6 | "dependencies": { 7 | "@aws-sdk/client-dynamodb": "^3.14.0", 8 | "@aws-sdk/util-dynamodb": "^3.14.0" 9 | } 10 | } -------------------------------------------------------------------------------- /src/handlers/on-disconnect/app.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * SPDX-License-Identifier: MIT-0 4 | */ 5 | 6 | const { DynamoDBClient, ScanCommand, BatchWriteItemCommand } = require("@aws-sdk/client-dynamodb"); 7 | const ddb = new DynamoDBClient({ region: process.env.AWS_REGION }); 8 | 9 | exports.handler = async event => { 10 | console.info(`On Connect Event Object: ${JSON.stringify(event)}`); 11 | 12 | const scanParams = { 13 | TableName: process.env.TABLE_NAME, 14 | ProjectionExpression: "executionArnId", 15 | FilterExpression: "#conn = :conn_id", 16 | ExpressionAttributeNames: { 17 | "#conn": "connectionId", 18 | }, 19 | ExpressionAttributeValues: { 20 | ":conn_id": { S: event.requestContext.connectionId } 21 | } 22 | }; 23 | 24 | const { Items } = await ddb.send(new ScanCommand(scanParams)); 25 | console.info(`Items from Scan: ${JSON.stringify(Items)}`); 26 | 27 | if (!Items || Items.length == 0) { 28 | return { statusCode: 204, body: JSON.stringify({ message: 'No connections to Delete..' }) }; 29 | } 30 | 31 | const batchWriteItemInput = { 32 | RequestItems: { 33 | [process.env.TABLE_NAME]: Items.map(item => { 34 | return { 35 | DeleteRequest: { 36 | Key: item 37 | } 38 | } 39 | }) 40 | } 41 | }; 42 | 43 | try { 44 | await ddb.send(new BatchWriteItemCommand(batchWriteItemInput)); 45 | } catch (err) { 46 | console.error(`Failed to disconnect.. ${JSON.stringify(err)}`); 47 | } 48 | 49 | return { statusCode: 200, body: JSON.stringify({ message: 'Disconnected.' }) }; 50 | }; -------------------------------------------------------------------------------- /src/handlers/on-disconnect/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "file-download-app", 3 | "description": "file-download-app", 4 | "version": "0.0.1", 5 | "private": true, 6 | "dependencies": { 7 | "@aws-sdk/client-dynamodb": "^3.14.0", 8 | "@aws-sdk/util-dynamodb": "^3.14.0" 9 | } 10 | } -------------------------------------------------------------------------------- /src/handlers/send-message/app.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * SPDX-License-Identifier: MIT-0 4 | */ 5 | 6 | const { ApiGatewayManagementApiClient, PostToConnectionCommand } = require('@aws-sdk/client-apigatewaymanagementapi'); 7 | 8 | exports.handler = async event => { 9 | const { connectionId, preSignedUrl, webSocketEndpoint } = event; 10 | const stage = webSocketEndpoint.split("/").pop(); 11 | 12 | const apiGatewayManagementAPIClient = new ApiGatewayManagementApiClient({ 13 | apiVersion: "2018-11-29", 14 | endpoint: webSocketEndpoint.replace("wss://", "https://") 15 | }); 16 | 17 | // Workaround for issue: https://github.com/aws/aws-sdk-js-v3/issues/1830 18 | apiGatewayManagementAPIClient.middlewareStack.add( 19 | (next) => 20 | async (args) => { 21 | args.request.path = stage + args.request.path; 22 | return await next(args); 23 | }, 24 | { step: "build" }, 25 | ); 26 | 27 | try { 28 | await apiGatewayManagementAPIClient.send(new PostToConnectionCommand({ ConnectionId: connectionId, Data: preSignedUrl })); 29 | } catch (e) { 30 | if (e.statusCode === 410) { 31 | console.error(`Found stale connection: ${connectionId}`); 32 | } else { 33 | throw e; 34 | } 35 | } 36 | 37 | return { statusCode: 200, body: 'Data sent.' }; 38 | }; -------------------------------------------------------------------------------- /src/handlers/send-message/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "file-download-app", 3 | "description": "file-download-app", 4 | "version": "0.0.1", 5 | "private": true, 6 | "dependencies": { 7 | "@aws-sdk/client-apigatewaymanagementapi": "^3.14.0" 8 | }, 9 | "devDependencies": { 10 | "@aws-sdk/client-apigatewaymanagementapi": "^3.14.0" 11 | } 12 | } -------------------------------------------------------------------------------- /statemachine/simple_workflow.asl.json: -------------------------------------------------------------------------------- 1 | { 2 | "Comment": "A simple state machine to mock an HTTP Endpoint call with a wait, then download a large file from S3, send pre-signed URL to WebSocket", 3 | "StartAt": "WaitForHTTPCall", 4 | "States": { 5 | "WaitForHTTPCall": { 6 | "Type": "Wait", 7 | "Seconds": 60, 8 | "Next": "DownloadS3Object" 9 | }, 10 | "DownloadS3Object": { 11 | "Type": "Task", 12 | "Resource": "${DownloadS3ObjectFunctionArn}", 13 | "ResultPath": "$.preSignedUrl", 14 | "Next": "GetWebSocketConnectionId" 15 | }, 16 | "GetWebSocketConnectionId": { 17 | "Type": "Task", 18 | "Resource": "${GetWebSocketConnectionIDFunctionArn}", 19 | "Parameters": { 20 | "Execution.$": "$$.Execution.Id" 21 | }, 22 | "ResultPath": "$.connectionId", 23 | "Next": "CheckIfConnectionIdAvailable" 24 | }, 25 | "CheckIfConnectionIdAvailable": { 26 | "Type": "Choice", 27 | "Choices": [ 28 | { 29 | "Variable": "$.connectionId", 30 | "IsNull": true, 31 | "Next": "WaitForConnectionIdToBeAvailable" 32 | } 33 | ], 34 | "Default": "NotifyWebSocketClient" 35 | }, 36 | "WaitForConnectionIdToBeAvailable": { 37 | "Type": "Wait", 38 | "Seconds": 60, 39 | "Next": "GetWebSocketConnectionId" 40 | }, 41 | "NotifyWebSocketClient": { 42 | "Type": "Task", 43 | "Resource": "${SendMessageFunctionArn}", 44 | "End": true 45 | } 46 | } 47 | } -------------------------------------------------------------------------------- /step-functions-ui.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-serverless-file-download/0673aeebffe0364d6094c6dfed218fd721b2c941/step-functions-ui.png -------------------------------------------------------------------------------- /template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: 2010-09-09 2 | Description: >- 3 | file-mgmt 4 | 5 | Transform: 6 | - AWS::Serverless-2016-10-31 7 | 8 | Globals: 9 | Function: 10 | Tracing: Active 11 | Layers: 12 | - !Sub "arn:aws:lambda:${AWS::Region}:580247275435:layer:LambdaInsightsExtension:14" 13 | 14 | Parameters: 15 | StageName: 16 | Type: String 17 | Default: dev 18 | BucketName: 19 | Type: String 20 | 21 | Resources: 22 | AsyncDownload: 23 | Type: AWS::Serverless::Function 24 | Properties: 25 | CodeUri: src/handlers/async-download/ 26 | Handler: app.handler 27 | Runtime: nodejs14.x 28 | MemorySize: 128 29 | Timeout: 120 30 | Description: A simple example includes a HTTP get method to call a Step Function 31 | Environment: 32 | Variables: 33 | WEB_SOCKET_ENDPOINT: !Join [ "/", [ !GetAtt SimpleAsyncResponseSocket.ApiEndpoint, !Ref Stage ] ] 34 | CONNECTIONS_TABLE: !Ref DDBConnectionsTable 35 | STATE_MACHINE_ARN: !Ref ObjectUploaderStateMachine 36 | Policies: 37 | - CloudWatchLambdaInsightsExecutionRolePolicy 38 | - DynamoDBWritePolicy: 39 | TableName: !Ref DDBConnectionsTable 40 | - StepFunctionsExecutionPolicy: 41 | StateMachineName: !GetAtt ObjectUploaderStateMachine.Name 42 | Events: 43 | DownloadApiEvent: 44 | Type: Api 45 | Properties: 46 | Path: /download 47 | Method: GET 48 | RestApiId: !Ref DownloadApi 49 | 50 | DownloadApi: 51 | Type: AWS::Serverless::Api 52 | Properties: 53 | Name: FileDownloadApi 54 | Description: The API which clients will use to initiate downloading a large file 55 | StageName: !Ref StageName 56 | TracingEnabled: true 57 | EndpointConfiguration: 58 | Type: REGIONAL 59 | 60 | ObjectUploaderStateMachine: 61 | Type: AWS::Serverless::StateMachine 62 | Properties: 63 | Tracing: 64 | Enabled: true 65 | DefinitionUri: statemachine/simple_workflow.asl.json 66 | DefinitionSubstitutions: 67 | DownloadS3ObjectFunctionArn: !GetAtt DownloadS3ObjectFunction.Arn 68 | GetWebSocketConnectionIDFunctionArn: !GetAtt GetWebSocketConnectionIDFunction.Arn 69 | SendMessageFunctionArn: !GetAtt SendMessageFunction.Arn 70 | Policies: 71 | - LambdaInvokePolicy: 72 | FunctionName: !Ref DownloadS3ObjectFunction 73 | - LambdaInvokePolicy: 74 | FunctionName: !Ref GetWebSocketConnectionIDFunction 75 | - LambdaInvokePolicy: 76 | FunctionName: !Ref SendMessageFunction 77 | 78 | DownloadS3ObjectFunction: 79 | Type: AWS::Serverless::Function 80 | Properties: 81 | CodeUri: src/handlers/download-s3-object/ 82 | Handler: app.handler 83 | Runtime: nodejs14.x 84 | MemorySize: 256 85 | Timeout: 900 86 | Environment: 87 | Variables: 88 | BUCKET_NAME: !Ref BucketName 89 | Policies: 90 | - CloudWatchLambdaInsightsExecutionRolePolicy 91 | - S3ReadPolicy: 92 | BucketName: !Ref BucketName 93 | - S3WritePolicy: 94 | BucketName: !Ref BucketName 95 | 96 | GetWebSocketConnectionIDFunction: 97 | Type: AWS::Serverless::Function 98 | Properties: 99 | CodeUri: src/handlers/get-websocket-connection/ 100 | Handler: app.handler 101 | Runtime: nodejs14.x 102 | Environment: 103 | Variables: 104 | CONNECTIONS_TABLE: !Ref DDBConnectionsTable 105 | Policies: 106 | - CloudWatchLambdaInsightsExecutionRolePolicy 107 | - DynamoDBReadPolicy: 108 | TableName: !Ref DDBConnectionsTable 109 | 110 | DDBConnectionsTable: 111 | Type: AWS::Serverless::SimpleTable 112 | Properties: 113 | PrimaryKey: 114 | Name: executionArnId 115 | Type: String 116 | ProvisionedThroughput: 117 | ReadCapacityUnits: 2 118 | WriteCapacityUnits: 2 119 | 120 | # WebSocket API Configurations 121 | SimpleAsyncResponseSocket: 122 | Type: AWS::ApiGatewayV2::Api 123 | Properties: 124 | Name: SimpleAsyncResponseSocket 125 | ProtocolType: WEBSOCKET 126 | RouteSelectionExpression: "$request.body.action" 127 | 128 | ConnectRoute: 129 | Type: AWS::ApiGatewayV2::Route 130 | Properties: 131 | ApiId: !Ref SimpleAsyncResponseSocket 132 | RouteKey: $connect 133 | AuthorizationType: NONE 134 | OperationName: ConnectRoute 135 | Target: !Join 136 | - '/' 137 | - - 'integrations' 138 | - !Ref ConnectIntegration 139 | 140 | ConnectIntegration: 141 | Type: AWS::ApiGatewayV2::Integration 142 | Properties: 143 | ApiId: !Ref SimpleAsyncResponseSocket 144 | Description: Connect Integration 145 | IntegrationType: AWS_PROXY 146 | IntegrationUri: 147 | Fn::Sub: 148 | arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${OnConnectFunction.Arn}/invocations 149 | 150 | DisconnectRoute: 151 | Type: AWS::ApiGatewayV2::Route 152 | Properties: 153 | ApiId: !Ref SimpleAsyncResponseSocket 154 | RouteKey: $disconnect 155 | AuthorizationType: NONE 156 | OperationName: DisconnectRoute 157 | Target: !Join 158 | - '/' 159 | - - 'integrations' 160 | - !Ref DisconnectIntegration 161 | 162 | DisconnectIntegration: 163 | Type: AWS::ApiGatewayV2::Integration 164 | Properties: 165 | ApiId: !Ref SimpleAsyncResponseSocket 166 | Description: Disconnect Integration 167 | IntegrationType: AWS_PROXY 168 | IntegrationUri: 169 | Fn::Sub: 170 | arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${OnDisconnectFunction.Arn}/invocations 171 | 172 | SendRoute: 173 | Type: AWS::ApiGatewayV2::Route 174 | Properties: 175 | ApiId: !Ref SimpleAsyncResponseSocket 176 | RouteKey: sendMessage 177 | AuthorizationType: NONE 178 | OperationName: SendRoute 179 | Target: !Join 180 | - '/' 181 | - - 'integrations' 182 | - !Ref SendIntegration 183 | 184 | SendIntegration: 185 | Type: AWS::ApiGatewayV2::Integration 186 | Properties: 187 | ApiId: !Ref SimpleAsyncResponseSocket 188 | Description: Send Integration 189 | IntegrationType: AWS_PROXY 190 | IntegrationUri: 191 | Fn::Sub: 192 | arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${SendMessageFunction.Arn}/invocations 193 | 194 | Deployment: 195 | Type: AWS::ApiGatewayV2::Deployment 196 | DependsOn: 197 | - ConnectRoute 198 | - SendRoute 199 | - DisconnectRoute 200 | Properties: 201 | ApiId: !Ref SimpleAsyncResponseSocket 202 | 203 | Stage: 204 | Type: AWS::ApiGatewayV2::Stage 205 | Properties: 206 | StageName: !Ref StageName 207 | DeploymentId: !Ref Deployment 208 | ApiId: !Ref SimpleAsyncResponseSocket 209 | 210 | OnConnectFunction: 211 | Type: AWS::Serverless::Function 212 | Properties: 213 | CodeUri: src/handlers/on-connect/ 214 | Handler: app.handler 215 | MemorySize: 256 216 | Runtime: nodejs14.x 217 | Environment: 218 | Variables: 219 | TABLE_NAME: !Ref DDBConnectionsTable 220 | Policies: 221 | - CloudWatchLambdaInsightsExecutionRolePolicy 222 | - DynamoDBWritePolicy: 223 | TableName: !Ref DDBConnectionsTable 224 | 225 | OnConnectPermission: 226 | Type: AWS::Lambda::Permission 227 | DependsOn: 228 | - SimpleAsyncResponseSocket 229 | Properties: 230 | Action: lambda:InvokeFunction 231 | FunctionName: !Ref OnConnectFunction 232 | Principal: apigateway.amazonaws.com 233 | 234 | OnDisconnectFunction: 235 | Type: AWS::Serverless::Function 236 | Properties: 237 | CodeUri: src/handlers/on-disconnect/ 238 | Handler: app.handler 239 | MemorySize: 256 240 | Runtime: nodejs14.x 241 | Environment: 242 | Variables: 243 | TABLE_NAME: !Ref DDBConnectionsTable 244 | Policies: 245 | - CloudWatchLambdaInsightsExecutionRolePolicy 246 | - DynamoDBCrudPolicy: 247 | TableName: !Ref DDBConnectionsTable 248 | 249 | OnDisconnectPermission: 250 | Type: AWS::Lambda::Permission 251 | DependsOn: 252 | - SimpleAsyncResponseSocket 253 | Properties: 254 | Action: lambda:InvokeFunction 255 | FunctionName: !Ref OnDisconnectFunction 256 | Principal: apigateway.amazonaws.com 257 | 258 | SendMessageFunction: 259 | Type: AWS::Serverless::Function 260 | Properties: 261 | CodeUri: src/handlers/send-message/ 262 | Handler: app.handler 263 | MemorySize: 256 264 | Runtime: nodejs14.x 265 | Policies: 266 | - CloudWatchLambdaInsightsExecutionRolePolicy 267 | - Statement: 268 | - Effect: Allow 269 | Action: 270 | - 'execute-api:ManageConnections' 271 | Resource: 272 | - !Sub 'arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${SimpleAsyncResponseSocket}/*' 273 | 274 | SendMessagePermission: 275 | Type: AWS::Lambda::Permission 276 | DependsOn: 277 | - SimpleAsyncResponseSocket 278 | Properties: 279 | Action: lambda:InvokeFunction 280 | FunctionName: !Ref SendMessageFunction 281 | Principal: apigateway.amazonaws.com 282 | 283 | Outputs: 284 | WebEndpoint: 285 | Description: "API Gateway endpoint URL" 286 | Value: !Sub "https://${DownloadApi}.execute-api.${AWS::Region}.amazonaws.com/${StageName}/" 287 | 288 | WebSocketEndpoint: 289 | Description: "WebSocket API Gateway endpoint URL" 290 | Value: !Join [ "/", [ !GetAtt SimpleAsyncResponseSocket.ApiEndpoint, !Ref Stage ] ] 291 | --------------------------------------------------------------------------------