├── .gitignore ├── Logs.png ├── Function.png ├── Schedule.png ├── ChangeSet.png ├── Resources.png ├── TagsAndRole.png ├── TickBoxes.png ├── CreatingStack.png ├── ConfigureStack.png ├── NOTICE ├── KinesisLambdaRuntime.png ├── KinesisLambdaPrewarming.png ├── .github └── PULL_REQUEST_TEMPLATE.md ├── CODE_OF_CONDUCT.md ├── src ├── main │ └── java │ │ └── com │ │ └── amazonaws │ │ └── services │ │ ├── lambda │ │ └── kinesis │ │ │ └── prewarming │ │ │ ├── ShardPrewarmMessage.java │ │ │ ├── PrewarmConfig.java │ │ │ ├── LambdaHandler.java │ │ │ ├── KinesisShardPrewarmer.java │ │ │ └── StreamUtils.java │ │ └── kinesis │ │ └── scaling │ │ ├── PercentDoubleSerialiser.java │ │ ├── AdjacentShards.java │ │ └── ShardHashInfo.java └── test │ └── java │ └── com │ └── amazonaws │ └── services │ └── lambda │ └── kinesis │ └── prewarming │ └── ShardPrewarmerTests.java ├── snippets.txt ├── pom.xml ├── deploy.yaml ├── CONTRIBUTING.md ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | target/* 2 | .settings 3 | .classpath 4 | .project 5 | -------------------------------------------------------------------------------- /Logs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/Logs.png -------------------------------------------------------------------------------- /Function.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/Function.png -------------------------------------------------------------------------------- /Schedule.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/Schedule.png -------------------------------------------------------------------------------- /ChangeSet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/ChangeSet.png -------------------------------------------------------------------------------- /Resources.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/Resources.png -------------------------------------------------------------------------------- /TagsAndRole.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/TagsAndRole.png -------------------------------------------------------------------------------- /TickBoxes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/TickBoxes.png -------------------------------------------------------------------------------- /CreatingStack.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/CreatingStack.png -------------------------------------------------------------------------------- /ConfigureStack.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/ConfigureStack.png -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | AWS Lambda Kinesis Prewarming 2 | Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /KinesisLambdaRuntime.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/KinesisLambdaRuntime.png -------------------------------------------------------------------------------- /KinesisLambdaPrewarming.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-lambda-kinesis-prewarming/HEAD/KinesisLambdaPrewarming.png -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/lambda/kinesis/prewarming/ShardPrewarmMessage.java: -------------------------------------------------------------------------------- 1 | package com.amazonaws.services.lambda.kinesis.prewarming; 2 | 3 | import lombok.Getter; 4 | import lombok.RequiredArgsConstructor; 5 | 6 | @RequiredArgsConstructor 7 | public class ShardPrewarmMessage { 8 | @Getter 9 | private final String shardId; 10 | @Getter 11 | private final String partitionKey; 12 | @Getter 13 | private final String explicitHashKey; 14 | @Getter 15 | private final String message; 16 | @Getter 17 | private final String sequenceNumber; 18 | } 19 | -------------------------------------------------------------------------------- /snippets.txt: -------------------------------------------------------------------------------- 1 | // generate the github table of cloudformation link images for launching stacks in all regions 2 | for r in `aws ec2 describe-regions --query Regions[*].RegionName --output text`; do echo "| [](https://console.aws.amazon.com/cloudformation/home?region=$r#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-$r.amazonaws.com/awslabs-code-$r/KinesisLambdaPrewarming/deploy.yaml) in $r |"; done 3 | 4 | // publish deploy.yaml to regional buckets 5 | for r in `aws ec2 describe-regions --query Regions[*].RegionName --output text`; do aws s3 cp deploy.yaml s3://awslabs-code-$r/KinesisLambdaPrewarming/deploy.yaml --acl public-read --region $r; done 6 | 7 | // publish lambda jar to regional buckets 8 | for r in `aws ec2 describe-regions --query Regions[*].RegionName --output text`; do aws s3 cp target/kinesis-prewarming-1.0.0.jar s3://awslabs-code-$r/KinesisLambdaPrewarming/kinesis-prewarming-1.0.0.jar --acl public-read --region $r; done -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/lambda/kinesis/prewarming/PrewarmConfig.java: -------------------------------------------------------------------------------- 1 | package com.amazonaws.services.lambda.kinesis.prewarming; 2 | 3 | import lombok.Getter; 4 | import lombok.Setter; 5 | 6 | public class PrewarmConfig { 7 | @Getter 8 | @Setter 9 | private String streamName = null; 10 | @Getter 11 | @Setter 12 | private String regionName = null; 13 | @Getter 14 | @Setter 15 | private String messagePrototype = null; 16 | 17 | public PrewarmConfig() { 18 | } 19 | 20 | public PrewarmConfig(String streamName, String regionName, String messagePrototype) { 21 | this.streamName = streamName; 22 | this.regionName = regionName; 23 | this.messagePrototype = messagePrototype; 24 | } 25 | 26 | protected Void validate() throws Exception { 27 | if (this.streamName == null) { 28 | throw new Exception("Configured Stream Name must not be null"); 29 | } 30 | 31 | if (this.regionName == null) { 32 | throw new Exception("Configured Region must not be null"); 33 | } 34 | 35 | return null; 36 | } 37 | 38 | @Override 39 | public String toString() { 40 | return String.format("%s,%s,%s", this.streamName, this.regionName, this.messagePrototype); 41 | } 42 | } 43 | -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/kinesis/scaling/PercentDoubleSerialiser.java: -------------------------------------------------------------------------------- 1 | /** 2 | * Amazon Kinesis Scaling Utility 3 | * 4 | * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved. 5 | * SPDX-License-Identifier: Apache-2.0 6 | */ 7 | package com.amazonaws.services.kinesis.scaling; 8 | 9 | import java.io.IOException; 10 | import java.text.DecimalFormat; 11 | 12 | import com.fasterxml.jackson.core.JsonGenerationException; 13 | import com.fasterxml.jackson.core.JsonGenerator; 14 | import com.fasterxml.jackson.databind.JsonSerializer; 15 | import com.fasterxml.jackson.databind.SerializerProvider; 16 | 17 | public class PercentDoubleSerialiser extends JsonSerializer { 18 | final DecimalFormat myFormatter = new DecimalFormat("#0.000%"); 19 | 20 | public PercentDoubleSerialiser() { 21 | } 22 | 23 | public void serialize(Double value, JsonGenerator jgen, SerializerProvider provider) 24 | throws IOException, JsonGenerationException { 25 | if (null == value) { 26 | // write the word 'null' if there's no value available 27 | jgen.writeNull(); 28 | } else { 29 | final String output = myFormatter.format(value); 30 | jgen.writeNumber(output); 31 | } 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/kinesis/scaling/AdjacentShards.java: -------------------------------------------------------------------------------- 1 | /** 2 | * Amazon Kinesis Scaling Utility 3 | * 4 | * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved. 5 | * SPDX-License-Identifier: Apache-2.0 6 | */ 7 | package com.amazonaws.services.kinesis.scaling; 8 | 9 | import java.math.BigInteger; 10 | 11 | /** 12 | * AdjacentShards are a transfer object for maintaining references between an 13 | * open shard, and it's lower and higher neighbours by partition hash value 14 | */ 15 | public class AdjacentShards { 16 | private String streamName; 17 | 18 | private ShardHashInfo lowerShard; 19 | 20 | private ShardHashInfo higherShard; 21 | 22 | public AdjacentShards(String streamName, ShardHashInfo lower, ShardHashInfo higher) throws Exception { 23 | // ensure that the shards are adjacent 24 | if (!new BigInteger(higher.getShard().getHashKeyRange().getStartingHashKey()) 25 | .subtract(new BigInteger(lower.getShard().getHashKeyRange().getEndingHashKey())) 26 | .equals(new BigInteger("1"))) { 27 | throw new Exception("Shards are not Adjacent"); 28 | } 29 | this.streamName = streamName; 30 | this.lowerShard = lower; 31 | this.higherShard = higher; 32 | } 33 | 34 | protected ShardHashInfo getLowerShard() { 35 | return lowerShard; 36 | } 37 | 38 | protected ShardHashInfo getHigherShard() { 39 | return higherShard; 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/lambda/kinesis/prewarming/LambdaHandler.java: -------------------------------------------------------------------------------- 1 | package com.amazonaws.services.lambda.kinesis.prewarming; 2 | 3 | import java.util.HashMap; 4 | import java.util.List; 5 | import java.util.Map; 6 | 7 | import com.amazonaws.regions.Region; 8 | import com.amazonaws.regions.Regions; 9 | import com.amazonaws.services.kinesis.AmazonKinesis; 10 | import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder; 11 | import com.amazonaws.services.lambda.runtime.Context; 12 | import com.amazonaws.services.lambda.runtime.RequestHandler; 13 | 14 | public class LambdaHandler implements RequestHandler { 15 | public static final String STREAM_NAME_PROPERTY = "StreamName"; 16 | private Map clientCache = new HashMap<>(); 17 | private KinesisShardPrewarmer prewarmer = new KinesisShardPrewarmer(); 18 | 19 | public Integer handleRequest(PrewarmConfig prewarmConfig, Context context) { 20 | try { 21 | prewarmConfig.validate(); 22 | 23 | Region region = Region.getRegion(Regions.fromName(prewarmConfig.getRegionName())); 24 | AmazonKinesis useKinesisClient = null; 25 | 26 | // get the connection from the cache 27 | if (clientCache.containsKey(region)) { 28 | useKinesisClient = clientCache.get(region); 29 | } else { 30 | // connect to kinesis and cache the connection 31 | AmazonKinesisClientBuilder builder = AmazonKinesisClientBuilder.standard() 32 | .withRegion(prewarmConfig.getRegionName()); 33 | useKinesisClient = builder.build(); 34 | clientCache.put(region, useKinesisClient); 35 | } 36 | 37 | List results = prewarmer.sendCanaryMessages(prewarmConfig.getStreamName(), 38 | prewarmConfig.getMessagePrototype(), useKinesisClient); 39 | results.forEach(item -> { 40 | System.out.println( 41 | String.format("Generated Kinesis canary message targeting Shard %s (Hash %s) sequence %s", 42 | item.getShardId(), item.getExplicitHashKey(), item.getSequenceNumber())); 43 | }); 44 | 45 | return results.size(); 46 | } catch (Exception e) { 47 | System.err.println(e.getMessage()); 48 | return -1; 49 | } 50 | } 51 | } 52 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4.0.0 4 | 5 | com.amazonaws.services.lambda 6 | kinesis-prewarming 7 | 1.0.0 8 | jar 9 | 10 | aws-lambda-kinesis-prewarming 11 | http://maven.apache.org 12 | 13 | 14 | UTF-8 15 | 16 | 17 | 18 | 19 | maven-compiler-plugin 20 | 3.1 21 | 22 | 1.8 23 | 1.8 24 | 25 | 26 | 27 | org.apache.maven.plugins 28 | maven-shade-plugin 29 | 2.3 30 | 31 | false 32 | 33 | 34 | 35 | package 36 | 37 | shade 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | junit 47 | junit 48 | 3.8.1 49 | test 50 | 51 | 52 | com.amazonaws 53 | aws-java-sdk-core 54 | 1.11.133 55 | 56 | 57 | com.amazonaws 58 | aws-java-sdk-kinesis 59 | 1.11.133 60 | 61 | 62 | com.amazonaws 63 | aws-lambda-java-core 64 | 1.1.0 65 | 66 | 67 | org.projectlombok 68 | lombok 69 | 1.16.10 70 | 71 | 72 | 73 | -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/lambda/kinesis/prewarming/KinesisShardPrewarmer.java: -------------------------------------------------------------------------------- 1 | package com.amazonaws.services.lambda.kinesis.prewarming; 2 | 3 | import java.nio.ByteBuffer; 4 | import java.util.ArrayList; 5 | import java.util.List; 6 | import java.util.Map; 7 | import java.util.Random; 8 | 9 | import com.amazonaws.services.kinesis.AmazonKinesis; 10 | import com.amazonaws.services.kinesis.model.PutRecordRequest; 11 | import com.amazonaws.services.kinesis.model.PutRecordResult; 12 | import com.amazonaws.services.kinesis.model.Shard; 13 | import com.amazonaws.services.kinesis.scaling.ShardHashInfo; 14 | 15 | import lombok.RequiredArgsConstructor; 16 | 17 | @RequiredArgsConstructor 18 | public class KinesisShardPrewarmer { 19 | private Random r = new Random(); 20 | 21 | private final byte[] randomBytes = new byte[16]; 22 | 23 | public List sendCanaryMessages(String streamName, String streamMessage, 24 | AmazonKinesis kinesisClient) throws Exception { 25 | Map openShards = StreamUtils.getOpenShards(kinesisClient, streamName, null); 26 | 27 | List results = new ArrayList<>(); 28 | 29 | // send a message to the stream on the start hash of each open shard 30 | openShards.keySet().forEach(shardId -> { 31 | // create a message to send based upon the provided stream message 32 | // this message will be set to use the explicit hash key for the 33 | // start of each shard, ensuring that we send a message to every 34 | // shard in the stream each time 35 | r.nextBytes(randomBytes); 36 | String thisPartitionKey = new String(randomBytes); 37 | String message = streamMessage == null ? new String() : streamMessage; 38 | Shard thisShard = openShards.get(shardId).getShard(); 39 | String thisHashKey = thisShard.getHashKeyRange().getStartingHashKey(); 40 | PutRecordRequest putRecordRequest = new PutRecordRequest().withStreamName(streamName) 41 | .withData(ByteBuffer.wrap(message.getBytes())).withPartitionKey(thisPartitionKey) 42 | .withExplicitHashKey(thisHashKey); 43 | 44 | // write the message to kinesis 45 | PutRecordResult response = kinesisClient.putRecord(putRecordRequest); 46 | 47 | // add result information 48 | results.add(new ShardPrewarmMessage(shardId, thisPartitionKey, thisHashKey, message, 49 | response.getSequenceNumber())); 50 | }); 51 | 52 | return results; 53 | } 54 | } 55 | -------------------------------------------------------------------------------- /src/test/java/com/amazonaws/services/lambda/kinesis/prewarming/ShardPrewarmerTests.java: -------------------------------------------------------------------------------- 1 | package com.amazonaws.services.lambda.kinesis.prewarming; 2 | 3 | import java.util.List; 4 | 5 | import com.amazonaws.services.kinesis.AmazonKinesis; 6 | import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder; 7 | 8 | import junit.framework.Test; 9 | import junit.framework.TestCase; 10 | import junit.framework.TestSuite; 11 | 12 | public class ShardPrewarmerTests extends TestCase { 13 | private LambdaHandler handler = new LambdaHandler(); 14 | 15 | /** 16 | * Create the test case 17 | * 18 | * @param testName 19 | * name of the test case 20 | */ 21 | public ShardPrewarmerTests(String testName) { 22 | super(testName); 23 | } 24 | 25 | /** 26 | * @return the suite of tests being tested 27 | */ 28 | public static Test suite() { 29 | return new TestSuite(ShardPrewarmerTests.class); 30 | } 31 | 32 | public void testHandler() throws Exception { 33 | PrewarmConfig config = new PrewarmConfig("IanTest", "eu-west-1", "Test Message"); 34 | Integer resultCount = handler.handleRequest(config, null); 35 | 36 | assertEquals("Send Message Count", 3, resultCount.intValue()); 37 | } 38 | 39 | public void testPrewarmer() throws Exception { 40 | // connect to kinesis and cache the connection 41 | AmazonKinesisClientBuilder builder = AmazonKinesisClientBuilder.standard().withRegion("eu-west-1"); 42 | AmazonKinesis useKinesisClient = builder.build(); 43 | KinesisShardPrewarmer prewarmer = new KinesisShardPrewarmer(); 44 | List results = prewarmer.sendCanaryMessages("IanTest", "Test Message", useKinesisClient); 45 | assertEquals("Message Count OK - with Prototype", 3, results.size()); 46 | 47 | results = prewarmer.sendCanaryMessages("IanTest", null, useKinesisClient); 48 | assertEquals("Message Count OK - no Prototype", 3, results.size()); 49 | } 50 | 51 | public void testMultiRegion() throws Exception { 52 | // connect to kinesis and cache the connection 53 | AmazonKinesisClientBuilder builder = AmazonKinesisClientBuilder.standard(); 54 | builder.withRegion("eu-west-1"); 55 | AmazonKinesis euWestClient = builder.build(); 56 | builder.withRegion("us-east-1"); 57 | AmazonKinesis usEastClient = builder.build(); 58 | KinesisShardPrewarmer prewarmer = new KinesisShardPrewarmer(); 59 | List results = prewarmer.sendCanaryMessages("IanTest", "Test Message", euWestClient); 60 | assertNotSame("Message Count OK", 0, results.size()); 61 | results = prewarmer.sendCanaryMessages("EnergyPipelineSensors", "Test Message", usEastClient); 62 | assertNotSame("Message Count OK", 0, results.size()); 63 | } 64 | } 65 | -------------------------------------------------------------------------------- /deploy.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: 2010-09-09 2 | Transform: AWS::Serverless-2016-10-31 3 | Parameters: 4 | StreamName: 5 | Default: My Stream 6 | Description: Name of the Kinesis Stream to send messages to 7 | Type: String 8 | AllowedPattern: .* 9 | MessageFrequency: 10 | Default: 60 11 | Description: Interval (minutes) at which to send messages 12 | Type: Number 13 | MinValue: 2 14 | StreamRegion: 15 | Default: us-east-1 16 | Description: Region where the Kinesis Stream exists 17 | Type: String 18 | AllowedValues: 19 | - ap-south-1 20 | - eu-west-2 21 | - eu-west-1 22 | - ap-northeast-2 23 | - ap-northeast-1 24 | - sa-east-1 25 | - ca-central-1 26 | - ap-southeast-1 27 | - ap-southeast-2 28 | - eu-central-1 29 | - us-east-1 30 | - us-east-2 31 | - us-west-1 32 | - us-west-2 33 | MessagePrototype: 34 | Default: Test Message 35 | Description: Message Template to use for sent messages 36 | Type: String 37 | Resources: 38 | ScheduledFunction: 39 | Type: AWS::Serverless::Function 40 | Properties: 41 | Handler: com.amazonaws.services.lambda.kinesis.prewarming.LambdaHandler::handleRequest 42 | Runtime: java8 43 | CodeUri: 44 | Bucket: !Sub awslabs-code-${AWS::Region} 45 | Key: KinesisLambdaPrewarming/kinesis-prewarming-1.0.0.jar 46 | MemorySize: 192 47 | Timeout: 60 48 | Tags: 49 | Name: KinesisLambdaPrewarmer 50 | Role: !GetAtt ScheduledServiceIAMRole.Arn 51 | Events: 52 | Timer: 53 | Type: Schedule 54 | Properties: 55 | Schedule: !Sub rate(${MessageFrequency} minutes) 56 | Input: 57 | !Sub | 58 | { 59 | "streamName":"${StreamName}", 60 | "regionName":"${StreamRegion}", 61 | "messagePrototype":"${MessagePrototype}" 62 | } 63 | ScheduledServiceIAMRole: 64 | Type: "AWS::IAM::Role" 65 | Properties: 66 | Path: "/" 67 | ManagedPolicyArns: 68 | - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" 69 | AssumeRolePolicyDocument: 70 | Version: "2012-10-17" 71 | Statement: 72 | - 73 | Sid: "AllowLambdaServiceToAssumeRole" 74 | Effect: "Allow" 75 | Action: 76 | - "sts:AssumeRole" 77 | Principal: 78 | Service: 79 | - "lambda.amazonaws.com" 80 | Policies: 81 | - 82 | PolicyName: "KinesisPermission" 83 | PolicyDocument: 84 | Version: "2012-10-17" 85 | Statement: 86 | - 87 | Effect: "Allow" 88 | Action: 89 | - "kinesis:DescribeStream" 90 | - "kinesis:ListStreams" 91 | - "kinesis:PutRecord" 92 | - "kinesis:PutRecords" 93 | Resource: "*" -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/awslabs/aws-lambda-kinesis-prewarming/issues), or [recently closed](https://github.com/awslabs/aws-lambda-kinesis-prewarming/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/aws-lambda-kinesis-prewarming/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/awslabs/aws-lambda-kinesis-prewarming/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/kinesis/scaling/ShardHashInfo.java: -------------------------------------------------------------------------------- 1 | /** 2 | * Amazon Kinesis Scaling Utility 3 | * 4 | * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved. 5 | * SPDX-License-Identifier: Apache-2.0 6 | */ 7 | package com.amazonaws.services.kinesis.scaling; 8 | 9 | import java.math.BigDecimal; 10 | import java.math.BigInteger; 11 | import java.math.RoundingMode; 12 | import java.text.DecimalFormat; 13 | import java.text.NumberFormat; 14 | 15 | import com.amazonaws.services.kinesis.model.Shard; 16 | import com.fasterxml.jackson.annotation.JsonProperty; 17 | import com.fasterxml.jackson.databind.annotation.JsonSerialize; 18 | 19 | /** 20 | * Immutable transfer object containing enhanced metadata about Shards in a 21 | * Stream, as well as utility methods for working with a Stream of Shards 22 | */ 23 | public class ShardHashInfo { 24 | private String streamName; 25 | 26 | @JsonProperty 27 | private BigInteger startHash; 28 | 29 | @JsonProperty 30 | private BigInteger endHash; 31 | 32 | @JsonProperty 33 | private BigInteger hashWidth; 34 | 35 | @JsonProperty 36 | @JsonSerialize(using = PercentDoubleSerialiser.class) 37 | private Double pctOfKeyspace; 38 | 39 | private Boolean matchesTargetResize; 40 | 41 | private Shard shard; 42 | 43 | private final NumberFormat pctFormat = NumberFormat.getPercentInstance(); 44 | 45 | private static final BigInteger maxHash = new BigInteger("340282366920938463463374607431768211455"); 46 | 47 | public ShardHashInfo(String streamName, Shard shard) { 48 | // prevent constructing a null object 49 | if (streamName == null || shard == null) { 50 | throw new ExceptionInInitializerError("Stream Name & Shard Required"); 51 | } 52 | this.shard = shard; 53 | this.streamName = streamName; 54 | this.endHash = new BigInteger(shard.getHashKeyRange().getEndingHashKey()); 55 | this.startHash = new BigInteger(shard.getHashKeyRange().getStartingHashKey()); 56 | this.hashWidth = getWidth(this.startHash, this.endHash); 57 | this.pctOfKeyspace = getPctOfKeyspace(this.hashWidth); 58 | } 59 | 60 | public static BigInteger getWidth(BigInteger startHash, BigInteger endHash) { 61 | return endHash.subtract(startHash); 62 | } 63 | 64 | public static BigInteger getWidth(String startHash, String endHash) { 65 | return getWidth(new BigInteger(endHash), new BigInteger(startHash)); 66 | } 67 | 68 | public static Double getPctOfKeyspace(BigInteger hashWidth) { 69 | return new BigDecimal(hashWidth).divide(new BigDecimal(maxHash), 10, RoundingMode.HALF_DOWN).doubleValue(); 70 | } 71 | 72 | @JsonProperty("shardID") 73 | protected String getShardId() { 74 | return this.shard.getShardId(); 75 | } 76 | 77 | public Shard getShard() { 78 | return this.shard; 79 | } 80 | 81 | protected BigInteger getStartHash() { 82 | return this.startHash; 83 | } 84 | 85 | protected BigInteger getEndHash() { 86 | return this.endHash; 87 | } 88 | 89 | protected BigInteger getHashWidth() { 90 | return this.hashWidth; 91 | } 92 | 93 | protected double getPctWidth() { 94 | return this.pctOfKeyspace; 95 | } 96 | 97 | protected Boolean getMatchesTargetResize() { 98 | return matchesTargetResize; 99 | } 100 | 101 | protected BigInteger getHashAtPctOffset(double pct) { 102 | return this.startHash.add(new BigDecimal(maxHash).multiply(BigDecimal.valueOf(pct)).toBigInteger()); 103 | } 104 | 105 | protected boolean isFirstShard() { 106 | return this.startHash.equals(BigInteger.valueOf(0l)); 107 | } 108 | 109 | protected boolean isLastShard() { 110 | return this.endHash.equals(maxHash); 111 | } 112 | 113 | public String getStreamName() { 114 | return streamName; 115 | } 116 | 117 | 118 | @Override 119 | public String toString() { 120 | return String.format("Shard %s - Start: %s, End: %s, Keyspace Width: %s (%s)\n", this.getShardId(), 121 | this.getStartHash().toString(), this.getEndHash().toString(), this.getHashWidth().toString(), 122 | new DecimalFormat("#0.000%").format(this.getPctWidth())); 123 | } 124 | } 125 | -------------------------------------------------------------------------------- /src/main/java/com/amazonaws/services/lambda/kinesis/prewarming/StreamUtils.java: -------------------------------------------------------------------------------- 1 | package com.amazonaws.services.lambda.kinesis.prewarming; 2 | 3 | import java.math.BigInteger; 4 | import java.util.ArrayList; 5 | import java.util.Collection; 6 | import java.util.Collections; 7 | import java.util.Comparator; 8 | import java.util.LinkedHashMap; 9 | import java.util.List; 10 | import java.util.Map; 11 | 12 | import com.amazonaws.services.kinesis.AmazonKinesis; 13 | import com.amazonaws.services.kinesis.model.DescribeStreamRequest; 14 | import com.amazonaws.services.kinesis.model.DescribeStreamResult; 15 | import com.amazonaws.services.kinesis.model.LimitExceededException; 16 | import com.amazonaws.services.kinesis.model.ResourceInUseException; 17 | import com.amazonaws.services.kinesis.model.Shard; 18 | import com.amazonaws.services.kinesis.model.StreamDescription; 19 | import com.amazonaws.services.kinesis.scaling.ShardHashInfo; 20 | 21 | public class StreamUtils { 22 | public static final int DESCRIBE_RETRIES = 10; 23 | // retry timeout set to 100ms as API's will potentially throttle > 10/sec 24 | public static final int RETRY_TIMEOUT_MS = 100; 25 | 26 | private static Object doOperation(AmazonKinesis kinesisClient, KinesisOperation operation, String streamName, 27 | int retries, boolean waitForActive) throws Exception { 28 | boolean done = false; 29 | int attempts = 0; 30 | Object result = null; 31 | do { 32 | attempts++; 33 | try { 34 | result = operation.run(kinesisClient); 35 | 36 | if (waitForActive) { 37 | waitForStreamStatus(kinesisClient, streamName, "ACTIVE"); 38 | } 39 | done = true; 40 | } catch (ResourceInUseException e) { 41 | // thrown when the Shard is mutating - wait until we are able to 42 | // do the modification or ResourceNotFoundException is thrown 43 | Thread.sleep(1000); 44 | } catch (LimitExceededException lee) { 45 | // API Throttling 46 | Thread.sleep(getTimeoutDuration(attempts)); 47 | } 48 | } while (!done && attempts < retries); 49 | 50 | if (!done) { 51 | throw new Exception(String.format("Unable to Complete Kinesis Operation after %s Retries", retries)); 52 | } else { 53 | return result; 54 | } 55 | } 56 | 57 | // calculate an exponential backoff based on the attempt count 58 | private static final long getTimeoutDuration(int attemptCount) { 59 | return new Double(Math.pow(2, attemptCount) * RETRY_TIMEOUT_MS).longValue(); 60 | } 61 | 62 | /** 63 | * Wait for a Stream to become available or transition to the indicated 64 | * status 65 | * 66 | * @param streamName 67 | * @param status 68 | * @throws Exception 69 | */ 70 | public static void waitForStreamStatus(AmazonKinesis kinesisClient, String streamName, String status) 71 | throws Exception { 72 | boolean ok = false; 73 | String streamStatus; 74 | // stream mutation takes around 30 seconds, so we'll start with 20 as 75 | // a timeout 76 | int waitTimeout = 20000; 77 | do { 78 | streamStatus = getStreamStatus(kinesisClient, streamName); 79 | if (!streamStatus.equals(status)) { 80 | Thread.sleep(waitTimeout); 81 | // reduce the wait timeout from the initial wait time 82 | waitTimeout = 1000; 83 | } else { 84 | ok = true; 85 | } 86 | } while (!ok); 87 | } 88 | 89 | /** 90 | * Get the status of a Stream 91 | * 92 | * @param streamName 93 | * @return 94 | */ 95 | protected static String getStreamStatus(AmazonKinesis kinesisClient, String streamName) throws Exception { 96 | return describeStream(kinesisClient, streamName, null).getStreamDescription().getStreamStatus(); 97 | } 98 | 99 | private static interface KinesisOperation { 100 | public Object run(AmazonKinesis client); 101 | } 102 | 103 | public static DescribeStreamResult describeStream(final AmazonKinesis kinesisClient, final String streamName, 104 | final String shardIdStart) throws Exception { 105 | KinesisOperation describe = new KinesisOperation() { 106 | public Object run(AmazonKinesis client) { 107 | DescribeStreamResult result = client.describeStream( 108 | new DescribeStreamRequest().withStreamName(streamName).withExclusiveStartShardId(shardIdStart)); 109 | 110 | return result; 111 | } 112 | }; 113 | return (DescribeStreamResult) doOperation(kinesisClient, describe, streamName, DESCRIBE_RETRIES, false); 114 | 115 | } 116 | 117 | public static Map getOpenShards(AmazonKinesis kinesisClient, String streamName, 118 | String lastShardId) throws Exception { 119 | StreamDescription stream = null; 120 | Collection openShardNames = new ArrayList<>(); 121 | Map shardMap = new LinkedHashMap<>(); 122 | 123 | // load all shards on the stream 124 | List allShards = new ArrayList<>(); 125 | 126 | do { 127 | stream = describeStream(kinesisClient, streamName, lastShardId).getStreamDescription(); 128 | for (Shard shard : stream.getShards()) { 129 | allShards.add(shard); 130 | lastShardId = shard.getShardId(); 131 | } 132 | } while (/* in some cases the describeStream call will return nothing */stream == null 133 | || stream.getShards() == null || stream.getShards().size() == 0 || stream.getHasMoreShards()); 134 | 135 | // load all the open shards on the Stream and sort if required 136 | for (Shard shard : allShards) { 137 | openShardNames.add(shard.getShardId()); 138 | shardMap.put(shard.getShardId(), new ShardHashInfo(streamName, shard)); 139 | 140 | // remove this Shard's parents from the set of active shards - they 141 | // are now closed and cannot be modified or written to 142 | if (shard.getParentShardId() != null) { 143 | openShardNames.remove(shard.getParentShardId()); 144 | shardMap.remove(shard.getParentShardId()); 145 | } 146 | if (shard.getAdjacentParentShardId() != null) { 147 | openShardNames.remove(shard.getAdjacentParentShardId()); 148 | shardMap.remove(shard.getAdjacentParentShardId()); 149 | } 150 | } 151 | 152 | // create a List of Open shards for sorting 153 | List sortShards = new ArrayList<>(); 154 | for (String s : openShardNames) { 155 | // paranoid null check in case we get a null map entry 156 | if (s != null) { 157 | sortShards.add(shardMap.get(s).getShard()); 158 | } 159 | } 160 | 161 | // sort the list into lowest start hash order 162 | Collections.sort(sortShards, new Comparator() { 163 | public int compare(Shard o1, Shard o2) { 164 | return compareShardsByStartHash(o1, o2); 165 | } 166 | }); 167 | 168 | // build the Shard map into the correct order 169 | shardMap.clear(); 170 | for (Shard s : sortShards) { 171 | shardMap.put(s.getShardId(), new ShardHashInfo(streamName, s)); 172 | } 173 | 174 | return shardMap; 175 | } 176 | 177 | private static final int compareShardsByStartHash(Shard o1, Shard o2) { 178 | return new BigInteger(o1.getHashKeyRange().getStartingHashKey()) 179 | .compareTo(new BigInteger(o2.getHashKeyRange().getStartingHashKey())); 180 | } 181 | } 182 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AWS Lambda pre-warming for Amazon Kinesis 2 | 3 | This project aims to give customers a simple ability to ensure that AWS Lambda functions are 'warm' and ready to process messages when they are received from Amazon Kinesis. When you create an AWS Lambda function with an Event Source which reads from Amazon Kinesis Streams, AWS Lambda will run up to 1 container per Shard in the Stream: 4 | 5 | ![kinesis lambda runtime](KinesisLambdaRuntime.png) 6 | 7 | These containers will be kept running as long as there are messages to be processed on the Stream. During very quiet periods, where there are no messages being received, then the Lambda containers [may be shut down or frozen](https://aws.amazon.com/blogs/compute/container-reuse-in-lambda). Depending up on the implementation language, you may find that the first time a container has to start up, it will take some time to load your code modules, attach ENI's, and so on. While extremely rare, in some cases our customers have said that they have extremly low latency requirements for execution, and this potential for increased latency due to cold starts needs to be minimised. There are some best practices to limit the impact of cold starts when they do happen: keep your deployed Lambda modules as small as possible, consider using Node.js or Python rather than Java (which has to start a JVM and classload certain dependencies), and run outside of VPC if possible. 8 | 9 | This module specifically focuses on those rare cases where we must do everything possible to limit the chances of Lambda containers being 'frozen' or stopped by periodically. We do this by sending 'canary' messages to the Stream, which will force AWS Lambda containers to stay running. However, given the mapping of Kinesis Shards to Lambda containers, we also ensure that canary messages are sent to every open Shard, which will then start a Lambda function container and ensure that real messages are less likely to observe cold start latency: 10 | 11 | ![kinesis lambda prewarming](KinesisLambdaPrewarming.png) 12 | 13 | A few key points are needed to understand the workflow: 14 | 15 | 1. The function can be invoked via an Amazon CloudWatch Scheduled Event, an AWS API Gateway call, or any other mechanism you'd like. The function invocation can provide the following parameters: 16 | 17 | ``` 18 | { 19 | "streamName": "MyStream", 20 | "regionName": "MyStreamRegion", 21 | "messagePrototype": "InputMessage" 22 | } 23 | ``` 24 | 25 | where you provide the `streamName` to send messages to, in the `regionName` where the Stream has been created. 26 | 27 | `messagePrototype` is a special optional field that allows you to inject special characters that your down-stream processing functions will recognise and know that they shouldn't attempt to deserialise. If you don't provide a `messagePrototype`, then an empty-string message will be sent. The module will send messages every 60 minutes by default, but you can change the frequency when deploying. 28 | 29 | 2. The function connects to Amazon Kinesis in the specified region, and extracts all the currently open [Shards](http://docs.aws.amazon.com/streams/latest/dev/key-concepts.html) that are being used for processing. 30 | 3. The function pushes messages to the Stream using PutRecords. It ensurese that every Shard receives a message by setting the [ExplicitHashKey](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html) for the message to the Shard's starting hash value 31 | 4. The architecture of AWS Lambda for processing Kinesis Streams is to have a single Lambda container/processor per Shard. This is to ensure that messages will be strictly ordered over time. By sending a Kinesis message to each Shard, we will force Lambda to start a container to process the message, even if there is no other traffic naturally coming across the Stream. 32 | 33 | 34 | Please note that a single function implementation can work across many [AWS Regions](http://docs.aws.amazon.com/general/latest/gr/rande.html) and can service a virtually unlimited number of Kinesis Streams. 35 | 36 | ## Getting Started 37 | 38 | This functionality is provided via the [Serverless Application Model](https://github.com/awslabs/serverless-application-model), which enables you to deploy Lambda applications, with their associated event sources and permissions, via AWS CloudFormation. 39 | 40 | You can launch the SAM stack which sets up the utility using the following templates: 41 | 42 | | | 43 | | --------------------------| 44 | | [](https://console.aws.amazon.com/cloudformation/home?region=ap-south-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-ap-south-1.amazonaws.com/awslabs-code-ap-south-1/KinesisLambdaPrewarming/deploy.yaml) in ap-south-1 | 45 | | [](https://console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-eu-west-2.amazonaws.com/awslabs-code-eu-west-2/KinesisLambdaPrewarming/deploy.yaml) in eu-west-2 | 46 | | [](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-eu-west-1.amazonaws.com/awslabs-code-eu-west-1/KinesisLambdaPrewarming/deploy.yaml) in eu-west-1 | 47 | | [](https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-2#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-ap-northeast-2.amazonaws.com/awslabs-code-ap-northeast-2/KinesisLambdaPrewarming/deploy.yaml) in ap-northeast-2 | 48 | | [](https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-ap-northeast-1.amazonaws.com/awslabs-code-ap-northeast-1/KinesisLambdaPrewarming/deploy.yaml) in ap-northeast-1 | 49 | | [](https://console.aws.amazon.com/cloudformation/home?region=sa-east-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-sa-east-1.amazonaws.com/awslabs-code-sa-east-1/KinesisLambdaPrewarming/deploy.yaml) in sa-east-1 | 50 | | [](https://console.aws.amazon.com/cloudformation/home?region=ca-central-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-ca-central-1.amazonaws.com/awslabs-code-ca-central-1/KinesisLambdaPrewarming/deploy.yaml) in ca-central-1 | 51 | | [](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-ap-southeast-1.amazonaws.com/awslabs-code-ap-southeast-1/KinesisLambdaPrewarming/deploy.yaml) in ap-southeast-1 | 52 | | [](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-ap-southeast-2.amazonaws.com/awslabs-code-ap-southeast-2/KinesisLambdaPrewarming/deploy.yaml) in ap-southeast-2 | 53 | | [](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-eu-central-1.amazonaws.com/awslabs-code-eu-central-1/KinesisLambdaPrewarming/deploy.yaml) in eu-central-1 | 54 | | [](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-us-east-1.amazonaws.com/awslabs-code-us-east-1/KinesisLambdaPrewarming/deploy.yaml) in us-east-1 | 55 | | [](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-us-east-2.amazonaws.com/awslabs-code-us-east-2/KinesisLambdaPrewarming/deploy.yaml) in us-east-2 | 56 | | [](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-us-west-1.amazonaws.com/awslabs-code-us-west-1/KinesisLambdaPrewarming/deploy.yaml) in us-west-1 | 57 | | [](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=AWSLambdaKinesisPrewarming&templateURL=https://s3-us-west-2.amazonaws.com/awslabs-code-us-west-2/KinesisLambdaPrewarming/deploy.yaml) in us-west-2 | 58 | 59 | This will load the template into the CloudFormation Web Console. Click 'Next' to go the the detail configuration: 60 | 61 | ![ConfigureStack](ConfigureStack.png) 62 | 63 | The stack name has been preconfigured to a descriptive value, but you can change this should you wish to. Then add the Message Prototype text, the name of the Kinesis Stream, the AWS Region where the Stream is deployed, and the frequency, in minutes, that the module should send messages. Please note that this configuration can run across AWS Regions. Click 'Next' to go to the next screen: 64 | 65 | ![TagsAndRole](TagsAndRole.png) 66 | 67 | You can add any relevant Tags, and then you MUST select an [AWS IAM Role which CloudFormation uses to modify AWS Resources on your behalf](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-iam-servicerole.html). Click 'Next' to go to the review and validation pages. You will have to tick two boxes: 68 | 69 | ![TickBoxes](TickBoxes.png) 70 | 71 | Which enable CloudFormation to create IAM resources on your behalf, in this case Permissions for CloudWatch to invoke the new Lambda function. Finally, the Serverless Application Model uses [CloudFormation Change Sets](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets-create.html) to transform the provided deployment model to a full CloudFormation Stack. You'll need to enable the creation of a Change Set by selecting 'Create Change Set': 72 | 73 | ![Create Change Set](ChangeSet.png) 74 | 75 | Once done, CloudFormation will give you a preview of the resources to be created: 76 | 77 | ![Resources](Resources.png) 78 | 79 | Finally, click 'Execute' to build the CloudFormation stack. You'll see that the console shows the stack being created: 80 | 81 | ![StackCreating](CreatingStack.png) 82 | 83 | Once, done, you'll see you have a new Lambda function called "AWSLambdaKinesisPrewarming-ScheduledFunction-\": 84 | 85 | ![Function](Function.png) 86 | 87 | And a CloudWatch Scheduled Event Rule to invoke it every 15 minutes: 88 | 89 | ![Schedule](Schedule.png) 90 | 91 | ## Checking that the module is running 92 | 93 | Once running, you can check the CloudWatch Logs for the Lambda function, which should show you detail about canary messages being sent: 94 | 95 | ![Logs](Logs.png) 96 | 97 | ## Making changes 98 | 99 | You are very welcome to make changes to the module, and we're always keen to accept pull requests. If you do make changes to the codebase, you can then build a new version of the Lambda jar with `mvn package`, which will create a new deployable jar in the `./target` folder. It is reocmmended if you do this that you update the version number in the `pom.xml` so that your jar has a new version. 100 | 101 | ---- 102 | 103 | AWS Lambda Kinesis Prewarming 104 | 105 | Copyright 2017-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 106 | 107 | This library is licensed under the Apache 2.0 License. --------------------------------------------------------------------------------