├── images
├── aws_services.PNG
├── aws_lambda_creation1.PNG
├── aws_lambda_creation2.PNG
├── aws_lambda_creation3.PNG
├── aws_lambda_creation4.PNG
├── aws_lambda_creation5.PNG
└── aws_lambda_creation6.PNG
├── README.md
├── code
├── lambda_s3_delete_objects.py
└── lambda_s3_copy_objects.py
├── s3_delete_data
└── README.md
└── s3_copy_data
└── README.md
/images/aws_services.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/prabhakar2020/aws_lambda_function/HEAD/images/aws_services.PNG
--------------------------------------------------------------------------------
/images/aws_lambda_creation1.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/prabhakar2020/aws_lambda_function/HEAD/images/aws_lambda_creation1.PNG
--------------------------------------------------------------------------------
/images/aws_lambda_creation2.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/prabhakar2020/aws_lambda_function/HEAD/images/aws_lambda_creation2.PNG
--------------------------------------------------------------------------------
/images/aws_lambda_creation3.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/prabhakar2020/aws_lambda_function/HEAD/images/aws_lambda_creation3.PNG
--------------------------------------------------------------------------------
/images/aws_lambda_creation4.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/prabhakar2020/aws_lambda_function/HEAD/images/aws_lambda_creation4.PNG
--------------------------------------------------------------------------------
/images/aws_lambda_creation5.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/prabhakar2020/aws_lambda_function/HEAD/images/aws_lambda_creation5.PNG
--------------------------------------------------------------------------------
/images/aws_lambda_creation6.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/prabhakar2020/aws_lambda_function/HEAD/images/aws_lambda_creation6.PNG
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # AWS Lambda Function
2 |
3 | - The code we run on AWS Lambda is called a **Lambda function**. After we create Lambda function it is always ready to run as soon as it is triggered, similar to a formula in a spreadsheet.
4 | - Each lambda function includes our code as well as some associated configuration information, including the function names and resource requirements.
5 | - AWS Lambda is a serverless compute service that runs our code when events are triggered and automatically manages the underlying compute resources for you.
6 | - We can use AWS Lambda functions to extend other AWS services with custom logic, or create our own backend services that operate at AWS scale, performance, and security.
7 | - AWS Lambda can automatically run code in response to multiple events, such as HTTP requests via Amazon API Gateway, modifications to objects in Amazon S3 buckets, EC2 updates, triggering the deployments and etc.
8 |
9 | In this example, we are going talk about -
10 | 1. [S3 delete data as soon as data uploaded on S3 bucket](s3_delete_data)
11 | 1. [Copy source S3 data to target S3 bucket as soon as data uploaded on Source S3 bucket](s3_copy_data)
12 |
13 | For more details about code examplanation, debugging and testing.
14 | Please checkout my youtube video -
15 |
--------------------------------------------------------------------------------
/code/lambda_s3_delete_objects.py:
--------------------------------------------------------------------------------
1 | from __future__ import print_function
2 | import boto3
3 | import time, urllib
4 | import json
5 | """Code snippet for deleting the objects from AWS S3 bucket as soon as objects uploaded on S3 bucket
6 | @author: Prabhakar G
7 | """
8 | print ("*"*80)
9 | print ("Initializing..")
10 | print ("*"*80)
11 |
12 | s3 = boto3.client('s3')
13 |
14 | def lambda_handler(event, context):
15 | # TODO implement
16 | bucket = event['Records'][0]['s3']['bucket']['name']
17 | object_key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'])
18 |
19 | try:
20 | print ("Using waiter to waiting for object to persist through s3 service")
21 | # It will till S3 service return the response
22 | waiter = s3.get_waiter('object_exists')
23 | waiter.wait(Bucket=bucket, Key=object_key)
24 | response = s3.head_object(Bucket=bucket, Key=object_key)
25 | print ("CONTENT TYPE : "+str(response['ContentType']))
26 | print ('ETag :' +str(response['ETag']))
27 | print ('Content-Length :'+str(response['ContentLength']))
28 | print ('KeyName :'+str(object_key))
29 | print ('Deleting object :'+str(object_key))
30 | # It will delete the objects/ data from S3 bucket as soon as trigger/ event is invoked
31 | s3.delete_object(Bucket=bucket, Key=object_key)
32 | return response['ContentType']
33 | except Exception as err:
34 | print ("Error -"+str(err))
35 | return err
36 |
--------------------------------------------------------------------------------
/code/lambda_s3_copy_objects.py:
--------------------------------------------------------------------------------
1 | from __future__ import print_function
2 | import boto3
3 | import time, urllib
4 | import json
5 | """Code snippet for copying the objects from AWS source S3 bucket to target S3 bucket as soon as objects uploaded on source S3 bucket
6 | @author: Prabhakar G
7 | """
8 | print ("*"*80)
9 | print ("Initializing..")
10 | print ("*"*80)
11 |
12 | s3 = boto3.client('s3')
13 |
14 | def lambda_handler(event, context):
15 | # TODO implement
16 | source_bucket = event['Records'][0]['s3']['bucket']['name']
17 | object_key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'])
18 | target_bucket = 'my-target-bucket'
19 | copy_source = {'Bucket': source_bucket, 'Key': object_key}
20 |
21 | print ("Source bucket : ", source_bucket)
22 | print ("Target bucket : ", target_bucket)
23 | print ("Log Stream name: ", context.log_stream_name)
24 | print ("Log Group name: ", context.log_group_name)
25 | print ("Request ID: ", context.aws_request_id)
26 | print ("Mem. limits(MB): ", context.memory_limit_in_mb)
27 |
28 | try:
29 | print ("Using waiter to waiting for object to persist through s3 service")
30 | waiter = s3.get_waiter('object_exists')
31 | waiter.wait(Bucket=source_bucket, Key=object_key)
32 | s3.copy_object(Bucket=target_bucket, Key=object_key, CopySource=copy_source)
33 | return response['ContentType']
34 | except Exception as err:
35 | print ("Error -"+str(err))
36 | return e
37 |
--------------------------------------------------------------------------------
/s3_delete_data/README.md:
--------------------------------------------------------------------------------
1 | # AWS Lambda Function
2 |
3 | In this example, we are going talk about -
4 | ```
5 | AWS lambda function for deleting uploaded objects/ data from S3 bucket as soon as object/ data uploaded on S3 bucket.
6 | ```
7 |
8 | Steps for creating the AWS Lambda function
9 | -------------
10 | 1. Login to AWS console https://console.aws.amazon.com (If you dont have AWS account, follow AWS console signup options)
11 | 1. Click on **Services** link on main menu and choose **Lambda** option under **compute** section
12 | 
13 | 1. Under Lambda functions page, click on **create function** button
14 | 
15 | 1. You can create lambda function from available pre-defined templates as well as from scratch. In this example, I am creating lambda function from scratch by selecting **Author from scratch**
16 | 
17 | 1. Fill the name of the lambda function and runtime (technology which you can write lambda function)
18 | 
19 | 1. Choose permissions to run/ execute the lambda function. Here I am selecting option as already existing role which I had created earlier. In your case you can choose option **Create a new role with basic Lambda permissions** and create the role accordingly. For demo purpose you can grant administrative role
20 | 
21 | 1. Click on **Create Function** button
22 | 1. Click on **Add Trigger** button, since we want to execute the lambda function for specific triggers/ events.
23 | 
24 | 1. Choose a trigger type and its configuration as bucket, event type and prefix and sufix values. After filling required details, click on **Add** button. As of now these prefix and sufix are not required for this demo, this these prefix and sufix are required for executing the lambda functions based on the filters with file types prefix and sufix.
25 | 
26 | 1. Now its time to write a code for first lambda function.
27 |
28 | As I described earlier, in this example we are going to delete the data/ objects from S3 bucket as soon as data uploaded on S3 bucket.
29 |
30 | The following code snippet (lambda function will be helpful for deleting uploaded objects/ data on S3 bucket
31 | ```python
32 | from __future__ import print_function
33 | import boto3
34 | import time, urllib
35 | import json
36 | """Code snippet for deleting the objects from AWS S3 bucket as soon as objects uploaded on S3 bucket
37 | @author: Prabhakar G
38 | """
39 | print ("*"*80)
40 | print ("Initializing..")
41 | print ("*"*80)
42 |
43 | s3 = boto3.client('s3')
44 |
45 | def lambda_handler(event, context):
46 | # TODO implement
47 | bucket = event['Records'][0]['s3']['bucket']['name']
48 | object_key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'])
49 |
50 | try:
51 | print ("Using waiter to waiting for object to persist through s3 service")
52 | # It will wait till s3 service return the output
53 | waiter = s3.get_waiter('object_exists')
54 | waiter.wait(Bucket=bucket, Key=object_key)
55 | response = s3.head_object(Bucket=bucket, Key=object_key)
56 | print ("Key :"+str(object_key))
57 | print ("Content Type : "+str(response['ContentType']))
58 | print ('ETag :' +str(response['ETag']))
59 | print ('Content-Length :'+str(response['ContentLength']))
60 | print ('KeyName :'+str(object_key))
61 | print ('Deleting object :'+str(object_key))
62 | # Delete the uploaded objects/ data from defined bucket
63 | s3.delete_object(Bucket=bucket, Key=object_key)
64 | return response['ContentType']
65 | except Exception as err:
66 | print ("Error -"+str(err))
67 | return err
68 | ```
69 |
70 | Once your lambda function creation is done, now go to your AWS S3 buckets page and click on the bucket name which you have selected on lambda function **Add triggers** page. Upload any file and refresh the page once file is uploaded. It will delete the file as soon as upload is completed on S3 bucket.
71 |
72 | For more details about code examplanation, debugging and testing.
73 |
74 | Please checkout my youtube video -
75 |
76 | [Download source code - S3 Delete data](../code)
--------------------------------------------------------------------------------
/s3_copy_data/README.md:
--------------------------------------------------------------------------------
1 | # AWS Lambda Function
2 |
3 | In this example, we are going talk about -
4 | ```
5 | AWS lambda function for copy source S3 bucket objects/ data on target S3 bucket as soon as objects/ data uploaded on source bucket.
6 | ```
7 | **Note**: First we have to create two S3 buckets one with name my-source-bucket and second one is my-target-bucket
8 |
9 | Steps for creating the AWS Lambda function
10 | -------------
11 | 1. Login to AWS console https://console.aws.amazon.com (If you dont have AWS account, follow AWS console signup options)
12 | 1. Click on **Services** link on main menu and choose **Lambda** option under **compute** section
13 | 
14 | 1. Under Lambda functions page, click on **create function** button
15 | 
16 | 1. You can create lambda function from available pre-defined templates as well as from scratch. In this example, I am creating lambda function from scratch by selecting **Author from scratch**
17 | 
18 | 1. Fill the name of the lambda function and runtime (technology which you can write lambda function)
19 | 
20 | 1. Choose permissions to run/ execute the lambda function. Here I am selecting option as already existing role which I had created earlier. In your case you can choose option **Create a new role with basic Lambda permissions** and create the role accordingly. For demo purpose you can grant administrative role
21 | 
22 | 1. Click on **Create Function** button
23 | 1. Click on **Add Trigger** button, since we want to execute the lambda function for specific triggers/ events.
24 | 
25 | 1. Choose a trigger type and its configuration as bucket, event type and prefix and sufix values. After filling required details, click on **Add** button. As of now these prefix and sufix are not required for this demo, this these prefix and sufix are required for executing the lambda functions based on the filters with file types prefix and sufix.
26 | 
27 | 1. Now its time to write a code for first lambda function.
28 |
29 | As I described earlier, in this example we are going to copy the objects/ data from source S3 bucket to target S3 bucket.
30 |
31 | The following code snippet (lambda function will be helpful for copying uploaded objects/ data from source S3 bucket to target S3 bucket.
32 | ```python
33 | from __future__ import print_function
34 | import boto3
35 | import time, urllib
36 | import json
37 | """Code snippet for copying the objects from AWS source S3 bucket to target S3 bucket as soon as objects uploaded on source S3 bucket
38 | @author: Prabhakar G
39 | """
40 | print ("*"*80)
41 | print ("Initializing..")
42 | print ("*"*80)
43 |
44 | s3 = boto3.client('s3')
45 |
46 | def lambda_handler(event, context):
47 | # TODO implement
48 | source_bucket = event['Records'][0]['s3']['bucket']['name']
49 | object_key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'])
50 | target_bucket = 'my-target-bucket'
51 | copy_source = {'Bucket': source_bucket, 'Key': object_key}
52 | print ("Source bucket : ", source_bucket)
53 | print ("Target bucket : ", target_bucket)
54 | print ("Log Stream name: ", context.log_stream_name)
55 | print ("Log Group name: ", context.log_group_name)
56 | print ("Request ID: ", context.aws_request_id)
57 | print ("Mem. limits(MB): ", context.memory_limit_in_mb)
58 | try:
59 | print ("Using waiter to waiting for object to persist through s3 service")
60 | waiter = s3.get_waiter('object_exists')
61 | waiter.wait(Bucket=source_bucket, Key=object_key)
62 | s3.copy_object(Bucket=target_bucket, Key=object_key, CopySource=copy_source)
63 | return response['ContentType']
64 | except Exception as err:
65 | print ("Error -"+str(err))
66 | return e
67 | ```
68 |
69 | Once your lambda function creation is done, now go to your AWS S3 buckets page and click on my-source-bucket name which you have selected on lambda function **Add triggers** page. Upload any file on my-source-bucket and then go to my-target-bucket and check. Both the bucket you can see the similar data.
70 |
71 | For more details about code examplanation, debugging and testing.
72 |
73 | Please checkout my youtube video -
74 |
75 | [Download source code - S3 Copy data](../code)
--------------------------------------------------------------------------------