├── README.md ├── aws_lambda_ftp_function.py ├── images ├── configure_function.png ├── lambda.png └── lambda_arn.png └── parse_and_email.py /README.md: -------------------------------------------------------------------------------- 1 | #AWS Lambda FTP Function 2 | 3 | The following How-To will demonstrate how to use AWS Lambda service to: 4 | 5 | - Connect to FTP 6 | - Download files 7 | - Archive files in FTP (moving them to `archive` folder) 8 | - Save files to S3 9 | 10 | Later this will trigger another event on Lambda to parse these files and send data to API Gateway. 11 | **Make sure that S3 and Lambda are on the same region, otherwise Lambda event will not work.** 12 | 13 | Flow diagram: 14 | 15 | ![](images/lambda.png) 16 | 17 | **This how-to assume that you have an AWS account with admin access.** 18 | 19 | 20 | ### Step 1 21 | **Create S3 bucket.** 22 | 23 | ### Step 2 24 | **Create Role to access this S3 bucket** 25 | 26 | 1. Click on Identity & Access Management (IAM) 27 | 2. Click on Roles 28 | 3. Create New Role 29 | 4. Name it `lambda_ftp_to_s3` 30 | 5. Select AWS Lambda. 31 | 6. Select `AWSLambdaFullAccess` and click Next. 32 | 7. This will then show you a review screen like below. 33 | ![](images/lambda_arn.png) 34 | 35 | ### Step 3 36 | Create Lambda function 37 | 38 | 1. Click on Get started now. 39 | 2. Skip the next screen as it just showing pre-defined functions 40 | 3. In configure function, please fill the form as below 41 | ![](images/configure_function.png) 42 | 43 | 4. In the code section, copy and paste the code in [aws_lambda_ftp_function.py](./aws_lambda_ftp_function.py) 44 | 5. **Please make sure to populate data with your own specific credentials for S3 bucket name and FTP connection. 45 | 6. Handler, keep it as it is 46 | 7. Role, use the one that we have created on step 2. 47 | 8. Timeout is quite improtant in this application as you may expect delays, slow FTP connection or large number of files to download. I would put it as 1 minute. 48 | 9. Click create function 49 | 50 | 51 | ### Step 4 52 | Now we need to setup the event to call this function to fetch FTP and download files. 53 | 54 | 1. In Lambda function page, go to `Event sources` 55 | 2. Add event source 56 | 3. Choose `CloudWatch Events - Schedule` 57 | 4. Use Rule name as you prefer and choose `Schedule expression` 58 | 5. You can either use pre-defined rates (1 minute, 5 minutes, 15 minutes, 1 hour or cron type schedule) 59 | 60 | 61 | ### Testing 62 | To test your function, run **Test** from function page and you should be able to see output at buttom. 63 | 64 | If everything was ok, you should be able to see files in your S3. 65 | 66 | ### TODO 67 | - ~~Add code to archive files in FTP~~ 68 | - Add code to read files from S3, parse them and send them to API gateway 69 | -------------------------------------------------------------------------------- /aws_lambda_ftp_function.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import boto3 3 | import os 4 | import ftplib 5 | 6 | # AWS S3 Bucket name 7 | bucket = "" 8 | 9 | # FTP Credentials 10 | ip = "" 11 | username = "" 12 | password = "" 13 | remote_directory = "" 14 | ftp_archive_folder = "" 15 | 16 | s3_client = boto3.resource('s3') 17 | 18 | # This function will check if a given name/path is a folder to avoid downloading it 19 | def is_ftp_folder(ftp, filename): 20 | try: 21 | res = ftp.sendcmd('MLST ' + filename) 22 | if 'type=dir;' in res: 23 | return True 24 | else: 25 | return False 26 | except: 27 | return False 28 | 29 | 30 | def lambda_handler(event, context): 31 | # Connecting to FTP 32 | try: 33 | ftp = ftplib.FTP(ip) 34 | ftp.login(username, password) 35 | except: 36 | print("Error connecting to FTP") 37 | 38 | try: 39 | ftp.cwd(remote_directory) 40 | except: 41 | print("Error changing to directory {}".format(remote_directory)) 42 | 43 | try: 44 | if not is_ftp_folder(ftp, ftp_archive_folder): 45 | print("Creating archive directory {}".format(ftp_archive_folder)) 46 | ftp.mkd(ftp_archive_folder) 47 | except: 48 | print("Error creting {} directory".format(ftp_archive_folder)) 49 | 50 | files = ftp.nlst() 51 | 52 | for file in files: 53 | if not is_ftp_folder(ftp, file): 54 | try: 55 | if os.path.isfile("/tmp/" + file): 56 | print("File {} exists locally, skip".format(file)) 57 | try: 58 | ftp.rename(file, ftp_archive_folder + "/" + file) 59 | except: 60 | print("Can not move file {} to archive folder".format(file)) 61 | 62 | else: 63 | print("Downloading {} ....".format(file)) 64 | ftp.retrbinary("RETR " + file, open("/tmp/" + file, 'wb').write) 65 | try: 66 | s3_client.meta.client.upload_file("/tmp/" + file, bucket, file) 67 | print("File {} uploaded to S3".format(file)) 68 | 69 | try: 70 | ftp.rename(file, ftp_archive_folder + "/" + file) 71 | except: 72 | print("Can not move file {} to archive folder".format(file)) 73 | except: 74 | print("Error uploading file {} !".format(file)) 75 | except: 76 | print("Error downloading file {}!".format(file)) 77 | -------------------------------------------------------------------------------- /images/configure_function.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/orasik/aws_lambda_ftp_function/8e9d183c6005ff863b6abbbc26e639c1ce7a2467/images/configure_function.png -------------------------------------------------------------------------------- /images/lambda.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/orasik/aws_lambda_ftp_function/8e9d183c6005ff863b6abbbc26e639c1ce7a2467/images/lambda.png -------------------------------------------------------------------------------- /images/lambda_arn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/orasik/aws_lambda_ftp_function/8e9d183c6005ff863b6abbbc26e639c1ce7a2467/images/lambda_arn.png -------------------------------------------------------------------------------- /parse_and_email.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | 3 | import urllib 4 | import boto3 5 | from email.mime.multipart import MIMEMultipart 6 | from email.mime.text import MIMEText 7 | 8 | print('Loading function') 9 | s3 = boto3.resource('s3') 10 | local_folder = "/tmp/" 11 | 12 | def sendMail(FROM,TO,SUBJECT,TEXT): 13 | import smtplib 14 | # your AWS SES smtp 15 | SERVER = '' 16 | EMAIL_USERNAME = '' 17 | EMAIL_PASSWORD = '' 18 | """this is some test documentation in the function""" 19 | msg = MIMEMultipart() 20 | msg['From'] = FROM 21 | msg['To'] = TO 22 | msg['Subject'] = SUBJECT 23 | part1 = MIMEText(TEXT, 'plain', 'utf-8') 24 | msg.attach(part1) 25 | # if you want to send HTML email, uncomment the following two lines 26 | # part2 = MIMEText(TEXT, 'html', 'utf-8') 27 | #msg.attach(part2) 28 | 29 | # Send the mail 30 | server = smtplib.SMTP(SERVER) 31 | "New part" 32 | server.starttls() 33 | server.login(EMAIL_USERNAME, EMAIL_PASSWORD) 34 | server.sendmail(FROM, TO, msg.as_string()) 35 | server.quit() 36 | 37 | 38 | def lambda_handler(event, context): 39 | 40 | # Get the object from the event and show its content type 41 | bucket = event['Records'][0]['s3']['bucket']['name'] 42 | key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8') 43 | try: 44 | print("Saving file {} to folder {}".format(key,local_folder)) 45 | s3.Bucket(bucket).download_file(key, local_folder+key) 46 | with open(local_folder+key, 'r') as content_file: 47 | content = content_file.read() 48 | print(content) 49 | 50 | try: 51 | sendMail('email@example.com','email@example.com', 'File '+ key , content) 52 | print("Email sent with subject {}".format(key)) 53 | except Exception as e: 54 | print(e) 55 | raise(e) 56 | except Exception as e: 57 | print(e) 58 | print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket)) 59 | raise e --------------------------------------------------------------------------------