├── README.md └── mylambdafunction.py /README.md: -------------------------------------------------------------------------------- 1 | # Lambda gdal-python 2 | 3 | Create a python Lambda function with [GDAL](http://gdal.org) python bindings 4 | 5 | Related to RemotePixel.ca [blog](http://remotepixel.ca/blog/landsat8-ndvi-20160212.html) 6 | 7 | I assume you know how to start an EC2 virtual server (with AMAZON Linux AMI) and to run some basics shell commands 8 | 9 | ###### Update and Install Python 10 | 11 | ```sh 12 | sudo yum update 13 | sudo yum install python27-devel python27-pip gcc libjpeg-devel zlib-devel gcc-c++ python-devel libpng-devel freetype-devel libcurl-devel 14 | 15 | ``` 16 | 17 | ###### Create a directory where to devellop function 18 | 19 | ```sh 20 | mkdir /home/ec2-user/lambda 21 | mkdir /home/ec2-user/lambda/local 22 | cd /home/ec2-user/lambda 23 | ``` 24 | 25 | ###### Create a virtualenv and activate it 26 | 27 | ```sh 28 | virtualenv env 29 | source env/bin/activate 30 | ``` 31 | 32 | ###### Install some Python modules 33 | 34 | ```sh 35 | pip --no-cache-dir install numpy 36 | ``` 37 | 38 | ###### Download and Install GDAL 39 | 40 | ```sh 41 | wget https://github.com/OSGeo/proj.4/archive/4.9.2.tar.gz 42 | tar -zvxf 4.9.2.tar.gz 43 | cd proj.4-4.9.2/ 44 | ./configure --prefix=/home/ec2-user/lambda/local 45 | make 46 | make install 47 | 48 | cd .. 49 | 50 | wget http://download.osgeo.org/gdal/1.11.3/gdal-1.11.3.tar.gz 51 | tar -xzvf gdal-1.11.3.tar.gz 52 | cd gdal-1.11.3 53 | ./configure --prefix=/home/ec2-user/lambda/local \ 54 | --with-geos=/home/ec2-user/lambda/local/bin/geos-config \ 55 | --with-static-proj4=/home/ec2-user/lambda/local \ 56 | --with-curl \ 57 | --with-python 58 | make 59 | make install 60 | cd .. 61 | ``` 62 | 63 | ###### Create you python function 64 | 65 | ```sh 66 | vi /home/ec2-user/lambda/mylambdafunction.py 67 | 68 | ``` 69 | 70 | ###### Edit your function 71 | 72 | In order to get osgeo.gdal to load you new to set the GDAL_DATA path in the environment variable and load some shared libraries 73 | 74 | ```python 75 | #!/usr/bin/env python 76 | # -*- coding: utf-8 -*- 77 | import os 78 | from ctypes import cdll 79 | 80 | #Set environment variables and load shared libraries 81 | path = os.path.dirname(os.path.realpath(__file__)) 82 | os.environ['GDAL_DATA'] = os.path.join(path, "local/share/gdal") 83 | lib2 = cdll.LoadLibrary(os.path.join(path, 'local/lib/libproj.so.9')) 84 | lib1 = cdll.LoadLibrary(os.path.join(path, 'local/lib/libgdal.so')) 85 | 86 | #Import gdal modules 87 | from osgeo import gdal 88 | 89 | def processing_func(event, context): 90 | "This is my worker function that responds to an API getaway call" 91 | 92 | #Whatever you want 93 | 94 | return results 95 | 96 | ``` 97 | 98 | ###### Creating a Deployment Package 99 | You have create a zip file with everything you need (shared libraries and python packages) 100 | 101 | First we need to move the osgeo package into the site-packages root directory 102 | ```sh 103 | cd $VIRTUAL_ENV/lib64/python2.7/site-packages/GDAL-1.11.3-py2.7-linux-x86_64.egg 104 | mv o* ../ 105 | mv g* ../ 106 | cd $VIRTUAL_ENV/.. 107 | ``` 108 | 109 | We can now zip everything 110 | 111 | ```sh 112 | zip -9 lambda.zip mylambdafunction.py 113 | zip -r9 lambda.zip local/lib/libgdal.so 114 | zip -r9 lambda.zip local/lib/libproj.so.9 115 | zip -r9 lambda.zip local/share 116 | cd $VIRTUAL_ENV/lib/python2.7/site-packages 117 | zip -r9 /home/ec2-user/lambda/lambda.zip * 118 | cd $VIRTUAL_ENV/lib64/python2.7/site-packages 119 | zip -r9 /home/ec2-user/lambda/lambda.zip * 120 | 121 | ``` 122 | 123 | We're done you can now upload your lambda.zip in AWS S3 and call your worker (mylambdafunction.processing_func) using API gateway for example) 124 | 125 | 126 | More 127 | ------- 128 | - Running Python with compiled code on AWS Lambda [PerryGeo Blog](http://www.perrygeo.com/running-python-with-compiled-code-on-aws-lambda.html) 129 | - AWS Lambda [Getting Started](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html) 130 | 131 | -------------------------------------------------------------------------------- /mylambdafunction.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | import os 4 | from ctypes import cdll 5 | 6 | #Import other Modules 7 | 8 | path = os.path.dirname(os.path.realpath(__file__)) 9 | os.environ['GDAL_DATA'] = os.path.join(path, "local/share/gdal") 10 | lib2 = cdll.LoadLibrary(os.path.join(path, 'local/lib/libproj.so.9')) 11 | lib1 = cdll.LoadLibrary(os.path.join(path, 'local/lib/libgdal.so')) 12 | from osgeo import gdal, ogr, osr 13 | 14 | #Import gdal modules 15 | from osgeo import gdal 16 | 17 | def processing_func(event, context): 18 | "This is my worker function that responds to an API getaway call" 19 | 20 | sceneid = event['scene'] 21 | 22 | try: 23 | WRSPath = sceneid[3:6] 24 | WRSRow = sceneid[6:9] 25 | landsat_address = 'http://landsat-pds.s3.amazonaws.com/L8/{0}/{1}/{2}/{2}'.format(WRSPath, WRSRow, sceneid) 26 | 27 | bqa = '/vsicurl/{0}_BQA.TIF'.format(landsat_address) 28 | meta_file = '{0}_MTL.txt'.format(landsat_address) 29 | try: 30 | meta_data = urllib2.urlopen(meta_file).readlines() 31 | except: 32 | return {'errorMessage': 'Landsat image {} not available on aws'.format(sceneid)} 33 | 34 | src_ds = gdal.Open(bqa, gdal.GA_ReadOnly) 35 | geoT = src_ds.GetGeoTransform() 36 | proj = src_ds.GetProjection() 37 | 38 | cols = master_ds.RasterXSize 39 | rows = master_ds.RasterYSize 40 | 41 | src_ds = None 42 | 43 | return {'landsatid': sceneid, 'xsize': cols, 'ysize': rows} 44 | except: 45 | return {'errorMessage': 'Could not open image {}'.format(sceneid)} 46 | --------------------------------------------------------------------------------