├── .gitignore ├── .terraform_version ├── LICENSE ├── README.md ├── main.tf ├── output.tf ├── scripts ├── cleanup.sh ├── dependencies_sha.sh ├── mktmp.sh ├── nodejs │ ├── build_environment.sh │ ├── build_payload.sh │ └── project_sha.sh ├── payload_exists.py ├── payload_hash.sh ├── python │ ├── build_environment.sh │ ├── build_payload.sh │ └── project_sha.sh └── tmpdir.sh ├── test ├── nodejs │ ├── main.tf │ ├── output │ │ └── .gitkeep │ └── src │ │ ├── index.js │ │ └── package.json └── python │ ├── main.tf │ ├── output │ └── .gitkeep │ └── src │ ├── main.py │ └── requirements.txt ├── tests ├── nodejs │ └── simple_build │ │ ├── .terraform.lock.hcl │ │ ├── lambda │ │ └── main.js │ │ ├── main.tf │ │ └── output │ │ └── .gitkeep └── python │ ├── complex_build │ ├── lambda │ │ ├── main.py │ │ └── requirements.txt │ ├── main.tf │ └── output │ │ └── .gitkeep │ ├── fail_build │ ├── lambda │ │ └── main.py │ ├── main.tf │ └── output │ │ └── .gitkeep │ └── simple_build │ ├── lambda │ └── main.py │ ├── main.tf │ └── output │ ├── .gitkeep │ ├── lambda │ └── main.py │ └── main.tf ├── variables.tf └── versions.tf /.gitignore: -------------------------------------------------------------------------------- 1 | tests/*/output/*.zip 2 | .terraform 3 | *.tfstate* 4 | *.zip -------------------------------------------------------------------------------- /.terraform_version: -------------------------------------------------------------------------------- 1 | latest:^0.11 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018, Eiara Limited 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Terraform-Lambda-Zip 2 | 3 | This is a Terraform module implements the necessary moving parts to take a path that contains a Python or Node project and into a `.zip` payload for deployment to AWS Lambda. 4 | 5 | It _requires_ that you have the following packages installed: 6 | 7 | - `jq` 8 | - `pyenv` 9 | - `openssl` 10 | - BSD `md5` or Linux `md5sum` 11 | - `python2.7`, `python3.6` and `python3.7`, selectable via `pyenv` 12 | - `node` and `npm` 13 | - `virtualenv`, in the selected python runtime, installed via `pip` 14 | - `terraform` v0.11.2 or higher. This project _may_ be usable with lower, but it is _untested._ It is currently _untested_ with v0.12.x. 15 | 16 | ## Impetus 17 | 18 | This module exists to make it easier to construct stable, long-lived zipped payloads for AWS Lambda functions, allowing for the Python or NodeJS Lambda functions to live in the same repository as the rest of the infracode. 19 | 20 | 21 | ## How It Works 22 | 23 | This module makes extensive use of `null_resource`s and temporary directories (from `$TMPDIR`) to construct a project directory that is zipped according to the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html). 24 | 25 | Because it uses temporary directories extensively, it _requires_ a user-provided output path to ensure that the written zip is not cleaned up during normal system maintenance. 26 | 27 | For Python, this project directory will include building a private virtualenv and running `python -m compileall`, in order to create the appropriate `.pyc` files. This is done to ensure faster startup time for the Lambda function. 28 | 29 | Installing `node_modules` is handled in a temporary work directory, to avoid cluttering the repository. 30 | 31 | Payload zip files are written in the form of `${var.name}_{epoch}_payload.zip`. This is done to provide a stable indicator of whether or not a file has been deleted, and if it needs to be re-created. 32 | 33 | ## Usage 34 | 35 | ``` 36 | 37 | module "zip_test" { 38 | source = "github.com/Eiara/terraform_lambda_zip" 39 | name = "test" 40 | project_path = "${path.module}/lambda" 41 | output_path = "${path.module}/output_path" 42 | runtime = "python3.6" 43 | dependencies_file = "${path.module}/lambda/requirements.txt" 44 | } 45 | ``` 46 | 47 | - `name`: the name of this zip. used to construct the payload zip filename. 48 | - `project_path`: The path to your Python lambda project, _not_ the `.py` file. 49 | - `output_path`: Where to write the final zip. 50 | - `runtime`: supports nodejs6.10, nodejs8.10, nodejs10.x, python2.7, python3.6, python3.7 51 | - `dependencies_file`: *Optional*. Not providing a dependency file will otherwise work normally. This largely exists to support Python projects that update the requirements file out of band. 52 | 53 | ## Outputs 54 | 55 | - `path`: The path to the final zip. This will be in the form of `${var.name}_{epoch}_payload.zip`, to allow for multiple uses of this module in a project 56 | - `sha256`: a `base64sha256()`-compatible sha256 representing the archive, for use in [source_code_hash](https://www.terraform.io/docs/providers/aws/r/lambda_function.html#source_code_hash). 57 | - `filename`: The filename being exported. 58 | - `md5`: Suitable for using in the S3 bucket object etag. 59 | 60 | ## Useful Knowledge 61 | 62 | This project _does not_ use the Terraform data source `archive_file`, even though it looks like it should. 63 | 64 | The reason it doesn't is that it uses a shasum of the lambda source to build a temporary directory to install packages and compile source. 65 | 66 | When the project sha changes, `archive_file` will error because it can't find the original source directory, which prevents Terraform from running. 67 | 68 | As a result, we use custom per-language archival scripts to mimic the `archive_file` functionality. 69 | 70 | ## License 71 | 72 | This project is Copyright 2018-2019 Eiara Limited, and licensed under the terms of the MIT license. -------------------------------------------------------------------------------- /main.tf: -------------------------------------------------------------------------------- 1 | locals { 2 | engine_lookup = { 3 | "python3.8" = "python" 4 | "python3.7" = "python" 5 | "python3.6" = "python" 6 | "nodejs14.x" = "nodejs" 7 | "nodejs12.x" = "nodejs" 8 | "nodejs10.x" = "nodejs" 9 | "nodejs8.10" = "nodejs" 10 | "nodejs6.10" = "nodejs" 11 | } 12 | 13 | custom = { 14 | install = join(" && ", compact(var.custom_install_commands)) 15 | } 16 | engine = local.engine_lookup[var.runtime] 17 | } 18 | 19 | # Sha the requirements file. This determines whether or not we need to 20 | # rebuild the dependencies (which will trigger whether or not we need to rebuild) 21 | # the payload 22 | 23 | data "external" "dependencies_sha" { 24 | program = ["bash", "${path.module}/scripts/dependencies_sha.sh"] 25 | 26 | query = { 27 | dependencies_file = local.dependencies_file != "" ? local.dependencies_file : "null" 28 | name = var.name 29 | } 30 | # returns 1 result, a sha 31 | } 32 | 33 | # Determines if the project has changed 34 | # If it has, we need to rebuild the project 35 | 36 | data "external" "project_sha" { 37 | program = ["bash", "${path.module}/scripts/${local.engine}/project_sha.sh"] 38 | 39 | query = { 40 | project_path = var.project_path 41 | } 42 | # returns 1 result, a sha 43 | } 44 | 45 | data "external" "payload_exists" { 46 | program = ["python", "${path.module}/scripts/payload_exists.py"] 47 | 48 | query = { 49 | name = var.name 50 | output_path = var.output_path 51 | } 52 | # Returns a stable identifier to determine whether or not 53 | # a payload archive actually exists, to provide a metadata 54 | # codepoint to tell if a user has, in fact, deleted the payload 55 | # file without changing the project or requirements 56 | # returns: identifier 57 | } 58 | 59 | # This will create a new work directory only if the requirements 60 | # has changed 61 | 62 | resource "null_resource" "make_project_work_dir" { 63 | triggers = { 64 | requirements = data.external.dependencies_sha.result["sha"] 65 | # the dependencies has been explicitly deleted already, by the cleanup code 66 | # later on, so if the project has changed we need to rebuild it 67 | project = data.external.project_sha.result["sha"] 68 | payload_exists = data.external.payload_exists.result["identifier"] 69 | } 70 | 71 | provisioner "local-exec" { 72 | command = "${path.module}/scripts/mktmp.sh project ${data.external.project_sha.result["sha"]}" 73 | } 74 | } 75 | 76 | resource "null_resource" "build_payload" { 77 | triggers = { 78 | build_dependencies = null_resource.build_dependencies.id 79 | payload_exists = data.external.payload_exists.result["identifier"] 80 | } 81 | 82 | depends_on = [ 83 | null_resource.make_project_work_dir, 84 | null_resource.build_dependencies, 85 | ] 86 | 87 | provisioner "local-exec" { 88 | # Which runtime we're using 89 | # Where we're building 90 | # our SHA, to tell where our work directory is 91 | # The requirements SHA, so we know where our environment is 92 | command = "${path.module}/scripts/${local.engine}/build_payload.sh" 93 | 94 | environment = { 95 | PAYLOAD_NAME = var.name 96 | PAYLOAD_RUNTIME = var.runtime 97 | PROJECT_PATH = var.project_path 98 | PROJECT_SHA = data.external.project_sha.result["sha"] 99 | WORK_SHA = data.external.project_sha.result["sha"] 100 | OUTPUT_PATH = var.output_path 101 | FILENAME = "${var.name}_${data.external.payload_exists.result["identifier"]}_payload.zip" 102 | } 103 | } 104 | } 105 | 106 | 107 | resource "null_resource" "build_dependencies" { 108 | triggers = { 109 | project_sha = data.external.project_sha.result["sha"] 110 | dependencies_sha = data.external.dependencies_sha.result["sha"] 111 | payload_exists = data.external.payload_exists.result["identifier"] 112 | } 113 | 114 | depends_on = [null_resource.make_project_work_dir] 115 | 116 | provisioner "local-exec" { 117 | command = "${path.module}/scripts/${local.engine}/build_environment.sh" 118 | 119 | environment = { 120 | PROJECT_PATH = var.project_path 121 | RUNTIME = var.runtime 122 | DEPENDENCIES_FILE = local.dependencies_file != "" ? local.dependencies_file : "null" 123 | WORK_SHA = data.external.project_sha.result["sha"] 124 | CUSTOM_COMMANDS = local.custom["install"] 125 | } 126 | } 127 | } 128 | 129 | resource "null_resource" "cleanup_environment_work_directory" { 130 | triggers = { 131 | project = null_resource.build_payload.id 132 | } 133 | 134 | depends_on = [ 135 | null_resource.build_payload, 136 | ] 137 | 138 | provisioner "local-exec" { 139 | command = "${path.module}/scripts/cleanup.sh ${data.external.dependencies_sha.result["sha"]}" 140 | } 141 | } 142 | 143 | resource "null_resource" "cleanup_project_work_directory" { 144 | triggers = { 145 | project = null_resource.build_payload.id 146 | } 147 | 148 | depends_on = [ 149 | null_resource.build_payload, 150 | ] 151 | 152 | provisioner "local-exec" { 153 | command = "${path.module}/scripts/cleanup.sh ${data.external.project_sha.result["sha"]}" 154 | } 155 | } 156 | 157 | data "external" "payload_sha" { 158 | program = ["bash", "${path.module}/scripts/payload_hash.sh"] 159 | 160 | depends_on = [ 161 | null_resource.build_payload, 162 | ] 163 | 164 | query = { 165 | filename = "${var.output_path}/${var.name}_${data.external.payload_exists.result["identifier"]}_payload.zip" 166 | id = null_resource.build_payload.id 167 | } 168 | } 169 | 170 | -------------------------------------------------------------------------------- /output.tf: -------------------------------------------------------------------------------- 1 | output "path" { 2 | value = "${var.output_path}/${var.name}_${data.external.payload_exists.result["identifier"]}_payload.zip" 3 | } 4 | 5 | output "filename" { 6 | value = "${var.name}_${data.external.payload_exists.result["identifier"]}_payload.zip" 7 | } 8 | 9 | output "sha256" { 10 | value = data.external.payload_sha.result["sha"] 11 | } 12 | 13 | output "md5" { 14 | value = data.external.payload_sha.result["md5"] 15 | } 16 | 17 | #output "md5" { 18 | # value = data.archive_file.payload.output_md5 19 | #} 20 | # 21 | #output "sha256" { 22 | # value = data.archive_file.payload.output_base64sha256 23 | #} -------------------------------------------------------------------------------- /scripts/cleanup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | WORK_DIR="${TMPDIR}${1}" 4 | 5 | # TODO 6 | # Improve defensive options here 7 | 8 | if [[ "${1} " = " " ]] ; then 9 | echo "Error: no SHA specified" 10 | exit 1 11 | fi 12 | 13 | if [ "${TMPDIR} " == " " ] || [ "${1}" == "/" ] ; then 14 | echo "ERROR: unsafe TMPDIR or SHA path" 15 | exit 1 16 | fi 17 | 18 | # THIS MAKES ME SO NERVOUS OMG 19 | if [[ -d $WORK_DIR ]] ; then 20 | rm -rf $WORK_DIR 21 | fi -------------------------------------------------------------------------------- /scripts/dependencies_sha.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | eval "$(jq -r '@sh "NAME=\(.name) DEPENDENCIES_FILE=\(.modules_file)"')" 5 | 6 | if [ "$DEPENDENCIES_FILE" != "null" ]; then 7 | current_sha=$(shasum -a 256 "${DEPENDENCIES_FILE}" | cut -d " " -f 1) 8 | 9 | else 10 | current_sha=$(echo "$NAME" | shasum -a 256 - | cut -d " " -f 1) 11 | fi 12 | 13 | jq -n --arg dep_sha "$current_sha" '{"sha":$dep_sha}' -------------------------------------------------------------------------------- /scripts/mktmp.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # $1 is a human-readable name to make error logs more useful 4 | # $2 is the requirements sha 5 | 6 | set -e 7 | 8 | WORK_DIR="${TMPDIR}${2}" 9 | echo "INFO: Attempting to make temporary directory $WORK_DIR" 10 | # TODO: Is this the best way of expressing the "is the variable null"? 11 | if [ "$TMPDIR " == " " ]; then 12 | echo "ERROR: while making $1, found null TMPDIR" 13 | exit 1 14 | fi 15 | 16 | if [ -d $WORK_DIR ]; then 17 | # It already exists 18 | # This means that: we haven't been given a requirements.txt 19 | # (since the sha256) isn't unique 20 | # And 21 | # something else is using the same name for the build 22 | # So we need to error 23 | echo "WARN: While making $1, temporary directory already exists" 24 | echo "WARN: Deleting!" 25 | # TODO: Make this a lot more defensive than it is 26 | rm -rf $WORK_DIR 27 | fi 28 | 29 | mkdir $WORK_DIR 30 | 31 | if ! [ -d $WORK_DIR ]; then 32 | echo "ERROR: $1 work directory could not be created" 33 | exit 1 34 | fi -------------------------------------------------------------------------------- /scripts/nodejs/build_environment.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | # $PROJECT_DIR 6 | # $DEPENDENCIES_SHA 7 | # $CUSTOM_COMMANDS 8 | 9 | WORK_DIR=${TMPDIR}${WORK_SHA} 10 | 11 | echo "INFO: Using work dir $WORK_DIR" 12 | 13 | if ! [ -d $WORK_DIR ]; then 14 | echo "ERROR: Work directory doesn't exist!" 15 | echo "ERROR: $WORK_DIR" 16 | exit 1 17 | fi 18 | 19 | # Copy everything to the work dir 20 | 21 | pushd $PROJECT_PATH 22 | cp -R * $WORK_DIR/ 23 | pushd $WORK_DIR 24 | 25 | # Install from npm 26 | 27 | npm install 28 | 29 | # Allows injection of things like 'npm run transpile' 30 | eval "${CUSTOM_COMMANDS}" 31 | 32 | popd # leave WORK_DIR 33 | popd # leave PROJECT_PATH 34 | -------------------------------------------------------------------------------- /scripts/nodejs/build_payload.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | NAME=$PAYLOAD_NAME 5 | RUNTIME=$PAYLOAD_RUNTIME 6 | # $PROJECT_PATH 7 | # $PROJECT_SHA 8 | # $DEPENDENCIES_SHA 9 | # $FILENAME 10 | # $OUTPUT_PATH 11 | 12 | WORK_DIR=${TMPDIR}${WORK_SHA} 13 | if ! [ -d $WORK_DIR ]; then 14 | echo "ERROR: Work directory doesn't exist!" 15 | echo "ERROR: $WORK_DIR" 16 | exit 1 17 | fi 18 | 19 | NODEJS_VERSIONS="nodejs6.10 nodejs8.10 nodejs10.x nodejs12.x nodejs14.x" 20 | 21 | if ! [[ $NODEJS_VERSIONS =~ (^|[[:space:]])$RUNTIME($|[[:space:]]) ]]; then 22 | echo "ERROR: Invalid nodejs runtime $RUNTIME" 23 | exit 1 24 | fi 25 | 26 | pushd $WORK_DIR 27 | 28 | zip -q -r payload.zip . 29 | 30 | popd # leave WORK_DIR 31 | 32 | mv $WORK_DIR/payload.zip ${OUTPUT_PATH}/${FILENAME} 33 | -------------------------------------------------------------------------------- /scripts/nodejs/project_sha.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | eval "$(jq -r '@sh "PROJECT=\(.project_path)"')" 5 | 6 | if ! [ -d $PROJECT ]; then 7 | >&2 echo "nodejs project $PROJECT does not exist!" 8 | exit 1 9 | fi 10 | 11 | PROJECT_HASH=$(find -s $PROJECT -type f | cpio -o --quiet | shasum -a 256 | cut -d " " -f 1) 12 | 13 | jq -n --arg sha "$PROJECT_HASH" '{"sha": $sha}' -------------------------------------------------------------------------------- /scripts/payload_exists.py: -------------------------------------------------------------------------------- 1 | 2 | import sys 3 | import re 4 | import json 5 | import glob, os 6 | import datetime 7 | import hashlib 8 | 9 | var = json.loads(sys.stdin.read()) 10 | # We expect there to be: 11 | # name 12 | # output path 13 | 14 | # let's check the output path for what we're looking for here 15 | 16 | path = "{0}/{1}".format( var["output_path"], "%s_*_payload.zip" % var["name"] ) 17 | 18 | files = glob.glob(path) 19 | 20 | if not files: 21 | # okay we have no files 22 | # We generate a new indicator and return that. Include the path in the 23 | # hash, so that if we have multiple instances of the module with empty 24 | # directories, we avoid clashes on the identifier by calling the script 25 | # in quick succession. 26 | identifier = int(datetime.datetime.now().strftime("%s")) +\ 27 | (int(hashlib.md5(path.encode()).hexdigest(), 16) % (10 ** 10)) 28 | sys.stdout.write(json.dumps({"identifier": str(identifier)}, indent=2)) 29 | sys.exit(0) 30 | 31 | if files: 32 | # Cool, we have an existing file. Go us! 33 | if len(files) > 1: 34 | print("ERROR: More than one payload file exists for %s" % var["name"]) 35 | sys.exit(1) 36 | r = re.compile("%s_([0-9]+)_payload.zip" % var["name"]) 37 | file_ = os.path.basename(files[0]) 38 | identifier = r.match(file_).groups()[0] 39 | if not identifier: 40 | print("ERROR: Couldn't find identifier in existing file") 41 | sys.exit(1) 42 | sys.stdout.write(json.dumps({"identifier": str(identifier)}, indent=2)) 43 | sys.exit(0) 44 | -------------------------------------------------------------------------------- /scripts/payload_hash.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | eval "$(jq -r '@sh "FILENAME=\(.filename)"')" 6 | 7 | if ! [ -f ${FILENAME} ]; then 8 | jq -n --arg sha "" --arg md5 "" '{"sha":$sha, "md5": $md5}' 9 | # echo "ERROR: No payload zip at ${FILENAME}!" >&2 10 | exit 0 11 | fi 12 | 13 | sha=$(openssl dgst -sha256 -binary ${FILENAME} | openssl enc -base64) 14 | 15 | if (which md5 > /dev/null 2>&1); then 16 | md5=$(md5 -q ${FILENAME}) 17 | else 18 | md5=($(md5sum ${FILENAME})) 19 | fi 20 | 21 | jq -n --arg sha "$sha" --arg md5 "$md5" '{"sha":$sha, "md5": $md5}' 22 | -------------------------------------------------------------------------------- /scripts/python/build_environment.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | PYTHON_RUNTIME=$RUNTIME 6 | REQUIREMENTS_FILE=$DEPENDENCIES_FILE 7 | # REQUIREMENTS_SHA=$DEPENDENCIES_SHA 8 | 9 | WORK_DIR=${TMPDIR}${WORK_SHA} 10 | 11 | echo "INFO: Using work dir $WORK_DIR" 12 | 13 | if ! [ -d $WORK_DIR ]; then 14 | echo "ERROR: Work directory doesn't exist!" 15 | echo "ERROR: $WORK_DIR" 16 | exit 1 17 | fi 18 | 19 | PYTHON_VERSIONS="python2.7 python3.6 python3.7 python3.8" 20 | 21 | if ! [[ $PYTHON_VERSIONS =~ (^|[[:space:]])$PYTHON_RUNTIME($|[[:space:]]) ]]; then 22 | echo "ERROR: Invalid python runtime $PYTHON_RUNTIME" 23 | exit 1 24 | fi 25 | 26 | # We need to grab pyenv 27 | # pyenv generally exists as a shell function, but that's not what we want 28 | # Instead, we want to reach to the explicit, expected install location of 29 | # pyenv, and, instead of initialising it into our subshell: 30 | # - list out the versions 31 | # - Get the shim path 32 | # - Directly use the pyenv binary 33 | # - directly use the pyenv-virtualenv binary 34 | 35 | # This should make it easier to control whether or not we're doing 36 | # something reasonable here. 37 | 38 | # This assumes that the user has the proper path set to the pyenv 39 | # executable, but does not require `eval "$(penv init -)"` to have been 40 | # run. If it has, however, this will still work by ensuring we run the 41 | # executable, not the shell function. 42 | PYENV="command pyenv" 43 | 44 | # eval "$(pyenv init -)" 45 | MAJOR_VERSION=$(echo $PYTHON_RUNTIME | sed 's/python//') 46 | 47 | VERSIONS=$(${PYENV} versions --bare | grep -e "$MAJOR_VERSION" | grep -e "[0-9]\.[0-9]\.[0-9]" | awk 'BEGIN { FS="/"; } {print $1}' | uniq | sort -r ) 48 | 49 | PYENV_ROOT=$(${PYENV} root) 50 | 51 | # Expand the VERSIONS string into a fully-fledged array 52 | # Not sure how else to do this 53 | for version in $VERSIONS; do 54 | VERSION=$version 55 | break 56 | done 57 | 58 | if ! [ -d $VERSION ]; then 59 | echo "ERROR: Python version not found, is it installed in pyenv?" 60 | echo "ERROR: Expected: $MAJOR_VERSION" 61 | exit 1 62 | fi 63 | 64 | echo "INFO: using python version $VERSION" 65 | # Versions should be an array now 66 | 67 | # pyenv shell $VERSION 68 | PYTHON=${PYENV_ROOT}/versions/${VERSION}/bin/python 69 | VIRTUALENV=${PYENV_ROOT}/versions/${VERSION}/bin/virtualenv 70 | 71 | # Okay cool let's build us a virtualenv! 72 | 73 | echo "INFO: Building virtualenv at $WORK_DIR" 74 | ${VIRTUALENV} --always-copy $WORK_DIR > /dev/null 2>&1 75 | 76 | if [ "$REQUIREMENTS_FILE" != "null" ]; then 77 | echo "INFO: Installing from pip" 78 | ${WORK_DIR}/bin/pip install -r ${REQUIREMENTS_FILE} 79 | fi 80 | 81 | # Okay, we're done building the virtualenv 82 | -------------------------------------------------------------------------------- /scripts/python/build_payload.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | NAME=$PAYLOAD_NAME 5 | RUNTIME=$PAYLOAD_RUNTIME 6 | PYTHON_PROJECT=$PROJECT_PATH 7 | # WORK_DIR_SHA=$PROJECT_SHA 8 | VIRTUALENV_SHA=$DEPENDENCIES_SHA 9 | # $WORK_SHA 10 | # OUTPUT_PATH=$6 11 | # FILENAME=$7 12 | 13 | 14 | if ! [ -d $OUTPUT_PATH ]; then 15 | echo "ERROR: Output path is missing" 16 | exit 1 17 | fi 18 | 19 | PYTHON_VERSIONS="python2.7 python3.6 python3.7 python3.8" 20 | 21 | if ! [[ $PYTHON_VERSIONS =~ (^|[[:space:]])$RUNTIME($|[[:space:]]) ]]; then 22 | echo "ERROR: Invalid python runtime $PYTHON_RUNTIME" 23 | exit 1 24 | fi 25 | 26 | VIRTUALENV="${TMPDIR}${WORK_SHA}" 27 | SITE_PACKAGES="${VIRTUALENV}/lib/${RUNTIME}/site-packages/" 28 | BIN="$VIRTUALENV/bin" 29 | WORK_DIR="${TMPDIR}${WORK_SHA}" 30 | 31 | if ! [ -d $SITE_PACKAGES ]; then 32 | echo "ERROR: Site packages missing!" 33 | echo "ERROR: $SITE_PACKAGES" 34 | exit 1 35 | fi 36 | 37 | if ! [ -d $PYTHON_PROJECT ]; then 38 | echo "ERROR: Missing python project directory" 39 | exit 1 40 | fi 41 | 42 | if ! [ -d $WORK_DIR ]; then 43 | echo "ERROR: Work directory doesn't exist!" 44 | echo "ERROR: $WORK_DIR" 45 | exit 1 46 | fi 47 | 48 | # Okay, let's make the virtualenv zip 49 | 50 | echo "INFO: Building virtualenv zip archive" 51 | pushd $SITE_PACKAGES 52 | ${BIN}/python -m compileall . > /dev/null 2>&1 53 | 54 | # Exclude all the default python stuff that's unnecessary in the default context 55 | 56 | zip -r -q virtualenv.zip . -x "pip*" -x "setuptools*" -x "wheel*" -x easy_install.py -x "__pycache__/easy_install*" -x "*.dist-info*" -x "boto3*" -x "botocore*" 57 | 58 | # zip -r -q virtualenv.zip . 59 | 60 | if ! [ -e $SITE_PACKAGES/virtualenv.zip ]; then 61 | # Uh 62 | # Well 63 | # that's bad? 64 | # Something went wrong? 65 | echo "ERROR: Missing virtualenv archive at $SITE_PACKAGES/virtualenv.zip" 66 | exit 1 67 | fi 68 | 69 | # Move back to where we started 70 | popd 71 | 72 | # Cool 73 | # There'll always be a virtualenv, because we need to create a clean build 74 | # point for our python package 75 | # There'll always be a project (why would there not be? That's just weird!) 76 | 77 | # Okay 78 | # Step 1: copy the project from where it is to the workdir 79 | 80 | echo "INFO: current working directory is ${PWD}" 81 | 82 | cp -r $PYTHON_PROJECT ${WORK_DIR} 83 | 84 | BASENAME=$(basename $PYTHON_PROJECT) 85 | 86 | pushd ${WORK_DIR}/${BASENAME} 87 | 88 | # Compile the python package into pycs and such 89 | # This improves startup time for lambda packages, since pycs are only valuable 90 | # for application startup times 91 | ${BIN}/python -m compileall . > /dev/null 2>&1 92 | 93 | cp $SITE_PACKAGES/virtualenv.zip . 94 | 95 | # Build the zipfile, exclude git stuff, and exclude the requirements.txt, if it exists 96 | echo "building payload zip" 97 | zip -q -r virtualenv.zip ./* -x .git -x requirements.txt 98 | 99 | popd 100 | 101 | # Output path is expected to be a fully qualified filename 102 | mv ${WORK_DIR}/${BASENAME}/virtualenv.zip ${OUTPUT_PATH}/${FILENAME} 103 | -------------------------------------------------------------------------------- /scripts/python/project_sha.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | eval "$(jq -r '@sh "PYTHON_PROJECT=\(.project_path)"')" 5 | 6 | if ! [ -d $PYTHON_PROJECT ]; then 7 | echo "ERROR: python project does not exist!" 8 | exit 1 9 | fi 10 | 11 | if (find -s > /dev/null 2>&1); then 12 | >&2 echo "DEBUG: Using find -s to identify python project" 13 | PROJECT_HASH=$(find -s $PYTHON_PROJECT -type f -not -iname requirements.txt | cpio -o --quiet | shasum -a 256 | cut -d " " -f 1) 14 | else 15 | >&2 echo "DEBUG: Using find | sort to identify python project" 16 | PROJECT_HASH=$(find $PYTHON_PROJECT -type f -not -iname requirements.txt | sort | cpio -o --quiet | shasum -a 256 | cut -d " " -f 1) 17 | fi 18 | 19 | jq -n --arg sha "$PROJECT_HASH" '{"sha": $sha}' 20 | -------------------------------------------------------------------------------- /scripts/tmpdir.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | jq -n --arg dir "$TMPDIR" '{"tmpdir":$dir}' -------------------------------------------------------------------------------- /test/nodejs/main.tf: -------------------------------------------------------------------------------- 1 | module "test" { 2 | source = "../../" 3 | 4 | name = "nodejs_test" 5 | project_path = "${path.module}/src" 6 | output_path = "${path.module}/output" 7 | runtime = "nodejs8.10" 8 | dependencies_file = "${path.module}/src/package.json" 9 | } -------------------------------------------------------------------------------- /test/nodejs/output/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eiara/terraform_lambda_zip/c18da0bd512ea978f87eea99b40f59eef303a884/test/nodejs/output/.gitkeep -------------------------------------------------------------------------------- /test/nodejs/src/index.js: -------------------------------------------------------------------------------- 1 | exports.myHandler = function(event, context, callback) { 2 | callback(null, "test success"); 3 | } -------------------------------------------------------------------------------- /test/nodejs/src/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "test", 3 | "version": "1.0.0", 4 | "description": "AWS Lambda Function test function", 5 | "main": "index.js", 6 | "author": "aurynn@eiara.nz", 7 | "license": "MIT", 8 | "dependencies": { 9 | "aws-sdk": "^2.4.6" 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /test/python/main.tf: -------------------------------------------------------------------------------- 1 | module "test" { 2 | source = "../../" 3 | 4 | name = "python_test" 5 | project_path = "${path.module}/src" 6 | output_path = "${path.module}/output" 7 | runtime = "python3.6" 8 | dependencies_file = "${path.module}/src/requirements.txt" 9 | } -------------------------------------------------------------------------------- /test/python/output/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eiara/terraform_lambda_zip/c18da0bd512ea978f87eea99b40f59eef303a884/test/python/output/.gitkeep -------------------------------------------------------------------------------- /test/python/src/main.py: -------------------------------------------------------------------------------- 1 | 2 | def main(event, context): 3 | return "hi" -------------------------------------------------------------------------------- /test/python/src/requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | -------------------------------------------------------------------------------- /tests/nodejs/simple_build/.terraform.lock.hcl: -------------------------------------------------------------------------------- 1 | # This file is maintained automatically by "terraform init". 2 | # Manual edits may be lost in future updates. 3 | 4 | provider "registry.terraform.io/hashicorp/external" { 5 | version = "2.1.0" 6 | hashes = [ 7 | "h1:LTl5CGW8wiIEe16AC4MtXN/95xWWNDbap70zJsBTk0w=", 8 | "zh:0d83ffb72fbd08986378204a7373d8c43b127049096eaf2765bfdd6b00ad9853", 9 | "zh:7577d6edc67b1e8c2cf62fe6501192df1231d74125d90e51d570d586d95269c5", 10 | "zh:9c669ded5d5affa4b2544952c4b6588dfed55260147d24ced02dca3a2829f328", 11 | "zh:a404d46f2831f90633947ab5d57e19dbfe35b3704104ba6ec80bcf50b058acfd", 12 | "zh:ae1caea1c936d459ceadf287bb5c5bd67b5e2a7819df6f5c4114b7305df7f822", 13 | "zh:afb4f805477694a4b9dde86b268d2c0821711c8aab1c6088f5f992228c4c06fb", 14 | "zh:b993b4a1de8a462643e78f4786789e44ce5064b332fee1cb0d6250ed085561b8", 15 | "zh:c84b2c13fa3ea2c0aa7291243006d560ce480a5591294b9001ce3742fc9c5791", 16 | "zh:c8966f69b7eccccb771704fd5335923692eccc9e0e90cb95d14538fe2e92a3b8", 17 | "zh:d5fe68850d449b811e633a300b114d0617df6d450305e8251643b4d143dc855b", 18 | "zh:ddebfd1e674ba336df09b1f27bbaa0e036c25b7a7087dc8081443f6e5954028b", 19 | ] 20 | } 21 | 22 | provider "registry.terraform.io/hashicorp/null" { 23 | version = "3.1.0" 24 | hashes = [ 25 | "h1:xhbHC6in3nQryvTQBWKxebi3inG5OCgHgc4fRxL0ymc=", 26 | "zh:02a1675fd8de126a00460942aaae242e65ca3380b5bb192e8773ef3da9073fd2", 27 | "zh:53e30545ff8926a8e30ad30648991ca8b93b6fa496272cd23b26763c8ee84515", 28 | "zh:5f9200bf708913621d0f6514179d89700e9aa3097c77dac730e8ba6e5901d521", 29 | "zh:9ebf4d9704faba06b3ec7242c773c0fbfe12d62db7d00356d4f55385fc69bfb2", 30 | "zh:a6576c81adc70326e4e1c999c04ad9ca37113a6e925aefab4765e5a5198efa7e", 31 | "zh:a8a42d13346347aff6c63a37cda9b2c6aa5cc384a55b2fe6d6adfa390e609c53", 32 | "zh:c797744d08a5307d50210e0454f91ca4d1c7621c68740441cf4579390452321d", 33 | "zh:cecb6a304046df34c11229f20a80b24b1603960b794d68361a67c5efe58e62b8", 34 | "zh:e1371aa1e502000d9974cfaff5be4cfa02f47b17400005a16f14d2ef30dc2a70", 35 | "zh:fc39cc1fe71234a0b0369d5c5c7f876c71b956d23d7d6f518289737a001ba69b", 36 | "zh:fea4227271ebf7d9e2b61b89ce2328c7262acd9fd190e1fd6d15a591abfa848e", 37 | ] 38 | } 39 | -------------------------------------------------------------------------------- /tests/nodejs/simple_build/lambda/main.js: -------------------------------------------------------------------------------- 1 | console.log('Hello World'); 2 | -------------------------------------------------------------------------------- /tests/nodejs/simple_build/main.tf: -------------------------------------------------------------------------------- 1 | module "zip_test" { 2 | source = "../../../" 3 | name = "simple_test" 4 | project_path = "${path.module}/lambda" 5 | output_path = "${path.module}/output" 6 | runtime = "nodejs10.x" 7 | } 8 | -------------------------------------------------------------------------------- /tests/nodejs/simple_build/output/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eiara/terraform_lambda_zip/c18da0bd512ea978f87eea99b40f59eef303a884/tests/nodejs/simple_build/output/.gitkeep -------------------------------------------------------------------------------- /tests/python/complex_build/lambda/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import requests 3 | 4 | print("Complex test") 5 | sys.exit(1) -------------------------------------------------------------------------------- /tests/python/complex_build/lambda/requirements.txt: -------------------------------------------------------------------------------- 1 | certifi==2019.3.9 2 | chardet==3.0.4 3 | idna==2.8 4 | requests==2.21.0 5 | urllib3==1.26.5 6 | virtualenv==16.5.0 7 | -------------------------------------------------------------------------------- /tests/python/complex_build/main.tf: -------------------------------------------------------------------------------- 1 | module "zip_test" { 2 | source = "../../../" 3 | name = "test" 4 | project_path = "${path.module}/lambda" 5 | output_path = "${path.module}/output" 6 | runtime = "python3.6" 7 | dependencies_file = "${path.module}/lambda/requirements.txt" 8 | } 9 | -------------------------------------------------------------------------------- /tests/python/complex_build/output/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eiara/terraform_lambda_zip/c18da0bd512ea978f87eea99b40f59eef303a884/tests/python/complex_build/output/.gitkeep -------------------------------------------------------------------------------- /tests/python/fail_build/lambda/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | print("Complex test") 4 | sys.exit(1) -------------------------------------------------------------------------------- /tests/python/fail_build/main.tf: -------------------------------------------------------------------------------- 1 | module "zip_test" { 2 | source = "../../../" 3 | name = "test" 4 | project_path = "${path.module}/lambda" 5 | output_path = "${path.module}/output" 6 | runtime = "python2.6" 7 | } 8 | -------------------------------------------------------------------------------- /tests/python/fail_build/output/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eiara/terraform_lambda_zip/c18da0bd512ea978f87eea99b40f59eef303a884/tests/python/fail_build/output/.gitkeep -------------------------------------------------------------------------------- /tests/python/simple_build/lambda/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | print("Simple test") 4 | sys.exit(1) -------------------------------------------------------------------------------- /tests/python/simple_build/main.tf: -------------------------------------------------------------------------------- 1 | module "zip_test" { 2 | source = "../../../" 3 | name = "simple_test" 4 | project_path = "${path.module}/lambda" 5 | output_path = "${path.module}/output" 6 | runtime = "python3.6" 7 | } 8 | -------------------------------------------------------------------------------- /tests/python/simple_build/output/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eiara/terraform_lambda_zip/c18da0bd512ea978f87eea99b40f59eef303a884/tests/python/simple_build/output/.gitkeep -------------------------------------------------------------------------------- /tests/python/simple_build/output/lambda/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | print("Simple test") 4 | sys.exit(1) -------------------------------------------------------------------------------- /tests/python/simple_build/output/main.tf: -------------------------------------------------------------------------------- 1 | module "zip_test" { 2 | source = "../../" 3 | name = "simple_test" 4 | project_path = "${path.module}/lambda" 5 | output_path = "${path.module}/output" 6 | runtime = "python3.6" 7 | } -------------------------------------------------------------------------------- /variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" { 2 | description = "Name of the function payload" 3 | } 4 | 5 | variable "project_path" { 6 | description = "path to the function" 7 | } 8 | 9 | variable "output_path" { 10 | description = "where to write the payload zip" 11 | } 12 | 13 | # Optional settings 14 | 15 | variable "runtime" { 16 | description = "What runtime. Currently supported: python, nodejs" 17 | } 18 | 19 | variable "dependencies_file" { 20 | default = "" 21 | description = "the path to the dependencies file. Can be empty." 22 | } 23 | 24 | variable "requirements_file" { 25 | default = "" 26 | description = "DEPRECATED: use dependencies_file" 27 | } 28 | 29 | variable "custom_install_commands" { 30 | type = list(string) 31 | description = "" 32 | default = [ 33 | "", 34 | ] 35 | } 36 | 37 | locals { 38 | dependencies_file = var.requirements_file != "" ? var.requirements_file : var.dependencies_file != "" ? var.dependencies_file : "" 39 | } 40 | 41 | -------------------------------------------------------------------------------- /versions.tf: -------------------------------------------------------------------------------- 1 | 2 | terraform { 3 | required_version = ">= 0.12" 4 | } 5 | --------------------------------------------------------------------------------