├── README.md └── sample-function ├── main.js ├── node_modules └── package.json /README.md: -------------------------------------------------------------------------------- 1 | # aws-esm-modules-layer-support 2 | 3 | ### TLDR: Symlink your layer into your deployment package, and include the symlink (NOT the symlinked directory) into your artifact. 4 | 5 | *nix example: 6 | ```bash 7 | cd directory-with-function-code 8 | ln -s /opt/nodejs/node_modules node_modules 9 | zip --symlinks -r function.zip . 10 | ``` 11 | 12 | Nodejs Example: 13 | ```javascript 14 | symlinkSync('/opt/nodejs/node_modules', 'node_modules', 'dir') 15 | process.chdir(cwd) 16 | 17 | spawnSync('zip', [ 18 | '--symlinks', '-r', `${artifactDirectory}/function.zip`, `.` 19 | ],{ 20 | cwd: functionPath, 21 | encoding: 'utf-8' 22 | }) 23 | ``` 24 | 25 | 26 | ## Background 27 | 28 | 29 | In early 2022, AWS released ES Module support for the the Node.js 14.x Lambda Runtime. 30 | 31 | To enable the ES Module support you simply have to include a `package.json` in your deployment with `type` set to `module` or simply use the `.mjs` extension. 32 | 33 | [Using Node.js ES modules and top-level await in AWS Lambda](https://aws.amazon.com/blogs/compute/using-node-js-es-modules-and-top-level-await-in-aws-lambda/) 34 | 35 | ## Problem 36 | 37 | Surprisingly, ES Module support was released without "support" for `AWS Layers`, which seems like slight oversight. 38 | 39 | This ultimately boils down to the fact, the [module resolution algorithm](https://nodejs.org/api/esm.html#resolution-algorithm) for ES Modules does not rely on `node_path`, which results in ES Modules failing to resolve modules from `node_modules`. 40 | 41 | A couple of goto suggestions that immediately come to mind... 42 | 43 | - Using a bundler 44 | - Include node_modules directly in the deployment package 45 | 46 | Both of these alternatives resolve around NOT using layers, however that introduces the limitions layers are used for. 47 | - Lack of sharability 48 | - Deployment package size limition 49 | - Console "file is too big to edit" errors 50 | - etc 51 | 52 | ## Solution 53 | 54 | Here's the thing, it's node.js all the way down. After I decompiled the [AWS Lambda Runtime](https://hub.docker.com/r/amazon/aws-lambda-nodejs) and reading the AWS specific code that bootstraps the environment, it's clear that all that needs to happen is for AWS Layer to provide additional [Layer Paths](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) for the node.js environment. 55 | 56 | Currently the bootstrap scripts that run before your lambda handler add the supported Layer paths into the `node_path` but instead what we need is the ability for `node_modules` to be mounted within the direct hireachy of the function code, since the module resolution alogrithm will look up `node_modules` starting at the function directory and work it's way up until it reaches the server root `/`. 57 | 58 | Working around the current limition is as simple as symlinking the layer path into your function directory. 59 | 60 | This is accomplished at your build/deployment step when generating your zip artifact that is uploaded to AWS. 61 | 62 | 1. Bundle your `node_modules` into a layer as normal 63 | 2. Create a symlink in your source code directory that points to whichever runtime Layer Path you are using ( `/opt/nodejs/node_modules` or `/opt/nodejs/node14/node_modules`) 64 | 3. Zip up your source code and include the symlink 65 | 4. Distribute your ZIP as normal 66 | 67 | During runtime, the symlink will essentially act as a proxy to your layer. 68 | 69 | Tada! ezpz. 70 | 71 | I use `cdk` in my projects, so here's an extracted snippet from my construct that creates AWS Lambda resources. 72 | 73 | ```javascript 74 | const dir = dirname(require.resolve('@whoami/sample-function')) 75 | const packageJson = join(dir, 'package.json') 76 | 77 | const directory = new Directory(this, 'directory', { 78 | baseDir: dir 79 | }) 80 | 81 | const pkg = JSON.parse(readFileSync(packageJson, { encoding: 'utf-8' })) 82 | const packageName = pkg.name.replace('@', '').replace('/', '_') 83 | const artifactName = `${packageName}-${directory.digests.md5}` 84 | const artifactDirectory = resolve(`cdktf.out/artifacts/${artifactName}`) 85 | 86 | const packageJsonExists = existsSync(packageJson) 87 | const tmp = mkdtempSync(join(tmpdir(), packageName)) 88 | 89 | if (packageJsonExists) { 90 | const dependencyPath = join(tmp, 'nodejs') 91 | 92 | mkdirSync(dependencyPath) 93 | copyFileSync(packageJson, join(dependencyPath, 'package.json')) 94 | 95 | 96 | spawnSync(process.platform === 'win32' ? 'npm.cmd' : 'npm', ['install', '--prod'], { 97 | cwd: dependencyPath 98 | }) 99 | 100 | 101 | const dependencyArtifact = new AdmZip() 102 | dependencyArtifact.addLocalFolder(dependencyPath, 'nodejs') 103 | dependencyArtifact.writeZip(join(artifactDirectory, 'layer.zip')) 104 | 105 | } 106 | 107 | const functionPath = join(tmp, 'function') 108 | 109 | buildSync({ 110 | entryPoints: [ 111 | config.code 112 | ], 113 | bundle: true, 114 | external: packageJson ? Object.keys(pkg.dependencies) : [], 115 | outdir: functionPath, 116 | target: ['es2022'], 117 | format: 'esm', 118 | platform: 'node' 119 | }) 120 | 121 | let cwd = process.cwd() 122 | process.chdir(functionPath) 123 | 124 | symlinkSync('/opt/nodejs/node_modules', 'node_modules', 'dir') 125 | process.chdir(cwd) 126 | 127 | spawnSync('zip', [ 128 | '--symlinks', '-r', `${artifactDirectory}/function.zip`, `.` 129 | ],{ 130 | cwd: functionPath, 131 | encoding: 'utf-8' 132 | } ) 133 | 134 | process.chdir(cwd) 135 | 136 | 137 | rmSync(tmp, { recursive: true }) 138 | 139 | ``` 140 | 141 | 142 | ## Other known workarounds. 143 | 144 | 145 | [Markus Tacker](https://twitter.com/coderbyheart/status/1487218393241563140) has an neat workaround which involves using dynamic async imports to load the modules from the layer. 146 | 147 | You find his example solution here.[AWS Lambda ESM with Layer](https://github.com/coderbyheart/aws-lambda-esm-with-layer) 148 | 149 | The downside to Tacker's solution is that you must include this boiler plate directly in every source file which can become a hassle. 150 | 151 | If you practice Infrastructure as Code, it's very easy to symlink the layer without each function having to be explicitly aware of the workaround. 152 | 153 | 154 | 155 | 156 | -------------------------------------------------------------------------------- /sample-function/main.js: -------------------------------------------------------------------------------- 1 | // ../../app/functions/sample-function/main.ts 2 | import { DynamoDBClient } from "@aws-sdk/client-dynamodb"; 3 | 4 | // ../../app/functions/sample-function/utils/test.ts 5 | function test() { 6 | console.log("hello test"); 7 | } 8 | 9 | // ../../app/functions/sample-function/main.ts 10 | function handler(event) { 11 | const dynamodb = new DynamoDBClient({}); 12 | test(); 13 | return { 14 | status: 200, 15 | body: JSON.stringify({ 16 | message: "Hello world" 17 | }) 18 | }; 19 | } 20 | export { 21 | handler 22 | }; -------------------------------------------------------------------------------- /sample-function/node_modules: -------------------------------------------------------------------------------- 1 | /opt/nodejs/node_modules -------------------------------------------------------------------------------- /sample-function/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "@whoami/sample-function", 3 | "version": "1.0.0", 4 | "description": "", 5 | "main": "main.js", 6 | "type": "module", 7 | "scripts": { 8 | "test": "echo \"Error: no test specified\" && exit 1" 9 | }, 10 | "author": "", 11 | "license": "ISC", 12 | "dependencies": { 13 | "@aws-sdk/client-dynamodb": "^3.54.0" 14 | } 15 | } 16 | --------------------------------------------------------------------------------