├── .gitignore ├── README.md ├── index.js ├── package-lock.json └── package.json /.gitignore: -------------------------------------------------------------------------------- 1 | # package directories 2 | node_modules 3 | 4 | .DS_Store -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # serverless-cloudside-plugin 2 | 3 | [Serverless](http://www.serverless.com) plugin for using _cloudside_ resources when developing functions locally. 4 | 5 | [![Serverless](http://public.serverless.com/badges/v3.svg)](http://www.serverless.com) 6 | [![npm](https://img.shields.io/npm/v/serverless-cloudside-plugin.svg)](https://www.npmjs.com/package/serverless-cloudside-plugin) 7 | [![npm](https://img.shields.io/npm/l/serverless-cloudside-plugin.svg)](https://www.npmjs.com/package/serverless-cloudside-plugin) 8 | 9 | This plugin allows you to use AWS CloudFormation intrinsic functions (such as `!Ref` and `!GetAtt`) to reference cloud resources during local development. When added to your `environment` variables, these values are replaced with the same identifiers used when deployed to the cloud. You can invoke your functions locally, use the `serverless-offline` plugin, or use a compatible test runner that uses the `serverless invoke test` command. You can now keep your `serverless.yml` files free from pseudo variables and other concatenated strings and simply use the built-in CloudFormation features. 10 | 11 | ## Installation 12 | 13 | #### Install using Serverless plugin manager 14 | 15 | ```bash 16 | serverless plugin install --name serverless-cloudside-plugin 17 | ``` 18 | 19 | #### Install using npm 20 | 21 | Install the module using npm: 22 | 23 | ```bash 24 | npm install serverless-cloudside-plugin --save-dev 25 | ``` 26 | 27 | Add `serverless-cloudside-plugin` to the plugin list of your `serverless.yml` file: 28 | 29 | ```yaml 30 | plugins: 31 | - serverless-cloudside-plugin 32 | ``` 33 | 34 | ## Usage 35 | 36 | When executing your function locally, the `serverless-cloudside-plugin` will replace any environment variable that contains either a `!Ref` or a `!GetAtt` that references a CloudFormation resource within your `serverless.yml` file. 37 | 38 | In the example below, we are creating an SQS Queue named `myQueue` and referencing it (using a CloudFormation intrinsic function) in an environment variable named `QUEUE`. 39 | 40 | ```yaml 41 | functions: 42 | myFunction: 43 | handler: myFunction.handler 44 | environment: 45 | QUEUE: !Ref myQueue 46 | 47 | resources: 48 | Resources: 49 | myQueue: 50 | Type: AWS::SQS::Queue 51 | Properties: 52 | QueueName: ${self:service}-${self:provider.stage}-myQueue 53 | ``` 54 | 55 | If we deploy this to the cloud, our `!Ref myQueue` will be replaced with a `QueueUrl` (e.g. _https://sqs.us-east-1.amazonaws.com/1234567890/sample-service-dev-myQueue_). We can then use that when invoking the AWS SDK and working with our queue. 56 | However, if we were to invoke this function locally using `sls invoke local -f myFunction`, our `QUEUE` environment variable would return `[object Object]` instead of our `QueueUrl`. This is because the Serverless Framework is actually replacing our `!Ref` with: `{ "Ref": "myQueue" }`. 57 | 58 | There are workarounds to this, typically involving using pseudo variables to construct our own URL. But this method is error prone and requires us to hardcode formats for the different service types. Using the `serverless-cloudside-plugin`, you can now use the simple reference format above, and always retrieve the correct `PhysicalResourceId` for the resource. 59 | 60 | #### Invoking a function locally 61 | 62 | Once the plugin is installed, you will have a new `invoke` option named `invoke cloudside`. Simply run this command with a function and it will resolve all of your cloud variables and then execute the standard `invoke local` command. 63 | 64 | ```bash 65 | sls invoke cloudside -f myFunction 66 | ``` 67 | 68 | **PLEASE NOTE** that in order for resources to be referenced, you must deploy your service to the cloud at least initially. References to non-deployed resources will be populated with **"RESOURCE NOT DEPLOYED"**. 69 | 70 | All `invoke local` parameters are supported such as `--stage` and `--path`, as well as the new `--docker` flag that lets you run your function locally in a Docker container. This mimics the Lambda environment much more closely than your local machine. 71 | 72 | By default, the plugin will reference resources from your current CloudFormation stack (including your "stage" if it is part of your stack name). You can change the cloudside stage by using the `--cloudStage` option and supplying the stage name that you'd like to use. For example, if you are developing in your `dev` stage locally, but want to use a DynamoDB table that is deployed to the `test` stage, you can do the following: 73 | 74 | ```bash 75 | sls invoke cloudside -f myFunction -s dev --cloudStage test 76 | ``` 77 | 78 | This will populate any `${opt:stage}` references with `dev`, but your `!Ref` values will use the ones from your `test` stage. 79 | 80 | You might also want to pull values from an entirely different CloudFormation stack. You can do this by using the `--stackName` option and supplying the complete stack name. For example: 81 | 82 | ```bash 83 | sls invoke cloudside -f myFunction --stackName someOtherStack-dev 84 | ``` 85 | 86 | #### Using with the serverless-offline plugin 87 | 88 | The `serverless-offline` plugin is a great tool for testing your serverless APIs locally, but it has the same problem referencing CloudFormation resources. The `serverless-cloudside-plugin` lets you run `serverless-offline` with all of your cloud variables correctly replaced. 89 | 90 | ```bash 91 | sls offline cloudside 92 | ``` 93 | 94 | To enable hot-reload when running the server, use the `--reloadHandler` flag: 95 | 96 | ```bash 97 | sls offline cloudside --reloadHandler 98 | ``` 99 | 100 | The above command will start the API Gateway emulator and allow you to test your functions locally. The `--cloudStage` and `--stackName` options are supported as well as all of the `serverless-offline` options. 101 | 102 | #### Using with a test runner 103 | 104 | You can use this plugin with other test runner plugins such as `serverless-mocha-plugin`. This will make it easier to run integration tests (including in your CI/CD systems) before deploying. Simply run the following when invoking your tests: 105 | 106 | ```bash 107 | sls invoke test cloudside -f myFunction 108 | ``` 109 | 110 | This plugin extends the `invoke test` command, so any test runner plugin that uses that format should work correctly. All plugin options should remain available. 111 | 112 | ## Available Functions 113 | 114 | This plugin currently supports the `!Ref` function that returns the `PhysicalResourceId` from CloudFormation. For most resources, this is the value you will need to interact with the corresponding service in the AWS SDK (e.g. `QueueUrl` for SQS, `TopicArn` for SNS, etc.). 115 | 116 | There is also initial (and limited) support for using `!GetAtt` to retrieve an **ARN**. For example, you may use `!GetAtt myQueue.Arn` to retrieve the ARN for `myQueue`. The plugin generates the ARN based on the service type. For supported types, it will return a properly formatted ARN. For others, it will replace the value with **"FUNCTION NOT SUPPORTED"**. In most cases, it should be possible to support generating an ARN for a resource, but the format will need to be added to the plugin. 117 | 118 | ## Contributions 119 | 120 | Contributions, ideas and bug reports are welcome and greatly appreciated. Please add [issues](https://github.com/jeremydaly/serverless-cloudside-plugin/issues) for suggestions and bug reports or create a pull request. 121 | -------------------------------------------------------------------------------- /index.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | const BbPromise = require('bluebird') 4 | 5 | class InvokeCloudside { 6 | constructor(serverless, options) { 7 | this.serverless = serverless 8 | this.options = options 9 | 10 | this.commands = { 11 | invoke: { 12 | commands: { 13 | cloudside: { 14 | usage: 'Invoke function locally using cloudside resources', 15 | lifecycleEvents: [ 16 | 'loadCloudsideEnvVars', 17 | 'invoke', 18 | ], 19 | options: { 20 | function: { 21 | usage: 'Name of the function', 22 | shortcut: 'f', 23 | required: true, 24 | type: 'string', 25 | }, 26 | cloudStage: { 27 | usage: 'Stage to use for cloudside resources', 28 | shortcut: 'S', 29 | type: 'string', 30 | }, 31 | stackName: { 32 | usage: 'CloudFormation stack to use for cloudside resources', 33 | shortcut: 'y', 34 | type: 'string', 35 | }, 36 | path: { 37 | usage: 'Path to JSON or YAML file holding input data', 38 | shortcut: 'p', 39 | type: 'string', 40 | }, 41 | data: { 42 | usage: 'input data', 43 | shortcut: 'd', 44 | type: 'string', 45 | }, 46 | raw: { 47 | usage: 'Flag to pass input data as a raw string', 48 | type: 'string', 49 | }, 50 | context: { 51 | usage: 'Context of the service', 52 | type: 'string', 53 | shortcut: 'c', 54 | }, 55 | contextPath: { 56 | usage: 'Path to JSON or YAML file holding context data', 57 | type: 'string', 58 | shortcut: 'x', 59 | }, 60 | env: { 61 | usage: 'Override environment variables. e.g. --env VAR1=val1 --env VAR2=val2', 62 | type: 'string', 63 | shortcut: 'e', 64 | }, 65 | docker: { 66 | usage: 'Flag to turn on docker use for node/python/ruby/java', 67 | type: 'string' 68 | }, 69 | 'docker-arg': { 70 | usage: 'Arguments to docker run command. e.g. --docker-arg "-p 9229:9229"', 71 | type: 'string', 72 | }, 73 | }, 74 | 75 | }, 76 | test: { 77 | usage: 'NOT INSTALLED - A testing plugin must be installed to use "invoke test cloudside"', 78 | commands: { 79 | cloudside: { 80 | usage: 'Invoke test(s) using cloudside resources', 81 | lifecycleEvents: [ 82 | 'loadCloudsideEnvVars', 83 | 'start' 84 | ], 85 | options: { 86 | cloudStage: { 87 | usage: 'Stage to use for cloudside resources', 88 | shortcut: 'S', 89 | type: 'string', 90 | }, 91 | stackName: { 92 | usage: 'CloudFormation stack to use for cloudside resources', 93 | shortcut: 'y', 94 | type: 'string', 95 | } 96 | } 97 | } 98 | } 99 | } 100 | }, 101 | }, 102 | offline: { 103 | lifecycleEvents: [ 104 | 'checkInstall' 105 | ], 106 | usage: 'NOT INSTALLED - This plugin must be installed to use "offline cloudside"', 107 | commands: { 108 | cloudside: { 109 | usage: 'Simulates API Gateway to call your lambda functions offline using cloudside resources', 110 | lifecycleEvents: [ 111 | 'checkInstall', 112 | 'loadCloudsideEnvVars', 113 | 'start' 114 | ], 115 | options: { 116 | cloudStage: { 117 | usage: 'Stage to use for cloudside resources', 118 | shortcut: 'S', 119 | type: 'string', 120 | }, 121 | reloadHandler: { 122 | usage: 'Enable serverless reload functionality', 123 | shortcut: 'r', 124 | type: 'boolean', 125 | }, 126 | stackName: { 127 | usage: 'CloudFormation stack to use for cloudside resources', 128 | shortcut: 'y', 129 | type: 'string', 130 | } 131 | } 132 | } 133 | } 134 | } 135 | }; 136 | 137 | this.hooks = { 138 | 'invoke:cloudside:loadCloudsideEnvVars': () => BbPromise.bind(this).then(this.loadCloudsideEnvVars), 139 | 'invoke:test:cloudside:loadCloudsideEnvVars': () => BbPromise.bind(this).then(this.loadCloudsideEnvVars), 140 | 'offline:cloudside:loadCloudsideEnvVars': () => BbPromise.bind(this).then(this.loadCloudsideEnvVars), 141 | 'after:invoke:cloudside:invoke': () => this.serverless.pluginManager.run(['invoke', 'local']), 142 | 'after:offline:cloudside:start': () => this.serverless.pluginManager.run(['offline', 'start']), 143 | 'offline:checkInstall': () => BbPromise.bind(this).then(this.checkInstall('serverless-offline')), 144 | 'offline:cloudside:checkInstall': () => BbPromise.bind(this).then(this.checkInstall('serverless-offline')), 145 | 'after:invoke:test:cloudside:start': () => this.serverless.pluginManager.run(['invoke', 'test']), 146 | }; 147 | } 148 | 149 | checkInstall(plugin) { 150 | if (!this.serverless.service.plugins.includes(plugin)) { 151 | throw Error(`You must install "${plugin}" to use the "offline cloudside" feature.\n\n You can install it by running "serverless plugin install --name ${plugin}"`) 152 | } 153 | return BbPromise.resolve() 154 | } 155 | 156 | getStackResources(result, params, options) { 157 | return this.serverless.getProvider('aws') 158 | .request('CloudFormation', 159 | 'listStackResources', 160 | params, 161 | options) 162 | .then(res => { 163 | result = [ ...result, ...res.StackResourceSummaries ] 164 | if(res.NextToken) { 165 | return this.getStackResources(result, { ...params, NextToken: res.NextToken }, options); 166 | } 167 | return result; 168 | }).catch(e => { 169 | console.log(e) 170 | }) 171 | }; 172 | 173 | // Set environment variables for "invoke cloudside" 174 | loadCloudsideEnvVars() { 175 | 176 | // Get the stage (use the cloudstage option first, then stage option, then provider) 177 | const stage = this.options.cloudStage ? this.options.cloudStage : 178 | this.options.stage ? this.options.stage : 179 | this.serverless.service.provider.stage 180 | 181 | // Get the stack name (use provided stackName or fallback to service name) 182 | const stackName = this.options.stackName ? this.options.stackName : 183 | this.serverless.service.provider.stackName ? 184 | this.serverless.service.provider.stackName : 185 | `${this.serverless.service.service}-${stage}` 186 | 187 | this.serverless.cli.log(`Loading cloudside resources for '${stackName}' stack.`) 188 | 189 | if (!this.serverless.service.provider.environment) { 190 | this.serverless.service.provider.environment = {} 191 | } 192 | 193 | this.serverless.service.provider.environment.IS_CLOUDSIDE = true 194 | this.serverless.service.provider.environment.CLOUDSIDE_STACK = stackName 195 | 196 | // Find all envs with CloudFormation Refs 197 | let cloudsideVars = parseEnvs(this.serverless.service.provider.environment) 198 | 199 | let functions = this.options.function ? 200 | { [this.options.function] : this.serverless.service.functions[this.options.function] } 201 | : this.serverless.service.functions 202 | 203 | Object.keys(functions).map(fn => { 204 | if (this.serverless.service.functions[fn].environment) { 205 | let vars = parseEnvs(this.serverless.service.functions[fn].environment,fn) 206 | for (let key in vars) { 207 | cloudsideVars[key] = cloudsideVars[key] ? cloudsideVars[key].concat(vars[key]) : vars[key] 208 | } 209 | } 210 | }) 211 | 212 | // If references need resolving, call CF 213 | if (Object.keys(cloudsideVars).length > 0) { 214 | 215 | const options = { useCache: true } 216 | 217 | const hander = (stackResources) => { 218 | if (stackResources) { 219 | // Loop through the returned StackResources 220 | for (let i = 0; i < stackResources.length; i++) { 221 | 222 | let resource = cloudsideVars[stackResources[i].LogicalResourceId] 223 | 224 | // If the logicial id exists, add the PhysicalResourceId to the ENV 225 | if (resource) { 226 | for (let j = 0; j < resource.length; j++) { 227 | 228 | let value = resource[j].type == 'Ref' ? stackResources[i].PhysicalResourceId 229 | : buildCloudValue(stackResources[i],resource[j].type) 230 | 231 | if (resource[j].fn) { 232 | this.serverless.service.functions[resource[j].fn].environment[ 233 | resource[j].env 234 | ] = value 235 | } else { 236 | this.serverless.service.provider.environment[ 237 | resource[j].env 238 | ] = value 239 | } 240 | 241 | } // end for 242 | // Remove the cloudside variable 243 | delete(cloudsideVars[stackResources[i].LogicalResourceId]) 244 | } // end if 245 | } // end for 246 | } // end if StackResources 247 | 248 | // Replace remaining variables with warning 249 | Object.keys(cloudsideVars).map(x => { 250 | for (let j = 0; j < cloudsideVars[x].length; j++) { 251 | if (cloudsideVars[x][j].fn) { 252 | this.serverless.service.functions[cloudsideVars[x][j].fn].environment[ 253 | cloudsideVars[x][j].env 254 | ] = '' 255 | } else { 256 | this.serverless.service.provider.environment[ 257 | cloudsideVars[x][j].env 258 | ] = '' 259 | } 260 | } 261 | }) 262 | 263 | return true 264 | } 265 | 266 | return this.getStackResources([], { StackName: stackName }, options).then(hander); 267 | } else { 268 | return BbPromise.resolve() 269 | } 270 | } 271 | } 272 | 273 | // Parse the environment variables and return formatted mappings 274 | const parseEnvs = (envs = {},fn) => Object.keys(envs).reduce((vars,key) => { 275 | let logicalId,ref 276 | 277 | if (envs[key].Ref) { 278 | logicalId = envs[key].Ref 279 | ref = { type: 'Ref', env: key, fn } 280 | } else if (envs[key]['Fn::GetAtt']) { 281 | logicalId = envs[key]['Fn::GetAtt'][0] 282 | ref = { type: envs[key]['Fn::GetAtt'][1], env: key, fn } 283 | } else { 284 | return vars 285 | } 286 | 287 | vars[logicalId] = vars[logicalId] ? 288 | vars[logicalId].concat([ref]) : [ref] 289 | 290 | return vars 291 | },{}) 292 | 293 | 294 | 295 | // Build the cloud value based on type 296 | const buildCloudValue = (resource,type) => { 297 | switch(type) { 298 | case 'Arn': 299 | return generateArn(resource) 300 | default: 301 | return '' 302 | } 303 | } 304 | 305 | const getRdsResourceType = resourceType => { 306 | switch(resourceType) { 307 | case 'dbcluster': 308 | return 'cluster' 309 | case 'dbinstance': 310 | return 'db' 311 | default: 312 | return null 313 | } 314 | } 315 | 316 | // Generate the ARN based on service type 317 | // TODO: add more service types or figure out a better way to do this 318 | const generateArn = resource => { 319 | 320 | let stack = resource.StackId.split(':').slice(0,5) 321 | let resourceType = resource.ResourceType.split('::') 322 | let serviceType = resourceType[1].toLowerCase() 323 | stack[2] = serviceType 324 | 325 | switch(serviceType) { 326 | case 'sqs': 327 | stack.push(resource.PhysicalResourceId.split('/').slice(-1)) 328 | break 329 | case 'dynamodb': 330 | case 'kinesis': 331 | stack.push(resourceType[2].toLowerCase()+'/'+resource.PhysicalResourceId) 332 | break 333 | case 'rds': 334 | const rdsResourceType = getRdsResourceType(resourceType[2].toLowerCase()) 335 | if (!rdsResourceType) { 336 | return '' 337 | } 338 | stack.push(rdsResourceType) 339 | stack.push(resource.PhysicalResourceId) 340 | break 341 | case 'secretsmanager': 342 | stack = resource.PhysicalResourceId.split('-').slice(0, -1).join('-').split(':') 343 | break 344 | case 's3': 345 | stack.splice(3,5,'','') 346 | stack.push(resource.PhysicalResourceId) 347 | break 348 | default: 349 | return '' 350 | } 351 | 352 | return stack.join(':') 353 | } 354 | 355 | module.exports = InvokeCloudside 356 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "serverless-cloudside-plugin", 3 | "version": "1.1.0", 4 | "description": "Serverless plugin for using cloudside resources during local development", 5 | "main": "index.js", 6 | "scripts": { 7 | "test": "echo \"Error: no test specified\" && exit 1" 8 | }, 9 | "repository": { 10 | "type": "git", 11 | "url": "git+https://github.com/jeremydaly/serverless-cloudside-plugin.git" 12 | }, 13 | "keywords": [ 14 | "serverless", 15 | "serverless-plugin", 16 | "aws", 17 | "nodejs", 18 | "local development", 19 | "offline" 20 | ], 21 | "author": "Jeremy Daly ", 22 | "license": "MIT", 23 | "bugs": { 24 | "url": "https://github.com/jeremydaly/serverless-cloudside-plugin/issues" 25 | }, 26 | "homepage": "https://github.com/jeremydaly/serverless-cloudside-plugin#readme", 27 | "dependencies": { 28 | "bluebird": "^3.5.4" 29 | }, 30 | "peerDependencies": { 31 | "serverless": "1 || 2 || 3" 32 | }, 33 | "files": [ 34 | "index.js" 35 | ] 36 | } 37 | --------------------------------------------------------------------------------