├── .gitignore ├── README.md ├── client.html ├── create_secure_vw.sql ├── handler.py ├── images ├── architecture.png ├── aws_account.png ├── client.png ├── deployed.png ├── environment.png ├── key_arn.png ├── private_key.png ├── profile.png └── response.png ├── keypair_auth.py ├── package-lock.json ├── package.json ├── requirements.txt ├── serverless.yml └── state_machine.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Distribution / packaging 2 | .DS_Store 3 | .Python 4 | node_modules/ 5 | env/ 6 | build/ 7 | develop-eggs/ 8 | dist/ 9 | downloads/ 10 | eggs/ 11 | .eggs/ 12 | lib/ 13 | lib64/ 14 | parts/ 15 | sdist/ 16 | var/ 17 | *.egg-info/ 18 | .installed.cfg 19 | *.egg 20 | 21 | # Serverless directories 22 | .serverless 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Snowflake API for Analytics Applications 2 | ======================================== 3 | 4 | This is an example of how to build a serverless API that leverages Snowflake's elastic data warehouse as DB engine for analytics applications. This example implements a read-only API that can run and paginate through the results of any view defined within a Snowflake database to which it has access to. It also leverages advanced Snowflake features like Multi-cluster Warehouses and multiple caching layers to build a truly scalable and performant analytics API at a fraction of the cost of legacy systems. 5 | 6 | Requirements 7 | ============ 8 | 9 | - Access to a Snowflake account. Get your free trial account at: https://trial.snowflake.com/ 10 | 11 | - Administrative access to an AWS account: https://aws.amazon.com/ 12 | 13 | 14 | Architecture 15 | =============== 16 | 17 | This API is implemented as a completely serverless solution leveraging 18 | various AWS services including Lambda, API Gateway and Step Functions. 19 | 20 | ![](images/architecture.png) 21 | 22 | 1. Typical REST endpoints timeout after a few minutes. So, we've 23 | implemented this analytics API on websockets which lends itself 24 | better for long running analytical queries. The client initiates a 25 | connection and sends a message: `{"action": "run_view", 26 | "view_name": "trip_weather_vw"}` 27 | 28 | 2. The request is routed to a Lambda function which, in turn, triggers 29 | an AWS Step Functions workflow. 30 | 31 | 3. The step functions workflow initiates the query/view in Snowflake 32 | using the Snowflake connector for Python. The workflow then checks 33 | the status of the job in Snowflake every 5 seconds. 34 | 35 | 4. Once the query/view finishes execution, the response (either results 36 | or error) is sent back to the client asynchronously. 37 | 38 | Setup & Configuration 39 | =============================== 40 | 41 | 1. Install the latest NodeJS: . You can check to see if you have NodeJS installed by: 42 | ``` 43 | node --version 44 | ``` 45 | 46 | 2. This lab uses the **Serverless** framework 47 | () which you can install globally using: 48 | ``` 49 | sudo npm install -g serverless 50 | ``` 51 | If installed successfully, you should be able to now check the installed version: 52 | ``` 53 | serverless --version 54 | ``` 55 | 56 | 3. The API is implemented using Python 3. Check to see if you Python 3 57 | installed on your machine: 58 | ``` 59 | python --version 60 | ``` 61 | OR 62 | ``` 63 | python3 --version 64 | ``` 65 | If not installed, download and install Python 3: 66 | 67 | 68 | 4. If you don't already, install the AWS CLI using pip3: 69 | ``` 70 | sudo pip3 install awscli --upgrade --user 71 | ``` 72 | You can use `aws --version` command to verify if the AWS CLI was 73 | correctly installed. If it wasn't, see this to troubleshoot: 74 | 75 | If installing for the first time, you will need to configure AWS CLI by using: 76 | ``` 77 | aws configure 78 | ``` 79 | Make sure you have your AWS credentials handy when you configure the AWS CLI. 80 | 81 | 5. You can check if you have Git installed already by: 82 | ``` 83 | git --version 84 | ``` 85 | If not, install Git: 86 | https://git-scm.com/book/en/v2/Getting-Started-Installing-Git 87 | 88 | 6. You will also need Docker to deploy the API. Download and install from here: https://www.docker.com/products/docker-desktop 89 | 90 | Snowflake Setup 91 | =============== 92 | 93 | Before we get into building the API, lets setup our backend Snowflake environment correctly 94 | so we have all the parameters ready when it comes time to edit the API code. 95 | 96 | 1. Create and save the RSA public and private keys using the procedure described here: 97 | https://docs.snowflake.net/manuals/user-guide/snowsql-start.html#using-key-pair-authentication 98 | 99 | Jot down the passphrase you used to encrypt the private key. 100 | 101 | 2. This example API is read-only and will get data by running a particular view within Snowflake. Lets go ahead and create a view that the API can run. Login to your Snowflake account and run the following SQL statements: 102 | 103 | ```sql 104 | use role accountadmin; 105 | create role if not exists snowflake_api_role; 106 | grant usage on database to role snowflake_api_role; 107 | grant usage on schema . to role snowflake_api_role; 108 | grant select on all views in schema . to role snowflake_api_role; 109 | grant select on future views in schema . to role snowflake_api_role; 110 | grant usage on warehouse to role snowflake_api_role; 111 | 112 | create user snowflake_api_user password='Snowfl*ke' default_role = snowflake_api_role must_change_password = false; 113 | alter user snowflake_api_user set rsa_public_key=''; --exclude the header and footer 114 | grant role snowflake_api_role to user snowflake_api_user; 115 | grant role snowflake_api_role to user ; 116 | ``` 117 | 118 | 3. Create a test view with some test data, switch to using the new `snowflake_api_role` and try a simple select to see if the permissions work: 119 | 120 | ```sql 121 | use role snowflake_api_role; 122 | select * from limit 10; 123 | ``` 124 | 125 | Clone, Modify and Deploy Code 126 | ======================================== 127 | 128 | 1. Clone this repo: 129 | ``` 130 | git clone https://github.com/filanthropic/snowflake-api.git 131 | ``` 132 | 133 | 2. Before the Serverless framework can deploy this code, it needs the `serverless-python-requirements` plugin so lets install that (dependency is declared in package.json) 134 | ``` 135 | cd snowflake-api/ 136 | npm install 137 | ``` 138 | 139 | 3. Open the AWS Secrets Manager and create a new secret that will hold the private key. Select 'Other type of secret' and then select `plaintext` and use `p_key` as the key and your private key that you generated in the Snowflake setup step 1 as the value. 140 | 141 | ![](images/private_key.png) 142 | 143 | Hit `Next`, give the secret a name and description. Hit `Next` again twice and then hit `Store`. Note the name you gave to the secret. 144 | 145 | 4. From within the `snowflake-api` directory, open `keypair_auth.py` and update the following line with the passphrase that you used when you created the key pair in Snowflake Setup step \#1: 146 | ``` 147 | passkey = "" 148 | ``` 149 | 150 | 5. Also within `keypair_auth.py`, update the `secret_name` variable with the name (not the ARN) of the secret you just created within AWS Secrets Manager: 151 | ``` 152 | secret_name = "" 153 | ``` 154 | 155 | 6. Open `serverless.yml`. At the top of this file contains the 'service' -> 'name' 156 | configuration. Go ahead and change the service name to whatever you want to name this project. 157 | 158 | 7. Change AWS account number in serverless.yml 159 | 160 | ![](images/aws_account.png) 161 | 162 | 8. If using the default AWS CLI profile, remove the `profile` attribute in `serverless.yml`. If using a named profile, change it to match the AWS CLI profile you want to use to deploy: 163 | 164 | ![](images/profile.png) 165 | 166 | 9. In serverless.yml, update the last part of the ARN (not the name) of the secret that holds the private key you previously created: 167 | 168 | ![](images/key_arn.png) 169 | 170 | 10. Update the rest of the environment variables to match your Snowflake account, warehouse name, database and schema name within `serverless.yml`. 171 | 172 | ![](images/environment.png) 173 | 174 | 11. Now we are ready to deploy the API to AWS. Go to the 'snowflake-api' 175 | folder and deploy the serverless stack: 176 | ``` 177 | serverless deploy 178 | ``` 179 | The command above will take all the code you cloned, package it up as 180 | AWS Lambda functions and deploys them. It also creates the AWS API 181 | Gateway endpoint with websockets and the AWS Step Functions state 182 | machine that orchestrates the Lambda functions. 183 | 184 | 12. Go ahead and make note of the API endpoint that you just created. 185 | ![](images/deployed.png) 186 | 187 | Using the API 188 | ============= 189 | 190 | The API is based on websockets because of the long running nature of analytics queries. The best way to understand how the client interacts with the API is to first install the "wscat" tool. 191 | 192 | 1. Install the "wscat" 193 | ``` 194 | sudo npm install -g wscat 195 | ``` 196 | 197 | 2. Connect to the API endpoint you created in step \#13: 198 | ``` 199 | wscat -c wss:// 200 | ``` 201 | 202 | 3. In the API code, we have implemented two websocket "routes" or types 203 | of actions that the API supports. First one is used to run a 204 | particular view and is called "**run_view**" and the other one 205 | called "**fetch_results**" is used to fetch cached results of an 206 | already run query and helps the client paginate through the results 207 | in an efficient manner. 208 | 209 | 4. Once connected, you can run the secure view you created previously 210 | by running: 211 | ``` 212 | {"action": "run_view", "view_name":""} 213 | ``` 214 | 215 | ------------------------------------------------------------------- 216 | 217 | The response should look something like this: 218 | 219 | ![](images/response.png) 220 | 221 | 5. The response of the previous command should give you a `query_id` 222 | which you can use to paginate through the results: 223 | ``` 224 | {"action": "fetch_results","query_id": "", "offset": "100"} 225 | ``` 226 | 227 | 6. Open up the 'client.html' file in a browser to see how a simple HTML 228 | client can interact with our Snowflake API. 229 | 230 | ![](images/client.png) 231 | 232 | 233 | 234 | Additional Resources 235 | ==================== 236 | 237 | - AWS API Gateway WebSocket Support: 238 | 239 | 240 | - Serverless Framework: 241 | 242 | - AWS Serverless Stack: 243 | 244 | - Snowflake Python Connector Keypair Auth: 245 | 246 | -------------------------------------------------------------------------------- /client.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | Snowflake API Client 12 | 13 | 14 | 15 |
16 |
17 |
18 |
19 |
20 |   21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |

Snowflake WebSocket API - Sample Client

30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 | 40 | e.g. wss://xxxyyzz.execute-api.us-east-1.amazonaws.com/dev 41 |
42 |
43 | 44 |
45 | 46 |
47 |
48 |
49 |
50 |
51 |
52 |   53 |
54 |
55 |
56 |
57 |
58 |
59 |
60 |
61 |
62 |
63 |
64 |
65 |
66 |
67 |
68 |   69 |
70 |
71 | 72 | 73 | 74 |
75 |
76 |
77 |
78 |
79 | 80 | 81 |
82 |
83 |
84 |
85 |
86 | 87 |
88 |
89 |
90 |
91 | 92 | 00:00.000 93 | 94 |
95 |
96 | 102 |
103 | 104 |
105 |
106 |
107 |
108 |
109 |   110 |
111 |
112 | 113 | 114 |
115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 226 | 227 | 228 | -------------------------------------------------------------------------------- /create_secure_vw.sql: -------------------------------------------------------------------------------- 1 | --use warehouse ; 2 | use schema citibike.public; 3 | --lets take a quick look at the trips and weather tables 4 | --the trips table contains trip info from the Citibike NYC bike sharing program 5 | select * from trips limit 100; 6 | --the weather table contains JSON weather data 7 | select * from weather limit 100; 8 | 9 | ---------------------------------------------------------------------------------- 10 | -- Create a secure view with trip (structured) and weather (semistructured) data 11 | ---------------------------------------------------------------------------------- 12 | -- **Make sure to give your view a unique name** 13 | create or replace secure view as 14 | select * 15 | from trips 16 | left outer join 17 | (select t as observation_time 18 | ,v:city.id::int as city_id 19 | ,v:city.name::string as city_name 20 | ,v:city.country::string as country 21 | ,v:city.coord.lat::float as city_lat 22 | ,v:city.coord.lon::float as city_lon 23 | ,v:clouds.all::int as clouds 24 | ,(v:main.temp::float)-273.15 as temp_avg_c 25 | ,(v:main.temp_min::float)-273.15 as temp_min_c 26 | ,(v:main.temp_max::float)-273.15 as temp_max_c 27 | ,(v:main.temp::float)*9/5-459.67 as temp_avg_f 28 | ,(v:main.temp_min::float)*9/5-459.67 as temp_min_f 29 | ,(v:main.temp_max::float)*9/5-459.67 as temp_max_f 30 | ,v:weather[0].id::int as weather_id 31 | ,v:weather[0].main::string as weather 32 | ,v:weather[0].description::string as weather_desc 33 | ,v:weather[0].icon::string as weather_icon 34 | ,v:wind.deg::float as wind_dir 35 | ,v:wind.speed::float as wind_speed 36 | from weather 37 | where city_id = 5128638) 38 | on date_trunc(HOUR, starttime) = date_trunc(HOUR, observation_time); 39 | 40 | --test the secure view 41 | select * from where date_part('year', observation_time)=2018 limit 20; 42 | -------------------------------------------------------------------------------- /handler.py: -------------------------------------------------------------------------------- 1 | import json 2 | import snowflake.connector 3 | import logging 4 | logger = logging.getLogger() 5 | logger.setLevel(logging.INFO) 6 | import urllib.parse 7 | import boto3 8 | import time 9 | from requests_aws4auth import AWS4Auth 10 | import requests 11 | import os 12 | from keypair_auth import get_snowflake_cursor 13 | 14 | def _get_response(status_code, body): 15 | if not isinstance(body, str): 16 | body = json.dumps(body) 17 | return {"statusCode": status_code, "body": body} 18 | 19 | def _get_body(event): 20 | try: 21 | return json.loads(event.get("body", "")) 22 | except: 23 | logger.debug("event body could not be JSON decoded.") 24 | return {} 25 | 26 | def connection_manager(event, context): 27 | """ 28 | Handles connecting and disconnecting for the Websocket. 29 | """ 30 | connectionID = event["requestContext"].get("connectionId") 31 | 32 | if event["requestContext"]["eventType"] == "CONNECT": 33 | print("Connect requested: ", connectionID) 34 | return _get_response(200, "Connect successful.") 35 | 36 | elif event["requestContext"]["eventType"] == "DISCONNECT": 37 | logger.info("Disconnect requested") 38 | return _get_response(200, "Disconnect successful.") 39 | 40 | else: 41 | logger.error("Connection manager received unrecognized eventType '{}'") 42 | return _get_response(500, "Unrecognized eventType.") 43 | 44 | def _get_postback_url(event): 45 | requestContext = event['requestContext'] 46 | domain = requestContext['domainName'] 47 | stage = requestContext['stage'] 48 | connectionId = requestContext['connectionId'] 49 | 50 | postbackURL = 'https://' + domain + '/' + stage + '/%40connections/' + urllib.parse.quote_plus(connectionId) 51 | 52 | return postbackURL 53 | 54 | def default_message(event, context): 55 | """ 56 | Send back error when unrecognized WebSocket action is received. 57 | """ 58 | logger.info("Unrecognized WebSocket action received.") 59 | return _get_response(400, "Unrecognized WebSocket action.") 60 | 61 | def run_view(event, context): 62 | #print("event: ", json.dumps(event)) 63 | postbackURL = _get_postback_url(event) 64 | print("postbackURL: ", postbackURL) 65 | message = postbackURL 66 | 67 | if 'body' in event: 68 | body = json.loads(event['body']) 69 | 70 | if 'action' in body: 71 | action = body['action'] 72 | if action == 'run_view': 73 | view_name = body['view_name'] 74 | 75 | input = {} 76 | input['view_name'] = view_name 77 | input['post_back_url'] = postbackURL 78 | input['wait_time'] = 5 79 | aws_account = boto3.client('sts').get_caller_identity()['Account'] 80 | 81 | client = boto3.client('stepfunctions') 82 | response = client.start_execution( 83 | stateMachineArn=os.environ['SNOWFLAKE_STATE_MACHINE_ARN'], 84 | name='execution-' + time.strftime("%Y%m%d%H%M%S"), 85 | input=json.dumps(input) 86 | ) 87 | message = "Request submitted. Please wait..." 88 | auth = AWS4Auth(os.environ['AWS_ACCESS_KEY_ID'], os.environ['AWS_SECRET_ACCESS_KEY'], 'us-east-1', 'execute-api', session_token = os.environ['AWS_SESSION_TOKEN']) 89 | r = requests.post(postbackURL, auth=auth, data=message) 90 | print(r.status_code) 91 | 92 | 93 | return _get_response(200, message) 94 | 95 | 96 | def fetch_results(event, context): 97 | if 'body' in event: 98 | body = json.loads(event['body']) 99 | 100 | 101 | if 'query_id' in body: 102 | query_id = body['query_id'] 103 | if 'offset' in body: 104 | offset = body['offset'] 105 | else: 106 | offset = "0" 107 | 108 | cs = get_snowflake_cursor() 109 | try: 110 | #cs.execute("use warehouse " + os.environ['SNOWFLAKE_WAREHOUSE'] + ";") 111 | #cs.execute("use schema " + os.environ['SNOWFLAKE_SCHEMA'] + ";") 112 | cs.execute("select * from table(result_scan('" + query_id + "')) limit 100 offset " + offset + ";") 113 | #print(','.join([col[0] for col in cs.description])) 114 | columns = [] 115 | for col in cs.description: 116 | columns.append(col[0]) 117 | results = cs.fetchall() 118 | json_results = [] 119 | for rec in results: 120 | json_rec = {} 121 | for col in columns: 122 | #print('%s: %s' % (col, rec[columns.index(col)])) 123 | json_rec[col] = str(rec[columns.index(col)]) 124 | print(json_rec) 125 | json_results.append(json_rec) 126 | 127 | json_root = {} 128 | json_root['query_id'] = query_id 129 | json_root['results'] = json_results 130 | print(json.dumps(json_root)) 131 | message = json.dumps(json_root) 132 | 133 | url = _get_postback_url(event) 134 | auth = AWS4Auth(os.environ['AWS_ACCESS_KEY_ID'], os.environ['AWS_SECRET_ACCESS_KEY'], 'us-east-1', 'execute-api', session_token = os.environ['AWS_SESSION_TOKEN']) 135 | r = requests.post(url, auth=auth, json=json_root) 136 | print(r.status_code) 137 | finally: 138 | cs.close() 139 | else: 140 | message = "No query_id provided." 141 | 142 | return _get_response(200, message) 143 | -------------------------------------------------------------------------------- /images/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/architecture.png -------------------------------------------------------------------------------- /images/aws_account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/aws_account.png -------------------------------------------------------------------------------- /images/client.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/client.png -------------------------------------------------------------------------------- /images/deployed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/deployed.png -------------------------------------------------------------------------------- /images/environment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/environment.png -------------------------------------------------------------------------------- /images/key_arn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/key_arn.png -------------------------------------------------------------------------------- /images/private_key.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/private_key.png -------------------------------------------------------------------------------- /images/profile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/profile.png -------------------------------------------------------------------------------- /images/response.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kbash09/snowflake-api/2c905889f1be9ef28b1e295345cc802927ab6a9d/images/response.png -------------------------------------------------------------------------------- /keypair_auth.py: -------------------------------------------------------------------------------- 1 | import snowflake.connector 2 | import json 3 | import os 4 | from cryptography.hazmat.backends import default_backend 5 | from cryptography.hazmat.primitives.asymmetric import rsa 6 | from cryptography.hazmat.primitives.asymmetric import dsa 7 | from cryptography.hazmat.primitives import serialization 8 | 9 | import boto3 10 | import base64 11 | from botocore.exceptions import ClientError 12 | 13 | 14 | def _get_secret(): 15 | 16 | secret_name = "" 17 | 18 | # Create a Secrets Manager client 19 | session = boto3.session.Session() 20 | client = session.client( 21 | service_name='secretsmanager', 22 | region_name=os.environ['API_REGION'] 23 | ) 24 | 25 | try: 26 | get_secret_value_response = client.get_secret_value( 27 | SecretId=secret_name 28 | ) 29 | except ClientError as e: 30 | if e.response['Error']['Code'] == 'DecryptionFailureException': 31 | # Secrets Manager can't decrypt the protected secret text using the provided KMS key. 32 | # Deal with the exception here, and/or rethrow at your discretion. 33 | raise e 34 | elif e.response['Error']['Code'] == 'InternalServiceErrorException': 35 | # An error occurred on the server side. 36 | # Deal with the exception here, and/or rethrow at your discretion. 37 | raise e 38 | elif e.response['Error']['Code'] == 'InvalidParameterException': 39 | # You provided an invalid value for a parameter. 40 | # Deal with the exception here, and/or rethrow at your discretion. 41 | raise e 42 | elif e.response['Error']['Code'] == 'InvalidRequestException': 43 | # You provided a parameter value that is not valid for the current state of the resource. 44 | # Deal with the exception here, and/or rethrow at your discretion. 45 | raise e 46 | elif e.response['Error']['Code'] == 'ResourceNotFoundException': 47 | # We can't find the resource that you asked for. 48 | # Deal with the exception here, and/or rethrow at your discretion. 49 | raise e 50 | else: 51 | # Decrypts secret using the associated KMS CMK. 52 | # Depending on whether the secret is a string or binary, one of these fields will be populated. 53 | if 'SecretString' in get_secret_value_response: 54 | secret = get_secret_value_response['SecretString'] 55 | else: 56 | decoded_binary_secret = base64.b64decode(get_secret_value_response['SecretBinary']) 57 | 58 | p_key = json.loads(secret, strict=False)['p_key'] 59 | return p_key 60 | 61 | def get_snowflake_cursor(): 62 | secret_key = _get_secret() 63 | passkey = "" 64 | 65 | p_key= serialization.load_pem_private_key(secret_key.encode(),password=passkey.encode(),backend=default_backend()) 66 | 67 | pkb = p_key.private_bytes( 68 | encoding=serialization.Encoding.DER, 69 | format=serialization.PrivateFormat.PKCS8, 70 | encryption_algorithm=serialization.NoEncryption()) 71 | 72 | ctx = snowflake.connector.connect( 73 | user=os.environ['SNOWFLAKE_USER'], 74 | account=os.environ['SNOWFLAKE_ACCOUNT'], 75 | private_key=pkb, 76 | warehouse=os.environ['SNOWFLAKE_WAREHOUSE'], 77 | database=os.environ['SNOWFLAKE_DATABASE'], 78 | schema=os.environ['SNOWFLAKE_SCHEMA'] 79 | ) 80 | 81 | cs = ctx.cursor() 82 | return cs 83 | -------------------------------------------------------------------------------- /package-lock.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "snowflake-api", 3 | "version": "1.0.0", 4 | "lockfileVersion": 1, 5 | "requires": true, 6 | "dependencies": { 7 | "appdirectory": { 8 | "version": "0.1.0", 9 | "resolved": "https://registry.npmjs.org/appdirectory/-/appdirectory-0.1.0.tgz", 10 | "integrity": "sha1-62yBYyDnsqsW9e2ZfyjYIF31Y3U=" 11 | }, 12 | "array-filter": { 13 | "version": "0.0.1", 14 | "resolved": "https://registry.npmjs.org/array-filter/-/array-filter-0.0.1.tgz", 15 | "integrity": "sha1-fajPLiZijtcygDWB/SH2fKzS7uw=" 16 | }, 17 | "array-map": { 18 | "version": "0.0.0", 19 | "resolved": "https://registry.npmjs.org/array-map/-/array-map-0.0.0.tgz", 20 | "integrity": "sha1-iKK6tz0c97zVwbEYoAP2b2ZfpmI=" 21 | }, 22 | "array-reduce": { 23 | "version": "0.0.0", 24 | "resolved": "https://registry.npmjs.org/array-reduce/-/array-reduce-0.0.0.tgz", 25 | "integrity": "sha1-FziZ0//Rx9k4PkR5Ul2+J4yrXys=" 26 | }, 27 | "balanced-match": { 28 | "version": "1.0.0", 29 | "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.0.tgz", 30 | "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c=" 31 | }, 32 | "bluebird": { 33 | "version": "3.5.5", 34 | "resolved": "https://registry.npmjs.org/bluebird/-/bluebird-3.5.5.tgz", 35 | "integrity": "sha512-5am6HnnfN+urzt4yfg7IgTbotDjIT/u8AJpEt0sIU9FtXfVeezXAPKswrG+xKUCOYAINpSdgZVDU6QFh+cuH3w==" 36 | }, 37 | "brace-expansion": { 38 | "version": "1.1.11", 39 | "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz", 40 | "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==", 41 | "requires": { 42 | "balanced-match": "^1.0.0", 43 | "concat-map": "0.0.1" 44 | } 45 | }, 46 | "concat-map": { 47 | "version": "0.0.1", 48 | "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", 49 | "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s=" 50 | }, 51 | "core-util-is": { 52 | "version": "1.0.2", 53 | "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz", 54 | "integrity": "sha1-tf1UIgqivFq1eqtxQMlAdUUDwac=" 55 | }, 56 | "fs-extra": { 57 | "version": "7.0.1", 58 | "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-7.0.1.tgz", 59 | "integrity": "sha512-YJDaCJZEnBmcbw13fvdAM9AwNOJwOzrE4pqMqBq5nFiEqXUqHwlK4B+3pUw6JNvfSPtX05xFHtYy/1ni01eGCw==", 60 | "requires": { 61 | "graceful-fs": "^4.1.2", 62 | "jsonfile": "^4.0.0", 63 | "universalify": "^0.1.0" 64 | } 65 | }, 66 | "fs.realpath": { 67 | "version": "1.0.0", 68 | "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz", 69 | "integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8=" 70 | }, 71 | "glob": { 72 | "version": "7.1.4", 73 | "resolved": "https://registry.npmjs.org/glob/-/glob-7.1.4.tgz", 74 | "integrity": "sha512-hkLPepehmnKk41pUGm3sYxoFs/umurYfYJCerbXEyFIWcAzvpipAgVkBqqT9RBKMGjnq6kMuyYwha6csxbiM1A==", 75 | "requires": { 76 | "fs.realpath": "^1.0.0", 77 | "inflight": "^1.0.4", 78 | "inherits": "2", 79 | "minimatch": "^3.0.4", 80 | "once": "^1.3.0", 81 | "path-is-absolute": "^1.0.0" 82 | } 83 | }, 84 | "glob-all": { 85 | "version": "3.1.0", 86 | "resolved": "https://registry.npmjs.org/glob-all/-/glob-all-3.1.0.tgz", 87 | "integrity": "sha1-iRPd+17hrHgSZWJBsD1SF8ZLAqs=", 88 | "requires": { 89 | "glob": "^7.0.5", 90 | "yargs": "~1.2.6" 91 | } 92 | }, 93 | "graceful-fs": { 94 | "version": "4.1.15", 95 | "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.1.15.tgz", 96 | "integrity": "sha512-6uHUhOPEBgQ24HM+r6b/QwWfZq+yiFcipKFrOFiBEnWdy5sdzYoi+pJeQaPI5qOLRFqWmAXUPQNsielzdLoecA==" 97 | }, 98 | "immediate": { 99 | "version": "3.0.6", 100 | "resolved": "https://registry.npmjs.org/immediate/-/immediate-3.0.6.tgz", 101 | "integrity": "sha1-nbHb0Pr43m++D13V5Wu2BigN5ps=" 102 | }, 103 | "inflight": { 104 | "version": "1.0.6", 105 | "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz", 106 | "integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=", 107 | "requires": { 108 | "once": "^1.3.0", 109 | "wrappy": "1" 110 | } 111 | }, 112 | "inherits": { 113 | "version": "2.0.3", 114 | "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz", 115 | "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=" 116 | }, 117 | "is-wsl": { 118 | "version": "1.1.0", 119 | "resolved": "https://registry.npmjs.org/is-wsl/-/is-wsl-1.1.0.tgz", 120 | "integrity": "sha1-HxbkqiKwTRM2tmGIpmrzxgDDpm0=" 121 | }, 122 | "isarray": { 123 | "version": "1.0.0", 124 | "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz", 125 | "integrity": "sha1-u5NdSFgsuhaMBoNJV6VKPgcSTxE=" 126 | }, 127 | "jsonfile": { 128 | "version": "4.0.0", 129 | "resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-4.0.0.tgz", 130 | "integrity": "sha1-h3Gq4HmbZAdrdmQPygWPnBDjPss=", 131 | "requires": { 132 | "graceful-fs": "^4.1.6" 133 | } 134 | }, 135 | "jsonify": { 136 | "version": "0.0.0", 137 | "resolved": "https://registry.npmjs.org/jsonify/-/jsonify-0.0.0.tgz", 138 | "integrity": "sha1-LHS27kHZPKUbe1qu6PUDYx0lKnM=" 139 | }, 140 | "jszip": { 141 | "version": "3.2.1", 142 | "resolved": "https://registry.npmjs.org/jszip/-/jszip-3.2.1.tgz", 143 | "integrity": "sha512-iCMBbo4eE5rb1VCpm5qXOAaUiRKRUKiItn8ah2YQQx9qymmSAY98eyQfioChEYcVQLh0zxJ3wS4A0mh90AVPvw==", 144 | "requires": { 145 | "lie": "~3.3.0", 146 | "pako": "~1.0.2", 147 | "readable-stream": "~2.3.6", 148 | "set-immediate-shim": "~1.0.1" 149 | } 150 | }, 151 | "lie": { 152 | "version": "3.3.0", 153 | "resolved": "https://registry.npmjs.org/lie/-/lie-3.3.0.tgz", 154 | "integrity": "sha512-UaiMJzeWRlEujzAuw5LokY1L5ecNQYZKfmyZ9L7wDHb/p5etKaxXhohBcrw0EYby+G/NA52vRSN4N39dxHAIwQ==", 155 | "requires": { 156 | "immediate": "~3.0.5" 157 | } 158 | }, 159 | "lodash.get": { 160 | "version": "4.4.2", 161 | "resolved": "https://registry.npmjs.org/lodash.get/-/lodash.get-4.4.2.tgz", 162 | "integrity": "sha1-LRd/ZS+jHpObRDjVNBSZ36OCXpk=" 163 | }, 164 | "lodash.set": { 165 | "version": "4.3.2", 166 | "resolved": "https://registry.npmjs.org/lodash.set/-/lodash.set-4.3.2.tgz", 167 | "integrity": "sha1-2HV7HagH3eJIFrDWqEvqGnYjCyM=" 168 | }, 169 | "lodash.uniqby": { 170 | "version": "4.7.0", 171 | "resolved": "https://registry.npmjs.org/lodash.uniqby/-/lodash.uniqby-4.7.0.tgz", 172 | "integrity": "sha1-2ZwHpmnp5tJOE2Lf4mbGdhavEwI=" 173 | }, 174 | "lodash.values": { 175 | "version": "4.3.0", 176 | "resolved": "https://registry.npmjs.org/lodash.values/-/lodash.values-4.3.0.tgz", 177 | "integrity": "sha1-o6bCsOvsxcLLocF+bmIP6BtT00c=" 178 | }, 179 | "md5-file": { 180 | "version": "4.0.0", 181 | "resolved": "https://registry.npmjs.org/md5-file/-/md5-file-4.0.0.tgz", 182 | "integrity": "sha512-UC0qFwyAjn4YdPpKaDNw6gNxRf7Mcx7jC1UGCY4boCzgvU2Aoc1mOGzTtrjjLKhM5ivsnhoKpQVxKPp+1j1qwg==" 183 | }, 184 | "minimatch": { 185 | "version": "3.0.4", 186 | "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz", 187 | "integrity": "sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA==", 188 | "requires": { 189 | "brace-expansion": "^1.1.7" 190 | } 191 | }, 192 | "minimist": { 193 | "version": "0.1.0", 194 | "resolved": "https://registry.npmjs.org/minimist/-/minimist-0.1.0.tgz", 195 | "integrity": "sha1-md9lelJXTCHJBXSX33QnkLK0wN4=" 196 | }, 197 | "once": { 198 | "version": "1.4.0", 199 | "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", 200 | "integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=", 201 | "requires": { 202 | "wrappy": "1" 203 | } 204 | }, 205 | "pako": { 206 | "version": "1.0.10", 207 | "resolved": "https://registry.npmjs.org/pako/-/pako-1.0.10.tgz", 208 | "integrity": "sha512-0DTvPVU3ed8+HNXOu5Bs+o//Mbdj9VNQMUOe9oKCwh8l0GNwpTDMKCWbRjgtD291AWnkAgkqA/LOnQS8AmS1tw==" 209 | }, 210 | "path-is-absolute": { 211 | "version": "1.0.1", 212 | "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", 213 | "integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18=" 214 | }, 215 | "process-nextick-args": { 216 | "version": "2.0.0", 217 | "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.0.tgz", 218 | "integrity": "sha512-MtEC1TqN0EU5nephaJ4rAtThHtC86dNN9qCuEhtshvpVBkAW5ZO7BASN9REnF9eoXGcRub+pFuKEpOHE+HbEMw==" 219 | }, 220 | "readable-stream": { 221 | "version": "2.3.6", 222 | "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.6.tgz", 223 | "integrity": "sha512-tQtKA9WIAhBF3+VLAseyMqZeBjW0AHJoxOtYqSUZNJxauErmLbVm2FW1y+J/YA9dUrAC39ITejlZWhVIwawkKw==", 224 | "requires": { 225 | "core-util-is": "~1.0.0", 226 | "inherits": "~2.0.3", 227 | "isarray": "~1.0.0", 228 | "process-nextick-args": "~2.0.0", 229 | "safe-buffer": "~5.1.1", 230 | "string_decoder": "~1.1.1", 231 | "util-deprecate": "~1.0.1" 232 | } 233 | }, 234 | "rimraf": { 235 | "version": "2.6.3", 236 | "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-2.6.3.tgz", 237 | "integrity": "sha512-mwqeW5XsA2qAejG46gYdENaxXjx9onRNCfn7L0duuP4hCuTIi/QO7PDK07KJfp1d+izWPrzEJDcSqBa0OZQriA==", 238 | "requires": { 239 | "glob": "^7.1.3" 240 | } 241 | }, 242 | "safe-buffer": { 243 | "version": "5.1.2", 244 | "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", 245 | "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==" 246 | }, 247 | "serverless-python-requirements": { 248 | "version": "4.3.0", 249 | "resolved": "https://registry.npmjs.org/serverless-python-requirements/-/serverless-python-requirements-4.3.0.tgz", 250 | "integrity": "sha512-VyXdEKNxUWoQDbssWZeR5YMaTDf1U4CO3yJH2953Y2Rt8zD6hG+vpTkVR490/Ws1PQsBopWuFfgDcLyvAppaRg==", 251 | "requires": { 252 | "appdirectory": "^0.1.0", 253 | "bluebird": "^3.0.6", 254 | "fs-extra": "^7.0.0", 255 | "glob-all": "^3.1.0", 256 | "is-wsl": "^1.1.0", 257 | "jszip": "^3.1.0", 258 | "lodash.get": "^4.4.2", 259 | "lodash.set": "^4.3.2", 260 | "lodash.uniqby": "^4.0.0", 261 | "lodash.values": "^4.3.0", 262 | "md5-file": "^4.0.0", 263 | "rimraf": "^2.6.2", 264 | "shell-quote": "^1.6.1" 265 | } 266 | }, 267 | "set-immediate-shim": { 268 | "version": "1.0.1", 269 | "resolved": "https://registry.npmjs.org/set-immediate-shim/-/set-immediate-shim-1.0.1.tgz", 270 | "integrity": "sha1-SysbJ+uAip+NzEgaWOXlb1mfP2E=" 271 | }, 272 | "shell-quote": { 273 | "version": "1.6.1", 274 | "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.6.1.tgz", 275 | "integrity": "sha1-9HgZSczkAmlxJ0MOo7PFR29IF2c=", 276 | "requires": { 277 | "array-filter": "~0.0.0", 278 | "array-map": "~0.0.0", 279 | "array-reduce": "~0.0.0", 280 | "jsonify": "~0.0.0" 281 | } 282 | }, 283 | "string_decoder": { 284 | "version": "1.1.1", 285 | "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz", 286 | "integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==", 287 | "requires": { 288 | "safe-buffer": "~5.1.0" 289 | } 290 | }, 291 | "universalify": { 292 | "version": "0.1.2", 293 | "resolved": "https://registry.npmjs.org/universalify/-/universalify-0.1.2.tgz", 294 | "integrity": "sha512-rBJeI5CXAlmy1pV+617WB9J63U6XcazHHF2f2dbJix4XzpUF0RS3Zbj0FGIOCAva5P/d/GBOYaACQ1w+0azUkg==" 295 | }, 296 | "util-deprecate": { 297 | "version": "1.0.2", 298 | "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", 299 | "integrity": "sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8=" 300 | }, 301 | "wrappy": { 302 | "version": "1.0.2", 303 | "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", 304 | "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8=" 305 | }, 306 | "yargs": { 307 | "version": "1.2.6", 308 | "resolved": "https://registry.npmjs.org/yargs/-/yargs-1.2.6.tgz", 309 | "integrity": "sha1-nHtKgv1dWVsr8Xq23MQxNUMv40s=", 310 | "requires": { 311 | "minimist": "^0.1.0" 312 | } 313 | } 314 | } 315 | } 316 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "requires": true, 3 | "lockfileVersion": 1, 4 | "dependencies": { 5 | "serverless-python-requirements": "^4.3.0" 6 | }, 7 | "name": "snowflake-api", 8 | "version": "1.0.0", 9 | "main": "index.js", 10 | "scripts": { 11 | "test": "echo \"Error: no test specified\" && exit 1" 12 | }, 13 | "repository": { 14 | "type": "git", 15 | "url": "git+https://github.com/filanthropic/snowflake-api.git" 16 | }, 17 | "author": "", 18 | "license": "ISC", 19 | "bugs": { 20 | "url": "https://github.com/filanthropic/snowflake-api/issues" 21 | }, 22 | "homepage": "https://github.com/filanthropic/snowflake-api#readme", 23 | "description": "" 24 | } 25 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | asn1crypto==0.24.0 2 | azure-common==1.1.22 3 | azure-storage-blob==2.0.1 4 | azure-storage-common==2.0.0 5 | boto3==1.9.164 6 | botocore==1.12.164 7 | certifi==2019.3.9 8 | cffi==1.12.3 9 | chardet==3.0.4 10 | cryptography==2.7 11 | docutils==0.14 12 | future==0.17.1 13 | idna==2.8 14 | ijson==2.3 15 | jmespath==0.9.4 16 | pycparser==2.19 17 | pycryptodomex==3.8.2 18 | PyJWT==1.7.1 19 | pyOpenSSL==19.0.0 20 | python-dateutil==2.8.0 21 | pytz==2019.1 22 | requests==2.22.0 23 | requests-aws4auth==0.9 24 | s3transfer==0.2.1 25 | six==1.12.0 26 | snowflake-connector-python==1.8.2 27 | urllib3==1.25.3 28 | -------------------------------------------------------------------------------- /serverless.yml: -------------------------------------------------------------------------------- 1 | service: 2 | name: snowflake-api-v1 3 | 4 | plugins: 5 | - serverless-python-requirements 6 | 7 | custom: 8 | aws_account: 9 | pythonRequirements: 10 | dockerizePip: non-linux 11 | 12 | provider: 13 | name: aws 14 | runtime: python3.7 15 | stackName: ${self:service}-${self:provider.stage} 16 | stage: ${opt:stage, 'dev'} 17 | region: ${opt:region, 'us-east-1'} 18 | profile: #remove if using the default profile for AWS CLI 19 | iamRoleStatements: 20 | - Effect: Allow 21 | Action: 22 | - "execute-api:ManageConnections" 23 | Resource: 24 | - "arn:aws:execute-api:*:*:**/@connections/*" 25 | - Effect: Allow 26 | Action: 27 | - "secretsmanager:GetSecretValue" 28 | Resource: 29 | - "arn:aws:secretsmanager:${opt:region, 'us-east-1'}:${self:custom.aws_account}:secret:" 30 | - Effect: Allow 31 | Action: 32 | - "states:StartExecution" 33 | Resource: 34 | - "arn:aws:states:${opt:region, 'us-east-1'}:${self:custom.aws_account}:stateMachine:StateMachine-${self:service}" 35 | environment: 36 | API_REGION: "${self:provider.region}" 37 | SNOWFLAKE_USER: "snowflake_api_user" 38 | SNOWFLAKE_ACCOUNT: "" 39 | SNOWFLAKE_WAREHOUSE: "" 40 | SNOWFLAKE_DATABASE: "" 41 | SNOWFLAKE_SCHEMA: "" 42 | SNOWFLAKE_STATE_MACHINE_ARN: "arn:aws:states:${opt:region, 'us-east-1'}:${self:custom.aws_account}:stateMachine:StateMachine-${self:service}" 43 | 44 | # optional 45 | websocketApiName: snowflake-websocket-api-${self:provider.stage} 46 | 47 | # required for websocket apis 48 | # this selects from your payload what your "routeKey" parameter is 49 | # from the websocket event types on your function 50 | websocketApiRouteSelectionExpression: $request.body.action 51 | 52 | functions: 53 | # manage connection and disconnection of clients 54 | connectionManager: 55 | handler: handler.connection_manager 56 | events: 57 | - websocket: 58 | route: $connect 59 | - websocket: 60 | route: $disconnect 61 | 62 | # default error message 63 | defaultMessages: 64 | handler: handler.default_message 65 | events: 66 | - websocket: 67 | route: $default 68 | 69 | # the main RunView handler 70 | RunView: 71 | handler: handler.run_view 72 | timeout: 300 73 | events: 74 | - websocket: 75 | route: run_view 76 | 77 | 78 | FetchResults: 79 | handler: handler.fetch_results 80 | timeout: 300 81 | events: 82 | - websocket: 83 | route: fetch_results 84 | 85 | # functions below are part of the state machine 86 | StartRun: 87 | handler: state_machine.start_run 88 | 89 | GetExecutionStatus: 90 | handler: state_machine.get_execution_status 91 | timeout: 300 92 | 93 | PostBackResults: 94 | handler: state_machine.post_back_results 95 | timeout: 300 96 | 97 | PostBackErrorMessage: 98 | handler: state_machine.post_back_error_message 99 | timeout: 300 100 | 101 | #-----OTHER RESOURCES-------- 102 | resources: 103 | Resources: 104 | StatesExecutionRoleLayers: 105 | Type: "AWS::IAM::Role" 106 | Properties: 107 | AssumeRolePolicyDocument: 108 | Version: "2012-10-17" 109 | Statement: 110 | - Effect: "Allow" 111 | Principal: 112 | Service: 113 | - !Sub states.${opt:region, 'us-east-1'}.amazonaws.com 114 | Action: "sts:AssumeRole" 115 | Path: "/" 116 | Policies: 117 | - PolicyName: StatesExecutionPolicy 118 | PolicyDocument: 119 | Version: "2012-10-17" 120 | Statement: 121 | - Effect: Allow 122 | Action: 123 | - "lambda:InvokeFunction" 124 | Resource: "*" 125 | 126 | StateMachine: 127 | Type: "AWS::StepFunctions::StateMachine" 128 | Properties: 129 | StateMachineName: "StateMachine-${self:service}" 130 | DefinitionString: 131 | |- 132 | { 133 | "Comment": "Runs a Snowflake view and waits for the execution to complete", 134 | "StartAt": "RUN_VIEW", 135 | "States": { 136 | "RUN_VIEW": { 137 | "Type": "Task", 138 | "Resource": "arn:aws:lambda:${opt:region, 'us-east-1'}:${self:custom.aws_account}:function:${self:service}-${self:provider.stage}-StartRun", 139 | "ResultPath": "$.query_id", 140 | "Next": "CHECK_STATUS" 141 | }, 142 | "CHECK_STATUS": { 143 | "Type": "Task", 144 | "Resource": "arn:aws:lambda:${opt:region, 'us-east-1'}:${self:custom.aws_account}:function:${self:service}-${self:provider.stage}-GetExecutionStatus", 145 | "ResultPath": "$.status", 146 | "Next": "EXECUTION_COMPLETE?" 147 | }, 148 | "EXECUTION_COMPLETE?": { 149 | "Type": "Choice", 150 | "Choices": [ 151 | { 152 | "Variable": "$.status", 153 | "StringEquals": "FAILED_WITH_ERROR", 154 | "Next": "POST_BACK_ERROR_MESSAGE" 155 | }, 156 | { 157 | "Variable": "$.status", 158 | "StringEquals": "SUCCESS", 159 | "Next": "POST_BACK_RESULTS" 160 | } 161 | ], 162 | "Default": "WAIT_FOR_EXECUTION" 163 | }, 164 | "WAIT_FOR_EXECUTION": { 165 | "Type": "Wait", 166 | "SecondsPath": "$.wait_time", 167 | "Next": "CHECK_STATUS" 168 | }, 169 | "POST_BACK_ERROR_MESSAGE": { 170 | "Type": "Task", 171 | "Resource": "arn:aws:lambda:${opt:region, 'us-east-1'}:${self:custom.aws_account}:function:${self:service}-${self:provider.stage}-PostBackErrorMessage", 172 | "ResultPath": "$.post_status", 173 | "Next": "EXECUTION_FAILED" 174 | }, 175 | "EXECUTION_FAILED": { 176 | "Type": "Fail", 177 | "Cause": "VIEW_EXECUTION_FAILED", 178 | "Error": "View execution failed" 179 | }, 180 | "POST_BACK_RESULTS": { 181 | "Type": "Task", 182 | "Resource": "arn:aws:lambda:${opt:region, 'us-east-1'}:${self:custom.aws_account}:function:${self:service}-${self:provider.stage}-PostBackResults", 183 | "ResultPath": "$.post_status", 184 | "Next": "SUCCESS" 185 | }, 186 | "SUCCESS": { 187 | "Type": "Succeed" 188 | } 189 | } 190 | } 191 | 192 | RoleArn: !GetAtt [ StatesExecutionRoleLayers, Arn ] 193 | -------------------------------------------------------------------------------- /state_machine.py: -------------------------------------------------------------------------------- 1 | import json 2 | import snowflake.connector 3 | import urllib.parse 4 | import requests 5 | from requests_aws4auth import AWS4Auth 6 | import os 7 | from keypair_auth import get_snowflake_cursor 8 | 9 | def start_run (event, context): 10 | 11 | if 'view_name' in event: 12 | snowflake_view = event['view_name'] 13 | print("now running snowflake view: ", snowflake_view) 14 | 15 | cs = get_snowflake_cursor() 16 | try: 17 | print("starting to execute query") 18 | # cs.execute("use warehouse " + os.environ['SNOWFLAKE_WAREHOUSE'] + ";") 19 | # cs.execute("use schema " + os.environ['SNOWFLAKE_SCHEMA'] + ";") 20 | cs.execute("SELECT * from " + snowflake_view, _no_results=True) 21 | query_id = cs.sfqid 22 | print("query id: ", query_id) 23 | 24 | if 'post_back_url' in event: 25 | url = event['post_back_url'] 26 | print("now trying to post back to: ", url) 27 | data = "Now running query_id: " + query_id 28 | auth = AWS4Auth(os.environ['AWS_ACCESS_KEY_ID'], os.environ['AWS_SECRET_ACCESS_KEY'], 'us-east-1', 'execute-api', session_token = os.environ['AWS_SESSION_TOKEN']) 29 | r = requests.post(url, auth=auth, data=data) 30 | finally: 31 | pass 32 | #cs.close() <-- purposely not closing the connection since we want this to be async 33 | 34 | return query_id 35 | 36 | 37 | def get_execution_status (event, context): 38 | if 'query_id' in event: 39 | query_id = event['query_id'] 40 | cs = get_snowflake_cursor() 41 | try: 42 | cs.execute("select execution_status from table(information_schema.query_history()) where query_id like '" + query_id + "';") 43 | status = cs.fetchone()[0] 44 | #print(one_row[0]) 45 | finally: 46 | cs.close() 47 | 48 | return status 49 | 50 | def post_back_error_message (event, context): 51 | if 'query_id' in event: 52 | query_id = event['query_id'] 53 | cs = get_snowflake_cursor() 54 | try: 55 | cs.execute("select error_message from table(information_schema.query_history()) where query_id like '" + query_id + "';") 56 | error_message = cs.fetchone()[0] 57 | 58 | if 'post_back_url' in event: 59 | url = event['post_back_url'] 60 | print("now trying to post back to: ", url) 61 | auth = AWS4Auth(os.environ['AWS_ACCESS_KEY_ID'], os.environ['AWS_SECRET_ACCESS_KEY'], 'us-east-1', 'execute-api', session_token = os.environ['AWS_SESSION_TOKEN']) 62 | r = requests.post(url, auth=auth, data=error_message) 63 | finally: 64 | cs.close() 65 | 66 | def post_back_results (event, context): 67 | if 'query_id' in event: 68 | query_id = event['query_id'] 69 | 70 | cs = get_snowflake_cursor() 71 | try: 72 | cs.execute("select * from table(result_scan('" + query_id + "')) limit 100;") 73 | results = cs.fetchall() 74 | finally: 75 | cs.close() 76 | 77 | columns = [] 78 | for col in cs.description: 79 | columns.append(col[0]) 80 | json_results = [] 81 | for rec in results: 82 | json_rec = {} 83 | for col in columns: 84 | #print('%s: %s' % (col, rec[columns.index(col)])) 85 | json_rec[col] = str(rec[columns.index(col)]) 86 | print(json_rec) 87 | json_results.append(json_rec) 88 | 89 | json_root = {} 90 | json_root['query_id'] = query_id 91 | json_root['results'] = json_results 92 | #print(json.dumps(json_root)) 93 | 94 | if 'post_back_url' in event: 95 | url = event['post_back_url'] 96 | print("now trying to post back to: ", url) 97 | auth = AWS4Auth(os.environ['AWS_ACCESS_KEY_ID'], os.environ['AWS_SECRET_ACCESS_KEY'], 'us-east-1', 'execute-api', session_token = os.environ['AWS_SESSION_TOKEN']) 98 | r = requests.post(url, auth=auth, json=json_root) 99 | print(r.status_code) 100 | --------------------------------------------------------------------------------