├── 00-ETL-Examples └── lab-01.sql ├── 00-Three-Layer └── images │ ├── Screenshot 2020-05-16 at 2.18.54 PM.png │ ├── Screenshot 2020-05-16 at 2.19.19 PM.png │ ├── Screenshot 2020-05-16 at 2.22.41 PM.png │ ├── Screenshot 2020-05-16 at 5.02.53 PM.png │ └── Screenshot 2020-05-16 at 5.03.34 PM.png ├── 01-database-table-view ├── db-tables-views.sql └── images │ ├── Screenshot 2020-05-18 at 1.12.10 PM.png │ ├── Screenshot 2020-05-18 at 1.13.03 PM.png │ ├── Screenshot 2020-05-18 at 1.52.08 PM.png │ ├── Screenshot 2020-05-18 at 12.28.00 PM.png │ ├── Screenshot 2020-05-18 at 12.30.39 PM.png │ ├── Screenshot 2020-05-18 at 12.35.38 PM.png │ ├── Screenshot 2020-05-18 at 12.38.36 PM.png │ ├── Screenshot 2020-05-18 at 12.43.49 PM.png │ ├── Screenshot 2020-05-18 at 12.49.45 PM.png │ ├── Screenshot 2020-05-18 at 12.59.52 PM.png │ └── Screenshot 2020-05-18 at 2.03.30 PM.png ├── 011-WebUI └── images │ ├── Screenshot 2020-05-21 at 9.34.49 AM.png │ ├── Screenshot 2020-05-21 at 9.35.29 AM.png │ ├── Screenshot 2020-05-21 at 9.36.42 AM.png │ └── Screenshot 2020-05-21 at 9.37.22 AM.png ├── 02-data-type-in-snowflake ├── data-type.sql └── images │ ├── Screenshot 2020-05-19 at 10.05.05 AM.png │ ├── Screenshot 2020-05-19 at 10.05.19 AM.png │ ├── Screenshot 2020-05-19 at 10.08.21 AM.png │ ├── Screenshot 2020-05-19 at 10.10.57 AM.png │ ├── Screenshot 2020-05-19 at 10.13.42 AM.png │ ├── Screenshot 2020-05-19 at 10.15.07 AM.png │ ├── Screenshot 2020-05-19 at 10.15.21 AM.png │ ├── Screenshot 2020-05-19 at 10.17.24 AM.png │ ├── Screenshot 2020-05-19 at 10.25.09 AM.png │ ├── Screenshot 2020-05-19 at 10.25.34 AM.png │ ├── Screenshot 2020-05-19 at 6.31.42 PM.png │ ├── Screenshot 2020-05-19 at 6.32.13 PM.png │ ├── Screenshot 2020-05-19 at 6.32.20 PM.png │ ├── Screenshot 2020-05-19 at 6.48.09 PM.png │ ├── Screenshot 2020-05-19 at 8.57.21 AM.png │ ├── Screenshot 2020-05-19 at 9.03.07 AM.png │ ├── Screenshot 2020-05-19 at 9.12.24 AM.png │ ├── Screenshot 2020-05-19 at 9.12.55 AM.png │ ├── Screenshot 2020-05-19 at 9.15.22 AM.png │ ├── Screenshot 2020-05-19 at 9.34.09 AM.png │ ├── Screenshot 2020-05-19 at 9.43.09 AM.png │ ├── Screenshot 2020-05-19 at 9.53.48 AM.png │ └── Screenshot 2020-05-19 at 9.54.40 AM.png ├── 03-variant-data-type └── variant-data-type.sql ├── 05-data-unloading-export ├── data-unloading-snowflake.sql └── images │ ├── Screenshot 2020-05-14 at 1.35.49 PM.png │ ├── Screenshot 2020-05-14 at 1.36.26 PM.png │ ├── Screenshot 2020-05-14 at 1.39.18 PM.png │ ├── Screenshot 2020-05-19 at 10.05.05 AM.png │ ├── Screenshot 2020-05-19 at 10.05.19 AM.png │ ├── Screenshot 2020-05-19 at 10.08.21 AM.png │ ├── Screenshot 2020-05-19 at 10.10.57 AM.png │ ├── Screenshot 2020-05-19 at 10.13.42 AM.png │ ├── Screenshot 2020-05-19 at 10.15.07 AM.png │ ├── Screenshot 2020-05-19 at 10.15.21 AM.png │ ├── Screenshot 2020-05-19 at 10.17.24 AM.png │ ├── Screenshot 2020-05-19 at 10.25.09 AM.png │ └── Screenshot 2020-05-19 at 10.25.34 AM.png ├── 06-snowflake-security-overview └── snowflake-security.sql ├── 09-data-cloning ├── data-cloning.sql └── images │ ├── Screenshot 2020-05-13 at 2.15.57 PM.png │ ├── Screenshot 2020-05-13 at 2.16.31 PM.png │ ├── Screenshot 2020-05-13 at 2.17.18 PM.png │ ├── Screenshot 2020-05-13 at 2.23.17 PM.png │ ├── Screenshot 2020-05-13 at 2.25.24 PM.png │ ├── Screenshot 2020-05-13 at 2.27.42 PM.png │ ├── Screenshot 2020-05-13 at 2.30.34 PM.png │ └── Screenshot 2020-05-13 at 2.34.27 PM.png ├── 10-account-usage ├── account-usage-snowflake.sql └── images │ ├── Screenshot 2020-05-12 at 2.12.30 PM.png │ ├── Screenshot 2020-05-12 at 2.12.43 PM.png │ ├── Screenshot 2020-05-12 at 2.13.03 PM.png │ ├── Screenshot 2020-05-12 at 2.36.01 PM.png │ ├── Screenshot 2020-05-12 at 2.37.11 PM.png │ ├── Screenshot 2020-05-12 at 2.40.56 PM.png │ ├── Screenshot 2020-05-12 at 2.42.55 PM.png │ ├── Screenshot 2020-05-12 at 2.46.00 PM.png │ ├── Screenshot 2020-05-12 at 2.50.53 PM.png │ ├── Screenshot 2020-05-12 at 2.52.05 PM.png │ ├── Screenshot 2020-05-12 at 2.52.59 PM.png │ └── Screenshot 2020-05-12 at 2.54.50 PM.png ├── 11-user-role-creation ├── images │ ├── Screenshot 2020-05-16 at 1.33.11 PM.png │ ├── Screenshot 2020-05-16 at 1.38.05 PM.png │ ├── Screenshot 2020-05-16 at 1.40.04 PM.png │ └── Screenshot 2020-05-16 at 1.45.40 PM.png └── user-role-snowflake.sql ├── README.md ├── grant-sql-example └── database-schema-table-grant.sql ├── images └── snowflake-name-stages-internal-external │ ├── Screenshot 2020-05-05 at 1.06.24 PM.png │ ├── Screenshot 2020-05-05 at 1.07.07 PM.png │ ├── Screenshot 2020-05-05 at 1.07.35 PM.png │ ├── Screenshot 2020-05-05 at 1.08.13 PM.png │ ├── Screenshot 2020-05-05 at 1.09.50 PM.png │ ├── Screenshot 2020-05-05 at 1.21.09 PM.png │ ├── Screenshot 2020-05-05 at 1.21.50 PM.png │ ├── Screenshot 2020-05-05 at 1.25.53 PM.png │ ├── Screenshot 2020-05-05 at 1.34.13 PM.png │ ├── Screenshot 2020-05-05 at 1.36.20 PM.png │ ├── Screenshot 2020-05-05 at 1.42.59 PM.png │ ├── Screenshot 2020-05-05 at 1.43.16 PM.png │ ├── Screenshot 2020-05-05 at 1.43.33 PM.png │ ├── Screenshot 2020-05-05 at 1.43.45 PM.png │ ├── Screenshot 2020-05-05 at 2.01.54 PM.png │ ├── Screenshot 2020-05-05 at 2.04.48 PM.png │ └── Screenshot 2020-05-05 at 2.10.30 PM.png ├── json-data-load-example.sql ├── role-sql-example └── role-assignment-snowflake.sql └── snowlfake-named-stages-internal-external.sql /00-ETL-Examples/lab-01.sql: -------------------------------------------------------------------------------- 1 | -- Lab 01 2 | 3 | -- Create a data base 4 | 5 | CREATE DATABASE TIPS_SALES_PROD 6 | COMMENT = 'This is topper tips sales production database'; 7 | 8 | 9 | -- Create Schema 10 | CREATE SCHEMA "TIPS_SALES_PROD"."SALES" 11 | COMMENT = 'This is TopperTips Sales Data'; 12 | 13 | -- Created a names internal stage 14 | CREATE STAGE "TIPS_SALES_PROD"."SALES".SALES_STAGE 15 | COMMENT = 'This is named internal SF manage stage'; 16 | 17 | --Creating a stage via clone (this operation may not work with WebUI) 18 | CREATE STAGE TIPS_SALES_PROD.SALES.S3 19 | CLONE USDA_NUTRIENT_STDREF.PUBLIC.S3_USDF_EXTERNAL_NAMED_STAGE 20 | COMMENT = 'Cloning USDF S3 Bucket'; 21 | 22 | 23 | -- Creating a part table 24 | CREATE TABLE TIPS_SALES_PROD.SALES.PART 25 | CLONE SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.PART 26 | COMMENT = 'This is cloned table from TPC AdHoc SF1'; 27 | 28 | -- SQL compilation error: Cannot clone from a table that was imported from a share. 29 | 30 | -- alternative is -- use create like 31 | CREATE TABLE TIPS_SALES_PROD.SALES.PART LIKE SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.PART 32 | COMMENT = 'This is create like table from TPC AdHoc SF1'; 33 | 34 | -- then insert 35 | insert into TIPS_SALES_PROD.SALES.PART select * from SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.PART 36 | 37 | 38 | https://zoom.us/j/7299924465 or Conference Room -------------------------------------------------------------------------------- /00-Three-Layer/images/Screenshot 2020-05-16 at 2.18.54 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/00-Three-Layer/images/Screenshot 2020-05-16 at 2.18.54 PM.png -------------------------------------------------------------------------------- /00-Three-Layer/images/Screenshot 2020-05-16 at 2.19.19 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/00-Three-Layer/images/Screenshot 2020-05-16 at 2.19.19 PM.png -------------------------------------------------------------------------------- /00-Three-Layer/images/Screenshot 2020-05-16 at 2.22.41 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/00-Three-Layer/images/Screenshot 2020-05-16 at 2.22.41 PM.png -------------------------------------------------------------------------------- /00-Three-Layer/images/Screenshot 2020-05-16 at 5.02.53 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/00-Three-Layer/images/Screenshot 2020-05-16 at 5.02.53 PM.png -------------------------------------------------------------------------------- /00-Three-Layer/images/Screenshot 2020-05-16 at 5.03.34 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/00-Three-Layer/images/Screenshot 2020-05-16 at 5.03.34 PM.png -------------------------------------------------------------------------------- /01-database-table-view/db-tables-views.sql: -------------------------------------------------------------------------------- 1 | -- --------------------- 2 | -- Important Notes 3 | -- --------------------- 4 | /* 5 | 1. No limit of db, db object in SF 6 | 2. default table is permanent table (temp, transient, external are other table types) 7 | 3. Non-permanent & transitory data can be stored in temporary & transient tables (short term data storage) 8 | 4. temporary & transient tables also called session specific data sets. 9 | 5. Temporary tables scope within the session in which they were created. 10 | (so where the data is stored during the session) 11 | - they don't support cloning 12 | - not accessible via other sessions 13 | - not recoverable (no fail safe) when session ends. 14 | - SF also support temporary stages (other objects too) 15 | - tmp data is part of storage cost and charges apply 16 | - tmp table data can live more than 24hrs if session is alive and it cost storage charges. 17 | - tmp table and permanent/transient table can have same name 18 | - tmp table gets precedence over permanent & transient table 19 | - this behaviour is given due to time travel (duplicate table name) 20 | - tmp tables can not be converted to any other table once created 21 | - retention period (time travel) is for 24 hours or the remainder of the session, whichever is shorter. 22 | 23 | 6. Transient Table 24 | - There is a field in table (view) called is_transient 25 | - Persist until explicitly dropped 26 | - available to all users with privileges 27 | - same as permanent table except fail safe 28 | - contribute to the overall storage & no fail safe cost 29 | - transient database and schemas are also possible 30 | - schemas created under transient db and/or schema are by default transient. 31 | - once created, can not be converted to other table type. 32 | - it support clone 33 | 34 | 35 | 7. Compare 36 | 37 | Type Persistent Cloning TimeTravel FailSafe(DR) 38 | Permanent(Std) Until Dropped Yes 0 or 1 7 39 | Permanent(Ent+) Until Dropped Yes 0 or 90 7 40 | Transient Until Dropped Yes 0 or 1 0 41 | Temporary Session Level Yes 0 or 1 0 42 | 43 | 8. timetravel can be specified for the table during creation or later via alter 44 | 9. The Fail-safe period is not configurable for any table type. 45 | 10. Transient and temporary tables have no Fail-safe period 46 | 11. External Table (ET) 47 | - s3, azure, google cloud storage 48 | - External tables store file-level metadata (file path, a version identifier, and partitioning information) 49 | - External tables support external (i.e. S3, Azure, or GCS) stages only; internal (i.e. Snowflake) stages are not supported. 50 | - Schema on read 51 | - No DML operation is allowed 52 | - External table can be used to perform join operation 53 | - Views can be created onn external table 54 | - it is bit slower than the native SF tables 55 | - materialized views on external table improves performance (Enterprise+ editionn) 56 | - Every ET has a column named VALUE of type VARIANT. Additional columns might be specified. All of the columns are treated as virtual columns. 57 | * value (a varient type) - represent single row in external file 58 | * METADATA$FILENAME - a metadata column 59 | - METADATA$FILENAME is a pseudocolumn, identifies stage data file and its path 60 | - virtual columns - it can be created using value and psedocolumn 61 | - No referential integrity constants on external tables are enforced by Snowflake 62 | - The default VALUE and METADATA$FILENAME columns cannot be dropped. 63 | - Not supported **** 64 | * Clustering Key 65 | * Cloning 66 | * Data Sharing 67 | * Data in XML format 68 | * Time travel 69 | - Recommendation for external table 70 | * partition the table 71 | * 72 | 73 | 12. Create External Table need following access 74 | - Database - usage 75 | - schema - usage/create stage/create external table 76 | - stage - usage 77 | */ 78 | 79 | -- Sample code to create a temp table and insert data .. all worked well 80 | use database TIPS_SALES_PROD; 81 | use schema public; 82 | create temporary table t1 (f1 numeric); 83 | desc table t1; 84 | insert into t1 select p_size from part_expoert limit 100; 85 | select * from t1; 86 | drop table t1; -- this also works 87 | 88 | -- if you run a show table, it shows the temporary tables too (refer image) 89 | 90 | -- if you try to run following command on temporary table, you will get error 91 | list @%t1 92 | SQL compilation error: Stage 'TIPS_SALES_PROD.PUBLIC."%T1"' does not exist or not authorized. 93 | 94 | 95 | 96 | drop table t2 97 | create transient table t2 (parts_name varchar(1000)); --Table T2 successfully created. 98 | insert into t2 select p_name from PART_EXPORT limit 1000; -- record inserted 99 | select * from t2 limit 100; 100 | show tables; 101 | list @%t2; --this works 102 | 103 | 104 | -- duplicate table name (refer images) 105 | select * from PART_EXPORT limit 10; --this is a permanent table 106 | create temporary table PART_EXPORT(parts_key number(38,0), parts_name varchar(55)); -- this works 107 | create transient table PART_EXPORT(parts_key number(38,0), parts_name varchar(55)); -- this does not 108 | drop table PART_EXPORT; -- this will drop the tmp table 109 | show tables; 110 | 111 | -- how to grant access to a database 112 | GRANT USAGE, CREATE SCHEMA ON DATABASE "TIPS_DATABASE" TO ROLE "USERADMIN"; 113 | 114 | -- --------------------- 115 | -- External Table Examples 116 | -- --------------------- 117 | -- additional key param is required 118 | create or replace stage mystage url='s3://mybucket/files/'; -- aws 119 | create or replace stage mystage url='gcs://mybucket/files'; -- gcp 120 | create or replace stage mystage url='azure://myaccount.blob.core.windows.net/mycontainer/files'; -- azure 121 | -- blob.core.windows.net endpoint for all supported types of Azure blob storage accounts, including Data Lake Storage Gen2. 122 | 123 | -- create external table with stage reference 124 | -- (Tip - create an internal stage reference and try to play with external table what happens) 125 | 126 | 127 | -- desc external table 128 | desc external table emp; 129 | 130 | -- show external table 131 | show external tables like 'line%' in tpch.public; 132 | 133 | -- manually refresh external table 134 | alter external table exttable_json refresh; 135 | 136 | -- Similar to the first example, but manually refresh only a path of the metadata for an external table: 137 | create or replace stage mystage 138 | url='://twitter_feed/logs/' 139 | .. ; 140 | 141 | -- Create the external table 142 | -- 'daily' path includes paths in format 143 | create or replace external table daily_tweets 144 | with location = @twitter_feed/daily/; 145 | 146 | -- Refresh the metadata for a single day of data files by date 147 | alter external table exttable_part refresh '2018/08/05/'; 148 | 149 | -- Add an explicit list of files to the external table metadata: 150 | alter external table exttable1 add files ('path1/sales4.json.gz', 'path1/sales5.json.gz'); 151 | 152 | --Remove an explicit list of files from the external table metadata: 153 | alter external table exttable1 remove files ('path1/sales4.json.gz', 'path1/sales5.json.gz'); 154 | 155 | -- drop external table 156 | drop external table t2; 157 | 158 | -- --------------------- 159 | /* 160 | Important Links 161 | 1. https://docs.snowflake.com/en/user-guide/databases.html 162 | 2. https://docs.snowflake.com/en/user-guide/tables-temp-transient.html#comparison-of-table-types 163 | 3. https://docs.snowflake.com/en/user-guide/tables-external.html 164 | 4. https://docs.snowflake.com/en/user-guide/tables-external-intro.html 165 | 5. https://docs.snowflake.com/en/sql-reference/sql/desc-external-table.html 166 | 6. https://docs.snowflake.com/en/sql-reference/sql/drop-external-table.html 167 | */ 168 | -- --------------------- 169 | 170 | 171 | /* 172 | https://s3.amazonaws.com/snowflake-docs/tutorials/json/server/2.6/2016/07/15/15/json_tutorial.json 173 | { 174 | "device_type": "server", 175 | "events": [ 176 | { 177 | "f": 83, 178 | "rv": "15219.64,783.63,48674.48,84679.52,27499.78,2178.83,0.42,74900.19", 179 | "t": 1437560931139, 180 | "v": { 181 | "ACHZ": 42869, 182 | "ACV": 709489, 183 | "DCA": 232, 184 | "DCV": 62287, 185 | "ENJR": 2599, 186 | "ERRS": 205, 187 | "MXEC": 487, 188 | "TMPI": 9 189 | }, 190 | "vd": 54, 191 | "z": 1437644222811 192 | }, 193 | { 194 | "f": 1000083, 195 | "rv": "8070.52,54470.71,85331.27,9.10,70825.85,65191.82,46564.53,29422.22", 196 | "t": 1437036965027, 197 | "v": { 198 | "ACHZ": 6953, 199 | "ACV": 346795, 200 | "DCA": 250, 201 | "DCV": 46066, 202 | "ENJR": 9033, 203 | "ERRS": 615, 204 | "MXEC": 0, 205 | "TMPI": 112 206 | }, 207 | "vd": 626, 208 | "z": 1437660796958 209 | } 210 | ], 211 | "version": 2.6 212 | } 213 | */ -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 1.12.10 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 1.12.10 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 1.13.03 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 1.13.03 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 1.52.08 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 1.52.08 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 12.28.00 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 12.28.00 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 12.30.39 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 12.30.39 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 12.35.38 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 12.35.38 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 12.38.36 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 12.38.36 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 12.43.49 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 12.43.49 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 12.49.45 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 12.49.45 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 12.59.52 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 12.59.52 PM.png -------------------------------------------------------------------------------- /01-database-table-view/images/Screenshot 2020-05-18 at 2.03.30 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/01-database-table-view/images/Screenshot 2020-05-18 at 2.03.30 PM.png -------------------------------------------------------------------------------- /011-WebUI/images/Screenshot 2020-05-21 at 9.34.49 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/011-WebUI/images/Screenshot 2020-05-21 at 9.34.49 AM.png -------------------------------------------------------------------------------- /011-WebUI/images/Screenshot 2020-05-21 at 9.35.29 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/011-WebUI/images/Screenshot 2020-05-21 at 9.35.29 AM.png -------------------------------------------------------------------------------- /011-WebUI/images/Screenshot 2020-05-21 at 9.36.42 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/011-WebUI/images/Screenshot 2020-05-21 at 9.36.42 AM.png -------------------------------------------------------------------------------- /011-WebUI/images/Screenshot 2020-05-21 at 9.37.22 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/011-WebUI/images/Screenshot 2020-05-21 at 9.37.22 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/data-type.sql: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | -- Summary of data type 7 | /* 8 | Legend 9 | ------- 10 | <***IMP***> : This is bit different in Snowflake over other Database system 11 | 12 | Snowflake has its internal way of storing data, so on surface it support many style but many of them are just alias or same like int and decimal are all numbers 13 | 14 | 1. Numeric - Number (38,0) (rest all numbers are stored with this data type no matter what definition you give) 15 | = decimal,float,real, int, big int, etc... 16 | 17 | 2. String/Binary Data Type 18 | = string/text/varchar 19 | = char -> varchar(1) 20 | = Binary = varbinary 21 | 22 | 3. Logical Data Type = Boolean (account supported post 25 Jan 2016) 23 | 4. Date/Time = Date 24 | Time 25 | DateTime 26 | TimeStamp 27 | TimeStamp_LTZ (Local Time Zone) 28 | TimeStamp_NTZ (with no timezone) 29 | TimeStamp_TZ (with timezone) 30 | 31 | 5. Semi Structured - Variant 32 | Object 33 | Array 34 | 35 | Note : Snowflake displays FLOAT, FLOAT4, FLOAT8, REAL, DOUBLE, and DOUBLE PRECISION as FLOAT 36 | 37 | 38 | Key Note about Numeric Data Type 39 | - Numeric Data types are fixed point (int, bigint etc), floating point (float/double), constant/literal 40 | - Number up to 38 with optional precision 41 | - Precition - total number of digit allowed 42 | - scale - totla # of digit right to the decimal point (37) 43 | - data converted from higher to lower and back to higher, it will lose precision 44 | - Example : convert a NUMBER(38,37) to DOUBLE (has 17 prevision), and then back to NUMBER, it lose its precision 45 | - Precision (total number of digits) does not impact storage <***IMP***> 46 | - SF Micropartition determine min/max precision for given column and then store <***IMP***> 47 | - Example: col values 1,2,3,4,5 -> 1 byte uncompressed 48 | col values 1,2,3,100000 -> 4 byte uncompressed 49 | - scale (number of digits following the decimal point) does have an impact on storage <***IMP***> 50 | - column of type NUMBER(10,5) consumes more space than NUMBER(5,0) + processing cost is higher <***IMP***> 51 | 52 | 53 | Key Note about Floating Point Numbers 54 | - SF uses double-precision (64 bit) IEEE 754 floating point numbers. 55 | - SF Supports following special values for Float 56 | 'NaN' -> Not a number 57 | 'inf' -> infinity 58 | '-inf' -> negative infinity 59 | ** these symbols must be single quote 60 | - IEEE 745 vs SF when apply condition check 61 | if( 'NaN' = 'NaN') .. in SF it is true but not in IEEE 62 | IF('NaN' > X) ........in SF it is true but not in IEEE 63 | 64 | Constant/Literals for Numbers 65 | - e/E indicate exponent (At least one digit must follow the exponent marker if present.) 66 | - Example 1.234E2 , 1.234E+2 , 15e-03 67 | - +/- indicate postive or negative, default is + 68 | 69 | 70 | Key Note about String & Binary Datatype 71 | - VARCHAR holds unicode characters. 72 | - max length is 16 MB (uncompressed) (single byte 16,777,216 and 2bypte/4byte per char - 8,388,608 to 4,194,304) 73 | - If a length is not specified, the default is the maximum length. <***IMP***> 74 | - char/character same as varchar and not defined it is 1 default. 75 | - for char datatype, no padding givein like standard database. 76 | 77 | Key Note about Binary 78 | - Max length = 8Mb 79 | - No notion of unicode charater, so length is always same. 80 | - varbinary = binary 81 | - internal representation - The BINARY data type holds a sequence of 8-bit bytes. 82 | - HELP is stored and will display as 48454C50 (H=48 in ascii) 83 | 84 | String Constant 85 | - String constants in SF must always be enclosed between delimiter characters. 86 | - SF supports using either single quotes or dollar signs to delimit string constants <***IMP***> 87 | Tip: in standard database, single quote is used for constant in select statement 88 | - select 'today' as today, $$t o d a y$$ as today_with_dollar 89 | - select $$wow \t wow$$ as today 90 | - Two single quotes is not the same as the double quote character 91 | - $$string with a ' character$$ 92 | - Note that the string constant cannot contain double-dollar signs. <***IMP***> 93 | 94 | Boolean Data Type & Key Note 95 | - BOOLEAN can have TRUE/FALSE values. 96 | - BOOLEAN can also have an “unknown” value, which is represented by NULL 97 | - Boolean Conversionn 98 | * String 99 | * true/t/yes/y/on/1 = True 100 | * false/f/no/f/off/0 = False 101 | * 102 | * Numeric 103 | * 0 - zero is false 104 | * any non zero (+/-) is true 105 | - String/Numeric convertionnn 106 | * to_string(True) -> true (lower case) -> 1 (for number) 107 | * to_string(False) -> false (lower case) -> 0 (for number) 108 | - Null is converted to NULL 109 | 110 | Quik Review - Ternary Logic - three-valued logic (3VL) 111 | - logic system with three truth values: TRUE, FALSE, and UNKNOWN 112 | - Snowflake, UNKNOWN is represented by NULL 113 | - When used in expressions (e.g. SELECT list), UNKNOWN results are returned as NULL values. 114 | - When used as a predicate (e.g. WHERE clause), UNKNOWN results evaluate to FALSE. 115 | 116 | Date/time key notes 117 | - single DATE data type for storing dates (with no time elements) 118 | - DATE supported formats for string constants (YYYY-MM-DD, DD-MON-YYYY, etc.) 119 | - When a date-only format is used, the associated time is assumed to be midnight on that day. 120 | - timestap = combined date + time 121 | - DATETIME is an alias for TIMESTAMP_NTZ 122 | - all accepted timestamps are valid inputs for dates 123 | -TIME data type for storing times in the form of HH:MI:SS 124 | - TIME supports an optional precision parameter for fractional seconds, e.g. TIME(3) 125 | - Time precision can range from 0 (seconds) to 9 (nanoseconds). The default precision is 9 126 | - All TIME values must be between 00:00:00 and 23:59:59.999999999 127 | - TIME internally stores “wallclock” time, and all operations on TIME values are performed without taking any time zone into consideration 128 | 129 | calendar support 130 | -Snowflake uses the Gregorian Calendar for all dates and timestamps 131 | 132 | Date and Time Constants 133 | - Snowflake supports using string constants to specify fixed date, time, or timestamp values 134 | - String constants must always be enclosed between delimiter characters 135 | - example 136 | date '2010-09-14' 137 | time '10:03:56' 138 | timestamp '2009-09-15 10:59:43' 139 | 140 | create table t1 (d1 date); 141 | insert into t1 (d1) values (date '2011-10-29'); 142 | value format can be set as alter session set date-input-format ='yyyy-MM-dd' 143 | 144 | Interval Constants 145 | -interval constants to add/subtract a period of time to/from a date, time, or timestamp 146 | - Example 147 | INTERVAL '1 YEAR' represents 1 year. 148 | INTERVAL '4 years, 5 months, 3 hours' represents 4 years, 5 months, and 3 hours 149 | INTERVAL '1 year, 1 day' first adds/subtracts a year and then a day. 150 | INTERVAL '1 day, 1 year' first adds/subtracts a day and then a year. 151 | -SQL 152 | select to_date ('2019-02-28') + INTERVAL '1 day, 1 year'; -- result 2020-03-01 153 | select to_date ('2019-02-28') + INTERVAL '1 year, 1 day'; -- result 2020-02-29 154 | - INTERVAL is not a data type and can be used only with date/time 155 | 156 | -- More example 157 | select to_date('2018-04-15') + INTERVAL '1 year'; -- 2019-04-15 158 | select to_time('04:15:29') + INTERVAL '3 hours, 18 minutes'; --07:33:29 159 | select name, hire_date from employees where hire_date > current_date - INTERVAL '2 y, 3 month'; 160 | select to_date('2025-01-17') + INTERVAL 161 | '1 y, 3 q, 4 mm, 5 w, 6 d, 7 h, 9 m, 8 s, 162 | 1000 ms, 445343232 us, 898498273498 ns' 163 | as complex_interval2; 164 | 165 | -- 2027-03-30 07:31:32.841 166 | 167 | Simple Arithmetic for Dates 168 | - TIME and TIMESTAMP values do not yet support simple arithmetic. 169 | - select to_date('2018-04-15') + 1; -- 2018-04-16 170 | -- select to_date('2018-04-15') - 4; -- 2018-04-11 171 | 172 | */ 173 | 174 | -- setting the timezone 175 | alter session set timestamp_type_mapping = timestamp_ntz; 176 | create or replace table ts_test(ts timestamp); 177 | desc table ts_test; 178 | 179 | create or replace table ts_test2(ts timestamp_ltz); 180 | desc table ts_test2; 181 | 182 | Use TIMESTAMP_LTZ with different time zones: 183 | /* 184 | Important links: 185 | 1.https://docs.snowflake.com/en/sql-reference/data-types.html 186 | 2. https://docs.snowflake.com/en/sql-reference/data-types-numeric.html 187 | 3. https://docs.snowflake.com/en/sql-reference/data-types-text.html 188 | 4. https://docs.snowflake.com/en/user-guide/binary-input-output.html 189 | 5. https://docs.snowflake.com/en/sql-reference/data-types-logical.html 190 | 6. https://docs.snowflake.com/en/sql-reference/ternary-logic.html 191 | */ -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.05.05 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.05.05 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.05.19 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.05.19 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.08.21 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.08.21 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.10.57 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.10.57 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.13.42 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.13.42 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.15.07 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.15.07 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.15.21 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.15.21 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.17.24 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.17.24 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.25.09 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.25.09 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.25.34 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 10.25.34 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.31.42 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.31.42 PM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.32.13 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.32.13 PM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.32.20 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.32.20 PM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.48.09 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 6.48.09 PM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 8.57.21 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 8.57.21 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.03.07 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.03.07 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.12.24 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.12.24 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.12.55 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.12.55 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.15.22 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.15.22 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.34.09 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.34.09 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.43.09 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.43.09 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.53.48 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.53.48 AM.png -------------------------------------------------------------------------------- /02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.54.40 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/02-data-type-in-snowflake/images/Screenshot 2020-05-19 at 9.54.40 AM.png -------------------------------------------------------------------------------- /03-variant-data-type/variant-data-type.sql: -------------------------------------------------------------------------------- 1 | -- ------------------------------ 2 | -- Variant/Array/Object Data Type 3 | -- ------------------------------ 4 | 5 | Introduction Key Note 6 | --------------------- 7 | Semi-structure data type: used to represent arbitrary data structures 8 | Can be used to import/operate on semi-structured data 9 | JSON 10 | Avro 11 | ORC 12 | Parquet 13 | XML 14 | Snowflake stores these types internally in an efficient compressed columnar binary representation 15 | 16 | Variant 17 | ------- 18 | universal type, can store values of any other type, including OBJECT and ARRAY 19 | maximum size of 16 MB compressed 20 | A value of any data type can be implicitly cast to a VARIANT value, subject to size restrictions 21 | example : (var:json_path >= 56) is cast to (var:json_path >= 56::VARIANT) for compare 22 | VARIANT value can be missing (contain SQL NULL), which is different from a VARIANT null value 23 | ** VARIANT columns in a relational table are stored as separate physical columns 24 | 25 | 26 | Object 27 | ------- 28 | Used to represent collections of key-value pairs, 29 | key is a non-empty string, and the value is a value of VARIANT type 30 | Snowflake does not currently support explicitly-typed objects. 31 | 32 | Array 33 | ------ 34 | Used to represent dense or sparse arrays of arbitrary size 35 | 36 | Unsupported Data Type 37 | ---------------------- 38 | LOB (Large Object) - BINARY can be used instead 39 | CLOB - VARCHAR can be used instead; 40 | ENUM 41 | User-defined data types 42 | 43 | 44 | Querying Semi-structured Data 45 | ------------------------------ 46 | Snowflake supports SQL queries that access semi-structured data 47 | JSON, Avro, ORC, and Parquet data (but not to xml) 48 | Traverse Pattern Insert a colon : between the VARIANT column name and any first-level element 49 | :. 50 | Query output is enclosed in double quotes because the query output is VARIANT, not VARCHAR 51 | Operators : and subsequent . and [] always return VARIANT values containing strings 52 | There are two ways to access elements in a JSON object: 53 | Dot Notation 54 | Bracket Notation 55 | Regardless of which notation you use, the column name is case-insensitive but element names are case-sensitive 56 | src:salesperson.name 57 | SRC:salesperson.name 58 | SRC:Salesperson.Name 59 | Dot notation 60 | select src:salesperson.name from car_sales; 61 | Bracker notation -- Enclose element names in single quotes 62 | select src['salesperson']['name'] from car_sales; 63 | Repeating elements 64 | select src:vehicle[0] from car_sales; 65 | select src:vehicle[0].price from car_sales; 66 | Explicitly Casting 67 | select src:salesperson.id::string from car_sales; 68 | FLATTEN Function to Parse Arrays 69 | FLATTEN is a table function that produces a lateral view of a VARIANT, OBJECT, or ARRAY 70 | Example: 71 | select 72 | value:name::string as "Customer Name", 73 | value:address::string as "Address" 74 | from 75 | car_sales 76 | , lateral flatten(input => src:customer); 77 | /* 78 | Important Links 79 | 1. https://docs.snowflake.com/en/sql-reference/data-types-semistructured.html 80 | 2. https://docs.snowflake.com/en/sql-reference/data-types-semistructured.html 81 | */ -------------------------------------------------------------------------------- /05-data-unloading-export/data-unloading-snowflake.sql: -------------------------------------------------------------------------------- 1 | -- Data Unloading & Export Function 2 | /* 3 | We all know that Snowflake is cloud native warehouse system, data unloading and export function cost. 4 | Hence it is very important to understand the data unloading practices and follow the guidelines. 5 | There are various ways data can be exported effectively (like cloud resion, buckets, local box etc) and following topics will help us to explore them in detail. 6 | 7 | Key concepts related to data unloading, as well as best practices. 8 | Overview of Data Unloading 9 | Summary of Data Unloading Features 10 | Data Unloading Considerations 11 | 12 | Overview of supported data file formats for unloading data. 13 | Preparing to Unload Data 14 | 15 | Detailed instructions for unloading data in bulk using the COPY command. 16 | Unloading into a Snowflake Storage 17 | Unloading into Amazon S3 18 | Unloading into Google Cloud Storage 19 | Unloading into Microsoft Azure 20 | */ 21 | 22 | /* 23 | 24 | -- Important links 25 | 1. https://docs.snowflake.com/en/user-guide-data-unload.html 26 | 2. https://docs.snowflake.com/en/user-guide/data-unload-overview.html 27 | 3. https://docs.snowflake.com/en/user-guide/intro-summary-unloading.html 28 | 4. https://docs.snowflake.com/en/user-guide/data-unload-considerations.html 29 | 5. https://docs.snowflake.com/en/user-guide/data-unload-snowflake.html (good one) 30 | 31 | -- Related Links 32 | 1. Data loading guide - https://docs.snowflake.com/en/user-guide-data-load.html 33 | 2. Loading Unloading DDLs - https://docs.snowflake.com/en/sql-reference/ddl-stage.html 34 | 3. DML commands - https://docs.snowflake.com/en/sql-reference/sql-dml.html 35 | 36 | */ 37 | 38 | /* 39 | Important Faqs which will be clarified later 40 | 1. Can data be loaded in different file formats like json/parque etc? or it is always csv? 41 | 2. Data can only be unloaded to stages? External/Internal? 42 | 3. What are the file size or naming patter or overwirte strategy to the stage location? 43 | 4. How does it chunk the file - what is the strategy? 44 | 4. How to download data from many tables using query? Is the COPY INTO only option? 45 | 5. Unloaded/exported data to internal or external stages are cloned when clone operation is performed - to be tried 46 | 6. 47 | */ 48 | 49 | /* 50 | Points to remember 51 | 1. Snowflake supports bulk export (i.e. unload) of data from a database table into flat, delimited text files 52 | 2. Use 'COPY INTO ' command (similar to load) the data from the Snowflake database table into one or more files in a Snowflake or external stage. 53 | (Tip: it is alwasy stages?) 54 | 3. Use the get command to download the file. 55 | (Tip: so it is 2 stage process.. like a actual warehouse.. get the item at stage area and then get it in your truck) 56 | 4. Select query can also be used instead of 'COPY INTO ' 57 | (Tip: look for code syntax) 58 | 5. Select query support full syntax including JSON style to download. 59 | 6. 'COPY INTO ' command provides option to specify single file or multiple (default SINGLE = FALSE) 60 | 7. Snowflake provides each file a unique name 61 | (Tip: try this out to reinformce the understanding and file name patters) 62 | 8. Snowflake prefixes the generated filenames with "data_". 63 | 9. Snowflake appends a suffix that ensures each file name is unique across parallel execution threads; e.g. data_stats_0_1_0. 64 | 10. MAX_FILE_SIZE copy option to specify each file 65 | (Tip: mb or gb or what?) 66 | 11. Compression/Encryption is decideds if stage is external or internal. 67 | 12. Location of stages - Internal (local), AWS s3, GCP bucket, Azure file 68 | 13. File formats - delited files (CSV/TSV), JSON, Parquet 69 | (Tip: not supported ORC, xml) 70 | 14. File encoding - UTF-8 71 | 15. Compression - gzip (default)/bzip2/Brotil/ZStandard 72 | 16. File Encryption - for internal stages - 128-bit (256-bit need configuration) 73 | 17. File Encryption - for external - user supplied key (possible only if it is provided) 74 | 75 | 76 | Super Important Notes 77 | 1. An empty string is a string with zero length or no characters, whereas NULL values represent an absence of data 78 | 2. For CSV (delimited files) 79 | 2.1 An empty string - ,'',, to indicate that the string contains zero characters 80 | 2.2 a NULL value is typically represented by two successive delimiters (e.g. ,,) 81 | (Tip: get your hands on to reinforce theory and also try with TSV) 82 | 3. FIELD_OPTIONALLY_ENCLOSED_BY = 'character' | NONE - use this option to define behaviour (PREFERRED ONE****) 83 | 4. EMPTY_FIELD_AS_NULL = TRUE|FALSE option set to FALSE 84 | 5. NULL_IF = ('',[,'']) : When unloading data from tables: Snowflake converts SQL NULL values to the first value in the list. 85 | (Tip: see all the option in play .. refer images) 86 | 6. ESCAPE_UNENCLOSED_FIELD = \\N (default) 87 | */ 88 | 89 | 90 | 91 | -- Setting the param for ui session with 92 | -- worksheet to work with specific role/warehouse/db/schema 93 | use role sysadmin; 94 | use warehouse compute_wh; 95 | use database TIPS_SALES_PROD; 96 | use schema public; 97 | 98 | -- populate data to a table from the example table 99 | CREATE TABLE part_export LIKE snowflake_sample_data.tpch_sf1.part 100 | INSERT INTO part_export select * from snowflake_sample_data.tpch_sf1.part 101 | 102 | -- copying data to stage in csv format 103 | copy into @export_stg/default_csv from part_export 104 | -- all records loaded to stage and md5 check is also generated 105 | -- the default_csv will be the prefix _0_0_0.csv.gz 106 | 107 | -- Note: if you don't give a name and if stage is having data followign error will come 108 | -- Files already existing at the unload destination: @export_stg. Use overwrite option to force unloading. 109 | -- use remove function as alterntively 110 | REMOVE @export_stg/export_stg/default_csv_0_0_0.csv.gz 111 | 112 | 113 | 114 | 115 | 116 | -- Creating a table 117 | create or replace table tips_export( 118 | id number(8) not null, 119 | first_name varchar(255) default null, 120 | last_name varchar(255) default null, 121 | city varchar(255), 122 | state varchar(255) 123 | ); 124 | 125 | 126 | -- Populate the table with data 127 | insert into tips_export (id,first_name,last_name,city,state) 128 | values 129 | (1,'Goutam','Shetty','Salt Lake City','UT'), 130 | (2,'Rue','Shane','Birmingham','AL'), 131 | (3,'John','Heck','Columbus','GA'), 132 | (3,'John','Heck','Columbus','GA'), 133 | (3,'Pick','Me','New York','NY'), 134 | (3,'Drop','You','Columbus','GA'), 135 | (3,'Hell','Nick','Columbus','GA'), 136 | (3,'Sim','Jack','Columbus','GA'), 137 | (3,'Jim','Raw','Columbus','GA'), 138 | (3,'Tium','Keo','New York','NY'); 139 | 140 | -- create a stage 141 | CREATE STAGE export_stg COMMENT = 'my_simple_stage'; 142 | 143 | -- initiate a copy command and load data in json using selecd query 144 | copy into @export_stg 145 | from (select object_construct('id', id, 'first_name', first_name, 'last_name', last_name, 'city', city, 'state', state) from tips_export) 146 | file_format = (type = json); 147 | 148 | 149 | -- Data unload in parquet format 150 | copy into @export_stg/myfile.parquet 151 | from (select id, first_name, last_name, city,state from tips_export) 152 | file_format=(type='parquet') 153 | header = true; 154 | 155 | 156 | 157 | -- export multiple & compression 158 | REMOVE @export_stg/file_size_0_0_0.csv.gz 159 | REMOVE @export_stg/file_size_0_0_1.csv.gz 160 | copy into @export_stg/file_size from part_export 161 | file_format = (type=csv compression='gzip') 162 | max_file_size=102400; -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-14 at 1.35.49 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-14 at 1.35.49 PM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-14 at 1.36.26 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-14 at 1.36.26 PM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-14 at 1.39.18 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-14 at 1.39.18 PM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.05.05 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.05.05 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.05.19 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.05.19 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.08.21 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.08.21 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.10.57 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.10.57 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.13.42 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.13.42 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.15.07 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.15.07 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.15.21 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.15.21 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.17.24 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.17.24 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.25.09 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.25.09 AM.png -------------------------------------------------------------------------------- /05-data-unloading-export/images/Screenshot 2020-05-19 at 10.25.34 AM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/05-data-unloading-export/images/Screenshot 2020-05-19 at 10.25.34 AM.png -------------------------------------------------------------------------------- /06-snowflake-security-overview/snowflake-security.sql: -------------------------------------------------------------------------------- 1 | 2 | 3 | VPC - Virtual Private Cloud (Virtual Private Snowflake VPS) 4 | 5 | Physical Security 6 | 24hr arm guards in data center 7 | video survilience data center 8 | no access to any unauthorized personality to data center 9 | not Snoflake personal nor Snowflake customer have access to these data centers 10 | 11 | Data Redundancy 12 | Provided by cloud provider 13 | 14 | Network Access 15 | Network Policy helps to control the SF access 16 | IP Whitelisting - policy can be created to allow or disallow IPs 17 | private link - private tunnel between customer and cloud provider (ESD or VPS customer) 18 | 19 | Account Access & Authentication 20 | MFA - Multi Factor Authentication can be implemented to increase Security 21 | MFA is provided by duo Security sevices 22 | Once MFA is installed, Duo app to be installed by user 23 | Each user must enable MFA by themself. 24 | All user with account admin role should have MFA enabled. 25 | SSO (SAML 2.0) allows user to access via federated services (IDP Identity provider) 26 | As long as IPD session is active, they can access SF 27 | SSO/IDP is available enterprise edition & + 28 | 29 | Object Security 30 | All the objects (warehouse/db/schema/table etc) can be controlled by DAC/RBAC 31 | DAC- Discretionary Access control 32 | RBAC - Role Based Access control 33 | SF implements hybrid model of DAC & RBAC 34 | DAC handle the ownership, each object has an owner and owner has full access to the object. 35 | RBAC - Handle all other access except ownership like object priviledge and role access 36 | Object priviledge assign to role which are intern assign to users. 37 | 38 | Data Security 39 | All data is encrypted uses AES-256 strong Encryption 40 | All files stored in stage area is automatically Encryption using AES-128 or AES-256 41 | Special edition of SF allows periodic re-key and customer manage encryption. 42 | Hi 43 | 44 | Connectivty Security 45 | All communication over internet is via HTTPS. 46 | All communication is secure and encrypted via TLS 1.2 or higher 47 | 48 | Compliance Security 49 | Third Party 50 | HIPPA 51 | PCI (Payment Card for Industry for data security) 52 | NIST 800-53 53 | SOC 54 | SOC-2 type II 55 | SIG Lite (SIG Assessment) (Standardized Information Gathering - SIG Questionnaire Tools allow organizations to build, customize, analyze and store vendor questionnaires) 56 | 57 | Application Activity Log 58 | History tab provide all historical commands 59 | All details including session id etc can be viewed. 60 | Each query has query id which helps for trouble shooting (no access to data to even SF users) 61 | 62 | User Access Audit Log 63 | Login_History family of table function can be use to query login attempts 64 | 65 | Query History 66 | Query History is available for 7 days, it can be stored in SF table or external system. 67 | 68 | 69 | 70 | Infrastructure Monitoring 71 | SF users Threat Stack & Sumo Logic to monitor production Infrastructure 72 | Lacework for behavioural Monitoring (use activity/network traffic/binaries) 73 | All alerts are viewed by SF security team. 74 | 75 | Penetration Testing 76 | SF performs 7-10 penetration testing per year 77 | Application Penetration Test 78 | Network Penetration Test 79 | Funtional Penetration Test 80 | All logs and findings are tracked to closure 81 | Test results are available with customer under NDA 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /09-data-cloning/data-cloning.sql: -------------------------------------------------------------------------------- 1 | -- data-cloning.sql 2 | 3 | 4 | -- Important links 5 | -- 1. https://docs.snowflake.com/en/user-guide/object-clone.html 6 | -- 2. 7 | 8 | 9 | 10 | -- Cloning a database 11 | CREATE DATABASE TIPS_DATABASE_QA 12 | CLONE "TIPS_DATABASE" 13 | COMMENT = 'This is cloned database from Tips Database'; 14 | 15 | 16 | 17 | -- Create like feature to copy table, but data is not copied 18 | CREATE TABLE "TIPS_DATABASE_QA"."LAB".DEVICE_DATA_CREATE_LIKE 19 | LIKE "TIPS_DATABASE_QA"."LAB"."DEVICE_DATA" 20 | COMMENT = 'Create like feature is used which is similar to clone'; 21 | 22 | 23 | 24 | -- Cloning a file format 25 | -- img exist 26 | CREATE FILE FORMAT "TIPS_DATABASE_QA"."LAB".CLONED_DEVICE_EVENT_FORMAT 27 | CLONE "TIPS_DATABASE_QA"."LAB"."DEVICE_EVENT_FORMAT" 28 | COMMENT = 'Cloning a file formate to the same schema'; 29 | 30 | 31 | -- Create a stage via clone. 32 | -- img exist 33 | CREATE STAGE "TIPS_DATABASE_QA"."LAB".CLONE_JSON_STAGE 34 | CLONE "TIPS_DATABASE_QA"."LAB"."JSON_STAGE" 35 | COMMENT = 'This is stage cloning for named external stage'; -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.15.57 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.15.57 PM.png -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.16.31 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.16.31 PM.png -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.17.18 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.17.18 PM.png -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.23.17 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.23.17 PM.png -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.25.24 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.25.24 PM.png -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.27.42 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.27.42 PM.png -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.30.34 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.30.34 PM.png -------------------------------------------------------------------------------- /09-data-cloning/images/Screenshot 2020-05-13 at 2.34.27 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/09-data-cloning/images/Screenshot 2020-05-13 at 2.34.27 PM.png -------------------------------------------------------------------------------- /10-account-usage/account-usage-snowflake.sql: -------------------------------------------------------------------------------- 1 | /* 2 | This page is all about account usage 3 | 4 | Reference Material 5 | 1. https://docs.snowflake.com/en/sql-reference/account-usage.html 6 | 2. https://docs.snowflake.com/en/sql-reference/account-usage.html#differences-between-account-usage-and-information-schema 7 | */ 8 | 9 | /* 10 | Important note 11 | - SNOWFLAKE is a system-defined, read-only shared database, provided by Snowflake 12 | - The database is automatically imported into each account from a share named ACCOUNT_USAGE 13 | - The SNOWFLAKE database is an example of Snowflake utilizing Secure Data Sharing to provide object metadata and other usage metrics for your account. 14 | - The SNOWFLAKE database contains three schemas (also read-only). Each schema contains a set of views (refer images) 15 | 16 | - ACCOUNT_USAGE: Views that display object metadata and usage metrics for your account. 17 | - Diff in ACCOUNT_USAGE vs INFORMATION_SCHEMA 18 | - Records for dropped objects included in each view. 19 | - Longer retention time for historical usage data. 20 | - Data latency. 21 | - By default, only account administrators (users with the ACCOUNTADMIN role) can access the SNOWFLAKE database and schemas within the database, or perform queries on the views 22 | */ 23 | 24 | -- Enabling Account Usage for Other Roles 25 | 26 | USE ROLE ACCOUNTADMIN; 27 | GRANT IMPORTED PRIVILEGES ON DATABASE snowflake TO ROLE SYSADMIN; 28 | GRANT IMPORTED PRIVILEGES ON DATABASE snowflake TO ROLE customrole1; 29 | 30 | USE ROLE customrole1; 31 | SELECT * FROM snowflake.account_usage.databases; 32 | 33 | 34 | -- Sample Queries 35 | 36 | -- Average number of seconds between failed login attempts by user (month-to-date): 37 | -- refer image for result and cost of the query 38 | USE ROLE ACCOUNTADMIN; 39 | select user_name, 40 | count(*) as failed_logins, 41 | avg(seconds_between_login_attempts) as average_seconds_between_login_attempts 42 | from ( 43 | select user_name, 44 | timediff(seconds, event_timestamp, lead(event_timestamp) 45 | over(partition by user_name order by event_timestamp)) as seconds_between_login_attempts 46 | from "SNOWFLAKE"."ACCOUNT_USAGE"."LOGIN_HISTORY" 47 | where event_timestamp > date_trunc(month, current_date) 48 | and is_success = 'NO' 49 | ) 50 | group by 1 51 | order by 3; 52 | 53 | --Failed logins by user (month-to-date): 54 | -- refer image for result 55 | select user_name, 56 | sum(iff(is_success = 'NO', 1, 0)) as failed_logins, 57 | count(*) as logins, 58 | sum(iff(is_success = 'NO', 1, 0)) / nullif(count(*), 0) as login_failure_rate 59 | from "SNOWFLAKE"."ACCOUNT_USAGE"."LOGIN_HISTORY" 60 | where event_timestamp > date_trunc(month, current_date) 61 | group by 1 62 | order by 4 desc; 63 | 64 | 65 | -- Failed logins by user and connecting client (month-to-date): 66 | select reported_client_type, 67 | user_name, 68 | sum(iff(is_success = 'NO', 1, 0)) as failed_logins, 69 | count(*) as logins, 70 | sum(iff(is_success = 'NO', 1, 0)) / nullif(count(*), 0) as login_failure_rate 71 | from "SNOWFLAKE"."ACCOUNT_USAGE"."LOGIN_HISTORY" 72 | where event_timestamp > date_trunc(month, current_date) 73 | group by 1,2 74 | order by 5 desc; 75 | 76 | 77 | -- Examples: Warehouse Credit Usage 78 | -- Credits used by each warehouse in your account (month-to-date): 79 | -- refer image for result 80 | select warehouse_name, 81 | sum(credits_used) as total_credits_used 82 | from "SNOWFLAKE"."ACCOUNT_USAGE"."WAREHOUSE_METERING_HISTORY" 83 | where start_time >= date_trunc(month, current_date) 84 | group by 1 85 | order by 2 desc; 86 | 87 | 88 | 89 | -- Credits used over time by each warehouse in your account (month-to-date): 90 | -- refer image for result 91 | 92 | select start_time::date as usage_date, 93 | warehouse_name, 94 | sum(credits_used) as total_credits_used 95 | from "SNOWFLAKE"."ACCOUNT_USAGE"."WAREHOUSE_METERING_HISTORY" 96 | where start_time >= date_trunc(month, current_date) 97 | group by 1,2 98 | order by 2,1; 99 | 100 | 101 | 102 | -- ---------------------------- 103 | -- Examples: Data Storage Usage 104 | ------------------------------- 105 | 106 | -- Billable terabytes stored in your account over time: 107 | select date_trunc(month, usage_date) as usage_month 108 | , avg(storage_bytes + stage_bytes + failsafe_bytes) / power(1024, 4) as billable_tb 109 | from "SNOWFLAKE"."ACCOUNT_USAGE"."STORAGE_USAGE" 110 | group by 1 111 | order by 1; 112 | 113 | 114 | 115 | ------------------------------- 116 | -- Examples: User Query Totals and Execution Times 117 | ------------------------------- 118 | -- Total jobs executed in your account (month-to-date): 119 | 120 | select count(*) as number_of_jobs 121 | from query_history 122 | where start_time >= date_trunc(month, current_date); 123 | 124 | -- Total jobs executed by each warehouse in your account (month-to-date): 125 | 126 | select warehouse_name, 127 | count(*) as number_of_jobs 128 | from query_history 129 | where start_time >= date_trunc(month, current_date) 130 | group by 1 131 | order by 2 desc; 132 | 133 | --Average query execution time by user (month-to-date): 134 | 135 | select user_name, 136 | avg(execution_time) as average_execution_time 137 | from query_history 138 | where start_time >= date_trunc(month, current_date) 139 | group by 1 140 | order by 2 desc; 141 | 142 | --Average query execution time by query type and warehouse size (month-to-date): 143 | 144 | select query_type, 145 | warehouse_size, 146 | avg(execution_time) as average_execution_time 147 | from query_history 148 | where start_time >= date_trunc(month, current_date) 149 | group by 1,2 150 | order by 3 desc; 151 | 152 | -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.12.30 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.12.30 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.12.43 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.12.43 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.13.03 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.13.03 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.36.01 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.36.01 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.37.11 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.37.11 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.40.56 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.40.56 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.42.55 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.42.55 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.46.00 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.46.00 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.50.53 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.50.53 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.52.05 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.52.05 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.52.59 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.52.59 PM.png -------------------------------------------------------------------------------- /10-account-usage/images/Screenshot 2020-05-12 at 2.54.50 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/10-account-usage/images/Screenshot 2020-05-12 at 2.54.50 PM.png -------------------------------------------------------------------------------- /11-user-role-creation/images/Screenshot 2020-05-16 at 1.33.11 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/11-user-role-creation/images/Screenshot 2020-05-16 at 1.33.11 PM.png -------------------------------------------------------------------------------- /11-user-role-creation/images/Screenshot 2020-05-16 at 1.38.05 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/11-user-role-creation/images/Screenshot 2020-05-16 at 1.38.05 PM.png -------------------------------------------------------------------------------- /11-user-role-creation/images/Screenshot 2020-05-16 at 1.40.04 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/11-user-role-creation/images/Screenshot 2020-05-16 at 1.40.04 PM.png -------------------------------------------------------------------------------- /11-user-role-creation/images/Screenshot 2020-05-16 at 1.45.40 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/11-user-role-creation/images/Screenshot 2020-05-16 at 1.45.40 PM.png -------------------------------------------------------------------------------- /11-user-role-creation/user-role-snowflake.sql: -------------------------------------------------------------------------------- 1 | 2 | 3 | --create a user (Blr@4950$) 4 | CREATE USER tips_jr_analyst 5 | PASSWORD = '**********' 6 | COMMENT = 'Tips Junior Analyst to run queries' 7 | MUST_CHANGE_PASSWORD = TRUE; 8 | 9 | -- with advance parameters 10 | CREATE USER tips_jr_analyst 11 | PASSWORD = '**********' 12 | COMMENT = 'Tips Junior Analyst to run queries' 13 | LOGIN_NAME = 'TipsJrAnalyst' 14 | DISPLAY_NAME = 'Junior Analyst' 15 | FIRST_NAME = 'Junior' 16 | LAST_NAME = 'Analyst' 17 | EMAIL = 'tips.analyst@toppertips.com' 18 | MUST_CHANGE_PASSWORD = TRUE; 19 | 20 | -- With advance params & default warehouse details 21 | -- also granting role 22 | CREATE USER tips_jr_analyst 23 | PASSWORD = '**********' 24 | COMMENT = 'Tips Junior Analyst to run queries' 25 | LOGIN_NAME = 'TipsJrAnalyst' 26 | DISPLAY_NAME = 'Junior Analyst' 27 | FIRST_NAME = 'Junior' 28 | LAST_NAME = 'Analyst' 29 | EMAIL = 'tips.analyst@toppertips.com' 30 | DEFAULT_ROLE = "USERADMIN" 31 | DEFAULT_WAREHOUSE = 'COMPUTE_WH' 32 | DEFAULT_NAMESPACE = 'TIPS_DB' 33 | MUST_CHANGE_PASSWORD = TRUE; 34 | 35 | GRANT ROLE "USERADMIN" TO USER tips_jr_analyst; 36 | 37 | -- for user to see the warehouse, it has to be assigend to that role 38 | GRANT USAGE, OPERATE, MONITOR ON WAREHOUSE "COMPUTE_WH" 39 | TO ROLE "USERADMIN"; 40 | 41 | -- grant the database usage 42 | GRANT USAGE, CREATE SCHEMA, MODIFY, MONITOR ON DATABASE "TIPS_SALES_PROD" 43 | TO ROLE "USERADMIN"; -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Snowflake-By-Example 2 | All the Snowflake Virtual Warehouse - Example 3 | -------------------------------------------------------------------------------- /grant-sql-example/database-schema-table-grant.sql: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | Reference Material 4 | 1. https://docs.snowflake.com/en/sql-reference/sql/grant-privilege.html 5 | 6 | */ 7 | -- Giving a usage grant on a table to public role 8 | GRANT USAGE ON DATABASE "DB_FOR_SECURITY" TO ROLE "PUBLIC" WITH GRANT OPTION; 9 | 10 | 11 | 12 | -- How To: Grant a role access to database objects in a schema 13 | /* 14 | To allow a role to use database objects in a specific schema, 15 | the owner of the database objects (typically a system administrator (SYSADMIN role)) 16 | must grant privileges on the database, schema, and objects. 17 | */ 18 | 19 | --Grant usage on the database: 20 | GRANT USAGE ON DATABASE TO ROLE ; 21 | 22 | 23 | --Grant usage on the schema: 24 | GRANT USAGE ON SCHEMA . TO ROLE ; 25 | 26 | --Grant the ability to query an existing table: 27 | GRANT SELECT ON TABLE .. TO ROLE ; 28 | 29 | /* 30 | The following table privileges are supported: 31 | Privilege Usage 32 | SELECT Execute a SELECT statement on the table. 33 | INSERT Execute an INSERT command on the table. 34 | UPDATE Execute an UPDATE command on the table. 35 | TRUNCATE Execute a TRUNCATE command on the table. 36 | DELETE Execute a DELETE command on the table. 37 | REFERENCES Reference the table as the unique/primary key table for a foreign key constraint. 38 | ALL [ PRIVILEGES ] Grant all privileges, except OWNERSHIP, on the table. 39 | OWNERSHIP Grant full control over a table. 40 | 41 | */ -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.06.24 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.06.24 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.07.07 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.07.07 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.07.35 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.07.35 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.08.13 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.08.13 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.09.50 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.09.50 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.21.09 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.21.09 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.21.50 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.21.50 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.25.53 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.25.53 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.34.13 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.34.13 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.36.20 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.36.20 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.42.59 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.42.59 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.43.16 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.43.16 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.43.33 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.43.33 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.43.45 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 1.43.45 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 2.01.54 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 2.01.54 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 2.04.48 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 2.04.48 PM.png -------------------------------------------------------------------------------- /images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 2.10.30 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TopperTips/Snowflake-By-Example/b84a365afd9cbe175613177d5f0399d2c60fd9be/images/snowflake-name-stages-internal-external/Screenshot 2020-05-05 at 2.10.30 PM.png -------------------------------------------------------------------------------- /json-data-load-example.sql: -------------------------------------------------------------------------------- 1 | 2 | #Sample JSON Data 3 | /* 4 | [{"time":"6:38:51.000 PM","customer":"Talya McCambrois","action":"power off","device":"Footbot Air Quality Monitor"}, 5 | {"time":"8:13:16.000 PM","customer":"Willi Jenkerson","action":"power off","device":"GreenIQ Controller"}, 6 | {"time":"2:21:46.000 AM","customer":"Kippy Roux","action":"power off","device":"Amazon Echo"}] 7 | */ 8 | CREATE DATABASE "TIPS_DATABASE" 9 | 10 | CREATE FILE FORMAT "TIPS_DATABASE"."LAB".IOT_JSON 11 | TYPE = 'JSON' 12 | COMPRESSION = 'AUTO' 13 | ENABLE_OCTAL = FALSE 14 | ALLOW_DUPLICATE = TRUE 15 | STRIP_OUTER_ARRAY = FALSE 16 | STRIP_NULL_VALUES = TRUE 17 | IGNORE_UTF8_ERRORS = FALSE 18 | COMMENT = 'JSON File Format'; 19 | 20 | 21 | -- Step 4: Creating an internal stage to load data 22 | CREATE STAGE "TIPS_DATABASE"."LAB".IOT_LAB_STAGE 23 | COMMENT = 'This is iot data loading stage'; 24 | 25 | 26 | 27 | -- Step 4: Creating a table 28 | CREATE TABLE "TIPS_DATABASE"."LAB"."DEVICE_DATA" ( 29 | "EVENT_TIME" TIMESTAMP NOT NULL, 30 | "CUSTOMER_NAME" VARCHAR (100) NOT NULL, 31 | "DEVICE_ACTION" VARCHAR (50) NOT NULL, 32 | "DEVICE_NAME" VARCHAR (200) NOT NULL) 33 | COMMENT = 'This is customer iot device data'; 34 | 35 | 36 | PUT file:///1.json @DEVICE_DATA/ui1589027218487 37 | 38 | COPY INTO "TIPS_DATABASE"."LAB"."DEVICE_DATA" 39 | FROM @/ui1589027218487 40 | FILE_FORMAT = '"TIPS_DATABASE"."LAB"."IOT_JSON"' 41 | ON_ERROR = 'CONTINUE' 42 | PURGE = TRUE; 43 | 44 | #since file has issue.. it threw following error 45 | /* 46 | Unable to copy files into table. 47 | SQL compilation error: JSON/XML/AVRO file format can produce 48 | one and only one column of type variant or object or array. 49 | Use CSV file format if you want to load more than one column. 50 | */ 51 | 52 | 53 | CREATE TABLE "TIPS_DATABASE"."LAB"."DEVICE_DATA_VARIANT" ("DEVICE_EVENT" VARIANT NOT NULL) COMMENT = 'This is single column data'; 54 | 55 | 56 | 57 | -- Data loaded into single row 58 | -- approach 2 to have in mutlipel rows 59 | CREATE TABLE "TIPS_DATABASE"."LAB"."DEVICE_DATA_VARIANT_2" ("DEVICE_EVENT" VARIANT NOT NULL) COMMENT = 'Trying to load on individual rows'; 60 | 61 | CREATE FILE FORMAT "TIPS_DATABASE"."LAB".Device_Event_Format TYPE = 'JSON' COMPRESSION = 'AUTO' ENABLE_OCTAL = FALSE ALLOW_DUPLICATE = TRUE STRIP_OUTER_ARRAY = TRUE STRIP_NULL_VALUES = TRUE IGNORE_UTF8_ERRORS = TRUE COMMENT = 'Each element as single row'; 62 | 63 | PUT file:///1.json @DEVICE_DATA_VARIANT_2/ui1589027881962 64 | 65 | COPY INTO "TIPS_DATABASE"."LAB"."DEVICE_DATA_VARIANT_2" FROM @/ui1589027881962 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."DEVICE_EVENT_FORMAT"' ON_ERROR = 'CONTINUE' PURGE = TRUE; 66 | 67 | 68 | -- table to load ORC file via structured schema 69 | CREATE TABLE "TIPS_DATABASE"."LAB"."ORC_DEVICE_DATA" ("EVENT_TIME" TIMESTAMP NOT NULL, "CUSTOMER_NAME" VARCHAR (100) NOT NULL, "EVENT_ACTION" VARCHAR (50) NOT NULL, "DEVICE_NAME" VARCHAR (200) NOT NULL) COMMENT = 'Storing Data from ORC File'; 70 | -- create file format to load data to table via put command 71 | CREATE FILE FORMAT "TIPS_DATABASE"."LAB".ORC_DATA_FORMAT TYPE = 'ORC' COMMENT = 'This is ORC data format file'; 72 | 73 | -- data loading command via internal unmammed stage 74 | PUT file:///part-00000-921b6fdc-41bb-4b06-b5d8-a8b760cb1558-c000.snappy.orc @ORC_DEVICE_DATA/ui1589034165777 75 | 76 | COPY INTO "TIPS_DATABASE"."LAB"."ORC_DEVICE_DATA" FROM @/ui1589034165777 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."DEVICE_EVENT_FORMAT"' ON_ERROR = 'ABORT_STATEMENT' PURGE = TRUE; 77 | 78 | -- looks it also gave an error like other data set 79 | Unable to copy files into table. 80 | SQL compilation error: JSON/XML/AVRO file format can produce one 81 | and only one column of type variant or object or array. 82 | Use CSV file format if you want to load more than one column. 83 | 84 | -- Now if you have lot of large data set then it will be an expensive operation 85 | 86 | 87 | CREATE TABLE "TIPS_DATABASE"."LAB"."ORC_DEVICE_DATA_VARIANT" ("DEVICE_DATA" VARIANT NOT NULL) COMMENT = 'This table will have only one column'; 88 | 89 | 90 | PUT file:///part-00000-921b6fdc-41bb-4b06-b5d8-a8b760cb1558-c000.snappy.orc @ORC_DEVICE_DATA_VARIANT/ui1589034420521 91 | 92 | COPY INTO "TIPS_DATABASE"."LAB"."ORC_DEVICE_DATA_VARIANT" FROM @/ui1589034420521 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."ORC_DATA_FORMAT"' ON_ERROR = 'CONTINUE' PURGE = TRUE; 93 | 94 | 95 | -- Now parquet type 96 | CREATE FILE FORMAT "TIPS_DATABASE"."LAB".PARQUET_DATA_FORMAT TYPE = 'PARQUET' COMPRESSION = 'SNAPPY' BINARY_AS_TEXT = TRUE COMMENT = 'File Format for Parquet Data'; 97 | 98 | 99 | -- load command 100 | PUT file:///part-00000-d5a70322-eb9d-4e02-a85f-e678c1029748-c000.snappy.parquet 101 | @ORC_DEVICE_DATA/ui1589034857768 102 | 103 | COPY INTO "TIPS_DATABASE"."LAB"."ORC_DEVICE_DATA" 104 | FROM @/ui1589034857768 105 | FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' 106 | ON_ERROR = 'CONTINUE' 107 | PURGE = TRUE; 108 | 109 | it also ended in errorUnable to copy files into table. 110 | SQL compilation error: JSON/XML/AVRO file format can produce one and only one column of type variant or object or array. Use CSV file format if you want to load more than one column. 111 | 112 | 113 | -- Now defineing a new table 114 | CREATE TABLE "TIPS_DATABASE"."LAB"."PARQUET_DEVIDE_DATA_VARIANT" ("DEVICE_EVENT" VARIANT) COMMENT = 'The single column data loaded via Variant'; 115 | 116 | 117 | -- different options 118 | PUT file:///part-00000-d5a70322-eb9d-4e02-a85f-e678c1029748-c000.snappy.parquet @PARQUET_DEVIDE_DATA_VARIANT/ui1589035032039 119 | 120 | --Do not load any data in the file 121 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_DEVIDE_DATA_VARIANT" FROM @/ui1589035032039 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'SKIP_FILE' PURGE = TRUE; 122 | --Stop loading, rollback and return the error 123 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_DEVIDE_DATA_VARIANT" FROM @/ui1589035032039 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'ABORT_STATEMENT' PURGE = TRUE; 124 | --Do not load any data in the file if the error count exceeds: 125 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_DEVIDE_DATA_VARIANT" FROM @/ui1589035032039 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'SKIP_FILE_100' PURGE = TRUE; 126 | --Continue loading valid data from the file 127 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_DEVIDE_DATA_VARIANT" FROM @/ui1589035032039 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'CONTINUE' PURGE = TRUE; 128 | 129 | 130 | 131 | -- Now trying to understand the object type 132 | CREATE TABLE "TIPS_DATABASE"."LAB"."PARQUET_DEVIDE_DATA_OBJECT" ("EVENT_RECORD" OBJECT NOT NULL) COMMENT = 'This table has one field with object as field type'; 133 | 134 | -- loading data via put and copy command 135 | PUT file:///part-00000-d5a70322-eb9d-4e02-a85f-e678c1029748-c000.snappy.parquet @PARQUET_DEVIDE_DATA_OBJECT/ui1589035396167 136 | 137 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_DEVIDE_DATA_OBJECT" FROM @/ui1589035396167 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'CONTINUE' PURGE = TRUE; 138 | 139 | -- ended up with error -- not sure why 140 | Unable to copy files into table. 141 | SQL execution internal error: Processing aborted due to error 300010:3163851172; incident 9323353. 142 | 143 | -- Now the final one is to use the array 144 | CREATE TABLE "TIPS_DATABASE"."LAB"."PARQUET_DEVICE_DATA_ARRAY" ("DEVICE_EVENT" ARRAY NOT NULL) COMMENT = 'This is an experiment with Parquet with array'; 145 | PUT file:///part-00000-d5a70322-eb9d-4e02-a85f-e678c1029748-c000.snappy.parquet @PARQUET_DEVICE_DATA_ARRAY/ui1589035563444 146 | 147 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_DEVICE_DATA_ARRAY" FROM @/ui1589035563444 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'CONTINUE' PURGE = TRUE; 148 | 149 | -- end up with error -- could be due to the role issue as format is created by different role 150 | Unable to copy files into table. 151 | SQL execution internal error: Processing aborted due to error 300010:3163851172; incident 5140068. 152 | 153 | -- Now following is given..lets see what happens 154 | GRANT UPDATE, INSERT, DELETE, REBUILD, REFERENCES, SELECT, TRUNCATE 155 | ON TABLE "TIPS_DATABASE"."LAB"."PARQUET_DEVICE_DATA_ARRAY" 156 | TO ROLE "SYSADMIN" WITH GRANT OPTION; 157 | 158 | 159 | -- Now role is changed 160 | PUT file:///tips.snappy.parquet @PARQUET_DEVICE_DATA_ARRAY/ui1589035944814 161 | 162 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_DEVICE_DATA_ARRAY" FROM @/ui1589035944814 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'CONTINUE' PURGE = TRUE; 163 | 164 | 165 | -- got the error like this 166 | There was an error while trying to stage "tips.snappy.parquet". 167 | Reason: SQL access control error: Insufficient privileges to operate on table stage 'PARQUET_DEVICE_DATA_ARRAY' 168 | 169 | -- looks the internal stage has issues ???? looks a big issue -- looks the int stage has access issue 170 | 171 | -- so createa a table again with sysadmin 172 | CREATE TABLE "TIPS_DATABASE"."LAB"."PARQUET_TYPE_ARRAY" ("DEVICE_EVENT" ARRAY NOT NULL) COMMENT = 'Parquet data loaded in array data type'; 173 | -- step-2 174 | PUT file:///tips.snappy.parquet @PARQUET_TYPE_ARRAY/ui1589036262859 175 | COPY INTO "TIPS_DATABASE"."LAB"."PARQUET_TYPE_ARRAY" FROM @/ui1589036262859 FILE_FORMAT = '"TIPS_DATABASE"."LAB"."PARQUET_DATA_FORMAT"' ON_ERROR = 'CONTINUE' PURGE = TRUE; 176 | 177 | -- step-3 looks error again 178 | Unable to copy files into table. 179 | SQL execution internal error: Processing aborted due to error 300010:3163851172; incident 5508373. 180 | 181 | -- @TOD - understant the object and array data type and why it is not not working with parquet. 182 | 183 | -------------------------------------------------------------------------------- /role-sql-example/role-assignment-snowflake.sql: -------------------------------------------------------------------------------- 1 | /* 2 | Important links 3 | 1.https://s3.amazonaws.com/snowflake-workshop-lab/Snowflake_free_trial_LabGuide.pdf 4 | 5 | */ 6 | 7 | 8 | /* 9 | Important Notes 10 | - For any role to function, we need at least one user assigned to it 11 | - SYSADMIN role by default cannot create new roles or users. role creation operation will fail 12 | */ 13 | 14 | create role tips_jr_dba; 15 | grant role tips_jr_dba to user tips_dba_user02; 16 | use role tips_jr_dba; 17 | 18 | -- Now we have to give usage access to this new role 19 | use role accountadmin; 20 | grant usage on database citibike to role tips_jr_dba; 21 | grant usage on database weather to role tips_jr_dba; -------------------------------------------------------------------------------- /snowlfake-named-stages-internal-external.sql: -------------------------------------------------------------------------------- 1 | -- http://toppertips.com 2 | -- This SQL scripts is primarily for all the stage related SQL Queries. 3 | 4 | /* 5 | 6 | Reference link 7 | 1. https://docs.snowflake.com/en/sql-reference/sql/create-stage.html 8 | 2. https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage.html 9 | 3. Alter Stages : https://docs.snowflake.com/en/sql-reference/sql/alter-stage.html 10 | */ 11 | 12 | -- How to show all the stages? 13 | show stages; 14 | list @S3_USDF_EXTERNAL_NAMED_STAGE; --loading all files inside the bucket 15 | list @S3_USDF_EXTERNAL_NAMED_STAGE/load; -- loading specific directory 16 | list @S3_USDF_EXTERNAL_NAMED_STAGE/load/D; -- loading specific directory and file within it (look for * is missing) 17 | 18 | 19 | -- Note : it is not stage, it is stages, make sure you remember the command. 20 | 21 | 22 | 23 | -- Create a name internal stage (a temporary loading area) 24 | -- This stage will be created within public schema if there is no other schema. 25 | -- you can change the schema and databaes names 26 | 27 | CREATE OR REPLACE STAGE "USDA_NUTRIENT_STDREF"."PUBLIC".Tips_Internal_Storage 28 | COMMENT = 'An internal staging area'; 29 | 30 | -- The grant query to be documented? 31 | 32 | -- Reference 33 | 34 | -- Create a named external stage (a temporary loading area) with AWS S3 (Simple Storage Services) 35 | -- This stage will be created within public schema if there is no other schema. 36 | -- you can change the schema and databaes names 37 | CREATE OR REPLACE STAGE "USDA_NUTRIENT_STDREF"."PUBLIC".S3_Tips_External_Named_Stage 38 | URL = 's3://my-sample-bucket' 39 | CREDENTIALS = (AWS_KEY_ID = 'myuser' AWS_SECRET_KEY = '********') 40 | ENCRYPTION = (MASTER_KEY = '*********') 41 | COMMENT = 'S3_Tips_External_Named_Stage'; 42 | 43 | -- Creating a temporary storage 44 | -- Not documentation is found how it is different from normal stage and what is the purpose 45 | -- This is to be explroed ??? 46 | CREATE OR REPLACE TEMPORARY STAGE "USDA_NUTRIENT_STDREF"."PUBLIC".S3_TEMPO_EXTERNAL_NAMED_STAGE 47 | URL = 's3://my-sample-bucket' 48 | CREDENTIALS = (AWS_KEY_ID = 'myuser' AWS_SECRET_KEY = '********') 49 | COMMENT = 'Checking what is this temporary stage is'; 50 | 51 | -- Alter Stage (Changing Paramters) 52 | ALTER STAGE "USDA_NUTRIENT_STDREF"."PUBLIC"."S3_USDF_EXTERNAL_NAMED_STAGE" 53 | SET URL = 's3://on-demand-files' 54 | COMMENT = 'S3_USDF_EXTERNAL_NAMED_STAGS'; 55 | 56 | -- Alter rename does not work, looks like it is not supported as UI also does not allow. 57 | ALTER STAGE IF EXISTS "USDA_NUTRIENT_STDREF"."PUBLIC".S3_TEMPO_EXTERNAL_NAMED_STAGE 58 | RENAME "USDA_NUTRIENT_STDREF"."PUBLIC".S3_TEMPO_EXTERNAL_NAMED_STAGE_RENAMED 59 | 60 | -- (this will end with error : SQL compilation error: syntax error line 2 at position 7 unexpected '"USDA_NUTRIENT_STDREF"'. ) 61 | 62 | -- Even unset comment does not seems to be working 63 | -- (error : Unsupported feature 'UNSET'.) 64 | 65 | -- Note-01: if you input incorrect AWS keys, Snowflake will throw error like this 66 | --Unable to create stage "S3_TIPS_EXTERNAL_NAMED_STAGE". 67 | --The provided master key has invalid length. It must be either 128 bits, 192 bits, or 256 bits long. 68 | 69 | -- Note-02: If the bucket(s3) is publickly available, then you don't need to specify the key at all. 70 | 71 | 72 | -- Creating External Named Stages with Azure 73 | CREATE OR REPLACE STAGE "USDA_NUTRIENT_STDREF"."PUBLIC".Azure_Tips_External_Named_Stage 74 | URL = 'azure://azurebucket' 75 | CREDENTIALS = (AZURE_SAS_TOKEN = '********************') 76 | ENCRYPTION = (TYPE = 'AZURE_CSE' MASTER_KEY = '****************************************') 77 | COMMENT = 'Azure_Tips_External_Named_Stage'; 78 | 79 | 80 | 81 | 82 | -- How to create a stage by cloing it 83 | -- Creating a clone works for external stage but does not work for internal stages 84 | CREATE STAGE "USDA_NUTRIENT_STDREF"."PUBLIC".Cloned_s3_tips 85 | CLONE "USDA_NUTRIENT_STDREF"."PUBLIC"."S3_TIPS_EXTERNAL_NAMED_STAGE" 86 | COMMENT = 'Cloning an s3 named stage'; 87 | 88 | -- Note-01 : If you try to create a clone using internal stages, it will throw error 89 | /* 90 | Unable to create stage "CLONE_INTERNAL_STAGE". 91 | Unsupported feature 'Cloning internal and temporary stages'. 92 | */ 93 | 94 | -- Note-02 : Grants are also not copied when you clone 95 | 96 | /* 97 | -- ------------------------------------ 98 | Different kind of error may appear if you are not having sufficient privileges 99 | so make sure you check the role using the role manu and then run the queries or operations. 100 | 101 | Check for the ICON called "Owner" on WebUI before operating on any staged object 102 | -- ------------------------------------ 103 | 104 | Unable to modify stage "S3_TIPS_EXTERNAL_NAMED_STAGE". 105 | SQL access control error: Insufficient privileges to operate on stage 'S3_TIPS_EXTERNAL_NAMED_STAGE' 106 | 107 | 108 | 109 | 110 | --------------------------------------------------------------------------------