├── README.md ├── Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task ├── SQL │ ├── 1. Table Creation.sql │ ├── 2. Data Loading.sql │ ├── 3. SCD 1.sql │ └── 4. SCD 2.sql ├── commands ├── docker-exp │ └── docker-compose.yml └── faker.ipynb └── SQL Code ├── 1. Stage Copy.sql ├── 10. Undrop.sql ├── 11. Table Types.sql ├── 12. Zero Copy Clone.sql ├── 13. Data Sharing.sql ├── 14. Materlized Views.sql ├── 15. Dynamic Data masking.sql ├── 2. File Format.sql ├── 3. JSON Data.sql ├── 4. Perfromance Optimization.sql ├── 5. Cache & Clustering.sql ├── 6. Storage Integration.sql ├── 7. Snowpipe.sql ├── 8. Time Travel.sql └── 9. Time Travel Restore.sql /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/README.md -------------------------------------------------------------------------------- /Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/1. Table Creation.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/1. Table Creation.sql -------------------------------------------------------------------------------- /Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/2. Data Loading.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/2. Data Loading.sql -------------------------------------------------------------------------------- /Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/3. SCD 1.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/3. SCD 1.sql -------------------------------------------------------------------------------- /Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/4. SCD 2.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/SQL/4. SCD 2.sql -------------------------------------------------------------------------------- /Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/commands: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/commands -------------------------------------------------------------------------------- /Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/docker-exp/docker-compose.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/docker-exp/docker-compose.yml -------------------------------------------------------------------------------- /Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/faker.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/Real-Time Data Streaming using Apache Nifi, AWS, Snowpipe, Stream & Task/faker.ipynb -------------------------------------------------------------------------------- /SQL Code/1. Stage Copy.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/1. Stage Copy.sql -------------------------------------------------------------------------------- /SQL Code/10. Undrop.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/10. Undrop.sql -------------------------------------------------------------------------------- /SQL Code/11. Table Types.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/11. Table Types.sql -------------------------------------------------------------------------------- /SQL Code/12. Zero Copy Clone.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/12. Zero Copy Clone.sql -------------------------------------------------------------------------------- /SQL Code/13. Data Sharing.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/13. Data Sharing.sql -------------------------------------------------------------------------------- /SQL Code/14. Materlized Views.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/14. Materlized Views.sql -------------------------------------------------------------------------------- /SQL Code/15. Dynamic Data masking.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/15. Dynamic Data masking.sql -------------------------------------------------------------------------------- /SQL Code/2. File Format.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/2. File Format.sql -------------------------------------------------------------------------------- /SQL Code/3. JSON Data.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/3. JSON Data.sql -------------------------------------------------------------------------------- /SQL Code/4. Perfromance Optimization.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/4. Perfromance Optimization.sql -------------------------------------------------------------------------------- /SQL Code/5. Cache & Clustering.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/5. Cache & Clustering.sql -------------------------------------------------------------------------------- /SQL Code/6. Storage Integration.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/6. Storage Integration.sql -------------------------------------------------------------------------------- /SQL Code/7. Snowpipe.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/7. Snowpipe.sql -------------------------------------------------------------------------------- /SQL Code/8. Time Travel.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/8. Time Travel.sql -------------------------------------------------------------------------------- /SQL Code/9. Time Travel Restore.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/darshilparmar/data-warehouse-snowflake-for-data-engineering/HEAD/SQL Code/9. Time Travel Restore.sql --------------------------------------------------------------------------------