├── BigQuery ├── BigQuery_partition_demo.sql ├── bq_procedures.sql ├── bq_table_clusters.sql ├── data_lineage_demo.sql ├── gcp_billingdata_analysis.sql └── table_function.sql ├── CloudFunctions ├── bq_events_to_gcs_cf.py ├── gcs_events_cf.py ├── pubsub_events_cf.py └── sheets_to_bigquery_dataload_cf.py ├── CloudSQL ├── cloudsql_python_connect.py └── federated_query_demo.sql ├── CloudSpanner ├── cloud_spanner_intro_demo.sql └── spanner_python_client_demo.py ├── Common_Realtime_Usecases ├── dataflow_spanner_demo │ ├── beam_dataflow-to-spanner.py │ ├── census-db-schema.sql │ ├── census_100_testing.csv │ ├── create-spanner-database-CLI.txt │ └── installations_beam.sh └── iam_snapshots.py ├── Composer ├── Demo_Input_sample_data │ └── customer_data.csv ├── airflow_dataproc_automate_dag.py ├── dataflow_python_operator_dag.py ├── ecom_customer_etl_pipeline.py ├── gcs_to_bq_and_bq_operators.py ├── https_operators_demo_dag.py └── python_bash_operators_dag.py ├── Dataflow ├── batch_etl_avro_data_cloudsql.py ├── beam_stream_data_process.py ├── dataflow_batch_demo.py ├── dataflow_batch_log_process.py └── process_nested_data_sql_demo.py ├── Dataproc └── pyspark_bq_to_gcs_demo.py ├── GCP_Data_Eng_concept_files ├── GCP_storage_db.png ├── Initfile.txt └── gcp_etl_services.png ├── GCS ├── object_lifecycle_mngmnt_cli.txt └── object_lifecylce_mngmnt.py ├── GoogleCloudStorage ├── gcs_python_client_demo.py └── object_versioning_cli.txt ├── README.md └── Security └── secretmanager_python_connect.py /BigQuery/BigQuery_partition_demo.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/BigQuery/BigQuery_partition_demo.sql -------------------------------------------------------------------------------- /BigQuery/bq_procedures.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/BigQuery/bq_procedures.sql -------------------------------------------------------------------------------- /BigQuery/bq_table_clusters.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/BigQuery/bq_table_clusters.sql -------------------------------------------------------------------------------- /BigQuery/data_lineage_demo.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/BigQuery/data_lineage_demo.sql -------------------------------------------------------------------------------- /BigQuery/gcp_billingdata_analysis.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/BigQuery/gcp_billingdata_analysis.sql -------------------------------------------------------------------------------- /BigQuery/table_function.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/BigQuery/table_function.sql -------------------------------------------------------------------------------- /CloudFunctions/bq_events_to_gcs_cf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudFunctions/bq_events_to_gcs_cf.py -------------------------------------------------------------------------------- /CloudFunctions/gcs_events_cf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudFunctions/gcs_events_cf.py -------------------------------------------------------------------------------- /CloudFunctions/pubsub_events_cf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudFunctions/pubsub_events_cf.py -------------------------------------------------------------------------------- /CloudFunctions/sheets_to_bigquery_dataload_cf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudFunctions/sheets_to_bigquery_dataload_cf.py -------------------------------------------------------------------------------- /CloudSQL/cloudsql_python_connect.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudSQL/cloudsql_python_connect.py -------------------------------------------------------------------------------- /CloudSQL/federated_query_demo.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudSQL/federated_query_demo.sql -------------------------------------------------------------------------------- /CloudSpanner/cloud_spanner_intro_demo.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudSpanner/cloud_spanner_intro_demo.sql -------------------------------------------------------------------------------- /CloudSpanner/spanner_python_client_demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/CloudSpanner/spanner_python_client_demo.py -------------------------------------------------------------------------------- /Common_Realtime_Usecases/dataflow_spanner_demo/beam_dataflow-to-spanner.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Common_Realtime_Usecases/dataflow_spanner_demo/beam_dataflow-to-spanner.py -------------------------------------------------------------------------------- /Common_Realtime_Usecases/dataflow_spanner_demo/census-db-schema.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Common_Realtime_Usecases/dataflow_spanner_demo/census-db-schema.sql -------------------------------------------------------------------------------- /Common_Realtime_Usecases/dataflow_spanner_demo/census_100_testing.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Common_Realtime_Usecases/dataflow_spanner_demo/census_100_testing.csv -------------------------------------------------------------------------------- /Common_Realtime_Usecases/dataflow_spanner_demo/create-spanner-database-CLI.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Common_Realtime_Usecases/dataflow_spanner_demo/create-spanner-database-CLI.txt -------------------------------------------------------------------------------- /Common_Realtime_Usecases/dataflow_spanner_demo/installations_beam.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Common_Realtime_Usecases/dataflow_spanner_demo/installations_beam.sh -------------------------------------------------------------------------------- /Common_Realtime_Usecases/iam_snapshots.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Common_Realtime_Usecases/iam_snapshots.py -------------------------------------------------------------------------------- /Composer/Demo_Input_sample_data/customer_data.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Composer/Demo_Input_sample_data/customer_data.csv -------------------------------------------------------------------------------- /Composer/airflow_dataproc_automate_dag.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Composer/airflow_dataproc_automate_dag.py -------------------------------------------------------------------------------- /Composer/dataflow_python_operator_dag.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Composer/dataflow_python_operator_dag.py -------------------------------------------------------------------------------- /Composer/ecom_customer_etl_pipeline.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Composer/ecom_customer_etl_pipeline.py -------------------------------------------------------------------------------- /Composer/gcs_to_bq_and_bq_operators.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Composer/gcs_to_bq_and_bq_operators.py -------------------------------------------------------------------------------- /Composer/https_operators_demo_dag.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Composer/https_operators_demo_dag.py -------------------------------------------------------------------------------- /Composer/python_bash_operators_dag.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Composer/python_bash_operators_dag.py -------------------------------------------------------------------------------- /Dataflow/batch_etl_avro_data_cloudsql.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Dataflow/batch_etl_avro_data_cloudsql.py -------------------------------------------------------------------------------- /Dataflow/beam_stream_data_process.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Dataflow/beam_stream_data_process.py -------------------------------------------------------------------------------- /Dataflow/dataflow_batch_demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Dataflow/dataflow_batch_demo.py -------------------------------------------------------------------------------- /Dataflow/dataflow_batch_log_process.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Dataflow/dataflow_batch_log_process.py -------------------------------------------------------------------------------- /Dataflow/process_nested_data_sql_demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Dataflow/process_nested_data_sql_demo.py -------------------------------------------------------------------------------- /Dataproc/pyspark_bq_to_gcs_demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Dataproc/pyspark_bq_to_gcs_demo.py -------------------------------------------------------------------------------- /GCP_Data_Eng_concept_files/GCP_storage_db.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/GCP_Data_Eng_concept_files/GCP_storage_db.png -------------------------------------------------------------------------------- /GCP_Data_Eng_concept_files/Initfile.txt: -------------------------------------------------------------------------------- 1 | Welcome!! 2 | -------------------------------------------------------------------------------- /GCP_Data_Eng_concept_files/gcp_etl_services.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/GCP_Data_Eng_concept_files/gcp_etl_services.png -------------------------------------------------------------------------------- /GCS/object_lifecycle_mngmnt_cli.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/GCS/object_lifecycle_mngmnt_cli.txt -------------------------------------------------------------------------------- /GCS/object_lifecylce_mngmnt.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/GCS/object_lifecylce_mngmnt.py -------------------------------------------------------------------------------- /GoogleCloudStorage/gcs_python_client_demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/GoogleCloudStorage/gcs_python_client_demo.py -------------------------------------------------------------------------------- /GoogleCloudStorage/object_versioning_cli.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/GoogleCloudStorage/object_versioning_cli.txt -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/README.md -------------------------------------------------------------------------------- /Security/secretmanager_python_connect.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/anjangcp/GCP-Data-Engineering-Demo-Codes/HEAD/Security/secretmanager_python_connect.py --------------------------------------------------------------------------------