├── .gitignore ├── README.md ├── appendix_a_example_code └── README.md ├── ch_01_gcp_account.md ├── ch_02_orchestration.md ├── ch_03_data_lake.md ├── ch_04_data_warehouse.md ├── ch_05_dags.md ├── ch_06_event_triggers.md ├── ch_07_parallel_processing.md ├── ch_08_streaming.md ├── ch_09_secrets.md ├── ch_10_infrastructure_as_code.md ├── ch_11_deployment_pipelines.md ├── ch_12_monitoring.md ├── ch_13_up_and_running.md └── images ├── airflow_dag_graph.png ├── airflow_dag_sample_1.png ├── airflow_web_ui_1.png ├── airflow_web_ui_2.png ├── airflow_web_ui_3.png ├── bigquery_console_1.png ├── composer_console.png ├── composer_console_2.png ├── gcp_create_key.png ├── gcp_create_project.png ├── gcp_dashboard.png ├── gcp_home_page.png ├── gcp_service_accounts_page.png ├── gcp_triggers_1.png ├── github-authorization.png ├── github-delete-repo.png ├── github_build_status.png └── github_repo.png /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/README.md -------------------------------------------------------------------------------- /appendix_a_example_code/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/appendix_a_example_code/README.md -------------------------------------------------------------------------------- /ch_01_gcp_account.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_01_gcp_account.md -------------------------------------------------------------------------------- /ch_02_orchestration.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_02_orchestration.md -------------------------------------------------------------------------------- /ch_03_data_lake.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_03_data_lake.md -------------------------------------------------------------------------------- /ch_04_data_warehouse.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_04_data_warehouse.md -------------------------------------------------------------------------------- /ch_05_dags.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_05_dags.md -------------------------------------------------------------------------------- /ch_06_event_triggers.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_06_event_triggers.md -------------------------------------------------------------------------------- /ch_07_parallel_processing.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_07_parallel_processing.md -------------------------------------------------------------------------------- /ch_08_streaming.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_08_streaming.md -------------------------------------------------------------------------------- /ch_09_secrets.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_09_secrets.md -------------------------------------------------------------------------------- /ch_10_infrastructure_as_code.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_10_infrastructure_as_code.md -------------------------------------------------------------------------------- /ch_11_deployment_pipelines.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_11_deployment_pipelines.md -------------------------------------------------------------------------------- /ch_12_monitoring.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_12_monitoring.md -------------------------------------------------------------------------------- /ch_13_up_and_running.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/ch_13_up_and_running.md -------------------------------------------------------------------------------- /images/airflow_dag_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/airflow_dag_graph.png -------------------------------------------------------------------------------- /images/airflow_dag_sample_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/airflow_dag_sample_1.png -------------------------------------------------------------------------------- /images/airflow_web_ui_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/airflow_web_ui_1.png -------------------------------------------------------------------------------- /images/airflow_web_ui_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/airflow_web_ui_2.png -------------------------------------------------------------------------------- /images/airflow_web_ui_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/airflow_web_ui_3.png -------------------------------------------------------------------------------- /images/bigquery_console_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/bigquery_console_1.png -------------------------------------------------------------------------------- /images/composer_console.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/composer_console.png -------------------------------------------------------------------------------- /images/composer_console_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/composer_console_2.png -------------------------------------------------------------------------------- /images/gcp_create_key.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/gcp_create_key.png -------------------------------------------------------------------------------- /images/gcp_create_project.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/gcp_create_project.png -------------------------------------------------------------------------------- /images/gcp_dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/gcp_dashboard.png -------------------------------------------------------------------------------- /images/gcp_home_page.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/gcp_home_page.png -------------------------------------------------------------------------------- /images/gcp_service_accounts_page.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/gcp_service_accounts_page.png -------------------------------------------------------------------------------- /images/gcp_triggers_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/gcp_triggers_1.png -------------------------------------------------------------------------------- /images/github-authorization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/github-authorization.png -------------------------------------------------------------------------------- /images/github-delete-repo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/github-delete-repo.png -------------------------------------------------------------------------------- /images/github_build_status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/github_build_status.png -------------------------------------------------------------------------------- /images/github_repo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nunie123/data_engineering_on_gcp_book/HEAD/images/github_repo.png --------------------------------------------------------------------------------