├── .astro ├── config.yaml └── test_dag_integrity_default.py ├── .devcontainer └── devcontainer.json ├── .dockerignore ├── .env ├── .gitignore ├── Dockerfile ├── README.md ├── dags ├── .DS_Store ├── TOOL_TEST_DAG.py ├── ingestion │ ├── in_climate_data.py │ └── in_local_weather.py ├── load │ └── load_data.py ├── report │ └── run_streamlit_app.py ├── start.py └── transform │ └── create_reporting_table.py ├── docker-compose.override.yml ├── include ├── .DS_Store ├── climate_data │ └── temp_global.csv ├── custom_operators │ └── minio.py ├── custom_task_groups │ └── create_bucket.py ├── global_variables │ └── global_variables.py ├── meterology_utils.py ├── streamlit_app │ └── weather_v_climate_app.py └── tool_testing │ └── streamlit_test.py ├── packages.txt ├── requirements.txt ├── src ├── click_on_run_streamlit.png ├── create_reporting_table.png ├── dataset_dag_dependency.png ├── fork_and_codespaces.png ├── in_climate_data_dag.png ├── in_local_weather_dag.png ├── load_data_dag.png ├── open_airflow_ui_codespaces.png ├── open_streamlit_codespaces.png ├── run_streamlit_app.png ├── start_dag.png └── streamlit_app.png └── tests └── dags └── test_dag_integrity.py /.astro/config.yaml: -------------------------------------------------------------------------------- 1 | project: 2 | name: codespaces -------------------------------------------------------------------------------- /.astro/test_dag_integrity_default.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/.astro/test_dag_integrity_default.py -------------------------------------------------------------------------------- /.devcontainer/devcontainer.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/.devcontainer/devcontainer.json -------------------------------------------------------------------------------- /.dockerignore: -------------------------------------------------------------------------------- 1 | astro 2 | .git 3 | .env 4 | airflow_settings.yaml 5 | logs/ -------------------------------------------------------------------------------- /.env: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/.env -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/.gitignore -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/Dockerfile -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/README.md -------------------------------------------------------------------------------- /dags/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/.DS_Store -------------------------------------------------------------------------------- /dags/TOOL_TEST_DAG.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/TOOL_TEST_DAG.py -------------------------------------------------------------------------------- /dags/ingestion/in_climate_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/ingestion/in_climate_data.py -------------------------------------------------------------------------------- /dags/ingestion/in_local_weather.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/ingestion/in_local_weather.py -------------------------------------------------------------------------------- /dags/load/load_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/load/load_data.py -------------------------------------------------------------------------------- /dags/report/run_streamlit_app.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/report/run_streamlit_app.py -------------------------------------------------------------------------------- /dags/start.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/start.py -------------------------------------------------------------------------------- /dags/transform/create_reporting_table.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/dags/transform/create_reporting_table.py -------------------------------------------------------------------------------- /docker-compose.override.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/docker-compose.override.yml -------------------------------------------------------------------------------- /include/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/.DS_Store -------------------------------------------------------------------------------- /include/climate_data/temp_global.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/climate_data/temp_global.csv -------------------------------------------------------------------------------- /include/custom_operators/minio.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/custom_operators/minio.py -------------------------------------------------------------------------------- /include/custom_task_groups/create_bucket.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/custom_task_groups/create_bucket.py -------------------------------------------------------------------------------- /include/global_variables/global_variables.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/global_variables/global_variables.py -------------------------------------------------------------------------------- /include/meterology_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/meterology_utils.py -------------------------------------------------------------------------------- /include/streamlit_app/weather_v_climate_app.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/streamlit_app/weather_v_climate_app.py -------------------------------------------------------------------------------- /include/tool_testing/streamlit_test.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/include/tool_testing/streamlit_test.py -------------------------------------------------------------------------------- /packages.txt: -------------------------------------------------------------------------------- 1 | build-essential -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/requirements.txt -------------------------------------------------------------------------------- /src/click_on_run_streamlit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/click_on_run_streamlit.png -------------------------------------------------------------------------------- /src/create_reporting_table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/create_reporting_table.png -------------------------------------------------------------------------------- /src/dataset_dag_dependency.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/dataset_dag_dependency.png -------------------------------------------------------------------------------- /src/fork_and_codespaces.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/fork_and_codespaces.png -------------------------------------------------------------------------------- /src/in_climate_data_dag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/in_climate_data_dag.png -------------------------------------------------------------------------------- /src/in_local_weather_dag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/in_local_weather_dag.png -------------------------------------------------------------------------------- /src/load_data_dag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/load_data_dag.png -------------------------------------------------------------------------------- /src/open_airflow_ui_codespaces.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/open_airflow_ui_codespaces.png -------------------------------------------------------------------------------- /src/open_streamlit_codespaces.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/open_streamlit_codespaces.png -------------------------------------------------------------------------------- /src/run_streamlit_app.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/run_streamlit_app.png -------------------------------------------------------------------------------- /src/start_dag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/start_dag.png -------------------------------------------------------------------------------- /src/streamlit_app.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/src/streamlit_app.png -------------------------------------------------------------------------------- /tests/dags/test_dag_integrity.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TJaniF/airflow-elt-blueprint/HEAD/tests/dags/test_dag_integrity.py --------------------------------------------------------------------------------