├── .gitignore ├── README.md ├── capstone_notebook.ipynb ├── dags ├── __init__.py └── etl.py ├── images ├── airflow.png ├── dag_graph.png ├── dask.png ├── map.png ├── redshift.png └── schema.png ├── plugins ├── __init__.py ├── operators │ ├── __init__.py │ └── data_quality.py └── utils │ ├── __init__.py │ ├── constants.py │ └── helper.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/.gitignore -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/README.md -------------------------------------------------------------------------------- /capstone_notebook.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/capstone_notebook.ipynb -------------------------------------------------------------------------------- /dags/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /dags/etl.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/dags/etl.py -------------------------------------------------------------------------------- /images/airflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/images/airflow.png -------------------------------------------------------------------------------- /images/dag_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/images/dag_graph.png -------------------------------------------------------------------------------- /images/dask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/images/dask.png -------------------------------------------------------------------------------- /images/map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/images/map.png -------------------------------------------------------------------------------- /images/redshift.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/images/redshift.png -------------------------------------------------------------------------------- /images/schema.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/images/schema.png -------------------------------------------------------------------------------- /plugins/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /plugins/operators/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /plugins/operators/data_quality.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/plugins/operators/data_quality.py -------------------------------------------------------------------------------- /plugins/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /plugins/utils/constants.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/plugins/utils/constants.py -------------------------------------------------------------------------------- /plugins/utils/helper.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danieldiamond/data-engineering-capstone/HEAD/plugins/utils/helper.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | regex 2 | pandas 3 | s3fs 4 | psycopg2-binary 5 | cryptography 6 | boto3 7 | --------------------------------------------------------------------------------