├── .gitignore ├── Dockerfile ├── LICENSE ├── README.md ├── airflow-data ├── airflow.cfg └── creds │ └── s3 ├── dags └── dagRun.py ├── docker-compose.yml ├── requirements.txt └── sparkFiles └── sparkProcess.py /.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | .vscode/ 3 | __pycache__/ 4 | .idea/ 5 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/Dockerfile -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/README.md -------------------------------------------------------------------------------- /airflow-data/airflow.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/airflow-data/airflow.cfg -------------------------------------------------------------------------------- /airflow-data/creds/s3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/airflow-data/creds/s3 -------------------------------------------------------------------------------- /dags/dagRun.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/dags/dagRun.py -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/docker-compose.yml -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pymongo 2 | boto3 3 | pyspark 4 | psycopg2-binary -------------------------------------------------------------------------------- /sparkFiles/sparkProcess.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/renatootescu/ETL-pipeline/HEAD/sparkFiles/sparkProcess.py --------------------------------------------------------------------------------