├── Dockerfile.Airflow ├── Dockerfile.Spark ├── Dockerfile.Spark_datamechanics ├── README.md ├── dags ├── spark_etl_script_docker.py └── spark_jobs_dag.py ├── docker-compose.Airflow.yaml ├── docker-compose.Spark.yaml ├── images ├── AirflowUI_Configuration.png ├── Airflow_UI_Result.png ├── S3_bucket_Result.png ├── Spark_Result.png ├── Spark_result.png ├── index.png └── index2.png ├── requirements.txt └── requirements_copy.txt /Dockerfile.Airflow: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/Dockerfile.Airflow -------------------------------------------------------------------------------- /Dockerfile.Spark: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/Dockerfile.Spark -------------------------------------------------------------------------------- /Dockerfile.Spark_datamechanics: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/Dockerfile.Spark_datamechanics -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/README.md -------------------------------------------------------------------------------- /dags/spark_etl_script_docker.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/dags/spark_etl_script_docker.py -------------------------------------------------------------------------------- /dags/spark_jobs_dag.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/dags/spark_jobs_dag.py -------------------------------------------------------------------------------- /docker-compose.Airflow.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/docker-compose.Airflow.yaml -------------------------------------------------------------------------------- /docker-compose.Spark.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/docker-compose.Spark.yaml -------------------------------------------------------------------------------- /images/AirflowUI_Configuration.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/images/AirflowUI_Configuration.png -------------------------------------------------------------------------------- /images/Airflow_UI_Result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/images/Airflow_UI_Result.png -------------------------------------------------------------------------------- /images/S3_bucket_Result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/images/S3_bucket_Result.png -------------------------------------------------------------------------------- /images/Spark_Result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/images/Spark_Result.png -------------------------------------------------------------------------------- /images/Spark_result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/images/Spark_result.png -------------------------------------------------------------------------------- /images/index.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/images/index.png -------------------------------------------------------------------------------- /images/index2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/images/index2.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yTek01/docker-spark-airflow/HEAD/requirements.txt -------------------------------------------------------------------------------- /requirements_copy.txt: -------------------------------------------------------------------------------- 1 | lxml 2 | python-decouple 3 | requests 4 | pandas 5 | --------------------------------------------------------------------------------