├── ETL_toll_data.py ├── README.md ├── configure_simulator.jpg ├── consolidate_data.jpg ├── create_toll_topic.jpg ├── dag_args.jpg ├── dag_definition.jpg ├── dag_runs.jpg ├── data_reader_output.jpg ├── download_simulator.jpg ├── exit ├── extract_data_from_csv.jpg ├── extract_data_from_fixed_width.jpg ├── extract_data_from_tsv.jpg ├── output_rows.jpg ├── simulator_output.jpg ├── start_kafka.jpg ├── start_zookeeper.jpg ├── streaming_reader_code.jpg ├── submit_dag.jpg ├── task_pipeline.jpg ├── transform.jpg ├── unpause_dag.jpg └── unzip_data.jpg /ETL_toll_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/ETL_toll_data.py -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/README.md -------------------------------------------------------------------------------- /configure_simulator.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/configure_simulator.jpg -------------------------------------------------------------------------------- /consolidate_data.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/consolidate_data.jpg -------------------------------------------------------------------------------- /create_toll_topic.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/create_toll_topic.jpg -------------------------------------------------------------------------------- /dag_args.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/dag_args.jpg -------------------------------------------------------------------------------- /dag_definition.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/dag_definition.jpg -------------------------------------------------------------------------------- /dag_runs.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/dag_runs.jpg -------------------------------------------------------------------------------- /data_reader_output.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/data_reader_output.jpg -------------------------------------------------------------------------------- /download_simulator.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/download_simulator.jpg -------------------------------------------------------------------------------- /exit: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/exit -------------------------------------------------------------------------------- /extract_data_from_csv.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/extract_data_from_csv.jpg -------------------------------------------------------------------------------- /extract_data_from_fixed_width.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/extract_data_from_fixed_width.jpg -------------------------------------------------------------------------------- /extract_data_from_tsv.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/extract_data_from_tsv.jpg -------------------------------------------------------------------------------- /output_rows.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/output_rows.jpg -------------------------------------------------------------------------------- /simulator_output.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/simulator_output.jpg -------------------------------------------------------------------------------- /start_kafka.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/start_kafka.jpg -------------------------------------------------------------------------------- /start_zookeeper.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/start_zookeeper.jpg -------------------------------------------------------------------------------- /streaming_reader_code.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/streaming_reader_code.jpg -------------------------------------------------------------------------------- /submit_dag.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/submit_dag.jpg -------------------------------------------------------------------------------- /task_pipeline.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/task_pipeline.jpg -------------------------------------------------------------------------------- /transform.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/transform.jpg -------------------------------------------------------------------------------- /unpause_dag.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/unpause_dag.jpg -------------------------------------------------------------------------------- /unzip_data.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mboccenti/ETL-and-Data-Pipelines-with-Shell-Airflow-and-Kafka/HEAD/unzip_data.jpg --------------------------------------------------------------------------------