├── .vscode ├── .ropeproject │ └── config.py └── settings.json ├── Blogs_Code ├── Airflow_Configuration.MD ├── DAG │ ├── Readme.md │ ├── lambda_hook_invoke.py │ └── lambda_invoke.py ├── New Text Document.txt ├── record_transform.py ├── sam_deploy.zip ├── send_records.py └── win_feature.PNG ├── Contents ├── Dataset.MD ├── Pipelines.MD ├── Tools.MD └── Tools_Detail.MD ├── Images ├── RBAC.png ├── daily_batch.jpg ├── data_view.PNG ├── dtree.PNG ├── packagetree.PNG ├── readme.MD ├── stream_processing.jpg ├── testresult.PNG ├── tools_used.jpg ├── ubuntu.PNG ├── ui1.png ├── ui2.png ├── ui3.png ├── visual.gif └── win_feature.PNG ├── Pipelines ├── README.md ├── README.md.template.txt ├── dynamo.py ├── kinesis_to_s3.py ├── s3_to_kinesis.py └── web_scrape_jobs.py /.vscode/.ropeproject/config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/.vscode/.ropeproject/config.py -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "python.linting.enabled": true 3 | } -------------------------------------------------------------------------------- /Blogs_Code/Airflow_Configuration.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/Airflow_Configuration.MD -------------------------------------------------------------------------------- /Blogs_Code/DAG/Readme.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/DAG/Readme.md -------------------------------------------------------------------------------- /Blogs_Code/DAG/lambda_hook_invoke.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/DAG/lambda_hook_invoke.py -------------------------------------------------------------------------------- /Blogs_Code/DAG/lambda_invoke.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/DAG/lambda_invoke.py -------------------------------------------------------------------------------- /Blogs_Code/New Text Document.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /Blogs_Code/record_transform.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/record_transform.py -------------------------------------------------------------------------------- /Blogs_Code/sam_deploy.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/sam_deploy.zip -------------------------------------------------------------------------------- /Blogs_Code/send_records.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/send_records.py -------------------------------------------------------------------------------- /Blogs_Code/win_feature.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Blogs_Code/win_feature.PNG -------------------------------------------------------------------------------- /Contents/Dataset.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Contents/Dataset.MD -------------------------------------------------------------------------------- /Contents/Pipelines.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Contents/Pipelines.MD -------------------------------------------------------------------------------- /Contents/Tools.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Contents/Tools.MD -------------------------------------------------------------------------------- /Contents/Tools_Detail.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Contents/Tools_Detail.MD -------------------------------------------------------------------------------- /Images/RBAC.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/RBAC.png -------------------------------------------------------------------------------- /Images/daily_batch.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/daily_batch.jpg -------------------------------------------------------------------------------- /Images/data_view.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/data_view.PNG -------------------------------------------------------------------------------- /Images/dtree.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/dtree.PNG -------------------------------------------------------------------------------- /Images/packagetree.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/packagetree.PNG -------------------------------------------------------------------------------- /Images/readme.MD: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Images/stream_processing.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/stream_processing.jpg -------------------------------------------------------------------------------- /Images/testresult.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/testresult.PNG -------------------------------------------------------------------------------- /Images/tools_used.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/tools_used.jpg -------------------------------------------------------------------------------- /Images/ubuntu.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/ubuntu.PNG -------------------------------------------------------------------------------- /Images/ui1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/ui1.png -------------------------------------------------------------------------------- /Images/ui2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/ui2.png -------------------------------------------------------------------------------- /Images/ui3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/ui3.png -------------------------------------------------------------------------------- /Images/visual.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/visual.gif -------------------------------------------------------------------------------- /Images/win_feature.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/Images/win_feature.PNG -------------------------------------------------------------------------------- /Pipelines: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/README.md -------------------------------------------------------------------------------- /README.md.template.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/README.md.template.txt -------------------------------------------------------------------------------- /dynamo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/dynamo.py -------------------------------------------------------------------------------- /kinesis_to_s3.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/kinesis_to_s3.py -------------------------------------------------------------------------------- /s3_to_kinesis.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/s3_to_kinesis.py -------------------------------------------------------------------------------- /web_scrape_jobs.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vijaykothareddy/Data-Engineering/HEAD/web_scrape_jobs.py --------------------------------------------------------------------------------