├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Config ├── LICENSE ├── README.md ├── app.py ├── cdk.json ├── config.json ├── deploy.sh ├── infrastructure ├── __init__.py ├── ecs_stack.py ├── emr_launch │ ├── README.md │ ├── __pycache__ │ │ ├── cluster_definition.cpython-37.pyc │ │ └── instance_group_config.cpython-37.pyc │ ├── bootstrap_actions │ │ └── install_boto3.sh │ ├── cluster_definition.py │ └── instance_group_config.py ├── emr_orchestration │ ├── __init__.py │ ├── __pycache__ │ │ ├── __init__.cpython-37.pyc │ │ └── stack.cpython-37.pyc │ └── stack.py ├── emr_trigger │ ├── __init__.py │ ├── __pycache__ │ │ ├── __init__.cpython-37.pyc │ │ └── stack.cpython-37.pyc │ ├── lambda_source │ │ └── trigger.py │ └── stack.py ├── infrastructure.egg-info │ ├── PKG-INFO │ ├── SOURCES.txt │ ├── dependency_links.txt │ ├── requires.txt │ └── top_level.txt └── lambda_function.py ├── requirements.txt ├── service ├── Dockerfile ├── app │ ├── main.py │ └── test.py └── requirements.txt ├── setup.py └── spark_scripts ├── detect_scenes.py └── synchronize_topics.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .env/ 3 | .idea/ 4 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/CODE_OF_CONDUCT.md -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/CONTRIBUTING.md -------------------------------------------------------------------------------- /Config: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/Config -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/README.md -------------------------------------------------------------------------------- /app.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/app.py -------------------------------------------------------------------------------- /cdk.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/cdk.json -------------------------------------------------------------------------------- /config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/config.json -------------------------------------------------------------------------------- /deploy.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/deploy.sh -------------------------------------------------------------------------------- /infrastructure/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infrastructure/ecs_stack.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/ecs_stack.py -------------------------------------------------------------------------------- /infrastructure/emr_launch/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_launch/README.md -------------------------------------------------------------------------------- /infrastructure/emr_launch/__pycache__/cluster_definition.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_launch/__pycache__/cluster_definition.cpython-37.pyc -------------------------------------------------------------------------------- /infrastructure/emr_launch/__pycache__/instance_group_config.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_launch/__pycache__/instance_group_config.cpython-37.pyc -------------------------------------------------------------------------------- /infrastructure/emr_launch/bootstrap_actions/install_boto3.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | sudo python3 -m pip install boto3 -------------------------------------------------------------------------------- /infrastructure/emr_launch/cluster_definition.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_launch/cluster_definition.py -------------------------------------------------------------------------------- /infrastructure/emr_launch/instance_group_config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_launch/instance_group_config.py -------------------------------------------------------------------------------- /infrastructure/emr_orchestration/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infrastructure/emr_orchestration/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_orchestration/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /infrastructure/emr_orchestration/__pycache__/stack.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_orchestration/__pycache__/stack.cpython-37.pyc -------------------------------------------------------------------------------- /infrastructure/emr_orchestration/stack.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_orchestration/stack.py -------------------------------------------------------------------------------- /infrastructure/emr_trigger/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infrastructure/emr_trigger/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_trigger/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /infrastructure/emr_trigger/__pycache__/stack.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_trigger/__pycache__/stack.cpython-37.pyc -------------------------------------------------------------------------------- /infrastructure/emr_trigger/lambda_source/trigger.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_trigger/lambda_source/trigger.py -------------------------------------------------------------------------------- /infrastructure/emr_trigger/stack.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/emr_trigger/stack.py -------------------------------------------------------------------------------- /infrastructure/infrastructure.egg-info/PKG-INFO: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/infrastructure.egg-info/PKG-INFO -------------------------------------------------------------------------------- /infrastructure/infrastructure.egg-info/SOURCES.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/infrastructure.egg-info/SOURCES.txt -------------------------------------------------------------------------------- /infrastructure/infrastructure.egg-info/dependency_links.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /infrastructure/infrastructure.egg-info/requires.txt: -------------------------------------------------------------------------------- 1 | aws-cdk.core==1.83.0 2 | -------------------------------------------------------------------------------- /infrastructure/infrastructure.egg-info/top_level.txt: -------------------------------------------------------------------------------- 1 | emr_orchestration 2 | emr_trigger 3 | -------------------------------------------------------------------------------- /infrastructure/lambda_function.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/infrastructure/lambda_function.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/requirements.txt -------------------------------------------------------------------------------- /service/Dockerfile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/service/Dockerfile -------------------------------------------------------------------------------- /service/app/main.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/service/app/main.py -------------------------------------------------------------------------------- /service/app/test.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/service/app/test.py -------------------------------------------------------------------------------- /service/requirements.txt: -------------------------------------------------------------------------------- 1 | boto3 2 | pandas 3 | bagpy 4 | fastparquet -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/setup.py -------------------------------------------------------------------------------- /spark_scripts/detect_scenes.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/spark_scripts/detect_scenes.py -------------------------------------------------------------------------------- /spark_scripts/synchronize_topics.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-scene-detection-pipeline/HEAD/spark_scripts/synchronize_topics.py --------------------------------------------------------------------------------