├── .gitattributes ├── .github └── workflows │ └── manual.yml ├── .gitignore ├── CODEOWNERS ├── Lesson 2 - Autonomous Vehicles and Computer Vision └── The OpenCV Library │ ├── .student_bashrc │ ├── OpenCV_exercises │ ├── CMakeLists.txt │ ├── images │ │ ├── img0005.jpg │ │ ├── img0005.png │ │ ├── img0006.jpg │ │ ├── img0006.png │ │ ├── img0007.jpg │ │ ├── img0007.png │ │ ├── img0008.jpg │ │ ├── img0008.png │ │ ├── img0009.jpg │ │ ├── img0009.png │ │ ├── img1.png │ │ ├── img1gray.png │ │ └── img2gray.png │ └── src │ │ ├── change_pixels.cpp │ │ ├── create_matrix.cpp │ │ ├── load_image_1.cpp │ │ ├── load_image_2.cpp │ │ └── load_image_3.cpp │ └── solutions │ ├── change_pixels.cpp │ ├── create_matrix.cpp │ └── load_image_3.cpp ├── Lesson 3 - Engineering a Collision Detection System ├── Estimating TTC with Camera │ ├── .student_bashrc │ ├── TTC_camera │ │ ├── CMakeLists.txt │ │ ├── dat │ │ │ ├── C23A5_KptMatches_AKAZE.dat │ │ │ ├── C23A5_KptMatches_SHI-BRISK.dat │ │ │ ├── C23A5_KptsRef_AKAZE.dat │ │ │ ├── C23A5_KptsRef_SHI-BRISK.dat │ │ │ ├── C23A5_KptsSource_AKAZE.dat │ │ │ └── C23A5_KptsSource_SHI-BRISK.dat │ │ └── src │ │ │ ├── compute_ttc_camera.cpp │ │ │ ├── dataStructures.h │ │ │ ├── structIO.cpp │ │ │ └── structIO.hpp │ └── solution │ │ └── compute_ttc_camera.cpp └── Estimating TTC with Lidar │ ├── .student_bashrc │ ├── TTC_lidar │ ├── CMakeLists.txt │ ├── dat │ │ ├── C22A5_currLidarPts.dat │ │ └── C22A5_prevLidarPts.dat │ └── src │ │ ├── compute_ttc_lidar.cpp │ │ ├── dataStructures.h │ │ ├── structIO.cpp │ │ └── structIO.hpp │ └── solution │ └── compute_ttc_lidar.cpp ├── Lesson 4 - Tracking Image Features ├── Descriptor Matching │ ├── .student_bashrc │ ├── descriptor_matching │ │ ├── CMakeLists.txt │ │ ├── dat │ │ │ ├── C35A5_DescRef_BRISK_large.dat │ │ │ ├── C35A5_DescRef_BRISK_small.dat │ │ │ ├── C35A5_DescRef_SIFT.dat │ │ │ ├── C35A5_DescSource_BRISK_large.dat │ │ │ ├── C35A5_DescSource_BRISK_small.dat │ │ │ ├── C35A5_DescSource_SIFT.dat │ │ │ ├── C35A5_KptsRef_BRISK_large.dat │ │ │ ├── C35A5_KptsRef_BRISK_small.dat │ │ │ ├── C35A5_KptsRef_SIFT.dat │ │ │ ├── C35A5_KptsSource_BRISK_large.dat │ │ │ ├── C35A5_KptsSource_BRISK_small.dat │ │ │ └── C35A5_KptsSource_SIFT.dat │ │ ├── images │ │ │ ├── img0005.jpg │ │ │ ├── img0005.png │ │ │ ├── img0006.jpg │ │ │ ├── img0006.png │ │ │ ├── img0007.jpg │ │ │ ├── img0007.png │ │ │ ├── img0008.jpg │ │ │ ├── img0008.png │ │ │ ├── img0009.jpg │ │ │ ├── img0009.png │ │ │ ├── img1.png │ │ │ ├── img1gray.png │ │ │ └── img2gray.png │ │ └── src │ │ │ ├── dataStructures.h │ │ │ ├── descriptor_matching.cpp │ │ │ ├── structIO.cpp │ │ │ └── structIO.hpp │ └── solution │ │ └── descriptor_matching.cpp ├── Gradient-based vs. Binary Descriptors │ ├── .student_bashrc │ ├── describe_keypoints │ │ ├── CMakeLists.txt │ │ ├── images │ │ │ ├── img0005.jpg │ │ │ ├── img0005.png │ │ │ ├── img0006.jpg │ │ │ ├── img0006.png │ │ │ ├── img0007.jpg │ │ │ ├── img0007.png │ │ │ ├── img0008.jpg │ │ │ ├── img0008.png │ │ │ ├── img0009.jpg │ │ │ ├── img0009.png │ │ │ ├── img1.png │ │ │ ├── img1gray.png │ │ │ └── img2gray.png │ │ └── src │ │ │ └── describe_keypoints.cpp │ └── solution │ │ └── describe_keypoints.cpp ├── Harris Corner Detection │ ├── .student_bashrc │ ├── cornerness_harris │ │ ├── CMakeLists.txt │ │ ├── images │ │ │ ├── img0005.jpg │ │ │ ├── img0005.png │ │ │ ├── img0006.jpg │ │ │ ├── img0006.png │ │ │ ├── img0007.jpg │ │ │ ├── img0007.png │ │ │ ├── img0008.jpg │ │ │ ├── img0008.png │ │ │ ├── img0009.jpg │ │ │ ├── img0009.png │ │ │ ├── img1.png │ │ │ ├── img1gray.png │ │ │ └── img2gray.png │ │ └── src │ │ │ └── cornerness_harris.cpp │ └── solution │ │ └── cornerness_harris.cpp ├── Intensity Gradient and Filtering │ ├── .student_bashrc │ ├── gradient_filtering │ │ ├── CMakeLists.txt │ │ ├── images │ │ │ ├── img0005.jpg │ │ │ ├── img0005.png │ │ │ ├── img0006.jpg │ │ │ ├── img0006.png │ │ │ ├── img0007.jpg │ │ │ ├── img0007.png │ │ │ ├── img0008.jpg │ │ │ ├── img0008.png │ │ │ ├── img0009.jpg │ │ │ ├── img0009.png │ │ │ ├── img1.png │ │ │ ├── img1gray.png │ │ │ └── img2gray.png │ │ └── src │ │ │ ├── gaussian_smoothing.cpp │ │ │ ├── gradient_sobel.cpp │ │ │ └── magnitude_sobel.cpp │ └── solutions │ │ ├── gaussian_smoothing.cpp │ │ └── magnitude_sobel.cpp └── Overview of Popular Keypoint Detectors │ ├── .student_bashrc │ ├── detect_keypoints │ ├── CMakeLists.txt │ ├── images │ │ ├── img0005.jpg │ │ ├── img0005.png │ │ ├── img0006.jpg │ │ ├── img0006.png │ │ ├── img0007.jpg │ │ ├── img0007.png │ │ ├── img0008.jpg │ │ ├── img0008.png │ │ ├── img0009.jpg │ │ ├── img0009.png │ │ ├── img1.png │ │ ├── img1gray.png │ │ └── img2gray.png │ └── src │ │ └── detect_keypoints.cpp │ └── solution │ └── detect_keypoints.cpp ├── Lesson 6 - Combining Camera and Lidar ├── Creating 3D-Objects │ ├── .student_bashrc │ ├── cluster_with_roi │ │ ├── CMakeLists.txt │ │ ├── dat │ │ │ ├── C53A3_currBoundingBoxes.dat │ │ │ └── C53A3_currLidarPts.dat │ │ └── src │ │ │ ├── cluster_with_roi.cpp │ │ │ ├── dataStructures.h │ │ │ ├── structIO.cpp │ │ │ └── structIO.hpp │ └── solution │ │ └── clusterLidarWithROI.cpp ├── Lidar-to-Camera Point Projection │ ├── .student_bashrc │ ├── lidar_to_camera │ │ ├── CMakeLists.txt │ │ ├── dat │ │ │ ├── C51_LidarPts_0000.dat │ │ │ ├── C51_LidarPts_0001.dat │ │ │ ├── C51_LidarPts_0002.dat │ │ │ ├── C51_LidarPts_0003.dat │ │ │ ├── C51_LidarPts_0004.dat │ │ │ ├── C51_LidarPts_0005.dat │ │ │ ├── C51_LidarPts_0006.dat │ │ │ ├── C51_LidarPts_0007.dat │ │ │ ├── C51_LidarPts_0008.dat │ │ │ ├── C51_LidarPts_0009.dat │ │ │ ├── calib_cam_to_cam.txt │ │ │ └── calib_velo_to_cam.txt │ │ ├── images │ │ │ ├── 0000000000.png │ │ │ └── s_thrun.jpg │ │ └── src │ │ │ ├── dataStructures.h │ │ │ ├── project_lidar_to_camera.cpp │ │ │ ├── show_lidar_top_view.cpp │ │ │ ├── structIO.cpp │ │ │ └── structIO.hpp │ └── solutions │ │ ├── filter_lidar_points.cpp │ │ ├── project_lidar_to_camera.cpp │ │ └── show_lidar_top_view.cpp └── Object Detection with YOLO │ ├── .student_bashrc │ └── detect_objects │ ├── CMakeLists.txt │ ├── dat │ └── yolo │ │ ├── coco.names │ │ ├── yolov3-tiny.cfg │ │ ├── yolov3-tiny.weights │ │ ├── yolov3.cfg │ │ └── yolov3.weights │ ├── images │ ├── 0000000000.png │ └── s_thrun.jpg │ └── src │ ├── dataStructures.h │ └── detect_objects_2.cpp └── README.md /.gitattributes: -------------------------------------------------------------------------------- 1 | yolov3.weights filter=lfs diff=lfs merge=lfs -text 2 | -------------------------------------------------------------------------------- /.github/workflows/manual.yml: -------------------------------------------------------------------------------- 1 | # Workflow to ensure whenever a Github PR is submitted, 2 | # a JIRA ticket gets created automatically. 3 | name: Manual Workflow 4 | 5 | # Controls when the action will run. 6 | on: 7 | # Triggers the workflow on pull request events but only for the master branch 8 | pull_request_target: 9 | types: [opened, reopened] 10 | 11 | # Allows you to run this workflow manually from the Actions tab 12 | workflow_dispatch: 13 | 14 | jobs: 15 | test-transition-issue: 16 | name: Convert Github Issue to Jira Issue 17 | runs-on: ubuntu-latest 18 | steps: 19 | - name: Checkout 20 | uses: actions/checkout@master 21 | 22 | - name: Login 23 | uses: atlassian/gajira-login@master 24 | env: 25 | JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }} 26 | JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }} 27 | JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }} 28 | 29 | - name: Create NEW JIRA ticket 30 | id: create 31 | uses: atlassian/gajira-create@master 32 | with: 33 | project: CONUPDATE 34 | issuetype: Task 35 | summary: | 36 | Github PR [Assign the ND component] | Repo: ${{ github.repository }} | PR# ${{github.event.number}} 37 | description: | 38 | Repo link: https://github.com/${{ github.repository }} 39 | PR no. ${{ github.event.pull_request.number }} 40 | PR title: ${{ github.event.pull_request.title }} 41 | PR description: ${{ github.event.pull_request.description }} 42 | In addition, please resolve other issues, if any. 43 | fields: '{"components": [{"name":"Github PR"}], "customfield_16449":"https://classroom.udacity.com/", "customfield_16450":"Resolve the PR", "labels": ["github"], "priority":{"id": "4"}}' 44 | 45 | - name: Log created issue 46 | run: echo "Issue ${{ steps.create.outputs.issue }} was created" 47 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | **.DS_Store 2 | **.vscode -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @udacity/active-public-content -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/OpenCV_exercises -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executable for create matrix exercise 17 | add_executable (create_matrix src/create_matrix.cpp) 18 | target_link_libraries (create_matrix ${OpenCV_LIBRARIES}) 19 | 20 | # Executable for change pixels exercise 21 | add_executable (change_pixels src/change_pixels.cpp) 22 | target_link_libraries (change_pixels ${OpenCV_LIBRARIES}) 23 | 24 | # Executables for load image exercises 25 | add_executable (load_image_1 src/load_image_1.cpp) 26 | target_link_libraries(load_image_1 ${OpenCV_LIBRARIES}) 27 | 28 | add_executable (load_image_2 src/load_image_2.cpp) 29 | target_link_libraries(load_image_2 ${OpenCV_LIBRARIES}) 30 | 31 | add_executable (load_image_3 src/load_image_3.cpp) 32 | target_link_libraries(load_image_3 ${OpenCV_LIBRARIES}) 33 | -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0005.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0005.jpg -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0005.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0005.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0006.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0006.jpg -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0006.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0007.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0007.jpg -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0007.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0008.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0008.jpg -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0008.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0008.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0009.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0009.jpg -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0009.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img0009.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img1.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img1gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img1gray.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img2gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/images/img2gray.png -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/src/change_pixels.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | 6 | using namespace std; 7 | 8 | void changePixels() 9 | { 10 | // create matrix 11 | int nrows = 480, ncols = 640; 12 | cv::Mat m1_8u; 13 | m1_8u.create(nrows, ncols, CV_8UC1); // two-channel matrix with 8bit unsigned elements 14 | m1_8u.setTo(cv::Scalar(0)); //black 15 | 16 | for (int r = 230; r < 250; r++) 17 | { 18 | // STUDENT TASK : loop over all columns and set matrix elements to 255 19 | 20 | } 21 | 22 | // show result 23 | string windowName = "First steps in OpenCV"; 24 | cv::namedWindow(windowName, 1); // create window 25 | cv::imshow(windowName, m1_8u); 26 | cv::waitKey(0); // wait for keyboard input before continuing 27 | } 28 | 29 | 30 | int main() 31 | { 32 | changePixels(); 33 | return 0; 34 | } -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/src/create_matrix.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | 6 | using namespace std; 7 | 8 | void createMatrix1() 9 | { 10 | // create matrix 11 | int nrows = 480, ncols = 640; 12 | cv::Mat m1_8u; 13 | m1_8u.create(nrows, ncols, CV_8UC1); // two-channel matrix with 8bit unsigned elements 14 | m1_8u.setTo(cv::Scalar(255)); // white 15 | 16 | // STUDENT TASK : 17 | // Create a variable of type cv::Mat* named m3_8u which has three channels with a 18 | // depth of 8bit per channel. Then, set the first channel to 255 and display the result. 19 | 20 | 21 | // show result 22 | string windowName = "First steps in OpenCV (m1_8u)"; 23 | cv::namedWindow(windowName, 1); // create window 24 | cv::imshow(windowName, m1_8u); 25 | cv::waitKey(0); // wait for keyboard input before continuing 26 | 27 | // STUDENT TASK : 28 | // Display the results from the STUDENT TASK above 29 | 30 | } 31 | 32 | 33 | int main() 34 | { 35 | createMatrix1(); 36 | return 0; 37 | } -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/src/load_image_1.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | using namespace std; 8 | 9 | void loadImage1() 10 | { 11 | // load image from file 12 | cv::Mat img; 13 | img = cv::imread("../images/img1.png"); 14 | 15 | // show result 16 | string windowName = "First steps in OpenCV"; 17 | cv::namedWindow(windowName, 1); // create window 18 | cv::imshow(windowName, img); 19 | cv::waitKey(0); // wait for keyboard input before continuing 20 | } 21 | 22 | int main() 23 | { 24 | loadImage1(); 25 | return 0; 26 | } -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/src/load_image_2.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | void loadImage2() 11 | { 12 | for (int i = 5; i <= 9; i++) 13 | { 14 | 15 | // create file name 16 | ostringstream imgNumber; // #include 17 | imgNumber << setfill('0') << setw(4) << i; // #include 18 | string filename = "../images/img" + imgNumber.str() + ".png"; 19 | 20 | // load and display image 21 | cv::Mat img; 22 | img = cv::imread(filename); 23 | string windowName = "First steps in OpenCV"; 24 | cv::namedWindow(windowName, 1); // create window 25 | cv::imshow(windowName, img); 26 | cv::waitKey(0); // wait for keyboard input before continuing 27 | } 28 | } 29 | 30 | int main() 31 | { 32 | loadImage2(); 33 | return 0; 34 | } -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/OpenCV_exercises/src/load_image_3.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | void loadImage3() 11 | { 12 | // load images 13 | vector imgList; 14 | for (int i = 5; i <= 9; i++) 15 | { 16 | // create file name 17 | ostringstream imgNumber; // #include 18 | imgNumber << setfill('0') << setw(4) << i; // #include 19 | string filename = "../images/img" + imgNumber.str() + ".jpg"; 20 | 21 | // load image and store it into a vector 22 | cv::Mat img; 23 | img = cv::imread(filename); 24 | imgList.push_back(img); // store pointer to current image in list 25 | } 26 | 27 | // display images from the vector 28 | string windowName = "First steps in OpenCV"; 29 | cv::namedWindow(windowName, 1); // create window 30 | for (auto it = imgList.begin(); it != imgList.end(); ++it) 31 | { 32 | 33 | // STUDENT TASK : Prevent image 7 from being displayed 34 | 35 | // display image 36 | cv::imshow(windowName, *it); 37 | cv::waitKey(0); // wait for keyboard input before continuing 38 | } 39 | } 40 | 41 | int main() 42 | { 43 | loadImage3(); 44 | return 0; 45 | } -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/solutions/change_pixels.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | 6 | using namespace std; 7 | 8 | void changePixels() 9 | { 10 | // create matrix 11 | int nrows = 480, ncols = 640; 12 | cv::Mat m1_8u; 13 | m1_8u.create(nrows, ncols, CV_8UC1); // two-channel matrix with 8bit unsigned elements 14 | m1_8u.setTo(cv::Scalar(0)); //black 15 | 16 | for (int r = 230; r < 250; r++) 17 | { 18 | // STUDENT TASK : loop over all columns and set matrix elements to 255 19 | for (int c = 0; c < m1_8u.cols; c++) 20 | { 21 | m1_8u.at(r, c) = 255; 22 | } 23 | } 24 | 25 | // show result 26 | string windowName = "First steps in OpenCV"; 27 | cv::namedWindow(windowName, 1); // create window 28 | cv::imshow(windowName, m1_8u); 29 | cv::waitKey(0); // wait for keyboard input before continuing 30 | } 31 | 32 | 33 | int main() 34 | { 35 | changePixels(); 36 | return 0; 37 | } -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/solutions/create_matrix.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | 6 | using namespace std; 7 | 8 | void createMatrix1() 9 | { 10 | // create matrix 11 | int nrows = 480, ncols = 640; 12 | cv::Mat m1_8u; 13 | m1_8u.create(nrows, ncols, CV_8UC1); // two-channel matrix with 8bit unsigned elements 14 | m1_8u.setTo(cv::Scalar(255)); // white 15 | 16 | // STUDENT TASK : 17 | // Create a variable of type cv::Mat* named m3_8u which has three channels with a 18 | // depth of 8bit per channel. Then, set the first channel to 255 and display the result. 19 | cv::Mat m3_8u; 20 | m3_8u.create(nrows, ncols, CV_8UC3); // three-channel matrix with 8bit unsigned elements 21 | m3_8u.setTo(cv::Scalar(255, 0, 0)); // blue 22 | 23 | // show result 24 | string windowName = "First steps in OpenCV (m1_8u)"; 25 | cv::namedWindow(windowName, 1); // create window 26 | cv::imshow(windowName, m1_8u); 27 | cv::waitKey(0); // wait for keyboard input before continuing 28 | 29 | // STUDENT TASK : 30 | // Display the results from the STUDENT TASK above 31 | windowName = "First steps in OpenCV (m3_8u)"; 32 | cv::namedWindow(windowName, 1); // create window 33 | cv::imshow(windowName, m3_8u); 34 | cv::waitKey(0); // wait for keyboard input before continuing 35 | } 36 | 37 | 38 | int main() 39 | { 40 | createMatrix1(); 41 | return 0; 42 | } -------------------------------------------------------------------------------- /Lesson 2 - Autonomous Vehicles and Computer Vision/The OpenCV Library/solutions/load_image_3.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | void loadImage3() 11 | { 12 | // load images 13 | vector imgList; 14 | for (int i = 5; i <= 9; i++) 15 | { 16 | // create file name 17 | ostringstream imgNumber; // #include 18 | imgNumber << setfill('0') << setw(4) << i; // #include 19 | string filename = "../images/img" + imgNumber.str() + ".jpg"; 20 | 21 | // load image and store it into a vector 22 | cv::Mat img; 23 | img = cv::imread(filename); 24 | imgList.push_back(img); // store pointer to current image in list 25 | } 26 | 27 | // display images from the vector 28 | string windowName = "First steps in OpenCV"; 29 | cv::namedWindow(windowName, 1); // create window 30 | for (auto it = imgList.begin(); it != imgList.end(); ++it) 31 | { 32 | 33 | // STUDENT TASK : Prevent image 7 from being displayed 34 | if (it == imgList.begin() + 2) 35 | { 36 | continue; 37 | } 38 | 39 | // display image 40 | cv::imshow(windowName, *it); 41 | cv::waitKey(0); // wait for keyboard input before continuing 42 | } 43 | } 44 | 45 | int main() 46 | { 47 | loadImage3(); 48 | return 0; 49 | } -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/TTC_camera -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executable for create matrix exercise 17 | add_executable (compute_ttc_camera src/compute_ttc_camera.cpp src/structIO.cpp) 18 | target_link_libraries (compute_ttc_camera ${OpenCV_LIBRARIES}) 19 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptMatches_AKAZE.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptMatches_AKAZE.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptMatches_SHI-BRISK.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptMatches_SHI-BRISK.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsRef_AKAZE.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsRef_AKAZE.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsRef_SHI-BRISK.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsRef_SHI-BRISK.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsSource_AKAZE.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsSource_AKAZE.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsSource_SHI-BRISK.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/dat/C23A5_KptsSource_SHI-BRISK.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/src/compute_ttc_camera.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | // these includes provide the data structures for managing 3D Lidar points and 2D keypoints 6 | #include "dataStructures.h" // you do not need to look into this file 7 | #include "structIO.hpp" // you do not need to look into this file 8 | 9 | using namespace std; 10 | 11 | // Purpose: Compute time-to-collision (TTC) based on keypoint correspondences in successive images 12 | // Notes: 13 | // - please take a look at the main()-function first 14 | // - kptsPrev and kptsCurr are the input keypoint sets, kptMatches are the matches between the two sets, 15 | // frameRate is required to compute the delta time between frames and TTC will hold the result of the computation 16 | void computeTTCCamera(std::vector &kptsPrev, std::vector &kptsCurr, 17 | std::vector kptMatches, double frameRate, double &TTC) 18 | { 19 | // compute distance ratios between all matched keypoints 20 | vector distRatios; // stores the distance ratios for all keypoints between curr. and prev. frame 21 | for (auto it1 = kptMatches.begin(); it1 != kptMatches.end() - 1; ++it1) 22 | { // outer keypoint loop 23 | 24 | // get current keypoint and its matched partner in the prev. frame 25 | cv::KeyPoint kpOuterCurr = kptsCurr.at(it1->trainIdx); 26 | cv::KeyPoint kpOuterPrev = kptsPrev.at(it1->queryIdx); 27 | 28 | for (auto it2 = kptMatches.begin() + 1; it2 != kptMatches.end(); ++it2) 29 | { // inner keypoint loop 30 | 31 | double minDist = 100.0; // min. required distance 32 | 33 | // get next keypoint and its matched partner in the prev. frame 34 | cv::KeyPoint kpInnerCurr = kptsCurr.at(it2->trainIdx); 35 | cv::KeyPoint kpInnerPrev = kptsPrev.at(it2->queryIdx); 36 | 37 | // compute distances and distance ratios 38 | double distCurr = cv::norm(kpOuterCurr.pt - kpInnerCurr.pt); 39 | double distPrev = cv::norm(kpOuterPrev.pt - kpInnerPrev.pt); 40 | 41 | if (distPrev > std::numeric_limits::epsilon() && distCurr >= minDist) 42 | { // avoid division by zero 43 | 44 | double distRatio = distCurr / distPrev; 45 | distRatios.push_back(distRatio); 46 | } 47 | } // eof inner loop over all matched kpts 48 | } // eof outer loop over all matched kpts 49 | 50 | // only continue if list of distance ratios is not empty 51 | if (distRatios.size() == 0) 52 | { 53 | TTC = NAN; 54 | return; 55 | } 56 | 57 | // compute camera-based TTC from distance ratios 58 | double meanDistRatio = std::accumulate(distRatios.begin(), distRatios.end(), 0.0) / distRatios.size(); 59 | 60 | double dT = 1 / frameRate; 61 | TTC = -dT / (1 - meanDistRatio); 62 | 63 | // TODO: STUDENT TASK (replacement for meanDistRatio) 64 | } 65 | 66 | int main() 67 | { 68 | // step 1: read pre-recorded keypoint sets from file 69 | // Note that the function "readKeypoints" is a helper function that is able to read pre-saved results from disk 70 | // so that you can focus on TTC computation based on a defined set of keypoints and matches. 71 | // The task you need to solve in this example does not require you to look into the data structures. 72 | vector kptsSource, kptsRef; 73 | readKeypoints("../dat/C23A5_KptsSource_AKAZE.dat", kptsSource); // readKeypoints("./dat/C23A5_KptsSource_SHI-BRISK.dat" 74 | readKeypoints("../dat/C23A5_KptsRef_AKAZE.dat", kptsRef); // readKeypoints("./dat/C23A5_KptsRef_SHI-BRISK.dat" 75 | 76 | // step 2: read pre-recorded keypoint matches from file 77 | vector matches; 78 | readKptMatches("../dat/C23A5_KptMatches_AKAZE.dat", matches); // readKptMatches("./dat/C23A5_KptMatches_SHI-BRISK.dat", matches); 79 | 80 | // step 3: compute the time-to-collision based on the pre-recorded data 81 | double ttc; 82 | computeTTCCamera(kptsSource, kptsRef, matches, 10.0, ttc); 83 | cout << "ttc = " << ttc << "s" << endl; 84 | } -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/src/dataStructures.h: -------------------------------------------------------------------------------- 1 | // 2 | // dataStructures.h 3 | // OpenCVTest 4 | // 5 | // Created by Andreas Haja on 01.04.19. 6 | // Copyright © 2019 Andreas Haja. All rights reserved. 7 | // 8 | 9 | #ifndef dataStructures_h 10 | #define dataStructures_h 11 | 12 | #include 13 | #include 14 | 15 | struct LidarPoint { // single lidar point in space 16 | double x,y,z,r; // x,y,z in [m], r is point reflectivity 17 | }; 18 | 19 | 20 | #endif /* dataStructures_h */ 21 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/src/structIO.cpp: -------------------------------------------------------------------------------- 1 | // 2 | // structIO.cpp 3 | // OpenCVTest 4 | // 5 | // Created by Prof. Dr. Andreas Haja on 24.04.19. 6 | // Copyright © 2019 Andreas Haja. All rights reserved. 7 | // 8 | 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include "structIO.hpp" 14 | 15 | using namespace std; 16 | 17 | 18 | /* TEMPLATES */ 19 | template void write_pod(std::ofstream& out, T& t) 20 | { 21 | out.write(reinterpret_cast(&t), sizeof(T)); 22 | } 23 | 24 | 25 | template void read_pod(std::ifstream& in, T& t) 26 | { 27 | in.read(reinterpret_cast(&t), sizeof(T)); 28 | } 29 | 30 | 31 | template void read_pod_vector(std::ifstream& in, std::vector& vect) 32 | { 33 | long size; 34 | 35 | read_pod(in, size); 36 | for(int i = 0;i < size;++i) 37 | { 38 | T t; 39 | read_pod(in, t); 40 | vect.push_back(t); 41 | } 42 | } 43 | 44 | template void write_pod_vector(std::ofstream& out, std::vector& vect) 45 | { 46 | long size = vect.size(); 47 | write_pod(out, size); 48 | for(auto it=vect.begin(); it!=vect.end(); ++it) 49 | { 50 | write_pod(out,*it); 51 | } 52 | } 53 | 54 | 55 | 56 | 57 | /* DATATYPE WRAPPERS */ 58 | 59 | void writeLidarPts(std::vector &input, const char* fileName) 60 | { 61 | std::ofstream out(fileName); 62 | write_pod_vector(out, input); 63 | out.close(); 64 | } 65 | 66 | 67 | void readLidarPts(const char* fileName, std::vector &output) 68 | { 69 | std::ifstream in(fileName); 70 | read_pod_vector(in, output); 71 | } 72 | 73 | 74 | void writeKeypoints(std::vector &input, const char* fileName) 75 | { 76 | std::ofstream out(fileName); 77 | write_pod_vector(out, input); 78 | out.close(); 79 | } 80 | 81 | 82 | void readKeypoints(const char* fileName, std::vector &output) 83 | { 84 | std::ifstream in(fileName); 85 | read_pod_vector(in, output); 86 | } 87 | 88 | 89 | void writeKptMatches(std::vector &input, const char* fileName) 90 | { 91 | std::ofstream out(fileName); 92 | write_pod_vector(out, input); 93 | out.close(); 94 | } 95 | 96 | 97 | void readKptMatches(const char* fileName, std::vector &output) 98 | { 99 | std::ifstream in(fileName); 100 | read_pod_vector(in, output); 101 | } 102 | 103 | 104 | 105 | void writeDescriptors(cv::Mat &input, const char* fileName) 106 | { 107 | cv::FileStorage opencv_file(fileName, cv::FileStorage::WRITE); 108 | opencv_file << "desc_matrix" << input; 109 | opencv_file.release(); 110 | } 111 | 112 | 113 | void readDescriptors(const char* fileName, cv::Mat &output) 114 | { 115 | cv::FileStorage opencv_file(fileName, cv::FileStorage::READ); 116 | opencv_file["desc_matrix"] >> output; 117 | opencv_file.release(); 118 | } 119 | 120 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/TTC_camera/src/structIO.hpp: -------------------------------------------------------------------------------- 1 | // 2 | // structIO.hpp 3 | // OpenCVTest 4 | // 5 | // Created by Prof. Dr. Andreas Haja on 24.04.19. 6 | // Copyright © 2019 Andreas Haja. All rights reserved. 7 | // 8 | 9 | #ifndef structIO_hpp 10 | #define structIO_hpp 11 | 12 | #include 13 | #include "dataStructures.h" 14 | 15 | void writeLidarPts(std::vector &input, const char* fileName); 16 | void readLidarPts(const char* fileName, std::vector &output); 17 | 18 | void writeKeypoints(std::vector &input, const char* fileName); 19 | void readKeypoints(const char* fileName, std::vector &output); 20 | 21 | void writeKptMatches(std::vector &input, const char* fileName); 22 | void readKptMatches(const char* fileName, std::vector &output); 23 | 24 | void writeDescriptors(cv::Mat &input, const char* fileName); 25 | void readDescriptors(const char* fileName, cv::Mat &output); 26 | 27 | 28 | template void write_pod(std::ofstream& out, T& t); 29 | template void read_pod(std::ifstream& in, T& t); 30 | template void read_pod_vector(std::ifstream& in, std::vector& vect); 31 | template void write_pod_vector(std::ofstream& out, std::vector& vect); 32 | 33 | #endif /* structIO_hpp */ 34 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Camera/solution/compute_ttc_camera.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | 6 | #include "dataStructures.h" 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | // Compute time-to-collision (TTC) based on keypoint correspondences in successive images 12 | void computeTTCCamera(std::vector &kptsPrev, std::vector &kptsCurr, 13 | std::vector kptMatches, double frameRate, double &TTC) 14 | { 15 | // compute distance ratios between all matched keypoints 16 | vector distRatios; // stores the distance ratios for all keypoints between curr. and prev. frame 17 | for (auto it1 = kptMatches.begin(); it1 != kptMatches.end() - 1; ++it1) 18 | { // outer kpt. loop 19 | 20 | // get current keypoint and its matched partner in the prev. frame 21 | cv::KeyPoint kpOuterCurr = kptsCurr.at(it1->trainIdx); 22 | cv::KeyPoint kpOuterPrev = kptsPrev.at(it1->queryIdx); 23 | 24 | for (auto it2 = kptMatches.begin() + 1; it2 != kptMatches.end(); ++it2) 25 | { // inner kpt.-loop 26 | 27 | double minDist = 100.0; // min. required distance 28 | 29 | // get next keypoint and its matched partner in the prev. frame 30 | cv::KeyPoint kpInnerCurr = kptsCurr.at(it2->trainIdx); 31 | cv::KeyPoint kpInnerPrev = kptsPrev.at(it2->queryIdx); 32 | 33 | // compute distances and distance ratios 34 | double distCurr = cv::norm(kpOuterCurr.pt - kpInnerCurr.pt); 35 | double distPrev = cv::norm(kpOuterPrev.pt - kpInnerPrev.pt); 36 | 37 | if (distPrev > std::numeric_limits::epsilon() && distCurr >= minDist) 38 | { // avoid division by zero 39 | 40 | double distRatio = distCurr / distPrev; 41 | distRatios.push_back(distRatio); 42 | } 43 | } // eof inner loop over all matched kpts 44 | } // eof outer loop over all matched kpts 45 | 46 | // only continue if list of distance ratios is not empty 47 | if (distRatios.size() == 0) 48 | { 49 | TTC = NAN; 50 | return; 51 | } 52 | 53 | 54 | // STUDENT TASK (replacement for meanDistRatio) 55 | std::sort(distRatios.begin(), distRatios.end()); 56 | long medIndex = floor(distRatios.size() / 2.0); 57 | double medDistRatio = distRatios.size() % 2 == 0 ? (distRatios[medIndex - 1] + distRatios[medIndex]) / 2.0 : distRatios[medIndex]; // compute median dist. ratio to remove outlier influence 58 | 59 | double dT = 1 / frameRate; 60 | TTC = -dT / (1 - medDistRatio); 61 | // EOF STUDENT TASK 62 | } 63 | 64 | int main() 65 | { 66 | vector kptsSource, kptsRef; 67 | readKeypoints("../dat/C23A5_KptsSource_AKAZE.dat", kptsSource); // readKeypoints("./dat/C23A5_KptsSource_SHI-BRISK.dat" 68 | readKeypoints("../dat/C23A5_KptsRef_AKAZE.dat", kptsRef); // readKeypoints("./dat/C23A5_KptsRef_SHI-BRISK.dat" 69 | 70 | vector matches; 71 | readKptMatches("../dat/C23A5_KptMatches_AKAZE.dat", matches); // readKptMatches("./dat/C23A5_KptMatches_SHI-BRISK.dat", matches); 72 | double ttc; 73 | computeTTCCamera(kptsSource, kptsRef, matches, 10.0, ttc); 74 | cout << "ttc = " << ttc << "s" << endl; 75 | } -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/TTC_lidar -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executable for create matrix exercise 17 | add_executable (compute_ttc_lidar src/compute_ttc_lidar.cpp src/structIO.cpp) 18 | target_link_libraries (compute_ttc_lidar ${OpenCV_LIBRARIES}) 19 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/dat/C22A5_currLidarPts.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/dat/C22A5_currLidarPts.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/dat/C22A5_prevLidarPts.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/dat/C22A5_prevLidarPts.dat -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/src/compute_ttc_lidar.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | 6 | #include "dataStructures.h" 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | void computeTTCLidar(std::vector &lidarPointsPrev, 12 | std::vector &lidarPointsCurr, double &TTC) 13 | { 14 | // auxiliary variables 15 | double dT = 0.1; // time between two measurements in seconds 16 | double laneWidth = 4.0; // assumed width of the ego lane 17 | 18 | // find closest distance to Lidar points within ego lane 19 | double minXPrev = 1e9, minXCurr = 1e9; 20 | for (auto it = lidarPointsPrev.begin(); it != lidarPointsPrev.end(); ++it) 21 | { 22 | minXPrev = minXPrev > it->x ? it->x : minXPrev; 23 | } 24 | 25 | for (auto it = lidarPointsCurr.begin(); it != lidarPointsCurr.end(); ++it) 26 | { 27 | minXCurr = minXCurr > it->x ? it->x : minXCurr; 28 | } 29 | 30 | // compute TTC from both measurements 31 | TTC = minXCurr * dT / (minXPrev - minXCurr); 32 | } 33 | 34 | int main() 35 | { 36 | 37 | std::vector currLidarPts, prevLidarPts; 38 | readLidarPts("../dat/C22A5_currLidarPts.dat", currLidarPts); 39 | readLidarPts("../dat/C22A5_prevLidarPts.dat", prevLidarPts); 40 | 41 | 42 | double ttc; 43 | computeTTCLidar(prevLidarPts, currLidarPts, ttc); 44 | cout << "ttc = " << ttc << "s" << endl; 45 | } -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/src/dataStructures.h: -------------------------------------------------------------------------------- 1 | #ifndef dataStructures_h 2 | #define dataStructures_h 3 | 4 | #include 5 | #include 6 | 7 | struct LidarPoint { // single lidar point in space 8 | double x,y,z,r; // x,y,z in [m], r is point reflectivity 9 | }; 10 | 11 | 12 | #endif /* dataStructures_h */ 13 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/src/structIO.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include "structIO.hpp" 6 | 7 | using namespace std; 8 | 9 | 10 | /* TEMPLATES */ 11 | template void write_pod(std::ofstream& out, T& t) 12 | { 13 | out.write(reinterpret_cast(&t), sizeof(T)); 14 | } 15 | 16 | 17 | template void read_pod(std::ifstream& in, T& t) 18 | { 19 | in.read(reinterpret_cast(&t), sizeof(T)); 20 | } 21 | 22 | 23 | template void read_pod_vector(std::ifstream& in, std::vector& vect) 24 | { 25 | long size; 26 | 27 | read_pod(in, size); 28 | for(int i = 0;i < size;++i) 29 | { 30 | T t; 31 | read_pod(in, t); 32 | vect.push_back(t); 33 | } 34 | } 35 | 36 | template void write_pod_vector(std::ofstream& out, std::vector& vect) 37 | { 38 | long size = vect.size(); 39 | write_pod(out, size); 40 | for(auto it=vect.begin(); it!=vect.end(); ++it) 41 | { 42 | write_pod(out,*it); 43 | } 44 | } 45 | 46 | 47 | 48 | 49 | /* DATATYPE WRAPPERS */ 50 | 51 | void writeLidarPts(std::vector &input, const char* fileName) 52 | { 53 | std::ofstream out(fileName); 54 | write_pod_vector(out, input); 55 | out.close(); 56 | } 57 | 58 | 59 | void readLidarPts(const char* fileName, std::vector &output) 60 | { 61 | std::ifstream in(fileName); 62 | read_pod_vector(in, output); 63 | } 64 | 65 | 66 | void writeKeypoints(std::vector &input, const char* fileName) 67 | { 68 | std::ofstream out(fileName); 69 | write_pod_vector(out, input); 70 | out.close(); 71 | } 72 | 73 | 74 | void readKeypoints(const char* fileName, std::vector &output) 75 | { 76 | std::ifstream in(fileName); 77 | read_pod_vector(in, output); 78 | } 79 | 80 | 81 | void writeKptMatches(std::vector &input, const char* fileName) 82 | { 83 | std::ofstream out(fileName); 84 | write_pod_vector(out, input); 85 | out.close(); 86 | } 87 | 88 | 89 | void readKptMatches(const char* fileName, std::vector &output) 90 | { 91 | std::ifstream in(fileName); 92 | read_pod_vector(in, output); 93 | } 94 | 95 | 96 | 97 | void writeDescriptors(cv::Mat &input, const char* fileName) 98 | { 99 | cv::FileStorage opencv_file(fileName, cv::FileStorage::WRITE); 100 | opencv_file << "desc_matrix" << input; 101 | opencv_file.release(); 102 | } 103 | 104 | 105 | void readDescriptors(const char* fileName, cv::Mat &output) 106 | { 107 | cv::FileStorage opencv_file(fileName, cv::FileStorage::READ); 108 | opencv_file["desc_matrix"] >> output; 109 | opencv_file.release(); 110 | } 111 | 112 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/TTC_lidar/src/structIO.hpp: -------------------------------------------------------------------------------- 1 | #ifndef structIO_hpp 2 | #define structIO_hpp 3 | 4 | #include 5 | #include "dataStructures.h" 6 | 7 | void writeLidarPts(std::vector &input, const char* fileName); 8 | void readLidarPts(const char* fileName, std::vector &output); 9 | 10 | void writeKeypoints(std::vector &input, const char* fileName); 11 | void readKeypoints(const char* fileName, std::vector &output); 12 | 13 | void writeKptMatches(std::vector &input, const char* fileName); 14 | void readKptMatches(const char* fileName, std::vector &output); 15 | 16 | void writeDescriptors(cv::Mat &input, const char* fileName); 17 | void readDescriptors(const char* fileName, cv::Mat &output); 18 | 19 | 20 | template void write_pod(std::ofstream& out, T& t); 21 | template void read_pod(std::ifstream& in, T& t); 22 | template void read_pod_vector(std::ifstream& in, std::vector& vect); 23 | template void write_pod_vector(std::ofstream& out, std::vector& vect); 24 | 25 | #endif /* structIO_hpp */ 26 | -------------------------------------------------------------------------------- /Lesson 3 - Engineering a Collision Detection System/Estimating TTC with Lidar/solution/compute_ttc_lidar.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | 6 | #include "dataStructures.h" 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | void computeTTCLidar(std::vector &lidarPointsPrev, 12 | std::vector &lidarPointsCurr, double &TTC) 13 | { 14 | // auxiliary variables 15 | double dT = 0.1; // time between two measurements in seconds 16 | double laneWidth = 4.0; // assumed width of the ego lane 17 | 18 | // find closest distance to Lidar points within ego lane 19 | double minXPrev = 1e9, minXCurr = 1e9; 20 | for (auto it = lidarPointsPrev.begin(); it != lidarPointsPrev.end(); ++it) 21 | { 22 | 23 | if (abs(it->y) <= laneWidth / 2.0) 24 | { // 3D point within ego lane? 25 | minXPrev = minXPrev > it->x ? it->x : minXPrev; 26 | } 27 | } 28 | 29 | for (auto it = lidarPointsCurr.begin(); it != lidarPointsCurr.end(); ++it) 30 | { 31 | 32 | if (abs(it->y) <= laneWidth / 2.0) 33 | { // 3D point within ego lane? 34 | minXCurr = minXCurr > it->x ? it->x : minXCurr; 35 | } 36 | } 37 | 38 | // compute TTC from both measurements 39 | TTC = minXCurr * dT / (minXPrev - minXCurr); 40 | } 41 | 42 | int main() 43 | { 44 | 45 | std::vector currLidarPts, prevLidarPts; 46 | readLidarPts("../dat/C22A5_currLidarPts.dat", currLidarPts); 47 | readLidarPts("../dat/C22A5_prevLidarPts.dat", prevLidarPts); 48 | 49 | 50 | double ttc; 51 | computeTTCLidar(prevLidarPts, currLidarPts, ttc); 52 | cout << "ttc = " << ttc << "s" << endl; 53 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/descriptor_matching -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercise 17 | add_executable (descriptor_matching src/descriptor_matching.cpp src/structIO.cpp) 18 | target_link_libraries (descriptor_matching ${OpenCV_LIBRARIES}) 19 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsRef_BRISK_large.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsRef_BRISK_large.dat -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsRef_BRISK_small.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsRef_BRISK_small.dat -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsRef_SIFT.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsRef_SIFT.dat -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsSource_BRISK_large.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsSource_BRISK_large.dat -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsSource_BRISK_small.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsSource_BRISK_small.dat -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsSource_SIFT.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/dat/C35A5_KptsSource_SIFT.dat -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0005.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0005.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0005.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0005.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0006.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0006.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0006.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0007.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0007.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0007.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0008.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0008.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0008.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0008.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0009.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0009.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0009.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img0009.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img1.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img1gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img1gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img2gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/images/img2gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/src/dataStructures.h: -------------------------------------------------------------------------------- 1 | #ifndef dataStructures_h 2 | #define dataStructures_h 3 | 4 | #include 5 | #include 6 | 7 | struct LidarPoint { // single lidar point in space 8 | double x,y,z,r; // x,y,z in [m], r is point reflectivity 9 | }; 10 | 11 | 12 | #endif /* dataStructures_h */ 13 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/src/descriptor_matching.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | #include "structIO.hpp" 9 | 10 | using namespace std; 11 | 12 | void matchDescriptors(cv::Mat &imgSource, cv::Mat &imgRef, vector &kPtsSource, vector &kPtsRef, cv::Mat &descSource, cv::Mat &descRef, 13 | vector &matches, string descriptorType, string matcherType, string selectorType) 14 | { 15 | 16 | // configure matcher 17 | bool crossCheck = false; 18 | cv::Ptr matcher; 19 | 20 | if (matcherType.compare("MAT_BF") == 0) 21 | { 22 | 23 | int normType = descriptorType.compare("DES_BINARY") == 0 ? cv::NORM_HAMMING : cv::NORM_L2; 24 | matcher = cv::BFMatcher::create(normType, crossCheck); 25 | cout << "BF matching cross-check=" << crossCheck; 26 | } 27 | else if (matcherType.compare("MAT_FLANN") == 0) 28 | { 29 | if (descSource.type() != CV_32F) 30 | { // OpenCV bug workaround : convert binary descriptors to floating point due to a bug in current OpenCV implementation 31 | descSource.convertTo(descSource, CV_32F); 32 | descRef.convertTo(descRef, CV_32F); 33 | } 34 | 35 | //... TODO : implement FLANN matching 36 | cout << "FLANN matching"; 37 | } 38 | 39 | // perform matching task 40 | if (selectorType.compare("SEL_NN") == 0) 41 | { // nearest neighbor (best match) 42 | 43 | double t = (double)cv::getTickCount(); 44 | matcher->match(descSource, descRef, matches); // Finds the best match for each descriptor in desc1 45 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 46 | cout << " (NN) with n=" << matches.size() << " matches in " << 1000 * t / 1.0 << " ms" << endl; 47 | } 48 | else if (selectorType.compare("SEL_KNN") == 0) 49 | { // k nearest neighbors (k=2) 50 | 51 | // TODO : implement k-nearest-neighbor matching 52 | 53 | // TODO : filter matches using descriptor distance ratio test 54 | } 55 | 56 | // visualize results 57 | cv::Mat matchImg = imgRef.clone(); 58 | cv::drawMatches(imgSource, kPtsSource, imgRef, kPtsRef, matches, 59 | matchImg, cv::Scalar::all(-1), cv::Scalar::all(-1), vector(), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 60 | 61 | string windowName = "Matching keypoints between two camera images (best 50)"; 62 | cv::namedWindow(windowName, 7); 63 | cv::imshow(windowName, matchImg); 64 | cv::waitKey(0); 65 | } 66 | 67 | int main() 68 | { 69 | cv::Mat imgSource = cv::imread("../images/img1gray.png"); 70 | cv::Mat imgRef = cv::imread("../images/img2gray.png"); 71 | 72 | vector kptsSource, kptsRef; 73 | readKeypoints("../dat/C35A5_KptsSource_BRISK_large.dat", kptsSource); 74 | readKeypoints("../dat/C35A5_KptsRef_BRISK_large.dat", kptsRef); 75 | 76 | cv::Mat descSource, descRef; 77 | readDescriptors("../dat/C35A5_DescSource_BRISK_large.dat", descSource); 78 | readDescriptors("../dat/C35A5_DescRef_BRISK_large.dat", descRef); 79 | 80 | vector matches; 81 | string matcherType = "MAT_BF"; 82 | string descriptorType = "DES_BINARY"; 83 | string selectorType = "SEL_NN"; 84 | matchDescriptors(imgSource, imgRef, kptsSource, kptsRef, descSource, descRef, matches, descriptorType, matcherType, selectorType); 85 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/src/structIO.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include "structIO.hpp" 6 | 7 | using namespace std; 8 | 9 | 10 | /* TEMPLATES */ 11 | template void write_pod(std::ofstream& out, T& t) 12 | { 13 | out.write(reinterpret_cast(&t), sizeof(T)); 14 | } 15 | 16 | 17 | template void read_pod(std::ifstream& in, T& t) 18 | { 19 | in.read(reinterpret_cast(&t), sizeof(T)); 20 | } 21 | 22 | 23 | template void read_pod_vector(std::ifstream& in, std::vector& vect) 24 | { 25 | long size; 26 | 27 | read_pod(in, size); 28 | for(int i = 0;i < size;++i) 29 | { 30 | T t; 31 | read_pod(in, t); 32 | vect.push_back(t); 33 | } 34 | } 35 | 36 | template void write_pod_vector(std::ofstream& out, std::vector& vect) 37 | { 38 | long size = vect.size(); 39 | write_pod(out, size); 40 | for(auto it=vect.begin(); it!=vect.end(); ++it) 41 | { 42 | write_pod(out,*it); 43 | } 44 | } 45 | 46 | 47 | 48 | 49 | /* DATATYPE WRAPPERS */ 50 | 51 | void writeLidarPts(std::vector &input, const char* fileName) 52 | { 53 | std::ofstream out(fileName); 54 | write_pod_vector(out, input); 55 | out.close(); 56 | } 57 | 58 | 59 | void readLidarPts(const char* fileName, std::vector &output) 60 | { 61 | std::ifstream in(fileName); 62 | read_pod_vector(in, output); 63 | } 64 | 65 | 66 | void writeKeypoints(std::vector &input, const char* fileName) 67 | { 68 | std::ofstream out(fileName); 69 | write_pod_vector(out, input); 70 | out.close(); 71 | } 72 | 73 | 74 | void readKeypoints(const char* fileName, std::vector &output) 75 | { 76 | std::ifstream in(fileName); 77 | read_pod_vector(in, output); 78 | } 79 | 80 | 81 | void writeKptMatches(std::vector &input, const char* fileName) 82 | { 83 | std::ofstream out(fileName); 84 | write_pod_vector(out, input); 85 | out.close(); 86 | } 87 | 88 | 89 | void readKptMatches(const char* fileName, std::vector &output) 90 | { 91 | std::ifstream in(fileName); 92 | read_pod_vector(in, output); 93 | } 94 | 95 | 96 | 97 | void writeDescriptors(cv::Mat &input, const char* fileName) 98 | { 99 | cv::FileStorage opencv_file(fileName, cv::FileStorage::WRITE); 100 | opencv_file << "desc_matrix" << input; 101 | opencv_file.release(); 102 | } 103 | 104 | 105 | void readDescriptors(const char* fileName, cv::Mat &output) 106 | { 107 | cv::FileStorage opencv_file(fileName, cv::FileStorage::READ); 108 | opencv_file["desc_matrix"] >> output; 109 | opencv_file.release(); 110 | } 111 | 112 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/descriptor_matching/src/structIO.hpp: -------------------------------------------------------------------------------- 1 | #ifndef structIO_hpp 2 | #define structIO_hpp 3 | 4 | #include 5 | #include "dataStructures.h" 6 | 7 | void writeLidarPts(std::vector &input, const char* fileName); 8 | void readLidarPts(const char* fileName, std::vector &output); 9 | 10 | void writeKeypoints(std::vector &input, const char* fileName); 11 | void readKeypoints(const char* fileName, std::vector &output); 12 | 13 | void writeKptMatches(std::vector &input, const char* fileName); 14 | void readKptMatches(const char* fileName, std::vector &output); 15 | 16 | void writeDescriptors(cv::Mat &input, const char* fileName); 17 | void readDescriptors(const char* fileName, cv::Mat &output); 18 | 19 | 20 | template void write_pod(std::ofstream& out, T& t); 21 | template void read_pod(std::ifstream& in, T& t); 22 | template void read_pod_vector(std::ifstream& in, std::vector& vect); 23 | template void write_pod_vector(std::ofstream& out, std::vector& vect); 24 | 25 | #endif /* structIO_hpp */ 26 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Descriptor Matching/solution/descriptor_matching.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | #include "structIO.hpp" 9 | 10 | using namespace std; 11 | 12 | void matchDescriptors(cv::Mat &imgSource, cv::Mat &imgRef, vector &kPtsSource, vector &kPtsRef, cv::Mat &descSource, cv::Mat &descRef, 13 | vector &matches, string descriptorType, string matcherType, string selectorType) 14 | { 15 | 16 | // configure matcher 17 | bool crossCheck = false; 18 | cv::Ptr matcher; 19 | 20 | if (matcherType.compare("MAT_BF") == 0) 21 | { 22 | 23 | int normType = descriptorType.compare("DES_BINARY") == 0 ? cv::NORM_HAMMING : cv::NORM_L2; 24 | matcher = cv::BFMatcher::create(normType, crossCheck); 25 | cout << "BF matching cross-check=" << crossCheck; 26 | } 27 | else if (matcherType.compare("MAT_FLANN") == 0) 28 | { 29 | if (descSource.type() != CV_32F) 30 | { // OpenCV bug workaround : convert binary descriptors to floating point due to a bug in current OpenCV implementation 31 | descSource.convertTo(descSource, CV_32F); 32 | descRef.convertTo(descRef, CV_32F); 33 | } 34 | 35 | matcher = cv::DescriptorMatcher::create(cv::DescriptorMatcher::FLANNBASED); 36 | cout << "FLANN matching"; 37 | } 38 | 39 | // perform matching task 40 | if (selectorType.compare("SEL_NN") == 0) 41 | { // nearest neighbor (best match) 42 | 43 | double t = (double)cv::getTickCount(); 44 | matcher->match(descSource, descRef, matches); // Finds the best match for each descriptor in desc1 45 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 46 | cout << " (NN) with n=" << matches.size() << " matches in " << 1000 * t / 1.0 << " ms" << endl; 47 | } 48 | else if (selectorType.compare("SEL_KNN") == 0) 49 | { // k nearest neighbors (k=2) 50 | 51 | vector> knn_matches; 52 | double t = (double)cv::getTickCount(); 53 | matcher->knnMatch(descSource, descRef, knn_matches, 2); // finds the 2 best matches 54 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 55 | cout << " (KNN) with n=" << knn_matches.size() << " matches in " << 1000 * t / 1.0 << " ms" << endl; 56 | 57 | // STUDENT TASK 58 | // filter matches using descriptor distance ratio test 59 | double minDescDistRatio = 0.8; 60 | for (auto it = knn_matches.begin(); it != knn_matches.end(); ++it) 61 | { 62 | 63 | if ((*it)[0].distance < minDescDistRatio * (*it)[1].distance) 64 | { 65 | matches.push_back((*it)[0]); 66 | } 67 | } 68 | cout << "# keypoints removed = " << knn_matches.size() - matches.size() << endl; 69 | // EOF STUDENT TASK 70 | } 71 | 72 | // visualize results 73 | cv::Mat matchImg = imgRef.clone(); 74 | cv::drawMatches(imgSource, kPtsSource, imgRef, kPtsRef, matches, 75 | matchImg, cv::Scalar::all(-1), cv::Scalar::all(-1), vector(), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 76 | 77 | string windowName = "Matching keypoints between two camera images (best 50)"; 78 | cv::namedWindow(windowName, 7); 79 | cv::imshow(windowName, matchImg); 80 | cv::waitKey(0); 81 | } 82 | 83 | int main() 84 | { 85 | cv::Mat imgSource = cv::imread("../images/img1gray.png"); 86 | cv::Mat imgRef = cv::imread("../images/img2gray.png"); 87 | 88 | vector kptsSource, kptsRef; 89 | readKeypoints("../dat/C35A5_KptsSource_BRISK_large.dat", kptsSource); 90 | readKeypoints("../dat/C35A5_KptsRef_BRISK_large.dat", kptsRef); 91 | 92 | cv::Mat descSource, descRef; 93 | readDescriptors("../dat/C35A5_DescSource_BRISK_large.dat", descSource); 94 | readDescriptors("../dat/C35A5_DescRef_BRISK_large.dat", descRef); 95 | 96 | vector matches; 97 | string matcherType = "MAT_BF"; // MAT_BF, MAT_FLANN 98 | string descriptorType = "DES_BINARY"; // DES_BINARY, DES_HOG 99 | string selectorType = "SEL_NN"; // SEL_NN, SEL_KNN 100 | matchDescriptors(imgSource, imgRef, kptsSource, kptsRef, descSource, descRef, matches, descriptorType, matcherType, selectorType); 101 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/describe_keypoints -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercise 17 | add_executable (describe_keypoints src/describe_keypoints.cpp) 18 | target_link_libraries (describe_keypoints ${OpenCV_LIBRARIES}) 19 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0005.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0005.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0005.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0005.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0006.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0006.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0006.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0007.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0007.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0007.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0008.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0008.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0008.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0008.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0009.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0009.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0009.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img0009.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img1.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img1gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img1gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img2gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/images/img2gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/describe_keypoints/src/describe_keypoints.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | using namespace std; 11 | 12 | void descKeypoints1() 13 | { 14 | // load image from file and convert to grayscale 15 | cv::Mat imgGray; 16 | cv::Mat img = cv::imread("../images/img1.png"); 17 | cv::cvtColor(img, imgGray, cv::COLOR_BGR2GRAY); 18 | 19 | // BRISK detector / descriptor 20 | cv::Ptr detector = cv::BRISK::create(); 21 | vector kptsBRISK; 22 | 23 | double t = (double)cv::getTickCount(); 24 | detector->detect(imgGray, kptsBRISK); 25 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 26 | cout << "BRISK detector with n= " << kptsBRISK.size() << " keypoints in " << 1000 * t / 1.0 << " ms" << endl; 27 | 28 | cv::Ptr descriptor = cv::BRISK::create(); 29 | cv::Mat descBRISK; 30 | t = (double)cv::getTickCount(); 31 | descriptor->compute(imgGray, kptsBRISK, descBRISK); 32 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 33 | cout << "BRISK descriptor in " << 1000 * t / 1.0 << " ms" << endl; 34 | 35 | // visualize results 36 | cv::Mat visImage = img.clone(); 37 | cv::drawKeypoints(img, kptsBRISK, visImage, cv::Scalar::all(-1), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 38 | string windowName = "BRISK Results"; 39 | cv::namedWindow(windowName, 1); 40 | imshow(windowName, visImage); 41 | cv::waitKey(0); 42 | 43 | // TODO: Add the SIFT detector / descriptor, compute the 44 | // time for both steps and compare both BRISK and SIFT 45 | // with regard to processing speed and the number and 46 | // visual appearance of keypoints. 47 | 48 | } 49 | 50 | int main() 51 | { 52 | descKeypoints1(); 53 | return 0; 54 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Gradient-based vs. Binary Descriptors/solution/describe_keypoints.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | using namespace std; 11 | 12 | void descKeypoints1() 13 | { 14 | // load image from file and convert to grayscale 15 | cv::Mat imgGray; 16 | cv::Mat img = cv::imread("../images/img1.png"); 17 | cv::cvtColor(img, imgGray, cv::COLOR_BGR2GRAY); 18 | 19 | // BRISK detector / descriptor 20 | cv::Ptr detector = cv::BRISK::create(); 21 | vector kptsBRISK; 22 | 23 | double t = (double)cv::getTickCount(); 24 | detector->detect(imgGray, kptsBRISK); 25 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 26 | cout << "BRISK detector with n= " << kptsBRISK.size() << " keypoints in " << 1000 * t / 1.0 << " ms" << endl; 27 | 28 | cv::Ptr descriptor = cv::BRISK::create(); 29 | cv::Mat descBRISK; 30 | t = (double)cv::getTickCount(); 31 | descriptor->compute(imgGray, kptsBRISK, descBRISK); 32 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 33 | cout << "BRISK descriptor in " << 1000 * t / 1.0 << " ms" << endl; 34 | 35 | // visualize results 36 | cv::Mat visImage = img.clone(); 37 | cv::drawKeypoints(img, kptsBRISK, visImage, cv::Scalar::all(-1), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 38 | string windowName = "BRISK Results"; 39 | cv::namedWindow(windowName, 1); 40 | imshow(windowName, visImage); 41 | 42 | // STUDENT CODE 43 | detector = cv::xfeatures2d::SIFT::create(); 44 | vector kptsSIFT; 45 | 46 | t = (double)cv::getTickCount(); 47 | detector->detect(imgGray, kptsSIFT); 48 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 49 | cout << "SIFT detector with n= " << kptsSIFT.size() << " keypoints in " << 1000 * t / 1.0 << " ms" << endl; 50 | 51 | descriptor = cv::xfeatures2d::SiftDescriptorExtractor::create(); 52 | cv::Mat descSIFT; 53 | t = (double)cv::getTickCount(); 54 | descriptor->compute(imgGray, kptsSIFT, descSIFT); 55 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 56 | cout << "SIFT descriptor in " << 1000 * t / 1.0 << " ms" << endl; 57 | 58 | visImage = img.clone(); 59 | cv::drawKeypoints(img, kptsSIFT, visImage, cv::Scalar::all(-1), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 60 | windowName = "SIFT Results"; 61 | cv::namedWindow(windowName, 2); 62 | imshow(windowName, visImage); 63 | cv::waitKey(0); 64 | 65 | // EOF STUDENT CODE 66 | } 67 | 68 | int main() 69 | { 70 | descKeypoints1(); 71 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/cornerness_harris -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercise 17 | add_executable (cornerness_harris src/cornerness_harris.cpp) 18 | target_link_libraries (cornerness_harris ${OpenCV_LIBRARIES}) 19 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0005.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0005.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0005.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0005.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0006.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0006.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0006.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0007.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0007.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0007.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0008.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0008.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0008.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0008.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0009.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0009.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0009.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img0009.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img1.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img1gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img1gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img2gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/images/img2gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/cornerness_harris/src/cornerness_harris.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | void cornernessHarris() 11 | { 12 | // load image from file 13 | cv::Mat img; 14 | img = cv::imread("../images/img1.png"); 15 | cv::cvtColor(img, img, cv::COLOR_BGR2GRAY); // convert to grayscale 16 | 17 | // Detector parameters 18 | int blockSize = 2; // for every pixel, a blockSize × blockSize neighborhood is considered 19 | int apertureSize = 3; // aperture parameter for Sobel operator (must be odd) 20 | int minResponse = 100; // minimum value for a corner in the 8bit scaled response matrix 21 | double k = 0.04; // Harris parameter (see equation for details) 22 | 23 | // Detect Harris corners and normalize output 24 | cv::Mat dst, dst_norm, dst_norm_scaled; 25 | dst = cv::Mat::zeros(img.size(), CV_32FC1); 26 | cv::cornerHarris(img, dst, blockSize, apertureSize, k, cv::BORDER_DEFAULT); 27 | cv::normalize(dst, dst_norm, 0, 255, cv::NORM_MINMAX, CV_32FC1, cv::Mat()); 28 | cv::convertScaleAbs(dst_norm, dst_norm_scaled); 29 | 30 | // visualize results 31 | string windowName = "Harris Corner Detector Response Matrix"; 32 | cv::namedWindow(windowName, 4); 33 | cv::imshow(windowName, dst_norm_scaled); 34 | cv::waitKey(0); 35 | 36 | // TODO: Your task is to locate local maxima in the Harris response matrix 37 | // and perform a non-maximum suppression (NMS) in a local neighborhood around 38 | // each maximum. The resulting coordinates shall be stored in a list of keypoints 39 | // of the type `vector`. 40 | 41 | } 42 | 43 | int main() 44 | { 45 | cornernessHarris(); 46 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Harris Corner Detection/solution/cornerness_harris.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | void cornernessHarris() 11 | { 12 | // load image from file 13 | cv::Mat img; 14 | img = cv::imread("../images/img1.png"); 15 | cv::cvtColor(img, img, cv::COLOR_BGR2GRAY); // convert to grayscale 16 | 17 | // Detector parameters 18 | int blockSize = 2; // for every pixel, a blockSize × blockSize neighborhood is considered 19 | int apertureSize = 3; // aperture parameter for Sobel operator (must be odd) 20 | int minResponse = 100; // minimum value for a corner in the 8bit scaled response matrix 21 | double k = 0.04; // Harris parameter (see equation for details) 22 | 23 | // Detect Harris corners and normalize output 24 | cv::Mat dst, dst_norm, dst_norm_scaled; 25 | dst = cv::Mat::zeros(img.size(), CV_32FC1); 26 | cv::cornerHarris(img, dst, blockSize, apertureSize, k, cv::BORDER_DEFAULT); 27 | cv::normalize(dst, dst_norm, 0, 255, cv::NORM_MINMAX, CV_32FC1, cv::Mat()); 28 | cv::convertScaleAbs(dst_norm, dst_norm_scaled); 29 | 30 | // visualize results 31 | string windowName = "Harris Corner Detector Response Matrix"; 32 | cv::namedWindow(windowName, 4); 33 | cv::imshow(windowName, dst_norm_scaled); 34 | cv::waitKey(0); 35 | 36 | // STUDENTS NEET TO ENTER THIS CODE (C3.2 Atom 4) 37 | 38 | // Look for prominent corners and instantiate keypoints 39 | vector keypoints; 40 | double maxOverlap = 0.0; // max. permissible overlap between two features in %, used during non-maxima suppression 41 | for (size_t j = 0; j < dst_norm.rows; j++) 42 | { 43 | for (size_t i = 0; i < dst_norm.cols; i++) 44 | { 45 | int response = (int)dst_norm.at(j, i); 46 | if (response > minResponse) 47 | { // only store points above a threshold 48 | 49 | cv::KeyPoint newKeyPoint; 50 | newKeyPoint.pt = cv::Point2f(i, j); 51 | newKeyPoint.size = 2 * apertureSize; 52 | newKeyPoint.response = response; 53 | 54 | // perform non-maximum suppression (NMS) in local neighbourhood around new key point 55 | bool bOverlap = false; 56 | for (auto it = keypoints.begin(); it != keypoints.end(); ++it) 57 | { 58 | double kptOverlap = cv::KeyPoint::overlap(newKeyPoint, *it); 59 | if (kptOverlap > maxOverlap) 60 | { 61 | bOverlap = true; 62 | if (newKeyPoint.response > (*it).response) 63 | { // if overlap is >t AND response is higher for new kpt 64 | *it = newKeyPoint; // replace old key point with new one 65 | break; // quit loop over keypoints 66 | } 67 | } 68 | } 69 | if (!bOverlap) 70 | { // only add new key point if no overlap has been found in previous NMS 71 | keypoints.push_back(newKeyPoint); // store new keypoint in dynamic list 72 | } 73 | } 74 | } // eof loop over cols 75 | } // eof loop over rows 76 | 77 | // visualize keypoints 78 | windowName = "Harris Corner Detection Results"; 79 | cv::namedWindow(windowName, 5); 80 | cv::Mat visImage = dst_norm_scaled.clone(); 81 | cv::drawKeypoints(dst_norm_scaled, keypoints, visImage, cv::Scalar::all(-1), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 82 | cv::imshow(windowName, visImage); 83 | cv::waitKey(0); 84 | // EOF STUDENT CODE 85 | } 86 | 87 | int main() 88 | { 89 | cornernessHarris(); 90 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/gradient_filtering -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercises 17 | add_executable (gaussian_smoothing src/gaussian_smoothing.cpp) 18 | target_link_libraries (gaussian_smoothing ${OpenCV_LIBRARIES}) 19 | 20 | add_executable (gradient_sobel src/gradient_sobel.cpp) 21 | target_link_libraries (gradient_sobel ${OpenCV_LIBRARIES}) 22 | 23 | add_executable (magnitude_sobel src/magnitude_sobel.cpp) 24 | target_link_libraries (magnitude_sobel ${OpenCV_LIBRARIES}) 25 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0005.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0005.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0005.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0005.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0006.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0006.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0006.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0007.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0007.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0007.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0008.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0008.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0008.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0008.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0009.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0009.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0009.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img0009.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img1.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img1gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img1gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img2gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/images/img2gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/src/gaussian_smoothing.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | using namespace std; 8 | 9 | void gaussianSmoothing1() 10 | { 11 | // load image from file 12 | cv::Mat img; 13 | img = cv::imread("../images/img1gray.png"); 14 | 15 | // create filter kernel 16 | float gauss_data[25] = {1, 4, 7, 4, 1, 17 | 4, 16, 26, 16, 4, 18 | 7, 26, 41, 26, 7, 19 | 4, 16, 26, 16, 4, 20 | 1, 4, 7, 4, 1}; 21 | cv::Mat kernel = cv::Mat(5, 5, CV_32F, gauss_data); 22 | 23 | // TODO: Divide each element of the kernel by the sum of all the values in the kernel. 24 | 25 | // apply filter 26 | cv::Mat result; 27 | cv::filter2D(img, result, -1, kernel, cv::Point(-1, -1), 0, cv::BORDER_DEFAULT); 28 | 29 | // show result 30 | string windowName = "Gaussian Blurring"; 31 | cv::namedWindow(windowName, 1); // create window 32 | cv::imshow(windowName, result); 33 | cv::waitKey(0); // wait for keyboard input before continuing 34 | } 35 | 36 | int main() 37 | { 38 | gaussianSmoothing1(); 39 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/src/gradient_sobel.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | using namespace std; 8 | 9 | void gradientSobel() 10 | { 11 | // Based on the image gradients in both x and y, compute an image 12 | // which contains the gradient magnitude according to the equation at the 13 | // beginning of this leesson (refer to the equations for gradient direction and magnitude) 14 | // for every pixel position. Also, apply different 15 | // levels of Gaussian blurring before applying the Sobel operator and compare the results. 16 | 17 | // Step 1 - load image from file 18 | cv::Mat img; 19 | img = cv::imread("./img1.png"); 20 | 21 | // Step 2 - convert image to grayscale 22 | cv::Mat imgGray; 23 | cv::cvtColor(img, imgGray, cv::COLOR_BGR2GRAY); 24 | 25 | // Step 3 - create filter kernel 26 | float sobel_x[9] = {-1, 0, +1, 27 | -2, 0, +2, 28 | -1, 0, +1}; 29 | cv::Mat kernel_x = cv::Mat(3, 3, CV_32F, sobel_x); 30 | 31 | // Step 4 - apply filter 32 | cv::Mat result_x; 33 | cv::filter2D(imgGray, result_x, -1, kernel_x, cv::Point(-1, -1), 0, cv::BORDER_DEFAULT); 34 | 35 | // Step 5 - show result 36 | string windowName = "Sobel operator (x-direction)"; 37 | cv::namedWindow( windowName, 1 ); // create window 38 | cv::imshow(windowName, result_x); 39 | cv::waitKey(0); // wait for keyboard input before continuing 40 | 41 | } 42 | 43 | int main() 44 | { 45 | gradientSobel(); 46 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/gradient_filtering/src/magnitude_sobel.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | using namespace std; 8 | 9 | void magnitudeSobel() 10 | { 11 | // load image from file 12 | cv::Mat img; 13 | img = cv::imread("./images/img1gray.png"); 14 | 15 | // convert image to grayscale 16 | cv::Mat imgGray; 17 | cv::cvtColor(img, imgGray, cv::COLOR_BGR2GRAY); 18 | 19 | // apply smoothing using the GaussianBlur() function from the OpenCV 20 | // ToDo : Add your code here 21 | 22 | // create filter kernels using the cv::Mat datatype both for x and y 23 | // ToDo : Add your code here 24 | 25 | // apply filter using the OpenCv function filter2D() 26 | // ToDo : Add your code here 27 | 28 | 29 | // compute magnitude image based on the equation presented in the lesson 30 | // ToDo : Add your code here 31 | 32 | 33 | // show result 34 | string windowName = "Gaussian Blurring"; 35 | cv::namedWindow(windowName, 1); // create window 36 | cv::imshow(windowName, magnitude); 37 | cv::waitKey(0); // wait for keyboard input before continuing 38 | } 39 | 40 | int main() 41 | { 42 | magnitudeSobel(); 43 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/solutions/gaussian_smoothing.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | using namespace std; 8 | 9 | void gaussianSmoothing1() 10 | { 11 | // load image from file 12 | cv::Mat img; 13 | img = cv::imread("../images/img1gray.png"); 14 | 15 | // create filter kernel 16 | float gauss_data[25] = {1, 4, 7, 4, 1, 17 | 4, 16, 26, 16, 4, 18 | 7, 26, 41, 26, 7, 19 | 4, 16, 26, 16, 4, 20 | 1, 4, 7, 4, 1}; 21 | cv::Mat kernel = cv::Mat(5, 5, CV_32F, gauss_data); 22 | 23 | // STUDENTS NEET TO ENTER THIS CODE 24 | for (int i = 0; i < 25; i++) 25 | { 26 | gauss_data[i] /= 273; 27 | } 28 | // EOF STUDENT CODE 29 | 30 | // apply filter 31 | cv::Mat result; 32 | cv::filter2D(img, result, -1, kernel, cv::Point(-1, -1), 0, cv::BORDER_DEFAULT); 33 | 34 | // show result 35 | string windowName = "Gaussian Blurring"; 36 | cv::namedWindow(windowName, 1); // create window 37 | cv::imshow(windowName, result); 38 | cv::waitKey(0); // wait for keyboard input before continuing 39 | } 40 | 41 | int main() 42 | { 43 | gaussianSmoothing1(); 44 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Intensity Gradient and Filtering/solutions/magnitude_sobel.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | using namespace std; 8 | 9 | void magnitudeSobel() 10 | { 11 | // load image from file 12 | cv::Mat img; 13 | img = cv::imread("../images/img1gray.png"); 14 | 15 | // convert image to grayscale 16 | cv::Mat imgGray; 17 | cv::cvtColor(img, imgGray, cv::COLOR_BGR2GRAY); 18 | 19 | // apply smoothing 20 | cv::Mat blurred = imgGray.clone(); 21 | int filterSize = 5; 22 | int stdDev = 2.0; 23 | cv::GaussianBlur(imgGray, blurred, cv::Size(filterSize, filterSize), stdDev); 24 | 25 | // create filter kernels 26 | float sobel_x[9] = {-1, 0, +1, -2, 0, +2, -1, 0, +1}; 27 | cv::Mat kernel_x = cv::Mat(3, 3, CV_32F, sobel_x); 28 | 29 | float sobel_y[9] = {-1, -2, -1, 0, 0, 0, +1, +2, +1}; 30 | cv::Mat kernel_y = cv::Mat(3, 3, CV_32F, sobel_y); 31 | 32 | // apply filter 33 | cv::Mat result_x, result_y; 34 | cv::filter2D(blurred, result_x, -1, kernel_x, cv::Point(-1, -1), 0, cv::BORDER_DEFAULT); 35 | cv::filter2D(blurred, result_y, -1, kernel_y, cv::Point(-1, -1), 0, cv::BORDER_DEFAULT); 36 | 37 | // compute magnitude image 38 | cv::Mat magnitude = imgGray.clone(); 39 | for (int r = 0; r < magnitude.rows; r++) 40 | { 41 | for (int c = 0; c < magnitude.cols; c++) 42 | { 43 | magnitude.at(r, c) = sqrt(pow(result_x.at(r, c), 2) + 44 | pow(result_y.at(r, c), 2)); 45 | } 46 | } 47 | 48 | // show result 49 | string windowName = "Gaussian Blurring"; 50 | // string windowName = "Sobel filter magnitude image"; 51 | cv::namedWindow(windowName, 1); // create window 52 | cv::imshow(windowName, magnitude); 53 | cv::waitKey(0); // wait for keyboard input before continuing 54 | } 55 | 56 | int main() 57 | { 58 | magnitudeSobel(); 59 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/detect_keypoints -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercise 17 | add_executable (detect_keypoints src/detect_keypoints.cpp) 18 | target_link_libraries (detect_keypoints ${OpenCV_LIBRARIES}) 19 | -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0005.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0005.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0005.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0005.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0006.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0006.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0006.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0007.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0007.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0007.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0008.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0008.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0008.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0008.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0009.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0009.jpg -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0009.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img0009.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img1.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img1gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img1gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img2gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/images/img2gray.png -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/detect_keypoints/src/detect_keypoints.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | void detKeypoints1() 11 | { 12 | // load image from file and convert to grayscale 13 | cv::Mat imgGray; 14 | cv::Mat img = cv::imread("../images/img1.png"); 15 | cv::cvtColor(img, imgGray, cv::COLOR_BGR2GRAY); 16 | 17 | // Shi-Tomasi detector 18 | int blockSize = 6; // size of a block for computing a derivative covariation matrix over each pixel neighborhood 19 | double maxOverlap = 0.0; // max. permissible overlap between two features in % 20 | double minDistance = (1.0 - maxOverlap) * blockSize; 21 | int maxCorners = img.rows * img.cols / max(1.0, minDistance); // max. num. of keypoints 22 | double qualityLevel = 0.01; // minimal accepted quality of image corners 23 | double k = 0.04; 24 | bool useHarris = false; 25 | 26 | vector kptsShiTomasi; 27 | vector corners; 28 | double t = (double)cv::getTickCount(); 29 | cv::goodFeaturesToTrack(imgGray, corners, maxCorners, qualityLevel, minDistance, cv::Mat(), blockSize, useHarris, k); 30 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 31 | cout << "Shi-Tomasi with n= " << corners.size() << " keypoints in " << 1000 * t / 1.0 << " ms" << endl; 32 | 33 | for (auto it = corners.begin(); it != corners.end(); ++it) 34 | { // add corners to result vector 35 | 36 | cv::KeyPoint newKeyPoint; 37 | newKeyPoint.pt = cv::Point2f((*it).x, (*it).y); 38 | newKeyPoint.size = blockSize; 39 | kptsShiTomasi.push_back(newKeyPoint); 40 | } 41 | 42 | // visualize results 43 | cv::Mat visImage = img.clone(); 44 | cv::drawKeypoints(img, kptsShiTomasi, visImage, cv::Scalar::all(-1), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 45 | string windowName = "Shi-Tomasi Results"; 46 | cv::namedWindow(windowName, 1); 47 | imshow(windowName, visImage); 48 | 49 | // TODO: use the OpenCV library to add the FAST detector 50 | // in addition to the already implemented Shi-Tomasi 51 | // detector and compare both algorithms with regard to 52 | // (a) number of keypoints, (b) distribution of 53 | // keypoints over the image and (c) processing speed. 54 | } 55 | 56 | int main() 57 | { 58 | detKeypoints1(); 59 | } -------------------------------------------------------------------------------- /Lesson 4 - Tracking Image Features/Overview of Popular Keypoint Detectors/solution/detect_keypoints.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | void detKeypoints1() 11 | { 12 | // load image from file and convert to grayscale 13 | cv::Mat imgGray; 14 | cv::Mat img = cv::imread("./images/img1.png"); 15 | cv::cvtColor(img, imgGray, cv::COLOR_BGR2GRAY); 16 | 17 | // Shi-Tomasi detector 18 | int blockSize = 6; // size of a block for computing a derivative covariation matrix over each pixel neighborhood 19 | double maxOverlap = 0.0; // max. permissible overlap between two features in % 20 | double minDistance = (1.0 - maxOverlap) * blockSize; 21 | int maxCorners = img.rows * img.cols / max(1.0, minDistance); // max. num. of keypoints 22 | double qualityLevel = 0.01; // minimal accepted quality of image corners 23 | double k = 0.04; 24 | bool useHarris = false; 25 | 26 | vector kptsShiTomasi; 27 | vector corners; 28 | double t = (double)cv::getTickCount(); 29 | cv::goodFeaturesToTrack(imgGray, corners, maxCorners, qualityLevel, minDistance, cv::Mat(), blockSize, useHarris, k); 30 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 31 | cout << "Shi-Tomasi with n= " << corners.size() << " keypoints in " << 1000 * t / 1.0 << " ms" << endl; 32 | 33 | for (auto it = corners.begin(); it != corners.end(); ++it) 34 | { // add corners to result vector 35 | 36 | cv::KeyPoint newKeyPoint; 37 | newKeyPoint.pt = cv::Point2f((*it).x, (*it).y); 38 | newKeyPoint.size = blockSize; 39 | kptsShiTomasi.push_back(newKeyPoint); 40 | } 41 | 42 | // visualize results 43 | cv::Mat visImage = img.clone(); 44 | cv::drawKeypoints(img, kptsShiTomasi, visImage, cv::Scalar::all(-1), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 45 | string windowName = "Shi-Tomasi Results"; 46 | cv::namedWindow(windowName, 1); 47 | imshow(windowName, visImage); 48 | 49 | // STUDENT CODE 50 | int threshold = 30; // difference between intensity of the central pixel and pixels of a circle around this pixel 51 | bool bNMS = true; // perform non-maxima suppression on keypoints 52 | cv::FastFeatureDetector::DetectorType type = cv::FastFeatureDetector::TYPE_9_16; // TYPE_9_16, TYPE_7_12, TYPE_5_8 53 | cv::Ptr detector = cv::FastFeatureDetector::create(threshold, bNMS, type); 54 | 55 | vector kptsFAST; 56 | t = (double)cv::getTickCount(); 57 | detector->detect(imgGray, kptsFAST); 58 | t = ((double)cv::getTickCount() - t) / cv::getTickFrequency(); 59 | cout << "FAST with n= " << kptsFAST.size() << " keypoints in " << 1000 * t / 1.0 << " ms" << endl; 60 | 61 | visImage = img.clone(); 62 | cv::drawKeypoints(img, kptsFAST, visImage, cv::Scalar::all(-1), cv::DrawMatchesFlags::DRAW_RICH_KEYPOINTS); 63 | windowName = "FAST Results"; 64 | cv::namedWindow(windowName, 2); 65 | imshow(windowName, visImage); 66 | cv::waitKey(0); 67 | 68 | // EOF STUDENT CODE 69 | } 70 | 71 | int main() 72 | { 73 | detKeypoints1(); 74 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/cluster_with_roi -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercises 17 | add_executable (cluster_with_roi src/cluster_with_roi.cpp src/structIO.cpp) 18 | target_link_libraries (cluster_with_roi ${OpenCV_LIBRARIES}) 19 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/dat/C53A3_currBoundingBoxes.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/dat/C53A3_currBoundingBoxes.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/dat/C53A3_currLidarPts.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/dat/C53A3_currLidarPts.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/src/cluster_with_roi.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | #include "structIO.hpp" 8 | #include "dataStructures.h" 9 | 10 | using namespace std; 11 | 12 | void loadCalibrationData(cv::Mat &P_rect_00, cv::Mat &R_rect_00, cv::Mat &RT) 13 | { 14 | RT.at(0,0) = 7.533745e-03; RT.at(0,1) = -9.999714e-01; RT.at(0,2) = -6.166020e-04; RT.at(0,3) = -4.069766e-03; 15 | RT.at(1,0) = 1.480249e-02; RT.at(1,1) = 7.280733e-04; RT.at(1,2) = -9.998902e-01; RT.at(1,3) = -7.631618e-02; 16 | RT.at(2,0) = 9.998621e-01; RT.at(2,1) = 7.523790e-03; RT.at(2,2) = 1.480755e-02; RT.at(2,3) = -2.717806e-01; 17 | RT.at(3,0) = 0.0; RT.at(3,1) = 0.0; RT.at(3,2) = 0.0; RT.at(3,3) = 1.0; 18 | 19 | R_rect_00.at(0,0) = 9.999239e-01; R_rect_00.at(0,1) = 9.837760e-03; R_rect_00.at(0,2) = -7.445048e-03; R_rect_00.at(0,3) = 0.0; 20 | R_rect_00.at(1,0) = -9.869795e-03; R_rect_00.at(1,1) = 9.999421e-01; R_rect_00.at(1,2) = -4.278459e-03; R_rect_00.at(1,3) = 0.0; 21 | R_rect_00.at(2,0) = 7.402527e-03; R_rect_00.at(2,1) = 4.351614e-03; R_rect_00.at(2,2) = 9.999631e-01; R_rect_00.at(2,3) = 0.0; 22 | R_rect_00.at(3,0) = 0; R_rect_00.at(3,1) = 0; R_rect_00.at(3,2) = 0; R_rect_00.at(3,3) = 1; 23 | 24 | P_rect_00.at(0,0) = 7.215377e+02; P_rect_00.at(0,1) = 0.000000e+00; P_rect_00.at(0,2) = 6.095593e+02; P_rect_00.at(0,3) = 0.000000e+00; 25 | P_rect_00.at(1,0) = 0.000000e+00; P_rect_00.at(1,1) = 7.215377e+02; P_rect_00.at(1,2) = 1.728540e+02; P_rect_00.at(1,3) = 0.000000e+00; 26 | P_rect_00.at(2,0) = 0.000000e+00; P_rect_00.at(2,1) = 0.000000e+00; P_rect_00.at(2,2) = 1.000000e+00; P_rect_00.at(2,3) = 0.000000e+00; 27 | 28 | } 29 | 30 | void showLidarTopview(std::vector &lidarPoints, cv::Size worldSize, cv::Size imageSize) 31 | { 32 | // create topview image 33 | cv::Mat topviewImg(imageSize, CV_8UC3, cv::Scalar(0, 0, 0)); 34 | 35 | // plot Lidar points into image 36 | for (auto it = lidarPoints.begin(); it != lidarPoints.end(); ++it) 37 | { 38 | float xw = (*it).x; // world position in m with x facing forward from sensor 39 | float yw = (*it).y; // world position in m with y facing left from sensor 40 | 41 | int y = (-xw * imageSize.height / worldSize.height) + imageSize.height; 42 | int x = (-yw * imageSize.height / worldSize.height) + imageSize.width / 2; 43 | 44 | float zw = (*it).z; // world position in m with y facing left from sensor 45 | if(zw > -1.40){ 46 | 47 | float val = it->x; 48 | float maxVal = worldSize.height; 49 | int red = min(255, (int)(255 * abs((val - maxVal) / maxVal))); 50 | int green = min(255, (int)(255 * (1 - abs((val - maxVal) / maxVal)))); 51 | cv::circle(topviewImg, cv::Point(x, y), 5, cv::Scalar(0, green, red), -1); 52 | } 53 | } 54 | 55 | // plot distance markers 56 | float lineSpacing = 2.0; // gap between distance markers 57 | int nMarkers = floor(worldSize.height / lineSpacing); 58 | for (size_t i = 0; i < nMarkers; ++i) 59 | { 60 | int y = (-(i * lineSpacing) * imageSize.height / worldSize.height) + imageSize.height; 61 | cv::line(topviewImg, cv::Point(0, y), cv::Point(imageSize.width, y), cv::Scalar(255, 0, 0)); 62 | } 63 | 64 | // display image 65 | string windowName = "Top-View Perspective of LiDAR data"; 66 | cv::namedWindow(windowName, 2); 67 | cv::imshow(windowName, topviewImg); 68 | cv::waitKey(0); // wait for key to be pressed 69 | } 70 | 71 | // TODO - Add your code inside this function 72 | void clusterLidarWithROI(std::vector &boundingBoxes, std::vector &lidarPoints) 73 | { 74 | // store calibration data in OpenCV matrices 75 | cv::Mat P_rect_xx(3,4,cv::DataType::type); // 3x4 projection matrix after rectification 76 | cv::Mat R_rect_xx(4,4,cv::DataType::type); // 3x3 rectifying rotation to make image planes co-planar 77 | cv::Mat RT(4,4,cv::DataType::type); // rotation matrix and translation vector 78 | loadCalibrationData(P_rect_xx, R_rect_xx, RT); 79 | 80 | // loop over all Lidar points and associate them to a 2D bounding box 81 | cv::Mat X(4, 1, cv::DataType::type); 82 | cv::Mat Y(3, 1, cv::DataType::type); 83 | 84 | for (auto it1 = lidarPoints.begin(); it1 != lidarPoints.end(); ++it1) 85 | { 86 | // assemble vector for matrix-vector-multiplication 87 | X.at(0, 0) = it1->x; 88 | X.at(1, 0) = it1->y; 89 | X.at(2, 0) = it1->z; 90 | X.at(3, 0) = 1; 91 | 92 | // project Lidar point into camera 93 | Y = P_rect_xx * R_rect_xx * RT * X; 94 | cv::Point pt; 95 | pt.x = Y.at(0, 0) / Y.at(0, 2); // pixel coordinates 96 | pt.y = Y.at(1, 0) / Y.at(0, 2); 97 | 98 | double shrinkFactor = 0.10; 99 | vector::iterator> enclosingBoxes; // pointers to all bounding boxes which enclose the current Lidar point 100 | for (vector::iterator it2 = boundingBoxes.begin(); it2 != boundingBoxes.end(); ++it2) 101 | { 102 | // shrink current bounding box slightly to avoid having too many outlier points around the edges 103 | cv::Rect smallerBox; 104 | smallerBox.x = (*it2).roi.x + shrinkFactor * (*it2).roi.width / 2.0; 105 | smallerBox.y = (*it2).roi.y + shrinkFactor * (*it2).roi.height / 2.0; 106 | smallerBox.width = (*it2).roi.width * (1 - shrinkFactor); 107 | smallerBox.height = (*it2).roi.height * (1 - shrinkFactor); 108 | 109 | // check wether point is within current bounding box 110 | if (smallerBox.contains(pt)) 111 | { 112 | it2->lidarPoints.push_back(*it1); 113 | lidarPoints.erase(it1); 114 | it1--; 115 | break; 116 | } 117 | } // eof loop over all bounding boxes 118 | 119 | // TODO - check wether point has been enclosed by one or by multiple boxes. 120 | // Accordingly, add Lidar point to bounding box 121 | 122 | } // eof loop over all Lidar points 123 | } 124 | 125 | int main() 126 | { 127 | std::vector lidarPoints; 128 | readLidarPts("../dat/C53A3_currLidarPts.dat", lidarPoints); 129 | 130 | std::vector boundingBoxes; 131 | readBoundingBoxes("../dat/C53A3_currBoundingBoxes.dat", boundingBoxes); 132 | 133 | clusterLidarWithROI(boundingBoxes, lidarPoints); 134 | for (auto it = boundingBoxes.begin(); it != boundingBoxes.end(); ++it) 135 | { 136 | if (it->lidarPoints.size() > 0) 137 | { 138 | showLidarTopview(it->lidarPoints, cv::Size(10.0, 25.0), cv::Size(1000, 2000)); 139 | } 140 | } 141 | 142 | return 0; 143 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/src/dataStructures.h: -------------------------------------------------------------------------------- 1 | #ifndef dataStructures_h 2 | #define dataStructures_h 3 | 4 | #include 5 | #include 6 | 7 | struct LidarPoint { // single lidar point in space 8 | double x,y,z,r; // x,y,z in [m], r is point reflectivity 9 | }; 10 | 11 | struct BoundingBox { // bounding box around a classified object (contains both 2D and 3D data) 12 | 13 | int boxID; // unique identifier for this bounding box 14 | int trackID; // unique identifier for the track to which this bounding box belongs 15 | 16 | cv::Rect roi; // 2D region-of-interest in image coordinates 17 | int classID; // ID based on class file provided to YOLO framework 18 | double confidence; // classification trust 19 | 20 | std::vector lidarPoints; // Lidar 3D points which project into 2D image roi 21 | std::vector keypoints; // keypoints enclosed by 2D roi 22 | std::vector kptMatches; // keypoint matches enclosed by 2D roi 23 | }; 24 | 25 | #endif /* dataStructures_h */ 26 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/src/structIO.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include "structIO.hpp" 6 | 7 | using namespace std; 8 | 9 | 10 | /* TEMPLATES */ 11 | template void write_pod(std::ofstream& out, T& t) 12 | { 13 | out.write(reinterpret_cast(&t), sizeof(T)); 14 | } 15 | 16 | 17 | template void read_pod(std::ifstream& in, T& t) 18 | { 19 | in.read(reinterpret_cast(&t), sizeof(T)); 20 | } 21 | 22 | 23 | template void read_pod_vector(std::ifstream& in, std::vector& vect) 24 | { 25 | long size; 26 | 27 | read_pod(in, size); 28 | for(int i = 0;i < size;++i) 29 | { 30 | T t; 31 | read_pod(in, t); 32 | vect.push_back(t); 33 | } 34 | } 35 | 36 | template void write_pod_vector(std::ofstream& out, std::vector& vect) 37 | { 38 | long size = vect.size(); 39 | write_pod(out, size); 40 | for(auto it=vect.begin(); it!=vect.end(); ++it) 41 | { 42 | write_pod(out,*it); 43 | } 44 | } 45 | 46 | 47 | 48 | 49 | /* DATATYPE WRAPPERS */ 50 | 51 | void writeBoundingBoxes(std::vector &input, const char* fileName) 52 | { 53 | std::ofstream out(fileName); 54 | write_pod_vector(out, input); 55 | out.close(); 56 | } 57 | 58 | void readBoundingBoxes(const char* fileName, std::vector &output) 59 | { 60 | std::ifstream in(fileName); 61 | read_pod_vector(in, output); 62 | } 63 | 64 | void writeLidarPts(std::vector &input, const char* fileName) 65 | { 66 | std::ofstream out(fileName); 67 | write_pod_vector(out, input); 68 | out.close(); 69 | } 70 | 71 | 72 | void readLidarPts(const char* fileName, std::vector &output) 73 | { 74 | std::ifstream in(fileName); 75 | read_pod_vector(in, output); 76 | } 77 | 78 | 79 | void writeKeypoints(std::vector &input, const char* fileName) 80 | { 81 | std::ofstream out(fileName); 82 | write_pod_vector(out, input); 83 | out.close(); 84 | } 85 | 86 | 87 | void readKeypoints(const char* fileName, std::vector &output) 88 | { 89 | std::ifstream in(fileName); 90 | read_pod_vector(in, output); 91 | } 92 | 93 | 94 | void writeKptMatches(std::vector &input, const char* fileName) 95 | { 96 | std::ofstream out(fileName); 97 | write_pod_vector(out, input); 98 | out.close(); 99 | } 100 | 101 | 102 | void readKptMatches(const char* fileName, std::vector &output) 103 | { 104 | std::ifstream in(fileName); 105 | read_pod_vector(in, output); 106 | } 107 | 108 | 109 | 110 | void writeDescriptors(cv::Mat &input, const char* fileName) 111 | { 112 | cv::FileStorage opencv_file(fileName, cv::FileStorage::WRITE); 113 | opencv_file << "desc_matrix" << input; 114 | opencv_file.release(); 115 | } 116 | 117 | 118 | void readDescriptors(const char* fileName, cv::Mat &output) 119 | { 120 | cv::FileStorage opencv_file(fileName, cv::FileStorage::READ); 121 | opencv_file["desc_matrix"] >> output; 122 | opencv_file.release(); 123 | } 124 | 125 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/cluster_with_roi/src/structIO.hpp: -------------------------------------------------------------------------------- 1 | #ifndef structIO_hpp 2 | #define structIO_hpp 3 | 4 | #include 5 | #include "dataStructures.h" 6 | 7 | void writeLidarPts(std::vector &input, const char* fileName); 8 | void readLidarPts(const char* fileName, std::vector &output); 9 | 10 | void writeKeypoints(std::vector &input, const char* fileName); 11 | void readKeypoints(const char* fileName, std::vector &output); 12 | 13 | void writeKptMatches(std::vector &input, const char* fileName); 14 | void readKptMatches(const char* fileName, std::vector &output); 15 | 16 | void writeDescriptors(cv::Mat &input, const char* fileName); 17 | void readDescriptors(const char* fileName, cv::Mat &output); 18 | 19 | void writeBoundingBoxes(std::vector &input, const char* fileName); 20 | void readBoundingBoxes(const char* fileName, std::vector &output); 21 | 22 | 23 | template void write_pod(std::ofstream& out, T& t); 24 | template void read_pod(std::ifstream& in, T& t); 25 | template void read_pod_vector(std::ifstream& in, std::vector& vect); 26 | template void write_pod_vector(std::ofstream& out, std::vector& vect); 27 | 28 | #endif /* structIO_hpp */ 29 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Creating 3D-Objects/solution/clusterLidarWithROI.cpp: -------------------------------------------------------------------------------- 1 | // Create groups of Lidar points whose projection into the camera falls into the same bounding box 2 | void clusterLidarWithROI(std::vector &boundingBoxes, std::vector &lidarPoints, float shrinkFactor, cv::Mat &P_rect_xx, cv::Mat &R_rect_xx, cv::Mat &RT) 3 | { 4 | // loop over all Lidar points and associate them to a 2D bounding box 5 | cv::Mat X(4, 1, cv::DataType::type); 6 | cv::Mat Y(3, 1, cv::DataType::type); 7 | 8 | for (auto it1 = lidarPoints.begin(); it1 != lidarPoints.end(); ++it1) 9 | { 10 | // assemble vector for matrix-vector-multiplication 11 | X.at(0, 0) = it1->x; 12 | X.at(1, 0) = it1->y; 13 | X.at(2, 0) = it1->z; 14 | X.at(3, 0) = 1; 15 | 16 | // project Lidar point into camera 17 | Y = P_rect_xx * R_rect_xx * RT * X; 18 | cv::Point pt; 19 | pt.x = Y.at(0, 0) / Y.at(0, 2); // pixel coordinates 20 | pt.y = Y.at(1, 0) / Y.at(0, 2); 21 | 22 | vector::iterator> enclosingBoxes; // pointers to all bounding boxes which enclose the current Lidar point 23 | for (vector::iterator it2 = boundingBoxes.begin(); it2 != boundingBoxes.end(); ++it2) 24 | { 25 | // shrink current bounding box slightly to avoid having too many outlier points around the edges 26 | cv::Rect smallerBox; 27 | smallerBox.x = (*it2).roi.x + shrinkFactor * (*it2).roi.width / 2.0; 28 | smallerBox.y = (*it2).roi.y + shrinkFactor * (*it2).roi.height / 2.0; 29 | smallerBox.width = (*it2).roi.width * (1 - shrinkFactor); 30 | smallerBox.height = (*it2).roi.height * (1 - shrinkFactor); 31 | 32 | // check wether point is within current bounding box 33 | if (smallerBox.contains(pt)) 34 | { 35 | enclosingBoxes.push_back(it2); 36 | } 37 | 38 | } // eof loop over all bounding boxes 39 | 40 | // STUDENTS NEED TO ADD THIS CODE - START 41 | // check wether point has been enclosed by one or by multiple boxes 42 | if (enclosingBoxes.size() == 1) 43 | { 44 | // add Lidar point to bounding box 45 | enclosingBoxes[0]->lidarPoints.push_back(*it1); 46 | } 47 | // STUDENTS NEED TO ADD THIS CODE - END 48 | 49 | } // eof loop over all Lidar points 50 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/lidar_to_camera -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercises 17 | add_executable (show_lidar_top_view src/show_lidar_top_view.cpp src/structIO.cpp) 18 | target_link_libraries (show_lidar_top_view ${OpenCV_LIBRARIES}) 19 | 20 | add_executable (project_lidar_to_camera src/project_lidar_to_camera.cpp src/structIO.cpp) 21 | target_link_libraries (project_lidar_to_camera ${OpenCV_LIBRARIES}) 22 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0000.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0000.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0001.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0001.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0002.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0002.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0003.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0003.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0004.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0004.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0005.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0005.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0006.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0006.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0007.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0007.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0008.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0008.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0009.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/C51_LidarPts_0009.dat -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/calib_cam_to_cam.txt: -------------------------------------------------------------------------------- 1 | calib_time: 09-Jan-2012 13:57:47 2 | corner_dist: 9.950000e-02 3 | S_00: 1.392000e+03 5.120000e+02 4 | K_00: 9.842439e+02 0.000000e+00 6.900000e+02 0.000000e+00 9.808141e+02 2.331966e+02 0.000000e+00 0.000000e+00 1.000000e+00 5 | D_00: -3.728755e-01 2.037299e-01 2.219027e-03 1.383707e-03 -7.233722e-02 6 | R_00: 1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 1.000000e+00 7 | T_00: 2.573699e-16 -1.059758e-16 1.614870e-16 8 | S_rect_00: 1.242000e+03 3.750000e+02 9 | R_rect_00: 9.999239e-01 9.837760e-03 -7.445048e-03 -9.869795e-03 9.999421e-01 -4.278459e-03 7.402527e-03 4.351614e-03 9.999631e-01 10 | P_rect_00: 7.215377e+02 0.000000e+00 6.095593e+02 0.000000e+00 0.000000e+00 7.215377e+02 1.728540e+02 0.000000e+00 0.000000e+00 0.000000e+00 1.000000e+00 0.000000e+00 11 | S_01: 1.392000e+03 5.120000e+02 12 | K_01: 9.895267e+02 0.000000e+00 7.020000e+02 0.000000e+00 9.878386e+02 2.455590e+02 0.000000e+00 0.000000e+00 1.000000e+00 13 | D_01: -3.644661e-01 1.790019e-01 1.148107e-03 -6.298563e-04 -5.314062e-02 14 | R_01: 9.993513e-01 1.860866e-02 -3.083487e-02 -1.887662e-02 9.997863e-01 -8.421873e-03 3.067156e-02 8.998467e-03 9.994890e-01 15 | T_01: -5.370000e-01 4.822061e-03 -1.252488e-02 16 | S_rect_01: 1.242000e+03 3.750000e+02 17 | R_rect_01: 9.996878e-01 -8.976826e-03 2.331651e-02 8.876121e-03 9.999508e-01 4.418952e-03 -2.335503e-02 -4.210612e-03 9.997184e-01 18 | P_rect_01: 7.215377e+02 0.000000e+00 6.095593e+02 -3.875744e+02 0.000000e+00 7.215377e+02 1.728540e+02 0.000000e+00 0.000000e+00 0.000000e+00 1.000000e+00 0.000000e+00 19 | S_02: 1.392000e+03 5.120000e+02 20 | K_02: 9.597910e+02 0.000000e+00 6.960217e+02 0.000000e+00 9.569251e+02 2.241806e+02 0.000000e+00 0.000000e+00 1.000000e+00 21 | D_02: -3.691481e-01 1.968681e-01 1.353473e-03 5.677587e-04 -6.770705e-02 22 | R_02: 9.999758e-01 -5.267463e-03 -4.552439e-03 5.251945e-03 9.999804e-01 -3.413835e-03 4.570332e-03 3.389843e-03 9.999838e-01 23 | T_02: 5.956621e-02 2.900141e-04 2.577209e-03 24 | S_rect_02: 1.242000e+03 3.750000e+02 25 | R_rect_02: 9.998817e-01 1.511453e-02 -2.841595e-03 -1.511724e-02 9.998853e-01 -9.338510e-04 2.827154e-03 9.766976e-04 9.999955e-01 26 | P_rect_02: 7.215377e+02 0.000000e+00 6.095593e+02 4.485728e+01 0.000000e+00 7.215377e+02 1.728540e+02 2.163791e-01 0.000000e+00 0.000000e+00 1.000000e+00 2.745884e-03 27 | S_03: 1.392000e+03 5.120000e+02 28 | K_03: 9.037596e+02 0.000000e+00 6.957519e+02 0.000000e+00 9.019653e+02 2.242509e+02 0.000000e+00 0.000000e+00 1.000000e+00 29 | D_03: -3.639558e-01 1.788651e-01 6.029694e-04 -3.922424e-04 -5.382460e-02 30 | R_03: 9.995599e-01 1.699522e-02 -2.431313e-02 -1.704422e-02 9.998531e-01 -1.809756e-03 2.427880e-02 2.223358e-03 9.997028e-01 31 | T_03: -4.731050e-01 5.551470e-03 -5.250882e-03 32 | S_rect_03: 1.242000e+03 3.750000e+02 33 | R_rect_03: 9.998321e-01 -7.193136e-03 1.685599e-02 7.232804e-03 9.999712e-01 -2.293585e-03 -1.683901e-02 2.415116e-03 9.998553e-01 34 | P_rect_03: 7.215377e+02 0.000000e+00 6.095593e+02 -3.395242e+02 0.000000e+00 7.215377e+02 1.728540e+02 2.199936e+00 0.000000e+00 0.000000e+00 1.000000e+00 2.729905e-03 35 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/dat/calib_velo_to_cam.txt: -------------------------------------------------------------------------------- 1 | calib_time: 15-Mar-2012 11:37:16 2 | R: 7.533745e-03 -9.999714e-01 -6.166020e-04 1.480249e-02 7.280733e-04 -9.998902e-01 9.998621e-01 7.523790e-03 1.480755e-02 3 | T: -4.069766e-03 -7.631618e-02 -2.717806e-01 4 | delta_f: 0.000000e+00 0.000000e+00 5 | delta_c: 0.000000e+00 0.000000e+00 6 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/images/0000000000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/images/0000000000.png -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/images/s_thrun.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/images/s_thrun.jpg -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/src/dataStructures.h: -------------------------------------------------------------------------------- 1 | #ifndef dataStructures_h 2 | #define dataStructures_h 3 | 4 | #include 5 | #include 6 | 7 | struct LidarPoint { // single lidar point in space 8 | double x,y,z,r; // x,y,z in [m], r is point reflectivity 9 | }; 10 | 11 | 12 | #endif /* dataStructures_h */ 13 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/src/project_lidar_to_camera.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | void loadCalibrationData(cv::Mat &P_rect_00, cv::Mat &R_rect_00, cv::Mat &RT) 12 | { 13 | RT.at(0,0) = 7.533745e-03; RT.at(0,1) = -9.999714e-01; RT.at(0,2) = -6.166020e-04; RT.at(0,3) = -4.069766e-03; 14 | RT.at(1,0) = 1.480249e-02; RT.at(1,1) = 7.280733e-04; RT.at(1,2) = -9.998902e-01; RT.at(1,3) = -7.631618e-02; 15 | RT.at(2,0) = 9.998621e-01; RT.at(2,1) = 7.523790e-03; RT.at(2,2) = 1.480755e-02; RT.at(2,3) = -2.717806e-01; 16 | RT.at(3,0) = 0.0; RT.at(3,1) = 0.0; RT.at(3,2) = 0.0; RT.at(3,3) = 1.0; 17 | 18 | R_rect_00.at(0,0) = 9.999239e-01; R_rect_00.at(0,1) = 9.837760e-03; R_rect_00.at(0,2) = -7.445048e-03; R_rect_00.at(0,3) = 0.0; 19 | R_rect_00.at(1,0) = -9.869795e-03; R_rect_00.at(1,1) = 9.999421e-01; R_rect_00.at(1,2) = -4.278459e-03; R_rect_00.at(1,3) = 0.0; 20 | R_rect_00.at(2,0) = 7.402527e-03; R_rect_00.at(2,1) = 4.351614e-03; R_rect_00.at(2,2) = 9.999631e-01; R_rect_00.at(2,3) = 0.0; 21 | R_rect_00.at(3,0) = 0; R_rect_00.at(3,1) = 0; R_rect_00.at(3,2) = 0; R_rect_00.at(3,3) = 1; 22 | 23 | P_rect_00.at(0,0) = 7.215377e+02; P_rect_00.at(0,1) = 0.000000e+00; P_rect_00.at(0,2) = 6.095593e+02; P_rect_00.at(0,3) = 0.000000e+00; 24 | P_rect_00.at(1,0) = 0.000000e+00; P_rect_00.at(1,1) = 7.215377e+02; P_rect_00.at(1,2) = 1.728540e+02; P_rect_00.at(1,3) = 0.000000e+00; 25 | P_rect_00.at(2,0) = 0.000000e+00; P_rect_00.at(2,1) = 0.000000e+00; P_rect_00.at(2,2) = 1.000000e+00; P_rect_00.at(2,3) = 0.000000e+00; 26 | 27 | } 28 | 29 | void projectLidarToCamera2() 30 | { 31 | // load image from file 32 | cv::Mat img = cv::imread("../images/0000000000.png"); 33 | 34 | // load Lidar points from file 35 | std::vector lidarPoints; 36 | readLidarPts("../dat/C51_LidarPts_0000.dat", lidarPoints); 37 | 38 | // store calibration data in OpenCV matrices 39 | cv::Mat P_rect_00(3,4,cv::DataType::type); // 3x4 projection matrix after rectification 40 | cv::Mat R_rect_00(4,4,cv::DataType::type); // 3x3 rectifying rotation to make image planes co-planar 41 | cv::Mat RT(4,4,cv::DataType::type); // rotation matrix and translation vector 42 | loadCalibrationData(P_rect_00, R_rect_00, RT); 43 | 44 | // TODO: project lidar points 45 | cv::Mat visImg = img.clone(); 46 | cv::Mat overlay = visImg.clone(); 47 | 48 | cv::Mat X(4,1,cv::DataType::type); 49 | cv::Mat Y(3,1,cv::DataType::type); 50 | 51 | // TODO 52 | for(auto it=lidarPoints.begin(); it!=lidarPoints.end(); ++it) { 53 | // 1. Convert current Lidar point into homogeneous coordinates and store it in the 4D variable X. 54 | 55 | // 2. Then, apply the projection equation as detailed in lesson 5.1 to map X onto the image plane of the camera. 56 | // Store the result in Y. 57 | 58 | // 3. Once this is done, transform Y back into Euclidean coordinates and store the result in the variable pt. 59 | cv::Point pt; 60 | 61 | float val = it->x; 62 | float maxVal = 20.0; 63 | int red = min(255, (int)(255 * abs((val - maxVal) / maxVal))); 64 | int green = min(255, (int)(255 * (1 - abs((val - maxVal) / maxVal)))); 65 | cv::circle(overlay, pt, 5, cv::Scalar(0, green, red), -1); 66 | } 67 | 68 | float opacity = 0.6; 69 | cv::addWeighted(overlay, opacity, visImg, 1 - opacity, 0, visImg); 70 | 71 | string windowName = "LiDAR data on image overlay"; 72 | cv::namedWindow( windowName, 3 ); 73 | cv::imshow( windowName, visImg ); 74 | cv::waitKey(0); // wait for key to be pressed 75 | } 76 | 77 | int main() 78 | { 79 | projectLidarToCamera2(); 80 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/src/show_lidar_top_view.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | void showLidarTopview() 12 | { 13 | std::vector lidarPoints; 14 | readLidarPts("../dat/C51_LidarPts_0000.dat", lidarPoints); 15 | 16 | cv::Size worldSize(10.0, 20.0); // width and height of sensor field in m 17 | cv::Size imageSize(1000, 2000); // corresponding top view image in pixel 18 | 19 | // create topview image 20 | cv::Mat topviewImg(imageSize, CV_8UC3, cv::Scalar(0, 0, 0)); 21 | 22 | // plot Lidar points into image 23 | for (auto it = lidarPoints.begin(); it != lidarPoints.end(); ++it) 24 | { 25 | float xw = (*it).x; // world position in m with x facing forward from sensor 26 | float yw = (*it).y; // world position in m with y facing left from sensor 27 | 28 | int y = (-xw * imageSize.height / worldSize.height) + imageSize.height; 29 | int x = (-yw * imageSize.width / worldSize.width) + imageSize.width / 2; 30 | 31 | 32 | cv::circle(topviewImg, cv::Point(x, y), 5, cv::Scalar(0, 0, 255), -1); 33 | 34 | // TODO: 35 | // 1. Change the color of the Lidar points such that 36 | // X=0.0m corresponds to red while X=20.0m is shown as green. 37 | // 2. Remove all Lidar points on the road surface while preserving 38 | // measurements on the obstacles in the scene. 39 | } 40 | 41 | // plot distance markers 42 | float lineSpacing = 2.0; // gap between distance markers 43 | int nMarkers = floor(worldSize.height / lineSpacing); 44 | for (size_t i = 0; i < nMarkers; ++i) 45 | { 46 | int y = (-(i * lineSpacing) * imageSize.height / worldSize.height) + imageSize.height; 47 | cv::line(topviewImg, cv::Point(0, y), cv::Point(imageSize.width, y), cv::Scalar(255, 0, 0)); 48 | } 49 | 50 | // display image 51 | string windowName = "Top-View Perspective of LiDAR data"; 52 | cv::namedWindow(windowName, 2); 53 | cv::imshow(windowName, topviewImg); 54 | cv::waitKey(0); // wait for key to be pressed 55 | } 56 | 57 | int main() 58 | { 59 | showLidarTopview(); 60 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/src/structIO.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include "structIO.hpp" 6 | 7 | using namespace std; 8 | 9 | 10 | /* TEMPLATES */ 11 | template void write_pod(std::ofstream& out, T& t) 12 | { 13 | out.write(reinterpret_cast(&t), sizeof(T)); 14 | } 15 | 16 | 17 | template void read_pod(std::ifstream& in, T& t) 18 | { 19 | in.read(reinterpret_cast(&t), sizeof(T)); 20 | } 21 | 22 | 23 | template void read_pod_vector(std::ifstream& in, std::vector& vect) 24 | { 25 | long size; 26 | 27 | read_pod(in, size); 28 | for(int i = 0;i < size;++i) 29 | { 30 | T t; 31 | read_pod(in, t); 32 | vect.push_back(t); 33 | } 34 | } 35 | 36 | template void write_pod_vector(std::ofstream& out, std::vector& vect) 37 | { 38 | long size = vect.size(); 39 | write_pod(out, size); 40 | for(auto it=vect.begin(); it!=vect.end(); ++it) 41 | { 42 | write_pod(out,*it); 43 | } 44 | } 45 | 46 | 47 | 48 | 49 | /* DATATYPE WRAPPERS */ 50 | 51 | void writeLidarPts(std::vector &input, const char* fileName) 52 | { 53 | std::ofstream out(fileName); 54 | write_pod_vector(out, input); 55 | out.close(); 56 | } 57 | 58 | 59 | void readLidarPts(const char* fileName, std::vector &output) 60 | { 61 | std::ifstream in(fileName); 62 | read_pod_vector(in, output); 63 | } 64 | 65 | 66 | void writeKeypoints(std::vector &input, const char* fileName) 67 | { 68 | std::ofstream out(fileName); 69 | write_pod_vector(out, input); 70 | out.close(); 71 | } 72 | 73 | 74 | void readKeypoints(const char* fileName, std::vector &output) 75 | { 76 | std::ifstream in(fileName); 77 | read_pod_vector(in, output); 78 | } 79 | 80 | 81 | void writeKptMatches(std::vector &input, const char* fileName) 82 | { 83 | std::ofstream out(fileName); 84 | write_pod_vector(out, input); 85 | out.close(); 86 | } 87 | 88 | 89 | void readKptMatches(const char* fileName, std::vector &output) 90 | { 91 | std::ifstream in(fileName); 92 | read_pod_vector(in, output); 93 | } 94 | 95 | 96 | 97 | void writeDescriptors(cv::Mat &input, const char* fileName) 98 | { 99 | cv::FileStorage opencv_file(fileName, cv::FileStorage::WRITE); 100 | opencv_file << "desc_matrix" << input; 101 | opencv_file.release(); 102 | } 103 | 104 | 105 | void readDescriptors(const char* fileName, cv::Mat &output) 106 | { 107 | cv::FileStorage opencv_file(fileName, cv::FileStorage::READ); 108 | opencv_file["desc_matrix"] >> output; 109 | opencv_file.release(); 110 | } 111 | 112 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/lidar_to_camera/src/structIO.hpp: -------------------------------------------------------------------------------- 1 | #ifndef structIO_hpp 2 | #define structIO_hpp 3 | 4 | #include 5 | #include "dataStructures.h" 6 | 7 | void writeLidarPts(std::vector &input, const char* fileName); 8 | void readLidarPts(const char* fileName, std::vector &output); 9 | 10 | void writeKeypoints(std::vector &input, const char* fileName); 11 | void readKeypoints(const char* fileName, std::vector &output); 12 | 13 | void writeKptMatches(std::vector &input, const char* fileName); 14 | void readKptMatches(const char* fileName, std::vector &output); 15 | 16 | void writeDescriptors(cv::Mat &input, const char* fileName); 17 | void readDescriptors(const char* fileName, cv::Mat &output); 18 | 19 | 20 | template void write_pod(std::ofstream& out, T& t); 21 | template void read_pod(std::ifstream& in, T& t); 22 | template void read_pod_vector(std::ifstream& in, std::vector& vect); 23 | template void write_pod_vector(std::ofstream& out, std::vector& vect); 24 | 25 | #endif /* structIO_hpp */ 26 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/solutions/filter_lidar_points.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | void loadCalibrationData(cv::Mat &P_rect_00, cv::Mat &R_rect_00, cv::Mat &RT) 12 | { 13 | RT.at(0,0) = 7.533745e-03; RT.at(0,1) = -9.999714e-01; RT.at(0,2) = -6.166020e-04; RT.at(0,3) = -4.069766e-03; 14 | RT.at(1,0) = 1.480249e-02; RT.at(1,1) = 7.280733e-04; RT.at(1,2) = -9.998902e-01; RT.at(1,3) = -7.631618e-02; 15 | RT.at(2,0) = 9.998621e-01; RT.at(2,1) = 7.523790e-03; RT.at(2,2) = 1.480755e-02; RT.at(2,3) = -2.717806e-01; 16 | RT.at(3,0) = 0.0; RT.at(3,1) = 0.0; RT.at(3,2) = 0.0; RT.at(3,3) = 1.0; 17 | 18 | R_rect_00.at(0,0) = 9.999239e-01; R_rect_00.at(0,1) = 9.837760e-03; R_rect_00.at(0,2) = -7.445048e-03; R_rect_00.at(0,3) = 0.0; 19 | R_rect_00.at(1,0) = -9.869795e-03; R_rect_00.at(1,1) = 9.999421e-01; R_rect_00.at(1,2) = -4.278459e-03; R_rect_00.at(1,3) = 0.0; 20 | R_rect_00.at(2,0) = 7.402527e-03; R_rect_00.at(2,1) = 4.351614e-03; R_rect_00.at(2,2) = 9.999631e-01; R_rect_00.at(2,3) = 0.0; 21 | R_rect_00.at(3,0) = 0; R_rect_00.at(3,1) = 0; R_rect_00.at(3,2) = 0; R_rect_00.at(3,3) = 1; 22 | 23 | P_rect_00.at(0,0) = 7.215377e+02; P_rect_00.at(0,1) = 0.000000e+00; P_rect_00.at(0,2) = 6.095593e+02; P_rect_00.at(0,3) = 0.000000e+00; 24 | P_rect_00.at(1,0) = 0.000000e+00; P_rect_00.at(1,1) = 7.215377e+02; P_rect_00.at(1,2) = 1.728540e+02; P_rect_00.at(1,3) = 0.000000e+00; 25 | P_rect_00.at(2,0) = 0.000000e+00; P_rect_00.at(2,1) = 0.000000e+00; P_rect_00.at(2,2) = 1.000000e+00; P_rect_00.at(2,3) = 0.000000e+00; 26 | 27 | } 28 | 29 | void filterLidarPoints() 30 | { 31 | // load image from file 32 | cv::Mat img = cv::imread("../images/0000000000.png"); 33 | 34 | // load Lidar points from file 35 | std::vector lidarPoints; 36 | readLidarPts("../dat/C51_LidarPts_0000.dat", lidarPoints); 37 | 38 | // store calibration data in OpenCV matrices 39 | cv::Mat P_rect_00(3,4,cv::DataType::type); // 3x4 projection matrix after rectification 40 | cv::Mat R_rect_00(4,4,cv::DataType::type); // 3x3 rectifying rotation to make image planes co-planar 41 | cv::Mat RT(4,4,cv::DataType::type); // rotation matrix and translation vector 42 | loadCalibrationData(P_rect_00, R_rect_00, RT); 43 | 44 | // project lidar points 45 | cv::Mat visImg = img.clone(); 46 | cv::Mat overlay = visImg.clone(); 47 | 48 | cv::Mat X(4,1,cv::DataType::type); 49 | cv::Mat Y(3,1,cv::DataType::type); 50 | for(auto it=lidarPoints.begin(); it!=lidarPoints.end(); ++it) { 51 | 52 | float maxX = 25.0, maxY = 6.0, minZ = -1.4; 53 | if(it->x > maxX || it->x < 0.0 || abs(it->y) > maxY || it->z < minZ || it->r<0.01 ) 54 | { 55 | continue; // skip to next point 56 | } 57 | 58 | X.at(0, 0) = it->x; 59 | X.at(1, 0) = it->y; 60 | X.at(2, 0) = it->z; 61 | X.at(3, 0) = 1; 62 | 63 | Y = P_rect_00 * R_rect_00 * RT * X; 64 | cv::Point pt; 65 | pt.x = Y.at(0, 0) / Y.at(2, 0); 66 | pt.y = Y.at(1, 0) / Y.at(2, 0); 67 | 68 | float val = it->x; 69 | float maxVal = 20.0; 70 | int red = min(255, (int)(255 * abs((val - maxVal) / maxVal))); 71 | int green = min(255, (int)(255 * (1 - abs((val - maxVal) / maxVal)))); 72 | cv::circle(overlay, pt, 5, cv::Scalar(0, green, red), -1); 73 | } 74 | 75 | float opacity = 0.6; 76 | cv::addWeighted(overlay, opacity, visImg, 1 - opacity, 0, visImg); 77 | 78 | string windowName = "LiDAR data on image overlay"; 79 | cv::namedWindow( windowName, 3 ); 80 | cv::imshow( windowName, visImg ); 81 | cv::waitKey(0); // wait for key to be pressed 82 | } 83 | 84 | int main() 85 | { 86 | filterLidarPoints(); 87 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/solutions/project_lidar_to_camera.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | void loadCalibrationData(cv::Mat &P_rect_00, cv::Mat &R_rect_00, cv::Mat &RT) 12 | { 13 | RT.at(0,0) = 7.533745e-03; RT.at(0,1) = -9.999714e-01; RT.at(0,2) = -6.166020e-04; RT.at(0,3) = -4.069766e-03; 14 | RT.at(1,0) = 1.480249e-02; RT.at(1,1) = 7.280733e-04; RT.at(1,2) = -9.998902e-01; RT.at(1,3) = -7.631618e-02; 15 | RT.at(2,0) = 9.998621e-01; RT.at(2,1) = 7.523790e-03; RT.at(2,2) = 1.480755e-02; RT.at(2,3) = -2.717806e-01; 16 | RT.at(3,0) = 0.0; RT.at(3,1) = 0.0; RT.at(3,2) = 0.0; RT.at(3,3) = 1.0; 17 | 18 | R_rect_00.at(0,0) = 9.999239e-01; R_rect_00.at(0,1) = 9.837760e-03; R_rect_00.at(0,2) = -7.445048e-03; R_rect_00.at(0,3) = 0.0; 19 | R_rect_00.at(1,0) = -9.869795e-03; R_rect_00.at(1,1) = 9.999421e-01; R_rect_00.at(1,2) = -4.278459e-03; R_rect_00.at(1,3) = 0.0; 20 | R_rect_00.at(2,0) = 7.402527e-03; R_rect_00.at(2,1) = 4.351614e-03; R_rect_00.at(2,2) = 9.999631e-01; R_rect_00.at(2,3) = 0.0; 21 | R_rect_00.at(3,0) = 0; R_rect_00.at(3,1) = 0; R_rect_00.at(3,2) = 0; R_rect_00.at(3,3) = 1; 22 | 23 | P_rect_00.at(0,0) = 7.215377e+02; P_rect_00.at(0,1) = 0.000000e+00; P_rect_00.at(0,2) = 6.095593e+02; P_rect_00.at(0,3) = 0.000000e+00; 24 | P_rect_00.at(1,0) = 0.000000e+00; P_rect_00.at(1,1) = 7.215377e+02; P_rect_00.at(1,2) = 1.728540e+02; P_rect_00.at(1,3) = 0.000000e+00; 25 | P_rect_00.at(2,0) = 0.000000e+00; P_rect_00.at(2,1) = 0.000000e+00; P_rect_00.at(2,2) = 1.000000e+00; P_rect_00.at(2,3) = 0.000000e+00; 26 | 27 | } 28 | 29 | void projectLidarToCamera2() 30 | { 31 | // load image from file 32 | cv::Mat img = cv::imread("../images/0000000000.png"); 33 | 34 | // load Lidar points from file 35 | std::vector lidarPoints; 36 | readLidarPts("../dat/C51_LidarPts_0000.dat", lidarPoints); 37 | 38 | // store calibration data in OpenCV matrices 39 | cv::Mat P_rect_00(3,4,cv::DataType::type); // 3x4 projection matrix after rectification 40 | cv::Mat R_rect_00(4,4,cv::DataType::type); // 3x3 rectifying rotation to make image planes co-planar 41 | cv::Mat RT(4,4,cv::DataType::type); // rotation matrix and translation vector 42 | loadCalibrationData(P_rect_00, R_rect_00, RT); 43 | // project lidar points 44 | cv::Mat visImg = img.clone(); 45 | cv::Mat overlay = visImg.clone(); 46 | 47 | cv::Mat X(4,1,cv::DataType::type); 48 | cv::Mat Y(3,1,cv::DataType::type); 49 | for(auto it=lidarPoints.begin(); it!=lidarPoints.end(); ++it) { 50 | 51 | X.at(0, 0) = it->x; 52 | X.at(1, 0) = it->y; 53 | X.at(2, 0) = it->z; 54 | X.at(3, 0) = 1; 55 | 56 | Y = P_rect_00 * R_rect_00 * RT * X; 57 | cv::Point pt; 58 | pt.x = Y.at(0, 0) / Y.at(2, 0); 59 | pt.y = Y.at(1, 0) / Y.at(2, 0); 60 | 61 | float val = it->x; 62 | float maxVal = 20.0; 63 | int red = min(255, (int)(255 * abs((val - maxVal) / maxVal))); 64 | int green = min(255, (int)(255 * (1 - abs((val - maxVal) / maxVal)))); 65 | cv::circle(overlay, pt, 5, cv::Scalar(0, green, red), -1); 66 | } 67 | 68 | float opacity = 0.6; 69 | cv::addWeighted(overlay, opacity, visImg, 1 - opacity, 0, visImg); 70 | 71 | string windowName = "LiDAR data on image overlay"; 72 | cv::namedWindow( windowName, 3 ); 73 | cv::imshow( windowName, visImg ); 74 | cv::waitKey(0); // wait for key to be pressed 75 | } 76 | 77 | int main() 78 | { 79 | projectLidarToCamera2(); 80 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Lidar-to-Camera Point Projection/solutions/show_lidar_top_view.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | #include "structIO.hpp" 8 | 9 | using namespace std; 10 | 11 | void showLidarTopview() 12 | { 13 | std::vector lidarPoints; 14 | readLidarPts("../dat/C51_LidarPts_0000.dat", lidarPoints); 15 | 16 | cv::Size worldSize(10.0, 20.0); // width and height of sensor field in m 17 | cv::Size imageSize(1000, 2000); // corresponding top view image in pixel 18 | 19 | // create topview image 20 | cv::Mat topviewImg(imageSize, CV_8UC3, cv::Scalar(0, 0, 0)); 21 | 22 | // plot Lidar points into image 23 | for (auto it = lidarPoints.begin(); it != lidarPoints.end(); ++it) 24 | { 25 | float xw = (*it).x; // world position in m with x facing forward from sensor 26 | float yw = (*it).y; // world position in m with y facing left from sensor 27 | 28 | int y = (-xw * imageSize.height / worldSize.height) + imageSize.height; 29 | int x = (-yw * imageSize.width / worldSize.width) + imageSize.width / 2; 30 | 31 | 32 | //cv::circle(topviewImg, cv::Point(x, y), 5, cv::Scalar(0, 0, 255), -1); 33 | 34 | // STUDENT EXERCISE 2 35 | float zw = (*it).z; // world position in m with y facing left from sensor 36 | if(zw > -1.40){ 37 | // EOF STUDENT EXERCISE 2 38 | 39 | // STUDENT EXERCISE 1 40 | float val = it->x; 41 | float maxVal = worldSize.height; 42 | int red = min(255, (int)(255 * abs((val - maxVal) / maxVal))); 43 | int green = min(255, (int)(255 * (1 - abs((val - maxVal) / maxVal)))); 44 | cv::circle(topviewImg, cv::Point(x, y), 5, cv::Scalar(0, green, red), -1); 45 | // EOF STUDENT EXERCISE 1 46 | } 47 | } 48 | 49 | // plot distance markers 50 | float lineSpacing = 2.0; // gap between distance markers 51 | int nMarkers = floor(worldSize.height / lineSpacing); 52 | for (size_t i = 0; i < nMarkers; ++i) 53 | { 54 | int y = (-(i * lineSpacing) * imageSize.height / worldSize.height) + imageSize.height; 55 | cv::line(topviewImg, cv::Point(0, y), cv::Point(imageSize.width, y), cv::Scalar(255, 0, 0)); 56 | } 57 | 58 | // display image 59 | string windowName = "Top-View Perspective of LiDAR data"; 60 | cv::namedWindow(windowName, 2); 61 | cv::imshow(windowName, topviewImg); 62 | cv::waitKey(0); // wait for key to be pressed 63 | } 64 | 65 | int main() 66 | { 67 | showLidarTopview(); 68 | } -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/.student_bashrc: -------------------------------------------------------------------------------- 1 | cd /home/workspace/detect_objects -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8 FATAL_ERROR) 2 | 3 | add_definitions(-std=c++11) 4 | 5 | set(CXX_FLAGS "-Wall" "-pedantic") 6 | set(CMAKE_CXX_FLAGS, "${CXX_FLAGS}") 7 | 8 | project(camera_fusion) 9 | 10 | find_package(OpenCV 4.1 REQUIRED) 11 | 12 | include_directories(${OpenCV_INCLUDE_DIRS}) 13 | link_directories(${OpenCV_LIBRARY_DIRS}) 14 | add_definitions(${OpenCV_DEFINITIONS}) 15 | 16 | # Executables for exercises 17 | add_executable (detect_objects src/detect_objects_2.cpp) 18 | target_link_libraries (detect_objects ${OpenCV_LIBRARIES}) -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/dat/yolo/coco.names: -------------------------------------------------------------------------------- 1 | person 2 | bicycle 3 | car 4 | motorbike 5 | aeroplane 6 | bus 7 | train 8 | truck 9 | boat 10 | traffic light 11 | fire hydrant 12 | stop sign 13 | parking meter 14 | bench 15 | bird 16 | cat 17 | dog 18 | horse 19 | sheep 20 | cow 21 | elephant 22 | bear 23 | zebra 24 | giraffe 25 | backpack 26 | umbrella 27 | handbag 28 | tie 29 | suitcase 30 | frisbee 31 | skis 32 | snowboard 33 | sports ball 34 | kite 35 | baseball bat 36 | baseball glove 37 | skateboard 38 | surfboard 39 | tennis racket 40 | bottle 41 | wine glass 42 | cup 43 | fork 44 | knife 45 | spoon 46 | bowl 47 | banana 48 | apple 49 | sandwich 50 | orange 51 | broccoli 52 | carrot 53 | hot dog 54 | pizza 55 | donut 56 | cake 57 | chair 58 | sofa 59 | pottedplant 60 | bed 61 | diningtable 62 | toilet 63 | tvmonitor 64 | laptop 65 | mouse 66 | remote 67 | keyboard 68 | cell phone 69 | microwave 70 | oven 71 | toaster 72 | sink 73 | refrigerator 74 | book 75 | clock 76 | vase 77 | scissors 78 | teddy bear 79 | hair drier 80 | toothbrush 81 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/dat/yolo/yolov3-tiny.cfg: -------------------------------------------------------------------------------- 1 | [net] 2 | # Testing 3 | batch=1 4 | subdivisions=1 5 | # Training 6 | # batch=64 7 | # subdivisions=2 8 | width=416 9 | height=416 10 | channels=3 11 | momentum=0.9 12 | decay=0.0005 13 | angle=0 14 | saturation = 1.5 15 | exposure = 1.5 16 | hue=.1 17 | 18 | learning_rate=0.001 19 | burn_in=1000 20 | max_batches = 500200 21 | policy=steps 22 | steps=400000,450000 23 | scales=.1,.1 24 | 25 | [convolutional] 26 | batch_normalize=1 27 | filters=16 28 | size=3 29 | stride=1 30 | pad=1 31 | activation=leaky 32 | 33 | [maxpool] 34 | size=2 35 | stride=2 36 | 37 | [convolutional] 38 | batch_normalize=1 39 | filters=32 40 | size=3 41 | stride=1 42 | pad=1 43 | activation=leaky 44 | 45 | [maxpool] 46 | size=2 47 | stride=2 48 | 49 | [convolutional] 50 | batch_normalize=1 51 | filters=64 52 | size=3 53 | stride=1 54 | pad=1 55 | activation=leaky 56 | 57 | [maxpool] 58 | size=2 59 | stride=2 60 | 61 | [convolutional] 62 | batch_normalize=1 63 | filters=128 64 | size=3 65 | stride=1 66 | pad=1 67 | activation=leaky 68 | 69 | [maxpool] 70 | size=2 71 | stride=2 72 | 73 | [convolutional] 74 | batch_normalize=1 75 | filters=256 76 | size=3 77 | stride=1 78 | pad=1 79 | activation=leaky 80 | 81 | [maxpool] 82 | size=2 83 | stride=2 84 | 85 | [convolutional] 86 | batch_normalize=1 87 | filters=512 88 | size=3 89 | stride=1 90 | pad=1 91 | activation=leaky 92 | 93 | [maxpool] 94 | size=2 95 | stride=1 96 | 97 | [convolutional] 98 | batch_normalize=1 99 | filters=1024 100 | size=3 101 | stride=1 102 | pad=1 103 | activation=leaky 104 | 105 | ########### 106 | 107 | [convolutional] 108 | batch_normalize=1 109 | filters=256 110 | size=1 111 | stride=1 112 | pad=1 113 | activation=leaky 114 | 115 | [convolutional] 116 | batch_normalize=1 117 | filters=512 118 | size=3 119 | stride=1 120 | pad=1 121 | activation=leaky 122 | 123 | [convolutional] 124 | size=1 125 | stride=1 126 | pad=1 127 | filters=255 128 | activation=linear 129 | 130 | 131 | 132 | [yolo] 133 | mask = 3,4,5 134 | anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 135 | classes=80 136 | num=6 137 | jitter=.3 138 | ignore_thresh = .7 139 | truth_thresh = 1 140 | random=1 141 | 142 | [route] 143 | layers = -4 144 | 145 | [convolutional] 146 | batch_normalize=1 147 | filters=128 148 | size=1 149 | stride=1 150 | pad=1 151 | activation=leaky 152 | 153 | [upsample] 154 | stride=2 155 | 156 | [route] 157 | layers = -1, 8 158 | 159 | [convolutional] 160 | batch_normalize=1 161 | filters=256 162 | size=3 163 | stride=1 164 | pad=1 165 | activation=leaky 166 | 167 | [convolutional] 168 | size=1 169 | stride=1 170 | pad=1 171 | filters=255 172 | activation=linear 173 | 174 | [yolo] 175 | mask = 0,1,2 176 | anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 177 | classes=80 178 | num=6 179 | jitter=.3 180 | ignore_thresh = .7 181 | truth_thresh = 1 182 | random=1 -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/dat/yolo/yolov3-tiny.weights: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/dat/yolo/yolov3-tiny.weights -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/dat/yolo/yolov3.cfg: -------------------------------------------------------------------------------- 1 | [net] 2 | # Testing 3 | # batch=1 4 | # subdivisions=1 5 | # Training 6 | batch=64 7 | subdivisions=16 8 | width=608 9 | height=608 10 | channels=3 11 | momentum=0.9 12 | decay=0.0005 13 | angle=0 14 | saturation = 1.5 15 | exposure = 1.5 16 | hue=.1 17 | 18 | learning_rate=0.001 19 | burn_in=1000 20 | max_batches = 500200 21 | policy=steps 22 | steps=400000,450000 23 | scales=.1,.1 24 | 25 | [convolutional] 26 | batch_normalize=1 27 | filters=32 28 | size=3 29 | stride=1 30 | pad=1 31 | activation=leaky 32 | 33 | # Downsample 34 | 35 | [convolutional] 36 | batch_normalize=1 37 | filters=64 38 | size=3 39 | stride=2 40 | pad=1 41 | activation=leaky 42 | 43 | [convolutional] 44 | batch_normalize=1 45 | filters=32 46 | size=1 47 | stride=1 48 | pad=1 49 | activation=leaky 50 | 51 | [convolutional] 52 | batch_normalize=1 53 | filters=64 54 | size=3 55 | stride=1 56 | pad=1 57 | activation=leaky 58 | 59 | [shortcut] 60 | from=-3 61 | activation=linear 62 | 63 | # Downsample 64 | 65 | [convolutional] 66 | batch_normalize=1 67 | filters=128 68 | size=3 69 | stride=2 70 | pad=1 71 | activation=leaky 72 | 73 | [convolutional] 74 | batch_normalize=1 75 | filters=64 76 | size=1 77 | stride=1 78 | pad=1 79 | activation=leaky 80 | 81 | [convolutional] 82 | batch_normalize=1 83 | filters=128 84 | size=3 85 | stride=1 86 | pad=1 87 | activation=leaky 88 | 89 | [shortcut] 90 | from=-3 91 | activation=linear 92 | 93 | [convolutional] 94 | batch_normalize=1 95 | filters=64 96 | size=1 97 | stride=1 98 | pad=1 99 | activation=leaky 100 | 101 | [convolutional] 102 | batch_normalize=1 103 | filters=128 104 | size=3 105 | stride=1 106 | pad=1 107 | activation=leaky 108 | 109 | [shortcut] 110 | from=-3 111 | activation=linear 112 | 113 | # Downsample 114 | 115 | [convolutional] 116 | batch_normalize=1 117 | filters=256 118 | size=3 119 | stride=2 120 | pad=1 121 | activation=leaky 122 | 123 | [convolutional] 124 | batch_normalize=1 125 | filters=128 126 | size=1 127 | stride=1 128 | pad=1 129 | activation=leaky 130 | 131 | [convolutional] 132 | batch_normalize=1 133 | filters=256 134 | size=3 135 | stride=1 136 | pad=1 137 | activation=leaky 138 | 139 | [shortcut] 140 | from=-3 141 | activation=linear 142 | 143 | [convolutional] 144 | batch_normalize=1 145 | filters=128 146 | size=1 147 | stride=1 148 | pad=1 149 | activation=leaky 150 | 151 | [convolutional] 152 | batch_normalize=1 153 | filters=256 154 | size=3 155 | stride=1 156 | pad=1 157 | activation=leaky 158 | 159 | [shortcut] 160 | from=-3 161 | activation=linear 162 | 163 | [convolutional] 164 | batch_normalize=1 165 | filters=128 166 | size=1 167 | stride=1 168 | pad=1 169 | activation=leaky 170 | 171 | [convolutional] 172 | batch_normalize=1 173 | filters=256 174 | size=3 175 | stride=1 176 | pad=1 177 | activation=leaky 178 | 179 | [shortcut] 180 | from=-3 181 | activation=linear 182 | 183 | [convolutional] 184 | batch_normalize=1 185 | filters=128 186 | size=1 187 | stride=1 188 | pad=1 189 | activation=leaky 190 | 191 | [convolutional] 192 | batch_normalize=1 193 | filters=256 194 | size=3 195 | stride=1 196 | pad=1 197 | activation=leaky 198 | 199 | [shortcut] 200 | from=-3 201 | activation=linear 202 | 203 | 204 | [convolutional] 205 | batch_normalize=1 206 | filters=128 207 | size=1 208 | stride=1 209 | pad=1 210 | activation=leaky 211 | 212 | [convolutional] 213 | batch_normalize=1 214 | filters=256 215 | size=3 216 | stride=1 217 | pad=1 218 | activation=leaky 219 | 220 | [shortcut] 221 | from=-3 222 | activation=linear 223 | 224 | [convolutional] 225 | batch_normalize=1 226 | filters=128 227 | size=1 228 | stride=1 229 | pad=1 230 | activation=leaky 231 | 232 | [convolutional] 233 | batch_normalize=1 234 | filters=256 235 | size=3 236 | stride=1 237 | pad=1 238 | activation=leaky 239 | 240 | [shortcut] 241 | from=-3 242 | activation=linear 243 | 244 | [convolutional] 245 | batch_normalize=1 246 | filters=128 247 | size=1 248 | stride=1 249 | pad=1 250 | activation=leaky 251 | 252 | [convolutional] 253 | batch_normalize=1 254 | filters=256 255 | size=3 256 | stride=1 257 | pad=1 258 | activation=leaky 259 | 260 | [shortcut] 261 | from=-3 262 | activation=linear 263 | 264 | [convolutional] 265 | batch_normalize=1 266 | filters=128 267 | size=1 268 | stride=1 269 | pad=1 270 | activation=leaky 271 | 272 | [convolutional] 273 | batch_normalize=1 274 | filters=256 275 | size=3 276 | stride=1 277 | pad=1 278 | activation=leaky 279 | 280 | [shortcut] 281 | from=-3 282 | activation=linear 283 | 284 | # Downsample 285 | 286 | [convolutional] 287 | batch_normalize=1 288 | filters=512 289 | size=3 290 | stride=2 291 | pad=1 292 | activation=leaky 293 | 294 | [convolutional] 295 | batch_normalize=1 296 | filters=256 297 | size=1 298 | stride=1 299 | pad=1 300 | activation=leaky 301 | 302 | [convolutional] 303 | batch_normalize=1 304 | filters=512 305 | size=3 306 | stride=1 307 | pad=1 308 | activation=leaky 309 | 310 | [shortcut] 311 | from=-3 312 | activation=linear 313 | 314 | 315 | [convolutional] 316 | batch_normalize=1 317 | filters=256 318 | size=1 319 | stride=1 320 | pad=1 321 | activation=leaky 322 | 323 | [convolutional] 324 | batch_normalize=1 325 | filters=512 326 | size=3 327 | stride=1 328 | pad=1 329 | activation=leaky 330 | 331 | [shortcut] 332 | from=-3 333 | activation=linear 334 | 335 | 336 | [convolutional] 337 | batch_normalize=1 338 | filters=256 339 | size=1 340 | stride=1 341 | pad=1 342 | activation=leaky 343 | 344 | [convolutional] 345 | batch_normalize=1 346 | filters=512 347 | size=3 348 | stride=1 349 | pad=1 350 | activation=leaky 351 | 352 | [shortcut] 353 | from=-3 354 | activation=linear 355 | 356 | 357 | [convolutional] 358 | batch_normalize=1 359 | filters=256 360 | size=1 361 | stride=1 362 | pad=1 363 | activation=leaky 364 | 365 | [convolutional] 366 | batch_normalize=1 367 | filters=512 368 | size=3 369 | stride=1 370 | pad=1 371 | activation=leaky 372 | 373 | [shortcut] 374 | from=-3 375 | activation=linear 376 | 377 | [convolutional] 378 | batch_normalize=1 379 | filters=256 380 | size=1 381 | stride=1 382 | pad=1 383 | activation=leaky 384 | 385 | [convolutional] 386 | batch_normalize=1 387 | filters=512 388 | size=3 389 | stride=1 390 | pad=1 391 | activation=leaky 392 | 393 | [shortcut] 394 | from=-3 395 | activation=linear 396 | 397 | 398 | [convolutional] 399 | batch_normalize=1 400 | filters=256 401 | size=1 402 | stride=1 403 | pad=1 404 | activation=leaky 405 | 406 | [convolutional] 407 | batch_normalize=1 408 | filters=512 409 | size=3 410 | stride=1 411 | pad=1 412 | activation=leaky 413 | 414 | [shortcut] 415 | from=-3 416 | activation=linear 417 | 418 | 419 | [convolutional] 420 | batch_normalize=1 421 | filters=256 422 | size=1 423 | stride=1 424 | pad=1 425 | activation=leaky 426 | 427 | [convolutional] 428 | batch_normalize=1 429 | filters=512 430 | size=3 431 | stride=1 432 | pad=1 433 | activation=leaky 434 | 435 | [shortcut] 436 | from=-3 437 | activation=linear 438 | 439 | [convolutional] 440 | batch_normalize=1 441 | filters=256 442 | size=1 443 | stride=1 444 | pad=1 445 | activation=leaky 446 | 447 | [convolutional] 448 | batch_normalize=1 449 | filters=512 450 | size=3 451 | stride=1 452 | pad=1 453 | activation=leaky 454 | 455 | [shortcut] 456 | from=-3 457 | activation=linear 458 | 459 | # Downsample 460 | 461 | [convolutional] 462 | batch_normalize=1 463 | filters=1024 464 | size=3 465 | stride=2 466 | pad=1 467 | activation=leaky 468 | 469 | [convolutional] 470 | batch_normalize=1 471 | filters=512 472 | size=1 473 | stride=1 474 | pad=1 475 | activation=leaky 476 | 477 | [convolutional] 478 | batch_normalize=1 479 | filters=1024 480 | size=3 481 | stride=1 482 | pad=1 483 | activation=leaky 484 | 485 | [shortcut] 486 | from=-3 487 | activation=linear 488 | 489 | [convolutional] 490 | batch_normalize=1 491 | filters=512 492 | size=1 493 | stride=1 494 | pad=1 495 | activation=leaky 496 | 497 | [convolutional] 498 | batch_normalize=1 499 | filters=1024 500 | size=3 501 | stride=1 502 | pad=1 503 | activation=leaky 504 | 505 | [shortcut] 506 | from=-3 507 | activation=linear 508 | 509 | [convolutional] 510 | batch_normalize=1 511 | filters=512 512 | size=1 513 | stride=1 514 | pad=1 515 | activation=leaky 516 | 517 | [convolutional] 518 | batch_normalize=1 519 | filters=1024 520 | size=3 521 | stride=1 522 | pad=1 523 | activation=leaky 524 | 525 | [shortcut] 526 | from=-3 527 | activation=linear 528 | 529 | [convolutional] 530 | batch_normalize=1 531 | filters=512 532 | size=1 533 | stride=1 534 | pad=1 535 | activation=leaky 536 | 537 | [convolutional] 538 | batch_normalize=1 539 | filters=1024 540 | size=3 541 | stride=1 542 | pad=1 543 | activation=leaky 544 | 545 | [shortcut] 546 | from=-3 547 | activation=linear 548 | 549 | ###################### 550 | 551 | [convolutional] 552 | batch_normalize=1 553 | filters=512 554 | size=1 555 | stride=1 556 | pad=1 557 | activation=leaky 558 | 559 | [convolutional] 560 | batch_normalize=1 561 | size=3 562 | stride=1 563 | pad=1 564 | filters=1024 565 | activation=leaky 566 | 567 | [convolutional] 568 | batch_normalize=1 569 | filters=512 570 | size=1 571 | stride=1 572 | pad=1 573 | activation=leaky 574 | 575 | [convolutional] 576 | batch_normalize=1 577 | size=3 578 | stride=1 579 | pad=1 580 | filters=1024 581 | activation=leaky 582 | 583 | [convolutional] 584 | batch_normalize=1 585 | filters=512 586 | size=1 587 | stride=1 588 | pad=1 589 | activation=leaky 590 | 591 | [convolutional] 592 | batch_normalize=1 593 | size=3 594 | stride=1 595 | pad=1 596 | filters=1024 597 | activation=leaky 598 | 599 | [convolutional] 600 | size=1 601 | stride=1 602 | pad=1 603 | filters=255 604 | activation=linear 605 | 606 | 607 | [yolo] 608 | mask = 6,7,8 609 | anchors = 10,13, 16,30, 33,23, 30,61, 62,45, 59,119, 116,90, 156,198, 373,326 610 | classes=80 611 | num=9 612 | jitter=.3 613 | ignore_thresh = .7 614 | truth_thresh = 1 615 | random=1 616 | 617 | 618 | [route] 619 | layers = -4 620 | 621 | [convolutional] 622 | batch_normalize=1 623 | filters=256 624 | size=1 625 | stride=1 626 | pad=1 627 | activation=leaky 628 | 629 | [upsample] 630 | stride=2 631 | 632 | [route] 633 | layers = -1, 61 634 | 635 | 636 | 637 | [convolutional] 638 | batch_normalize=1 639 | filters=256 640 | size=1 641 | stride=1 642 | pad=1 643 | activation=leaky 644 | 645 | [convolutional] 646 | batch_normalize=1 647 | size=3 648 | stride=1 649 | pad=1 650 | filters=512 651 | activation=leaky 652 | 653 | [convolutional] 654 | batch_normalize=1 655 | filters=256 656 | size=1 657 | stride=1 658 | pad=1 659 | activation=leaky 660 | 661 | [convolutional] 662 | batch_normalize=1 663 | size=3 664 | stride=1 665 | pad=1 666 | filters=512 667 | activation=leaky 668 | 669 | [convolutional] 670 | batch_normalize=1 671 | filters=256 672 | size=1 673 | stride=1 674 | pad=1 675 | activation=leaky 676 | 677 | [convolutional] 678 | batch_normalize=1 679 | size=3 680 | stride=1 681 | pad=1 682 | filters=512 683 | activation=leaky 684 | 685 | [convolutional] 686 | size=1 687 | stride=1 688 | pad=1 689 | filters=255 690 | activation=linear 691 | 692 | 693 | [yolo] 694 | mask = 3,4,5 695 | anchors = 10,13, 16,30, 33,23, 30,61, 62,45, 59,119, 116,90, 156,198, 373,326 696 | classes=80 697 | num=9 698 | jitter=.3 699 | ignore_thresh = .7 700 | truth_thresh = 1 701 | random=1 702 | 703 | 704 | 705 | [route] 706 | layers = -4 707 | 708 | [convolutional] 709 | batch_normalize=1 710 | filters=128 711 | size=1 712 | stride=1 713 | pad=1 714 | activation=leaky 715 | 716 | [upsample] 717 | stride=2 718 | 719 | [route] 720 | layers = -1, 36 721 | 722 | 723 | 724 | [convolutional] 725 | batch_normalize=1 726 | filters=128 727 | size=1 728 | stride=1 729 | pad=1 730 | activation=leaky 731 | 732 | [convolutional] 733 | batch_normalize=1 734 | size=3 735 | stride=1 736 | pad=1 737 | filters=256 738 | activation=leaky 739 | 740 | [convolutional] 741 | batch_normalize=1 742 | filters=128 743 | size=1 744 | stride=1 745 | pad=1 746 | activation=leaky 747 | 748 | [convolutional] 749 | batch_normalize=1 750 | size=3 751 | stride=1 752 | pad=1 753 | filters=256 754 | activation=leaky 755 | 756 | [convolutional] 757 | batch_normalize=1 758 | filters=128 759 | size=1 760 | stride=1 761 | pad=1 762 | activation=leaky 763 | 764 | [convolutional] 765 | batch_normalize=1 766 | size=3 767 | stride=1 768 | pad=1 769 | filters=256 770 | activation=leaky 771 | 772 | [convolutional] 773 | size=1 774 | stride=1 775 | pad=1 776 | filters=255 777 | activation=linear 778 | 779 | 780 | [yolo] 781 | mask = 0,1,2 782 | anchors = 10,13, 16,30, 33,23, 30,61, 62,45, 59,119, 116,90, 156,198, 373,326 783 | classes=80 784 | num=9 785 | jitter=.3 786 | ignore_thresh = .7 787 | truth_thresh = 1 788 | random=1 789 | 790 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/dat/yolo/yolov3.weights: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:523e4e69e1d015393a1b0a441cef1d9c7659e3eb2d7e15f793f060a21b32f297 3 | size 248007048 4 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/images/0000000000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/images/0000000000.png -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/images/s_thrun.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udacity/SFND_Camera/dfdfe7962bb72b299da9073185ca95f5b0297ee4/Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/images/s_thrun.jpg -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/src/dataStructures.h: -------------------------------------------------------------------------------- 1 | #ifndef dataStructures_h 2 | #define dataStructures_h 3 | 4 | #include 5 | #include 6 | 7 | struct LidarPoint { // single lidar point in space 8 | double x,y,z,r; // x,y,z in [m], r is point reflectivity 9 | }; 10 | 11 | struct BoundingBox { // bounding box around a classified object (contains both 2D and 3D data) 12 | 13 | int boxID; // unique identifier for this bounding box 14 | int trackID; // unique identifier for the track to which this bounding box belongs 15 | 16 | cv::Rect roi; // 2D region-of-interest in image coordinates 17 | int classID; // ID based on class file provided to YOLO framework 18 | double confidence; // classification trust 19 | 20 | std::vector lidarPoints; // Lidar 3D points which project into 2D image roi 21 | std::vector keypoints; // keypoints enclosed by 2D roi 22 | std::vector kptMatches; // keypoint matches enclosed by 2D roi 23 | }; 24 | 25 | #endif /* dataStructures_h */ 26 | -------------------------------------------------------------------------------- /Lesson 6 - Combining Camera and Lidar/Object Detection with YOLO/detect_objects/src/detect_objects_2.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #include "dataStructures.h" 10 | 11 | using namespace std; 12 | 13 | void detectObjects2() 14 | { 15 | // load image from file 16 | cv::Mat img = cv::imread("../images/s_thrun.jpg"); 17 | 18 | // load class names from file 19 | string yoloBasePath = "../dat/yolo/"; 20 | string yoloClassesFile = yoloBasePath + "coco.names"; 21 | string yoloModelConfiguration = yoloBasePath + "yolov3.cfg"; 22 | string yoloModelWeights = yoloBasePath + "yolov3.weights"; 23 | 24 | vector classes; 25 | ifstream ifs(yoloClassesFile.c_str()); 26 | string line; 27 | while (getline(ifs, line)) classes.push_back(line); 28 | 29 | // load neural network 30 | cv::dnn::Net net = cv::dnn::readNetFromDarknet(yoloModelConfiguration, yoloModelWeights); 31 | net.setPreferableBackend(cv::dnn::DNN_BACKEND_OPENCV); 32 | net.setPreferableTarget(cv::dnn::DNN_TARGET_CPU); 33 | 34 | // generate 4D blob from input image 35 | cv::Mat blob; 36 | double scalefactor = 1/255.0; 37 | cv::Size size = cv::Size(416, 416); 38 | cv::Scalar mean = cv::Scalar(0,0,0); 39 | bool swapRB = false; 40 | bool crop = false; 41 | cv::dnn::blobFromImage(img, blob, scalefactor, size, mean, swapRB, crop); 42 | 43 | // Get names of output layers 44 | vector names; 45 | vector outLayers = net.getUnconnectedOutLayers(); // get indices of output layers, i.e. layers with unconnected outputs 46 | vector layersNames = net.getLayerNames(); // get names of all layers in the network 47 | 48 | names.resize(outLayers.size()); 49 | for (size_t i = 0; i < outLayers.size(); ++i) // Get the names of the output layers in names 50 | { 51 | names[i] = layersNames[outLayers[i] - 1]; 52 | } 53 | 54 | // invoke forward propagation through network 55 | vector netOutput; 56 | net.setInput(blob); 57 | net.forward(netOutput, names); 58 | 59 | // Scan through all bounding boxes and keep only the ones with high confidence 60 | float confThreshold = 0.20; 61 | vector classIds; 62 | vector confidences; 63 | vector boxes; 64 | for (size_t i = 0; i < netOutput.size(); ++i) 65 | { 66 | float* data = (float*)netOutput[i].data; 67 | for (int j = 0; j < netOutput[i].rows; ++j, data += netOutput[i].cols) 68 | { 69 | cv::Mat scores = netOutput[i].row(j).colRange(5, netOutput[i].cols); 70 | cv::Point classId; 71 | double confidence; 72 | 73 | // Get the value and location of the maximum score 74 | cv::minMaxLoc(scores, 0, &confidence, 0, &classId); 75 | if (confidence > confThreshold) 76 | { 77 | cv::Rect box; int cx, cy; 78 | cx = (int)(data[0] * img.cols); 79 | cy = (int)(data[1] * img.rows); 80 | box.width = (int)(data[2] * img.cols); 81 | box.height = (int)(data[3] * img.rows); 82 | box.x = cx - box.width/2; // left 83 | box.y = cy - box.height/2; // top 84 | 85 | boxes.push_back(box); 86 | classIds.push_back(classId.x); 87 | confidences.push_back((float)confidence); 88 | } 89 | } 90 | } 91 | 92 | // perform non-maxima suppression 93 | float nmsThreshold = 0.4; // Non-maximum suppression threshold 94 | vector indices; 95 | cv::dnn::NMSBoxes(boxes, confidences, confThreshold, nmsThreshold, indices); 96 | std::vector bBoxes; 97 | for (auto it = indices.begin(); it != indices.end(); ++it) 98 | { 99 | BoundingBox bBox; 100 | bBox.roi = boxes[*it]; 101 | bBox.classID = classIds[*it]; 102 | bBox.confidence = confidences[*it]; 103 | bBox.boxID = (int)bBoxes.size(); // zero-based unique identifier for this bounding box 104 | 105 | bBoxes.push_back(bBox); 106 | } 107 | 108 | 109 | // show results 110 | cv::Mat visImg = img.clone(); 111 | for (auto it = bBoxes.begin(); it != bBoxes.end(); ++it) 112 | { 113 | // Draw rectangle displaying the bounding box 114 | int top, left, width, height; 115 | top = (*it).roi.y; 116 | left = (*it).roi.x; 117 | width = (*it).roi.width; 118 | height = (*it).roi.height; 119 | cv::rectangle(visImg, cv::Point(left, top), cv::Point(left + width, top + height), cv::Scalar(0, 255, 0), 2); 120 | 121 | string label = cv::format("%.2f", (*it).confidence); 122 | label = classes[((*it).classID)] + ":" + label; 123 | 124 | // Display label at the top of the bounding box 125 | int baseLine; 126 | cv::Size labelSize = getTextSize(label, cv::FONT_ITALIC, 0.5, 1, &baseLine); 127 | top = max(top, labelSize.height); 128 | rectangle(visImg, cv::Point(left, top - round(1.5 * labelSize.height)), cv::Point(left + round(1.5 * labelSize.width), top + baseLine), cv::Scalar(255, 255, 255), cv::FILLED); 129 | cv::putText(visImg, label, cv::Point(left, top), cv::FONT_ITALIC, 0.75, cv::Scalar(0, 0, 0), 1); 130 | } 131 | 132 | string windowName = "Object classification"; 133 | cv::namedWindow( windowName, 1 ); 134 | cv::imshow( windowName, visImg ); 135 | cv::waitKey(0); // wait for key to be pressed 136 | } 137 | 138 | int main() 139 | { 140 | detectObjects2(); 141 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Sensor Fusion NanoDegree- Camera Course 2 | Camera is the second course in the Sensor Fusion ND. The purpose of this repo is to provide the exercise code to the students, so that they can practice in local system. 3 | 4 | This repo contains lesson-wise exercises and corresponding solutions for Udacity's Sensor Fusion ND. 5 | 6 | ## A. List of Lesson-wise Exercises 7 | 1. Lesson 2: Autonomous Vehicles and Computer Vision 8 | - The OpenCV Library 9 | 1. Lesson 3: Engineering a Collision Detection System 10 | - Estimating TTC with Camera 11 | - Estimating TTC with Lidar 12 | 1. Lesson 4: Tracking Image Features 13 | - Descriptor Matching 14 | - Gradient-based vs. Binary Descriptors 15 | - Haris Corner Detection 16 | - Intensity Gradient and Filtering 17 | - Overview of Popular Keypoint Detectors 18 | 1. Lesson 5: Starter code for "Project: Camera Based 2D Feature Tracking" is available here - https://github.com/udacity/SFND_2D_Feature_Tracking 19 | 1. Lesson 6: Combining Camera and Lidar 20 | - Creating 3D-Objects 21 | - Lidar-to-Camera Point Projection 22 | - Object Detection with YOLO 23 | 1. Lesson 7: Starter code for "Project: Track an Object in 3D Space" is available here - https://github.com/udacity/SFND_3D_Object_Tracking 24 | 25 | 26 | ## B. Dependencies for Running Locally 27 | 1. cmake >= 2.8 28 | * All OSes: [click here for installation instructions](https://cmake.org/install/) 29 | 30 | 31 | 2. make >= 4.1 (Linux, Mac), 3.81 (Windows) 32 | * Linux: make is installed by default on most Linux distros 33 | * Mac: [install Xcode command line tools to get make](https://developer.apple.com/xcode/features/) 34 | * Windows: [Click here for installation instructions](http://gnuwin32.sourceforge.net/packages/make.htm) 35 | 36 | 37 | 3. OpenCV >= 4.1 38 | * This must be compiled from source using the `-D OPENCV_ENABLE_NONFREE=ON` cmake flag for testing the SIFT and SURF detectors. 39 | * The OpenCV 4.1.0 source code can be found [here](https://github.com/opencv/opencv/tree/4.1.0) 40 | 41 | 42 | 4. gcc/g++ >= 5.4 43 | * Linux: gcc / g++ is installed by default on most Linux distros 44 | * Mac: same deal as make - [install Xcode command line tools](https://developer.apple.com/xcode/features/) 45 | * Windows: recommend using [MinGW](http://www.mingw.org/) 46 | 47 | 48 | ## C. Build Instructions 49 | 1. Fork this repo to your Github account 50 | 2. Clone your Github repo. 51 | 3. Go to the top level directory for an exercise, and run the following commands on your terminal: 52 | ``` 53 | mkdir build && cd build 54 | cmake .. 55 | make 56 | ./ 57 | ``` 58 | 4. Update back the remote (online) repo so that you can use the updated code in the classroom workspace. 59 | 60 | ## D. Pushing Large Files (>100MB) [Optional] 61 | The file `Combining Camera and Lidar/Object Detection with YOLO/detect_objects/dat/yolo/yolov3.weights` is 236.52 MB; this exceeds GitHub's file size limit of 100.00 MB 62 | 63 | Github blocks the files that have size>100MB, while pushing them. To push large files, Git provides an option called Git Large File Storage (LFS). See the instructions at [https://git-lfs.github.com/](https://git-lfs.github.com/) to use Git LFS. See [http://git.io/iEPt8g](http://git.io/iEPt8g) for more information. 64 | 65 | You can push large file only to unprotected remote branches. Master branch is by default protected. Read more [here](https://docs.github.com/en/github/administering-a-repository/about-protected-branches). Therefore, you'll have to push to a new unprotected branch, and later, merge it with the remote master. Use the commands below: 66 | 67 | * Create a new local branch 68 | ``` 69 | git checkout -b 70 | ``` 71 | 72 | * Write the solution to your exercise. 73 | 74 | * Install Git Large File Storage (LFS). For MacOS, use 75 | ``` 76 | brew install git-lfs 77 | git lfs install 78 | git lfs track "" 79 | git add .gitattributes 80 | git add 81 | git config --global lfs.contenttype 0 82 | 83 | ``` 84 | 85 | * Add the modified files to the index area, and commit the changes 86 | ``` 87 | git add . --all 88 | git commit -m "your comment" 89 | ``` 90 | * To push the current local branch and set the remote as upstream: 91 | ``` 92 | git push --set-upstream origin 93 | ``` 94 | 95 | * Next, create a PR and merge the new branch with the remote master. 96 | 97 | --------------------------------------------------------------------------------