├── media ├── rosgraph.png ├── terminal.png ├── see_think_act.png ├── Navigation_stack.png └── RGB_Deep_D_visual_SLAM.png ├── record_run_as_rosbag.sh └── README.md /media/rosgraph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/surfii3z/jy_master_thesis/HEAD/media/rosgraph.png -------------------------------------------------------------------------------- /media/terminal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/surfii3z/jy_master_thesis/HEAD/media/terminal.png -------------------------------------------------------------------------------- /media/see_think_act.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/surfii3z/jy_master_thesis/HEAD/media/see_think_act.png -------------------------------------------------------------------------------- /media/Navigation_stack.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/surfii3z/jy_master_thesis/HEAD/media/Navigation_stack.png -------------------------------------------------------------------------------- /media/RGB_Deep_D_visual_SLAM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/surfii3z/jy_master_thesis/HEAD/media/RGB_Deep_D_visual_SLAM.png -------------------------------------------------------------------------------- /record_run_as_rosbag.sh: -------------------------------------------------------------------------------- 1 | rosbag record /camera/color/image_raw /packnet/color/image_raw /packnet/depth/image_raw /orb_slam/pose /vrpn_client_node/tello/pose /sdf_map/occupancy /sdf_map/occupancy_inflate /planning/position_cmd_vis /planning_vis/trajectory /map_generator/global_cloud /odom_visualization/robot /orb_slam/fast_planner_pose -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Master thesis 2 | This thesis work was done as a part of research at [Unmanned System Research Group (USRG)](http://unmanned.kaist.ac.kr/) led by Professor [David Hyunchul Shim](http://unmanned.kaist.ac.kr/bio.html), KAIST, South Korea. 3 | ## General information 4 | ### Important materials 5 | 1) [Presentation](https://drive.google.com/file/d/1pFDCmQryXQwL2Fri5JCQbI7lf9-F2rCw/view?usp=sharing) 6 | 2) [Dissertation](https://drive.google.com/file/d/1lgIQOYwCE5o1goZx6O_hlfct3TAkHzP4/view?usp=sharing) 7 | 8 | ### Title 9 | The development of the scale-aware monocular depth estimation aided monocular visual SLAM system for real-time robot navigation 10 | 11 | ### Motivation 12 | Human pilots could fly any drone by relying only on the monocular visual signal in the goggle. In this work, we want to use high performance computer to replace the human brain to navigate the drone by relying only on the monocular camera sensor. 13 | 14 | ### Objective 15 | To enable the mobile robot to autonomously navigate in an unknown environment using only a monocular camera as its sensor 16 | 17 | ### Approaches 18 | 1) `RGB-Deep D visual SLAM`: Augmented the `monocular depth estimation` (Packnet-sfm) to provide the pseudo depth information to the monocular visual SLAM (ORB_SLAM2) to solves its inherent problems: scale-ambiguity, tracking robustness, and map initialization delay (See presentation file page 5). 19 | 20 | ![RGB_Deep_D_visual_SLAM](https://github.com/surfii3z/jy_master_thesis/blob/main/media/RGB_Deep_D_visual_SLAM.png) 21 | 22 | 2) `Navigation stack`: Following the `see-think-act` scheme for autonomous mobile robot, we integrated visual SLAM (see, `RGB-Deep D SLAM`, _localization/ mapping_), local path planner (think, `Fast-Planner`, _planning/ obstacle avoidance_), and waypoint tracking controller (act, `PD controller`, _control_). 23 | 24 | Module | Employed system 25 | ------------ | ------------- 26 | Visual SLAM | [ORB_SLAM2](https://github.com/raulmur/ORB_SLAM2) 27 | Monocular Depth Estimation | [packnet-sfm](https://github.com/TRI-ML/packnet-sfm) 28 | Local path planner | [Fast-Planner](https://github.com/HKUST-Aerial-Robotics/Fast-Planner) 29 | Controller | [PD controller](https://github.com/surfii3z/drone_controller/tree/thesis) 30 | 31 | ![see_think_act](https://github.com/surfii3z/jy_master_thesis/blob/main/media/see_think_act.png) 32 | 33 | ![Navigation_stack](https://github.com/surfii3z/jy_master_thesis/blob/main/media/Navigation_stack.png) 34 | 35 | ## Dependencies 36 | We make `3` seperated workspaces. The installation instruction for each package is in the correspoding link. 37 | 1) catkin_ws 38 | ```bash 39 | catkin_ws/src 40 | ├── [drone_controller](https://github.com/surfii3z/drone_controller/tree/thesis) 41 | ├── [Fast-Planner](https://github.com/surfii3z/Fast-Planner/tree/thesis) 42 | ├── [image_proc](https://github.com/surfii3z/image_proc/tree/thesis) 43 | ├── [image_undistort](https://github.com/surfii3z/image_undistort/tree/thesis) 44 | └── [tello_driver](https://github.com/surfii3z/tello_driver/tree/thesis) 45 | ``` 46 | 2) packnet_ros_ws 47 | 48 | ```bash 49 | packnet_ros_ws/src/ 50 | ├── [packnet_sfm_ros](https://github.com/surfii3z/packnet_sfm_ros) 51 | └── vision_opencv 52 | ``` 53 | 54 | 3) ORB_SLAM2 55 | 56 | ```bash 57 | [ORB_SLAM2](https://github.com/surfii3z/ORB_SLAM2/tree/thesis) 58 | ```` 59 | 60 | ## How to run 61 | I experimented this system on `DGX-Station (V100)`, tested with `Ubuntu 18.04 LTS`, `python 3.6.9`, `TensorRT-7.1.3.4`, `PyTorch 1.4.0`, `Cuda 11.0` and `CuDNN 8.0.5`. 62 | We need to use 8 terminals to run 63 | ```bash 64 | # ------------------------- 65 | # 00_roscore 66 | roscore 67 | 68 | # ------------------------- 69 | # 01_depth_estimation_module 70 | # pwd: /path/to/packnet_ros_ws 71 | source install/setup.bash --extend 72 | rosrun packnet_sfm_ros trt_packnet_node 73 | 74 | # ------------------------- 75 | # 02_tello_driver 76 | # pwd: /path/to/catkin_ws 77 | source devel/setup.bash 78 | roslaunch tello_driver thesis_tello_node.launch 79 | 80 | # ------------------------- 81 | # 03_visual_SLAM 82 | # pwd: /path/to/ORB_SLAM2/ros 83 | source devel/setup.bash 84 | roslaunch orb_slam2 rgbd_tello_crop.launch 85 | 86 | # ------------------------- 87 | # 04_path_planner_visualization 88 | # pwd: /path/to/catkin_ws 89 | source devel/setup.bash 90 | roslaunch plan_manage rviz.launch 91 | 92 | # ------------------------- 93 | # 05_path_planner 94 | # pwd: /path/to/catkin_ws 95 | source devel/setup.bash 96 | roslaunch plan_manage tello_kino_replan.launch 97 | 98 | 99 | # ------------------------- 100 | # 06_drone_controller 101 | # pwd: /path/to/catkin_ws 102 | source devel/setup.bash 103 | roslaunch drone_controller waypoint_controller.launch 104 | 105 | # ------------------------- 106 | # 07_mission 107 | # pwd: /path/to/catkin_ws 108 | source devel/setup.bash 109 | rosrun drone_controller thesis_mission.py 110 | ``` 111 | ![terminal](https://github.com/surfii3z/jy_master_thesis/blob/main/media/terminal.png) 112 | 113 | ![rosgraph](https://github.com/surfii3z/jy_master_thesis/blob/main/media/rosgraph.png) 114 | 115 | NOTE: The monocular depth estimation network must be trained based on the application. 116 | 117 | ## Results 118 | Please see this [playlist](https://www.youtube.com/watch?v=TcpziH_DZm0&list=PLy765YYpYmKxbRtEeM9Om__sBgPfIyyZO) on Youtube 119 | --------------------------------------------------------------------------------