├── figures ├── RMU.png ├── RM-Depth.png ├── motion network.png └── demo video thumbnail.png ├── Supplementary Material for RM-Depth Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes_CVPR22.pdf ├── LICENSE └── README.md /figures/RMU.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/RM-Depth/HEAD/figures/RMU.png -------------------------------------------------------------------------------- /figures/RM-Depth.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/RM-Depth/HEAD/figures/RM-Depth.png -------------------------------------------------------------------------------- /figures/motion network.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/RM-Depth/HEAD/figures/motion network.png -------------------------------------------------------------------------------- /figures/demo video thumbnail.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/RM-Depth/HEAD/figures/demo video thumbnail.png -------------------------------------------------------------------------------- /Supplementary Material for RM-Depth Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes_CVPR22.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/RM-Depth/HEAD/Supplementary Material for RM-Depth Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes_CVPR22.pdf -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2022 Tak-Wai Hui. All rights reserved. 2 | 3 | This software and the associated documentation files (the "Software"), and the research paper 4 | RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes (the "Paper") 5 | including but not limited to the figures and tables are provided for non-commercial use only and 6 | without any warranty. Any commercial use requires a prior arrangement with the author (Tak-Wai Hui). 7 | When using any parts of the Software or the Paper in your work, please cite the following paper: 8 | 9 | @InProceedings{hui22rmdepth, 10 | author = {Tak-Wai Hui}, 11 | title = {RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes}, 12 | booktitle = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 13 | pages = {1675--1684}, 14 | year = {2022} 15 | } 16 | 17 | The above copyright notice and this permission notice shall be included in all copies or substantial 18 | portions of the Software. 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/rm-depth-unsupervised-learning-of-recurrent-1/unsupervised-monocular-depth-estimation-on)](https://paperswithcode.com/sota/unsupervised-monocular-depth-estimation-on?p=rm-depth-unsupervised-learning-of-recurrent-1) 2 | 3 | # RM-Depth 4 | 5 | This repository (https://github.com/twhui/RM-Depth) is the offical project page for my paper RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes published in CVPR 2022. The up-to-date version of the paper is available on arXiv. The supplementary material is available here. 6 | 7 | 8 |

9 | 10 | 11 | # Overview 12 |

13 | A new unsupervised CNN is proposed to predict single-image depth map and complete 3D motion (motions of moving objects and camera itself) in dynamic scenes without requiring scene rigidity and semantic labels. Optical flow and moving object segementations are also recovered. 14 |

15 | Major contributions: (1) Recurrent modulation units (RMU) are proposed to adaptively and iteratively combine encoder and decoder features. (2) Residual upsampling is proposed for fast and efficient resizing of feature maps while sharp depth can be resulted. (3) A warping-based network is proposed to estimate a motion field of moving objects without using semantic priors. The motion field is further regularized by an outlier-aware training loss. 16 |

17 | Despite the depth model just uses a single image in test time and 2.97M parameters, it achieves state-of-the-art results on the KITTI and Cityscapes benchmarks (AbsRel = 0.107 and 0.090, respectively). Besides, It can run at 40FPS (image size: 640 x 192) on a NVIDIA 1080 GPU. 18 | 19 | # Recurrent Modulation Unit (RMU) 20 |

21 | 22 | Fusion of feature maps across encoder and decoder often appears in depth estimation. In RM-Depth, the depth decoder consists of RMUs. The fusion is iteratively refined by adaptive modulating the encoder features using the hidden state of RMU. This in turn improves the performance of single-image depth inference. 23 | 24 | # Residual upsampling 25 | Conventionally, feature maps are upsampled using a single set of filters. In this work, multiple sets of filters are proposed such that each set of them is specifically trained for upsampling some of the spectral components. This effectively improves upsampling along edges. 26 | 27 | # Motion Network 28 |

29 | Besides camera motion, a 3D motion field of moving objects is recovered in a coarse-to-fine framework through a warping approach. The unsupervised learning of motion field is further improved by introducing an outlieraware regularization loss. 30 | 31 | # Depth Prediction Results 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 |
Semantic PriorKITTI Testing Set (Eigen split)Cityscapes Testing SetModel Size (M)
Monodepth2 (ICCV19)0.115-14.84
PackNet (CVPR20)0.111-128.29
Lee et al. (ICCV21)•0.1140.11622.77
Lee et al. (AAAI21)•0.1120.11114.84
RM-Depth (CVPR22),
updated results		0.107 (trained on K) (predictions),
0.105 (trained on CS+K) (predictions)		0.090 (predictions)2.97
RM-Depth (CVPR22),
1024 x 320		0.106 (predictions)0.088 (predictions)2.97
85 | 86 | # Code Package 87 | Please contact Dr. T.-W. Hui (e-mail provided in the first page of the paper) for academic research or commerical collaborations. 88 | 89 | # License and Citation 90 | This software and associated documentation files (the "Software"), and the research paper (RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes) including but not limited to the figures, and tables (the "Paper") are provided for academic research purposes only and without any warranty. Any commercial use requires my consent. When using any parts of the Software or the Paper in your work, please cite the following paper: 91 | 92 | 
@InProceedings{hui22rmdepth,
93 |  author = {Tak-Wai Hui},
94 |  title = {RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes},
95 |  booktitle = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
96 |  pages = {1675--1684},
97 |  year = {2022}
98 | }
99 | --------------------------------------------------------------------------------