├── LICENSE ├── README.md ├── figures ├── LiteFlowNet3.png ├── cost_volume_modulation.png ├── demo video thumbnail.png ├── flow_field_deformation.png ├── flow_field_deformation_for_ICCV19_submission.png └── toy_example_of_FD.png └── models └── testing ├── deploy_LiteFlowNet3-S-ft-kitti.prototxt ├── deploy_LiteFlowNet3-S-ft-sintel.prototxt ├── deploy_LiteFlowNet3-ft-kitti.prototxt ├── deploy_LiteFlowNet3-ft-sintel.prototxt ├── test_batch.py └── test_iter.py /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2020 Tak-Wai Hui. All rights reserved. 2 | 3 | This software and the associated documentation files (the "Software"), and the research paper 4 | LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation (the "Paper") 5 | including but not limited to the figures and tables are provided for non-commercial use only and 6 | without any warranty. Any commercial use requires a prior arrangement with the author (Tak-Wai Hui). 7 | When using any parts of the Software or the Paper in your work, please cite the following papers: 8 | 9 | @InProceedings{hui20liteflownet3, 10 | author = {Tak-Wai Hui and Chen Change Loy}, 11 | title = {{LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation}}, 12 | journal = {{Proceedings of the European Conference on Computer Vision (ECCV)}}, 13 | pages = {169--184}, 14 | year = {2020}, 15 | } 16 | 17 | @InProceedings{hui20liteflownet2, 18 | author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy}, 19 | title = {{A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization}, 20 | journal = {{IEEE Transactions on Pattern Analysis and Machine Intelligence}}, 21 | year = {2020}, 22 | url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 23 | } 24 | 25 | @InProceedings{hui18liteflownet, 26 | author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy}, 27 | title = {{LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation}}, 28 | booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}}, 29 | year = {2018}, 30 | pages = {8981--8989}, 31 | url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 32 | } 33 | 34 | The above copyright notice and this permission notice shall be included in all copies or 35 | substantial portions of the Software. 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # LiteFlowNet3 2 | 3 | [Important] Declaration // Overview // Cost Volume Modulation (CM) // Flow Field Deformation (FD) // Performance // License and Citation // Prerequisite and Compiling // Training // Trained Models // Testing 4 | 5 | This repository (https://github.com/twhui/LiteFlowNet3) provides the offical release of the code package for my paper "LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation" published in ECCV 2020. The pre-print is available on ECVA or 6 | arXiv (July 2020). Supplementary material is released on ECVA. 7 | 8 | 9 |

10 | 11 | 12 | # Overview 13 |

14 | LiteFlowNet3 is built upon our previous work LiteFlowNet2 (TPAMI 2020) with the incorporation of cost volume modulation (CM) and flow field deformation (FD) for improving the flow accuracy further. For the ease of presentation, only a 2-level encoder-decoder structure is shown. The proposed modules are applicable to other levels but not limited to level 1. 15 | 16 | # Cost Volume Modulation (CM) 17 |

18 | Given a pair of images, the existence of partial occlusion and homogeneous regions makes the establishment of correspondence very challenging. This situation also occurs on feature space because simply transforming images into feature maps does not resolve the correspondence ambiguity. In this way, a cost volume is corrupted and the subsequent flow decoding is seriously affected. To address this problem, we propose to filter outliers in a cost volume by using an adaptive modulation before performing the flow decoding. Besides, a confidence map is introduced to facilitate generating modulation parameters.

19 | 20 | # Flow Field Deformation (FD) 21 |

22 |

23 | In coarse-to-fine flow estimation, a flow estimate from the previous level is used as the flow initialization for the next level. This highly demands the previous estimate to be accurate. Otherwise, erroneous optical flow is propagated to the subsequent levels. Due to local flow consistency, neighboring image points that have similar feature vectors have similar optical flow. With this motivation, we propose to refine a given flow field by replacing each inaccurate optical flow with an accurate one from a nearby position. The refinement can be easily achieved by meta-warping of the flow field according to a displacement field. An auto-correlation (self-similarity) cost volume of feature map is used to store the similarity score of neighboring image points. To avoid trivial solution, a confidence map associated with the given flow field is used to guide the displacement decoding from the cost volume.

24 | 25 | # Performance 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 |
Sintel Clean Testing SetSintel Final Testing SetKITTI12 Testing Set (Avg-All)KITTI15 Testing Set (Fl-fg)Model Size (M)Runtime* (ms) GTX 1080
LiteFlowNet (CVPR18)4.545.381.67.99%5.488
LiteFlowNet2 (TPAMI20)3.484.691.47.64%6.440
HD3 (CVPR19)4.794.671.49.02%39.9128
IRR-PWC (CVPR19)3.844.581.67.52%6.4180
LiteFlowNet3 (ECCV20)3.034.531.36.96%5.259
84 | 85 | Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436. 86 | 87 | # Declaration 88 | Before LiteFlowNet3 was published in ECCV 2020, it was submitted to ICCV 2019 and CVPR 2020. The old version of Flow Field Deformation (ICCV 2019 submission) is not well-designed. Masking of deformed flow field in some image positions is not necessary as this is equivalent to have zero displacement in the displacement field d, i.e. d(x0) = 0 for some image positions x0. 89 |

90 | 91 | Two recent works [a, b] also claimed very similar contributions (including the motivations and technical details) as my Flow Field Deformation. In fact, I am earlier than them to propose this contribution for improving optical flow estimation. The technology of Flow Field Deformation is owned by me. 92 | 93 | [a] CoT-AMFlow: Adaptive Modulation Network with Co-Teaching Strategy for Unsupervised Optical Flow Estimation, CoRL 2020. 94 | 95 | Note: The motivation and technical detail of Flow Modulation Module (FMM) (Sect. 3.1) in their CoRL 2020 paper are basically the same as Flow Field Deformation (FD) (Sect. 3.3) in my ECCV 2020 paper. The only difference between FMM and FD is that their confidence map is implictly computed to adapt for the unsupervised training while LiteFlowNet3 explictly learn it from the training labels. 96 | 97 | [b] UPFlow: Upsampling Pyramid for Unsupervised Optical Flow Learning, CVPR 2021. 98 | 99 | Note: The motivation and some parts of the technical detail of Self-guided Upsample Module (Sect. 3.2) in their CVPR 2021 paper are similar to Flow Field Deformation (FD) (Sect. 3.3) in my ECCV 2020 paper. In fact, their Self-guided Upsample Module is very simialr to the early design in my ICCV 2019 submission. 100 | 101 | # License and Citation  102 | This software and associated documentation files (the "Software"), and the research paper (LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation) including but not limited to the figures, and tables (the "Paper") are provided for academic research purposes only and without any warranty. Any commercial use requires my consent. When using any parts of the Software or the Paper in your work, please cite the following papers: 103 | 104 | 
@InProceedings{hui20liteflownet3,    
105 |  author = {Tak-Wai Hui and Chen Change Loy},    
106 |  title = {{LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation}}, 
107 |  journal = {{Proceedings of the European Conference on Computer Vision (ECCV)}},
108 |  year = {2020},    
109 |  pages = {169--184},
110 | }
111 | 112 | 
@InProceedings{hui20liteflownet2,    
113 |  author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
114 |  title = {{A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization}, 
115 |  journal = {{IEEE Transactions on Pattern Analysis and Machine Intelligence}},
116 |  year = {2020},    
117 |  url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
118 | }
119 | 120 | 
@InProceedings{hui18liteflownet,    
121 |  author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
122 |  title = {{LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation}},    
123 |  booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},    
124 |  year = {2018},    
125 |  pages = {8981--8989},
126 |  url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
127 | }
128 | 129 | # Prerequisite and Compiling 130 | LiteFlowNet3 uses the same Caffe package as LiteFlowNet. Please refer to the details in LiteFlowNet GitHub repository. 131 | 132 | # Training 133 | Please refer to the training steps in LiteFlowNet GitHub repository and adopt the training protocols in LiteFlowNet3 paper. 134 | 135 | # Trained Models 136 | Download the models (LiteFlowNet3-ft-sintel, LiteFlowNet3-ft-kitti, LiteFlowNet3-S-ft-sintel, LiteFlowNet3-S-ft-kitti) and then place the models in the folder models/trained. 137 | 138 | # Testing 139 | 1. Open the testing folder 140 | 
$ cd LiteFlowNet3/models/testing
141 | 142 | 2. Create a soft link in the folder /testing 143 | 
$ ln -s ../../build/tools bin
144 | 145 | 3. Replace MODE in ./test_MODE.py to batch if all the images has the same resolution (e.g. Sintel dataset), otherwise replace it to iter (e.g. KITTI dataset). 146 | 147 | 4. Replace MODEL in lines 9 and 10 of test_MODE.py to one of the trained models (e.g. LiteFlowNet3-ft-sintel). 148 | 149 | 5. Run the testing script. Flow fields (MODEL-0000000.flo, MODEL-0000001.flo, ... etc) are stored in the folder /testing/results having the same order as the image pair sequence. 150 | 
$ test_MODE.py img1_pathList.txt img2_pathList.txt results
151 | -------------------------------------------------------------------------------- /figures/LiteFlowNet3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/LiteFlowNet3/e9f21c8281fc1328f969e40b19dcb46532091def/figures/LiteFlowNet3.png -------------------------------------------------------------------------------- /figures/cost_volume_modulation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/LiteFlowNet3/e9f21c8281fc1328f969e40b19dcb46532091def/figures/cost_volume_modulation.png -------------------------------------------------------------------------------- /figures/demo video thumbnail.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/LiteFlowNet3/e9f21c8281fc1328f969e40b19dcb46532091def/figures/demo video thumbnail.png -------------------------------------------------------------------------------- /figures/flow_field_deformation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/LiteFlowNet3/e9f21c8281fc1328f969e40b19dcb46532091def/figures/flow_field_deformation.png -------------------------------------------------------------------------------- /figures/flow_field_deformation_for_ICCV19_submission.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/LiteFlowNet3/e9f21c8281fc1328f969e40b19dcb46532091def/figures/flow_field_deformation_for_ICCV19_submission.png -------------------------------------------------------------------------------- /figures/toy_example_of_FD.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/twhui/LiteFlowNet3/e9f21c8281fc1328f969e40b19dcb46532091def/figures/toy_example_of_FD.png -------------------------------------------------------------------------------- /models/testing/deploy_LiteFlowNet3-ft-sintel.prototxt: -------------------------------------------------------------------------------- 1 | # LiteFlowNet3 (published in ECCV 2020) 2 | # Copyright (c) 2020 Tak-Wai Hui. All rights reserved. 3 | # 4 | # This software is licensed under the terms of the LiteFlowNet3 licence 5 | # which allows for non-commercial use only unless a prior arrangement 6 | # has been made with the author (Tak-Wai Hui), the full terms of which 7 | # are made available in the LICENSE file. 8 | 9 | layer { 10 | name: "Image1" 11 | type: "ImageData" 12 | top: "img0" 13 | top: "dummy1" 14 | image_data_param { 15 | source: "tmp/img1.txt" 16 | batch_size: 1 17 | } 18 | } 19 | layer { 20 | name: "Image2" 21 | type: "ImageData" 22 | top: "img1" 23 | top: "dummy2" 24 | image_data_param { 25 | source: "tmp/img2.txt" 26 | batch_size: 1 27 | } 28 | } 29 | layer { 30 | name: "dummy1" 31 | type: "Silence" 32 | bottom: "dummy1" 33 | } 34 | 35 | layer { 36 | name: "dummy2" 37 | type: "Silence" 38 | bottom: "dummy2" 39 | } 40 | layer { 41 | name: "Eltwise1" 42 | type: "Eltwise" 43 | bottom: "img0" 44 | top: "img0s" 45 | eltwise_param { 46 | operation: SUM 47 | coeff: 0.00392156862745 48 | } 49 | } 50 | layer { 51 | name: "Eltwise2" 52 | type: "Eltwise" 53 | bottom: "img1" 54 | top: "img1s" 55 | eltwise_param { 56 | operation: SUM 57 | coeff: 0.00392156862745 58 | } 59 | } 60 | layer { 61 | name: "img0s_aug" 62 | type: "DataAugmentation" 63 | bottom: "img0s" 64 | top: "img0_nomean" 65 | augmentation_param { 66 | augment_during_test: true 67 | recompute_mean: 3000 68 | mean_per_pixel: false 69 | } 70 | } 71 | layer { 72 | name: "img1s_aug" 73 | type: "DataAugmentation" 74 | bottom: "img1s" 75 | top: "img1_nomean" 76 | augmentation_param { 77 | augment_during_test: true 78 | recompute_mean: 3000 79 | mean_per_pixel: false 80 | } 81 | } 82 | layer { 83 | name: "Resample1" 84 | type: "Resample" 85 | bottom: "img0_nomean" 86 | top: "img0_nomean_resize" 87 | resample_param { 88 | width: $ADAPTED_WIDTH 89 | height: $ADAPTED_HEIGHT 90 | type: LINEAR 91 | antialias: true 92 | } 93 | } 94 | layer { 95 | name: "Resample2" 96 | type: "Resample" 97 | bottom: "img1_nomean" 98 | top: "img1_nomean_resize" 99 | resample_param { 100 | width: $ADAPTED_WIDTH 101 | height: $ADAPTED_HEIGHT 102 | type: LINEAR 103 | antialias: true 104 | } 105 | } 106 | 107 | # Feature Encoder 108 | layer { 109 | name: "conv1" 110 | type: "Convolution" 111 | bottom: "img0_nomean_resize" 112 | bottom: "img1_nomean_resize" 113 | top: "F0_L1" 114 | top: "F1_L1" 115 | param { lr_mult: 1 decay_mult: 1 } 116 | param { lr_mult: 1 decay_mult: 0 } 117 | convolution_param { 118 | num_output: 32 119 | pad: 3 120 | kernel_size: 7 121 | stride: 1 122 | engine: CUDNN 123 | } 124 | } 125 | layer { 126 | name: "ReLU1a" 127 | type: "ReLU" 128 | bottom: "F0_L1" 129 | top: "F0_L1" 130 | relu_param { negative_slope: 0.1 } 131 | } 132 | layer { 133 | name: "ReLU1b" 134 | type: "ReLU" 135 | bottom: "F1_L1" 136 | top: "F1_L1" 137 | relu_param { negative_slope: 0.1 } 138 | } 139 | 140 | layer { 141 | name: "conv2_1" 142 | type: "Convolution" 143 | bottom: "F0_L1" 144 | bottom: "F1_L1" 145 | top: "F0_1_L2" 146 | top: "F1_1_L2" 147 | param { lr_mult: 1 decay_mult: 1 } 148 | param { lr_mult: 1 decay_mult: 0 } 149 | convolution_param { 150 | num_output: 32 151 | pad: 1 152 | kernel_size: 3 153 | stride: 2 154 | engine: CUDNN 155 | } 156 | } 157 | layer { 158 | name: "ReLU2a_1" 159 | type: "ReLU" 160 | bottom: "F0_1_L2" 161 | top: "F0_1_L2" 162 | relu_param { negative_slope: 0.1 } 163 | } 164 | layer { 165 | name: "ReLU2b_1" 166 | type: "ReLU" 167 | bottom: "F1_1_L2" 168 | top: "F1_1_L2" 169 | relu_param { negative_slope: 0.1 } 170 | } 171 | layer { 172 | name: "conv2_2" 173 | type: "Convolution" 174 | bottom: "F0_1_L2" 175 | bottom: "F1_1_L2" 176 | top: "F0_2_L2" 177 | top: "F1_2_L2" 178 | param { lr_mult: 1 decay_mult: 1 } 179 | param { lr_mult: 1 decay_mult: 0 } 180 | convolution_param { 181 | num_output: 32 182 | pad: 1 183 | kernel_size: 3 184 | stride: 1 185 | engine: CUDNN 186 | } 187 | } 188 | layer { 189 | name: "ReLU2a_2" 190 | type: "ReLU" 191 | bottom: "F0_2_L2" 192 | top: "F0_2_L2" 193 | relu_param { negative_slope: 0.1 } 194 | } 195 | layer { 196 | name: "ReLU2b_2" 197 | type: "ReLU" 198 | bottom: "F1_2_L2" 199 | top: "F1_2_L2" 200 | relu_param { negative_slope: 0.1 } 201 | } 202 | layer { 203 | name: "conv2_3" 204 | type: "Convolution" 205 | bottom: "F0_2_L2" 206 | bottom: "F1_2_L2" 207 | top: "F0_L2" 208 | top: "F1_L2" 209 | param { lr_mult: 1 decay_mult: 1 } 210 | param { lr_mult: 1 decay_mult: 0 } 211 | convolution_param { 212 | num_output: 32 213 | pad: 1 214 | kernel_size: 3 215 | stride: 1 216 | engine: CUDNN 217 | } 218 | } 219 | layer { 220 | name: "ReLU2a_3" 221 | type: "ReLU" 222 | bottom: "F0_L2" 223 | top: "F0_L2" 224 | relu_param { negative_slope: 0.1 } 225 | } 226 | layer { 227 | name: "ReLU2b_3" 228 | type: "ReLU" 229 | bottom: "F1_L2" 230 | top: "F1_L2" 231 | relu_param { negative_slope: 0.1 } 232 | } 233 | 234 | 235 | layer { 236 | name: "conv3_1" 237 | type: "Convolution" 238 | bottom: "F0_L2" 239 | bottom: "F1_L2" 240 | top: "F0_1_L3" 241 | top: "F1_1_L3" 242 | param { lr_mult: 1 decay_mult: 1 } 243 | param { lr_mult: 1 decay_mult: 0 } 244 | convolution_param { 245 | num_output: 64 246 | pad: 1 247 | kernel_size: 3 248 | stride: 2 249 | engine: CUDNN 250 | } 251 | } 252 | layer { 253 | name: "ReLU3a_1" 254 | type: "ReLU" 255 | bottom: "F0_1_L3" 256 | top: "F0_1_L3" 257 | relu_param { negative_slope: 0.1 } 258 | } 259 | layer { 260 | name: "ReLU3b_1" 261 | type: "ReLU" 262 | bottom: "F1_1_L3" 263 | top: "F1_1_L3" 264 | relu_param { negative_slope: 0.1 } 265 | } 266 | layer { 267 | name: "conv3_2" 268 | type: "Convolution" 269 | bottom: "F0_1_L3" 270 | bottom: "F1_1_L3" 271 | top: "F0_L3" 272 | top: "F1_L3" 273 | param { lr_mult: 1 decay_mult: 1 } 274 | param { lr_mult: 1 decay_mult: 0 } 275 | convolution_param { 276 | num_output: 64 277 | pad: 1 278 | kernel_size: 3 279 | stride: 1 280 | engine: CUDNN 281 | } 282 | } 283 | layer { 284 | name: "ReLU3a_2" 285 | type: "ReLU" 286 | bottom: "F0_L3" 287 | top: "F0_L3" 288 | relu_param { negative_slope: 0.1 } 289 | } 290 | layer { 291 | name: "ReLU3b_2" 292 | type: "ReLU" 293 | bottom: "F1_L3" 294 | top: "F1_L3" 295 | relu_param { negative_slope: 0.1 } 296 | } 297 | 298 | layer { 299 | name: "conv4_1" 300 | type: "Convolution" 301 | bottom: "F0_L3" 302 | bottom: "F1_L3" 303 | top: "F0_1_L4" 304 | top: "F1_1_L4" 305 | param { lr_mult: 1 decay_mult: 1 } 306 | param { lr_mult: 1 decay_mult: 0 } 307 | convolution_param { 308 | num_output: 96 309 | pad: 1 310 | kernel_size: 3 311 | stride: 2 312 | engine: CUDNN 313 | } 314 | } 315 | layer { 316 | name: "ReLU4a_1" 317 | type: "ReLU" 318 | bottom: "F0_1_L4" 319 | top: "F0_1_L4" 320 | relu_param { negative_slope: 0.1 } 321 | } 322 | layer { 323 | name: "ReLU4b_1" 324 | type: "ReLU" 325 | bottom: "F1_1_L4" 326 | top: "F1_1_L4" 327 | relu_param { negative_slope: 0.1 } 328 | } 329 | layer { 330 | name: "conv4_2" 331 | type: "Convolution" 332 | bottom: "F0_1_L4" 333 | bottom: "F1_1_L4" 334 | top: "F0_L4" 335 | top: "F1_L4" 336 | param { lr_mult: 1 decay_mult: 1 } 337 | param { lr_mult: 1 decay_mult: 0 } 338 | convolution_param { 339 | num_output: 96 340 | pad: 1 341 | kernel_size: 3 342 | stride: 1 343 | engine: CUDNN 344 | } 345 | } 346 | layer { 347 | name: "ReLU4a_2" 348 | type: "ReLU" 349 | bottom: "F0_L4" 350 | top: "F0_L4" 351 | relu_param { negative_slope: 0.1 } 352 | } 353 | layer { 354 | name: "ReLU4b_2" 355 | type: "ReLU" 356 | bottom: "F1_L4" 357 | top: "F1_L4" 358 | relu_param { negative_slope: 0.1 } 359 | } 360 | 361 | layer { 362 | name: "conv5" 363 | type: "Convolution" 364 | bottom: "F0_L4" 365 | bottom: "F1_L4" 366 | top: "F0_L5" 367 | top: "F1_L5" 368 | param { lr_mult: 1 decay_mult: 1 } 369 | param { lr_mult: 1 decay_mult: 0 } 370 | convolution_param { 371 | num_output: 128 372 | pad: 1 373 | kernel_size: 3 374 | stride: 2 375 | engine: CUDNN 376 | } 377 | } 378 | layer { 379 | name: "ReLU5a" 380 | type: "ReLU" 381 | bottom: "F0_L5" 382 | top: "F0_L5" 383 | relu_param { negative_slope: 0.1 } 384 | } 385 | layer { 386 | name: "ReLU5b" 387 | type: "ReLU" 388 | bottom: "F1_L5" 389 | top: "F1_L5" 390 | relu_param { negative_slope: 0.1 } 391 | } 392 | 393 | layer { 394 | name: "conv6" 395 | type: "Convolution" 396 | bottom: "F0_L5" 397 | bottom: "F1_L5" 398 | top: "F0_L6" 399 | top: "F1_L6" 400 | param { lr_mult: 1 decay_mult: 1 } 401 | param { lr_mult: 1 decay_mult: 0 } 402 | convolution_param { 403 | num_output: 192 404 | pad: 1 405 | kernel_size: 3 406 | stride: 2 407 | engine: CUDNN 408 | } 409 | } 410 | layer { 411 | name: "ReLU6a" 412 | type: "ReLU" 413 | bottom: "F0_L6" 414 | top: "F0_L6" 415 | relu_param { negative_slope: 0.1 } 416 | } 417 | layer { 418 | name: "ReLU6b" 419 | type: "ReLU" 420 | bottom: "F1_L6" 421 | top: "F1_L6" 422 | relu_param { negative_slope: 0.1 } 423 | } 424 | 425 | # Downsample images 426 | layer { 427 | name: "Downsample_img0_aug_L3" 428 | type: "Downsample" 429 | bottom: "img0_nomean_resize" 430 | bottom: "F0_L3" 431 | top: "img0_aug_L3" 432 | propagate_down: false 433 | propagate_down: false 434 | } 435 | layer { 436 | name: "Downsample_img1_aug_L3" 437 | type: "Downsample" 438 | bottom: "img1_nomean_resize" 439 | bottom: "F1_L3" 440 | top: "img1_aug_L3" 441 | propagate_down: false 442 | propagate_down: false 443 | } 444 | 445 | layer { 446 | name: "Downsample_img0_aug_L4" 447 | type: "Downsample" 448 | bottom: "img0_nomean_resize" 449 | bottom: "F0_L4" 450 | top: "img0_aug_L4" 451 | propagate_down: false 452 | propagate_down: false 453 | } 454 | layer { 455 | name: "Downsample_img1_aug_L4" 456 | type: "Downsample" 457 | bottom: "img1_nomean_resize" 458 | bottom: "F1_L4" 459 | top: "img1_aug_L4" 460 | propagate_down: false 461 | propagate_down: false 462 | } 463 | 464 | layer { 465 | name: "Downsample_img0_aug_L5" 466 | type: "Downsample" 467 | bottom: "img0_aug_L4" 468 | bottom: "F0_L5" 469 | top: "img0_aug_L5" 470 | propagate_down: false 471 | propagate_down: false 472 | } 473 | layer { 474 | name: "Downsample_img1_aug_L5" 475 | type: "Downsample" 476 | bottom: "img1_aug_L4" 477 | bottom: "F1_L5" 478 | top: "img1_aug_L5" 479 | propagate_down: false 480 | propagate_down: false 481 | } 482 | 483 | layer { 484 | name: "Downsample_img0_aug_L6" 485 | type: "Downsample" 486 | bottom: "img0_aug_L4" 487 | bottom: "F0_L6" 488 | top: "img0_aug_L6" 489 | propagate_down: false 490 | propagate_down: false 491 | } 492 | layer { 493 | name: "Downsample_img1_aug_L6" 494 | type: "Downsample" 495 | bottom: "img1_aug_L4" 496 | bottom: "F1_L6" 497 | top: "img1_aug_L6" 498 | propagate_down: false 499 | propagate_down: false 500 | } 501 | 502 | # Feature Matching: L6 503 | layer { 504 | name: "corr_L6" 505 | type: "Correlation" 506 | bottom: "F0_L6" 507 | bottom: "F1_L6" 508 | top: "corr_L6" 509 | correlation_param { 510 | pad: 4 511 | kernel_size: 1 512 | max_displacement: 4 513 | stride_1: 1 514 | stride_2: 1 515 | } 516 | } 517 | layer { 518 | name: "ReLU_corr_L6" 519 | type: "ReLU" 520 | bottom: "corr_L6" 521 | top: "corr_L6" 522 | relu_param { 523 | negative_slope: 0.1 524 | } 525 | } 526 | layer { 527 | name: "conv1_D1_L6" 528 | type: "Convolution" 529 | bottom: "corr_L6" 530 | top: "conv1_D1_L6" 531 | param { lr_mult: 1 decay_mult: 1 } 532 | param { lr_mult: 1 decay_mult: 0 } 533 | convolution_param { 534 | num_output: 128 535 | pad: 1 536 | kernel_size: 3 537 | stride: 1 538 | engine: CUDNN 539 | } 540 | } 541 | layer { 542 | name: "ReLU1_D1_L6" 543 | type: "ReLU" 544 | bottom: "conv1_D1_L6" 545 | top: "conv1_D1_L6" 546 | relu_param { negative_slope: 0.1 } 547 | } 548 | layer { 549 | name: "conv2_D1_L6" 550 | type: "Convolution" 551 | bottom: "conv1_D1_L6" 552 | top: "conv2_D1_L6" 553 | param { lr_mult: 1 decay_mult: 1 } 554 | param { lr_mult: 1 decay_mult: 0 } 555 | convolution_param { 556 | num_output: 128 557 | pad: 1 558 | kernel_size: 3 559 | stride: 1 560 | engine: CUDNN 561 | } 562 | } 563 | layer { 564 | name: "ReLU2_D1_L6" 565 | type: "ReLU" 566 | bottom: "conv2_D1_L6" 567 | top: "conv2_D1_L6" 568 | relu_param { negative_slope: 0.1 } 569 | } 570 | layer { 571 | name: "conv3_D1_L6" 572 | type: "Convolution" 573 | bottom: "conv2_D1_L6" 574 | top: "conv3_D1_L6" 575 | param { lr_mult: 1 decay_mult: 1 } 576 | param { lr_mult: 1 decay_mult: 0 } 577 | convolution_param { 578 | num_output: 96 579 | pad: 1 580 | kernel_size: 3 581 | stride: 1 582 | engine: CUDNN 583 | } 584 | } 585 | layer { 586 | name: "ReLU3_D1_L6" 587 | type: "ReLU" 588 | bottom: "conv3_D1_L6" 589 | top: "conv3_D1_L6" 590 | relu_param { negative_slope: 0.1 } 591 | } 592 | layer { 593 | name: "conv4_D1_L6" 594 | type: "Convolution" 595 | bottom: "conv3_D1_L6" 596 | top: "conv4_D1_L6" 597 | param { lr_mult: 1 decay_mult: 1 } 598 | param { lr_mult: 1 decay_mult: 0 } 599 | convolution_param { 600 | num_output: 64 601 | pad: 1 602 | kernel_size: 3 603 | stride: 1 604 | engine: CUDNN 605 | } 606 | } 607 | layer { 608 | name: "ReLU4_D1_L6" 609 | type: "ReLU" 610 | bottom: "conv4_D1_L6" 611 | top: "conv4_D1_L6" 612 | relu_param { negative_slope: 0.1 } 613 | } 614 | layer { 615 | name: "conv5_D1_L6" 616 | type: "Convolution" 617 | bottom: "conv4_D1_L6" 618 | top: "conv5_D1_L6" 619 | param { lr_mult: 1 decay_mult: 1 } 620 | param { lr_mult: 1 decay_mult: 0 } 621 | convolution_param { 622 | num_output: 32 623 | pad: 1 624 | kernel_size: 3 625 | stride: 1 626 | engine: CUDNN 627 | } 628 | } 629 | layer { 630 | name: "ReLU5_D1_L6" 631 | type: "ReLU" 632 | bottom: "conv5_D1_L6" 633 | top: "conv5_D1_L6" 634 | relu_param { negative_slope: 0.1 } 635 | } 636 | layer { 637 | name: "scaled_flow_D1_L6" 638 | type: "Convolution" 639 | bottom: "conv5_D1_L6" 640 | top: "scaled_flow_D1_L6" 641 | param { lr_mult: 1 decay_mult: 1 } 642 | param { lr_mult: 1 decay_mult: 0 } 643 | convolution_param { 644 | num_output: 2 645 | pad: 1 646 | kernel_size: 3 647 | stride: 1 648 | engine: CUDNN 649 | } 650 | } 651 | 652 | # Sub-Pixel Refinement: L6 653 | layer { 654 | name: "FlowUnscaling_L6_D2" 655 | type: "Eltwise" 656 | bottom: "scaled_flow_D1_L6" 657 | top: "flow_D1_L6" 658 | eltwise_param { 659 | operation: SUM 660 | coeff: 0.625 661 | } 662 | } 663 | layer { 664 | name: "gxy_L6" 665 | type: "Grid" 666 | top: "gxy_L6" 667 | bottom: "flow_D1_L6" 668 | propagate_down: false 669 | } 670 | layer { 671 | name: "coords_D1_L6" 672 | type: "Eltwise" 673 | bottom: "flow_D1_L6" 674 | bottom: "gxy_L6" 675 | top: "coords_D1_L6" 676 | eltwise_param { coeff: 1 coeff: 1 } 677 | } 678 | layer { 679 | name: "warped_F1_L6" 680 | type: "Warp" 681 | bottom: "F1_L6" 682 | bottom: "coords_D1_L6" 683 | top: "warped_D1_F1_L6" 684 | } 685 | layer { 686 | name: "F_D2_L6" 687 | bottom: "F0_L6" 688 | bottom: "warped_D1_F1_L6" 689 | bottom: "scaled_flow_D1_L6" 690 | top: "F_D2_L6" 691 | type: "Concat" 692 | concat_param { axis: 1 } 693 | } 694 | layer { 695 | name: "conv1_D2_L6" 696 | type: "Convolution" 697 | bottom: "F_D2_L6" 698 | top: "conv1_D2_L6" 699 | param { lr_mult: 1 decay_mult: 1 } 700 | param { lr_mult: 1 decay_mult: 0 } 701 | convolution_param { 702 | num_output: 128 703 | pad: 1 704 | kernel_size: 3 705 | stride: 1 706 | engine: CUDNN 707 | } 708 | } 709 | layer { 710 | name: "ReLU1_D2_L6" 711 | type: "ReLU" 712 | bottom: "conv1_D2_L6" 713 | top: "conv1_D2_L6" 714 | relu_param { negative_slope: 0.1 } 715 | } 716 | layer { 717 | name: "conv2_D2_L6" 718 | type: "Convolution" 719 | bottom: "conv1_D2_L6" 720 | top: "conv2_D2_L6" 721 | param { lr_mult: 1 decay_mult: 1 } 722 | param { lr_mult: 1 decay_mult: 0 } 723 | convolution_param { 724 | num_output: 128 725 | pad: 1 726 | kernel_size: 3 727 | stride: 1 728 | engine: CUDNN 729 | } 730 | } 731 | layer { 732 | name: "ReLU2_D2_L6" 733 | type: "ReLU" 734 | bottom: "conv2_D2_L6" 735 | top: "conv2_D2_L6" 736 | relu_param { negative_slope: 0.1 } 737 | } 738 | layer { 739 | name: "conv3_D2_L6" 740 | type: "Convolution" 741 | bottom: "conv2_D2_L6" 742 | top: "conv3_D2_L6" 743 | param { lr_mult: 1 decay_mult: 1 } 744 | param { lr_mult: 1 decay_mult: 0 } 745 | convolution_param { 746 | num_output: 96 747 | pad: 1 748 | kernel_size: 3 749 | stride: 1 750 | engine: CUDNN 751 | } 752 | } 753 | layer { 754 | name: "ReLU3_D2_L6" 755 | type: "ReLU" 756 | bottom: "conv3_D2_L6" 757 | top: "conv3_D2_L6" 758 | relu_param { negative_slope: 0.1 } 759 | } 760 | layer { 761 | name: "conv4_D2_L6" 762 | type: "Convolution" 763 | bottom: "conv3_D2_L6" 764 | top: "conv4_D2_L6" 765 | param { lr_mult: 1 decay_mult: 1 } 766 | param { lr_mult: 1 decay_mult: 0 } 767 | convolution_param { 768 | num_output: 64 769 | pad: 1 770 | kernel_size: 3 771 | stride: 1 772 | engine: CUDNN 773 | } 774 | } 775 | layer { 776 | name: "ReLU4_D2_L6" 777 | type: "ReLU" 778 | bottom: "conv4_D2_L6" 779 | top: "conv4_D2_L6" 780 | relu_param { negative_slope: 0.1 } 781 | } 782 | layer { 783 | name: "conv5_D2_L6" 784 | type: "Convolution" 785 | bottom: "conv4_D2_L6" 786 | top: "conv5_D2_L6" 787 | param { lr_mult: 1 decay_mult: 1 } 788 | param { lr_mult: 1 decay_mult: 0 } 789 | convolution_param { 790 | num_output: 32 791 | pad: 1 792 | kernel_size: 3 793 | stride: 1 794 | engine: CUDNN 795 | } 796 | } 797 | layer { 798 | name: "ReLU5_D2_L6" 799 | type: "ReLU" 800 | bottom: "conv5_D2_L6" 801 | top: "conv5_D2_L6" 802 | relu_param { negative_slope: 0.1 } 803 | } 804 | layer { 805 | name: "scaled_flow_D2_res_L6" 806 | type: "Convolution" 807 | bottom: "conv5_D2_L6" 808 | top: "scaled_flow_D2_res_L6" 809 | param { lr_mult: 1 decay_mult: 1 } 810 | param { lr_mult: 1 decay_mult: 0 } 811 | convolution_param { 812 | num_output: 2 813 | pad: 1 814 | kernel_size: 3 815 | stride: 1 816 | engine: CUDNN 817 | } 818 | } 819 | layer { 820 | name: "scaled_flow_D2_L6" 821 | type: "Eltwise" 822 | bottom: "scaled_flow_D1_L6" 823 | bottom: "scaled_flow_D2_res_L6" 824 | top: "scaled_flow_D2_L6" 825 | eltwise_param { operation: SUM } 826 | } 827 | 828 | # Flow Regularization: L6 829 | layer { 830 | name: "slice_scaled_flow_D_L6" 831 | type: "Slice" 832 | bottom: "scaled_flow_D2_L6" 833 | top: "scaled_flow_D_L6_x" 834 | top: "scaled_flow_D_L6_y" 835 | slice_param { axis: 1 slice_point: 1 } 836 | } 837 | layer { 838 | name: "reshaped_scaled_flow_D_L6_x" 839 | type: "Im2col" 840 | bottom: "scaled_flow_D_L6_x" 841 | top: "reshaped_scaled_flow_D_L6_x" 842 | convolution_param { pad: 1 kernel_size: 3 stride: 1 } 843 | } 844 | layer { 845 | name: "reshaped_scaled_flow_D_L6_y" 846 | type: "Im2col" 847 | bottom: "scaled_flow_D_L6_y" 848 | top: "reshaped_scaled_flow_D_L6_y" 849 | convolution_param { pad: 1 kernel_size: 3 stride: 1 } 850 | } 851 | layer { 852 | name: "mean_scaled_flow_D_L6_x" 853 | type: "Reduction" 854 | bottom: "scaled_flow_D_L6_x" 855 | top: "mean_scaled_flow_D_L6_x" 856 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 857 | } 858 | layer { 859 | name: "scaled_flow_D_nomean_L6_x" 860 | type: "Bias" 861 | bottom: "scaled_flow_D_L6_x" 862 | bottom: "mean_scaled_flow_D_L6_x" 863 | top: "scaled_flow_D_nomean_L6_x" 864 | bias_param { axis: 0 } 865 | } 866 | layer { 867 | name: "mean_scaled_flow_D_L6_y" 868 | type: "Reduction" 869 | bottom: "scaled_flow_D_L6_y" 870 | top: "mean_scaled_flow_D_L6_y" 871 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 872 | } 873 | layer { 874 | name: "scaled_flow_D_nomean_L6_y" 875 | type: "Bias" 876 | bottom: "scaled_flow_D_L6_y" 877 | bottom: "mean_scaled_flow_D_L6_y" 878 | top: "scaled_flow_D_nomean_L6_y" 879 | bias_param { axis: 0 } 880 | } 881 | layer { 882 | name: "FlowUnscaling_L6_R" 883 | type: "Eltwise" 884 | bottom: "scaled_flow_D2_L6" 885 | top: "flow_D2_L6" 886 | eltwise_param { 887 | operation: SUM 888 | coeff: 0.625 889 | } 890 | } 891 | layer { 892 | name: "coords_R_L6" 893 | type: "Eltwise" 894 | bottom: "flow_D2_L6" 895 | bottom: "gxy_L6" 896 | top: "coords_R_L6" 897 | eltwise_param { coeff: 1 coeff: 1 } 898 | } 899 | layer { 900 | name: "warped_img1_aug_L6" 901 | type: "Warp" 902 | bottom: "img1_aug_L6" 903 | bottom: "coords_R_L6" 904 | top: "warped_img1_aug_L6" 905 | } 906 | layer { 907 | name: "img_diff_L6" 908 | type: "Eltwise" 909 | bottom: "img0_aug_L6" 910 | bottom: "warped_img1_aug_L6" 911 | top: "img_diff_L6" 912 | eltwise_param { 913 | operation: SUM 914 | coeff: 1.0 915 | coeff: -1.0 916 | } 917 | } 918 | layer { 919 | name: "channelNorm_L6" 920 | type: "ChannelNorm" 921 | bottom: "img_diff_L6" 922 | top: "channelNorm_L6" 923 | } 924 | layer { 925 | name: "concat_F0_R_L6" 926 | type: "Concat" 927 | bottom: "channelNorm_L6" 928 | bottom: "scaled_flow_D_nomean_L6_x" 929 | bottom: "scaled_flow_D_nomean_L6_y" 930 | bottom: "F0_L6" 931 | top: "concat_F0_R_L6" 932 | concat_param { axis: 1 } 933 | } 934 | layer { 935 | name: "conv1_R_L6" 936 | type: "Convolution" 937 | bottom: "concat_F0_R_L6" 938 | top: "conv1_R_L6" 939 | param { lr_mult: 1 decay_mult: 1 } 940 | param { lr_mult: 1 decay_mult: 0 } 941 | convolution_param { 942 | num_output: 128 943 | pad: 1 944 | kernel_size: 3 945 | stride: 1 946 | engine: CUDNN 947 | } 948 | } 949 | layer { 950 | name: "relu_conv1_R_L6" 951 | type: "ReLU" 952 | bottom: "conv1_R_L6" 953 | top: "conv1_R_L6" 954 | relu_param { negative_slope: 0.1 } 955 | } 956 | layer { 957 | name: "conv2_R_L6" 958 | type: "Convolution" 959 | bottom: "conv1_R_L6" 960 | top: "conv2_R_L6" 961 | param { lr_mult: 1 decay_mult: 1 } 962 | param { lr_mult: 1 decay_mult: 0 } 963 | convolution_param { 964 | num_output: 128 965 | pad: 1 966 | kernel_size: 3 967 | stride: 1 968 | engine: CUDNN 969 | } 970 | } 971 | layer { 972 | name: "relu_conv2_R_L6" 973 | type: "ReLU" 974 | bottom: "conv2_R_L6" 975 | top: "conv2_R_L6" 976 | relu_param { negative_slope: 0.1 } 977 | } 978 | layer { 979 | name: "conv3_R_L6" 980 | type: "Convolution" 981 | bottom: "conv2_R_L6" 982 | top: "conv3_R_L6" 983 | param { lr_mult: 1 decay_mult: 1 } 984 | param { lr_mult: 1 decay_mult: 0 } 985 | convolution_param { 986 | num_output: 64 987 | pad: 1 988 | kernel_size: 3 989 | stride: 1 990 | engine: CUDNN 991 | } 992 | } 993 | layer { 994 | name: "relu_conv3_R_L6" 995 | type: "ReLU" 996 | bottom: "conv3_R_L6" 997 | top: "conv3_R_L6" 998 | relu_param { negative_slope: 0.1 } 999 | } 1000 | layer { 1001 | name: "conv4_R_L6" 1002 | type: "Convolution" 1003 | bottom: "conv3_R_L6" 1004 | top: "conv4_R_L6" 1005 | param { lr_mult: 1 decay_mult: 1 } 1006 | param { lr_mult: 1 decay_mult: 0 } 1007 | convolution_param { 1008 | num_output: 64 1009 | pad: 1 1010 | kernel_size: 3 1011 | stride: 1 1012 | engine: CUDNN 1013 | } 1014 | } 1015 | layer { 1016 | name: "relu_conv4_R_L6" 1017 | type: "ReLU" 1018 | bottom: "conv4_R_L6" 1019 | top: "conv4_R_L6" 1020 | relu_param { negative_slope: 0.1 } 1021 | } 1022 | layer { 1023 | name: "conv5_R_L6" 1024 | type: "Convolution" 1025 | bottom: "conv4_R_L6" 1026 | top: "conv5_R_L6" 1027 | param { lr_mult: 1 decay_mult: 1 } 1028 | param { lr_mult: 1 decay_mult: 0 } 1029 | convolution_param { 1030 | num_output: 32 1031 | pad: 1 1032 | kernel_size: 3 1033 | stride: 1 1034 | engine: CUDNN 1035 | } 1036 | } 1037 | layer { 1038 | name: "relu_conv5_R_L6" 1039 | type: "ReLU" 1040 | bottom: "conv5_R_L6" 1041 | top: "conv5_R_L6" 1042 | relu_param { negative_slope: 0.1 } 1043 | } 1044 | layer { 1045 | name: "conv6_R_L6" 1046 | type: "Convolution" 1047 | bottom: "conv5_R_L6" 1048 | top: "conv6_R_L6" 1049 | param { lr_mult: 1 decay_mult: 1 } 1050 | param { lr_mult: 1 decay_mult: 0 } 1051 | convolution_param { 1052 | num_output: 32 1053 | pad: 1 1054 | kernel_size: 3 1055 | stride: 1 1056 | engine: CUDNN 1057 | } 1058 | } 1059 | layer { 1060 | name: "relu_conv6_R_L6" 1061 | type: "ReLU" 1062 | bottom: "conv6_R_L6" 1063 | top: "conv6_R_L6" 1064 | relu_param { negative_slope: 0.1 } 1065 | } 1066 | layer { 1067 | name: "dist_R_L6" 1068 | type: "Convolution" 1069 | bottom: "conv6_R_L6" 1070 | top: "dist_R_L6" 1071 | param { lr_mult: 1 decay_mult: 1 } 1072 | param { lr_mult: 0 } 1073 | convolution_param { 1074 | num_output: 9 1075 | pad: 1 1076 | kernel_size: 3 1077 | stride: 1 1078 | engine: CUDNN 1079 | } 1080 | } 1081 | layer { 1082 | name: "neg_sq_dist_R_L6" 1083 | type: "NegSquare" 1084 | bottom: "dist_R_L6" 1085 | top: "neg_sq_dist_R_L6" 1086 | } 1087 | layer { 1088 | name: "exp_kernel_R_L6" 1089 | type: "ExpMax" 1090 | bottom: "neg_sq_dist_R_L6" 1091 | top: "exp_kernel_R_L6" 1092 | } 1093 | layer { 1094 | name: "sum_exp_kernel_R_L6" 1095 | type: "Convolution" 1096 | bottom: "exp_kernel_R_L6" 1097 | top: "sum_exp_kernel_R_L6" 1098 | param { lr_mult: 0 } 1099 | param { lr_mult: 0 } 1100 | convolution_param { 1101 | num_output: 1 1102 | kernel_size: 1 1103 | weight_filler { type: "constant" value: 1 } 1104 | } 1105 | } 1106 | layer { 1107 | name: "divisor_R_L6" 1108 | bottom: "sum_exp_kernel_R_L6" 1109 | top: "divisor_R_L6" 1110 | type: "Power" 1111 | power_param { power: -1 scale: 1 shift: 0 } 1112 | } 1113 | layer { 1114 | name: "f-lconv_L6_x" 1115 | type: "Eltwise" 1116 | bottom: "reshaped_scaled_flow_D_L6_x" 1117 | bottom: "exp_kernel_R_L6" 1118 | top: "f-lconv_L6_x" 1119 | eltwise_param { operation: PROD } 1120 | } 1121 | layer { 1122 | name: "scaled_flow0_R_L6_x" 1123 | type: "Convolution" 1124 | bottom: "f-lconv_L6_x" 1125 | top: "scaled_flow0_R_L6_x" 1126 | param { lr_mult: 0 } 1127 | param { lr_mult: 0 } 1128 | convolution_param { 1129 | num_output: 1 1130 | kernel_size: 1 1131 | weight_filler { type: "constant" value: 1 } 1132 | } 1133 | } 1134 | layer { 1135 | name: "scaled_flow1_R_L6_x" 1136 | type: "Eltwise" 1137 | bottom: "scaled_flow0_R_L6_x" 1138 | bottom: "divisor_R_L6" 1139 | top: "scaled_flow1_R_L6_x" 1140 | eltwise_param { 1141 | operation: PROD 1142 | } 1143 | } 1144 | layer { 1145 | name: "f-lconv_L6_y" 1146 | type: "Eltwise" 1147 | bottom: "reshaped_scaled_flow_D_L6_y" 1148 | bottom: "exp_kernel_R_L6" 1149 | top: "f-lconv_L6_y" 1150 | eltwise_param { operation: PROD } 1151 | } 1152 | layer { 1153 | name: "scaled_flow0_R_L6_y" 1154 | type: "Convolution" 1155 | bottom: "f-lconv_L6_y" 1156 | top: "scaled_flow0_R_L6_y" 1157 | param { lr_mult: 0 } 1158 | param { lr_mult: 0 } 1159 | convolution_param { 1160 | num_output: 1 1161 | kernel_size: 1 1162 | weight_filler { type: "constant" value: 1 } 1163 | } 1164 | } 1165 | layer { 1166 | name: "scaled_flow1_R_L6_y" 1167 | type: "Eltwise" 1168 | bottom: "scaled_flow0_R_L6_y" 1169 | bottom: "divisor_R_L6" 1170 | top: "scaled_flow1_R_L6_y" 1171 | eltwise_param { 1172 | operation: PROD 1173 | } 1174 | } 1175 | layer { 1176 | name: "scaled_flow_R_L6" 1177 | bottom: "scaled_flow1_R_L6_x" 1178 | bottom: "scaled_flow1_R_L6_y" 1179 | top: "scaled_flow_R_L6" 1180 | type: "Concat" 1181 | concat_param { axis: 1 } 1182 | } 1183 | 1184 | # Feature Matching: L5 1185 | layer { 1186 | name: "scaled_flow_R_L6to5" 1187 | type: "Deconvolution" 1188 | bottom: "scaled_flow_R_L6" 1189 | top: "scaled_flow_R_L6to5" 1190 | param { lr_mult: 0 decay_mult: 0 } 1191 | convolution_param { 1192 | num_output: 2 1193 | group: 2 1194 | pad: 1 1195 | kernel_size: 4 1196 | stride: 2 1197 | weight_filler { type: "bilinear" } 1198 | bias_term: false 1199 | engine: CUDNN 1200 | } 1201 | } 1202 | layer { 1203 | name: "FlowUnscaling_L5_D1" 1204 | type: "Eltwise" 1205 | bottom: "scaled_flow_R_L6to5" 1206 | top: "flow_R_L6to5" 1207 | eltwise_param { 1208 | operation: SUM 1209 | coeff: 1.25 1210 | } 1211 | } 1212 | layer { 1213 | name: "gxy_L5" 1214 | type: "Grid" 1215 | top: "gxy_L5" 1216 | bottom: "flow_R_L6to5" 1217 | propagate_down: false 1218 | } 1219 | layer { 1220 | name: "coords_L5" 1221 | type: "Eltwise" 1222 | bottom: "flow_R_L6to5" 1223 | bottom: "gxy_L5" 1224 | top: "coords_L5" 1225 | eltwise_param { coeff: 1 coeff: 1 } 1226 | } 1227 | layer { 1228 | name: "warped_F1_L5" 1229 | type: "Warp" 1230 | bottom: "F1_L5" 1231 | bottom: "coords_L5" 1232 | top: "warped_F1_L5" 1233 | } 1234 | layer { 1235 | name: "corr_L5" 1236 | type: "Correlation" 1237 | bottom: "F0_L5" 1238 | bottom: "warped_F1_L5" 1239 | top: "corr_L5" 1240 | correlation_param { 1241 | pad: 4 1242 | kernel_size: 1 1243 | max_displacement: 4 1244 | stride_1: 1 1245 | stride_2: 1 1246 | } 1247 | } 1248 | layer { 1249 | name: "ReLU_corr_L5" 1250 | type: "ReLU" 1251 | bottom: "corr_L5" 1252 | top: "corr_L5" 1253 | relu_param { 1254 | negative_slope: 0.1 1255 | } 1256 | } 1257 | layer { 1258 | name: "conv1_D1_L5" 1259 | type: "Convolution" 1260 | bottom: "corr_L5" 1261 | top: "conv1_D1_L5" 1262 | param { lr_mult: 1 decay_mult: 1 } 1263 | param { lr_mult: 1 decay_mult: 0 } 1264 | convolution_param { 1265 | num_output: 128 1266 | pad: 1 1267 | kernel_size: 3 1268 | stride: 1 1269 | engine: CUDNN 1270 | } 1271 | } 1272 | layer { 1273 | name: "ReLU1_D1_L5" 1274 | type: "ReLU" 1275 | bottom: "conv1_D1_L5" 1276 | top: "conv1_D1_L5" 1277 | relu_param { negative_slope: 0.1 } 1278 | } 1279 | layer { 1280 | name: "conv2_D1_L5" 1281 | type: "Convolution" 1282 | bottom: "conv1_D1_L5" 1283 | top: "conv2_D1_L5" 1284 | param { lr_mult: 1 decay_mult: 1 } 1285 | param { lr_mult: 1 decay_mult: 0 } 1286 | convolution_param { 1287 | num_output: 128 1288 | pad: 1 1289 | kernel_size: 3 1290 | stride: 1 1291 | engine: CUDNN 1292 | } 1293 | } 1294 | layer { 1295 | name: "ReLU2_D1_L5" 1296 | type: "ReLU" 1297 | bottom: "conv2_D1_L5" 1298 | top: "conv2_D1_L5" 1299 | relu_param { negative_slope: 0.1 } 1300 | } 1301 | layer { 1302 | name: "conv3_D1_L5" 1303 | type: "Convolution" 1304 | bottom: "conv2_D1_L5" 1305 | top: "conv3_D1_L5" 1306 | param { lr_mult: 1 decay_mult: 1 } 1307 | param { lr_mult: 1 decay_mult: 0 } 1308 | convolution_param { 1309 | num_output: 96 1310 | pad: 1 1311 | kernel_size: 3 1312 | stride: 1 1313 | engine: CUDNN 1314 | } 1315 | } 1316 | layer { 1317 | name: "ReLU3_D1_L5" 1318 | type: "ReLU" 1319 | bottom: "conv3_D1_L5" 1320 | top: "conv3_D1_L5" 1321 | relu_param { negative_slope: 0.1 } 1322 | } 1323 | layer { 1324 | name: "conv4_D1_L5" 1325 | type: "Convolution" 1326 | bottom: "conv3_D1_L5" 1327 | top: "conv4_D1_L5" 1328 | param { lr_mult: 1 decay_mult: 1 } 1329 | param { lr_mult: 1 decay_mult: 0 } 1330 | convolution_param { 1331 | num_output: 64 1332 | pad: 1 1333 | kernel_size: 3 1334 | stride: 1 1335 | engine: CUDNN 1336 | } 1337 | } 1338 | layer { 1339 | name: "ReLU4_D1_L5" 1340 | type: "ReLU" 1341 | bottom: "conv4_D1_L5" 1342 | top: "conv4_D1_L5" 1343 | relu_param { negative_slope: 0.1 } 1344 | } 1345 | layer { 1346 | name: "conv5_D1_L5" 1347 | type: "Convolution" 1348 | bottom: "conv4_D1_L5" 1349 | top: "conv5_D1_L5" 1350 | param { lr_mult: 1 decay_mult: 1 } 1351 | param { lr_mult: 1 decay_mult: 0 } 1352 | convolution_param { 1353 | num_output: 32 1354 | pad: 1 1355 | kernel_size: 3 1356 | stride: 1 1357 | engine: CUDNN 1358 | } 1359 | } 1360 | layer { 1361 | name: "ReLU5_D1_L5" 1362 | type: "ReLU" 1363 | bottom: "conv5_D1_L5" 1364 | top: "conv5_D1_L5" 1365 | relu_param { negative_slope: 0.1 } 1366 | } 1367 | layer { 1368 | name: "scaled_flow_D1_res_L5" 1369 | type: "Convolution" 1370 | bottom: "conv5_D1_L5" 1371 | top: "scaled_flow_D1_res_L5" 1372 | param { lr_mult: 1 decay_mult: 1 } 1373 | param { lr_mult: 1 decay_mult: 0 } 1374 | convolution_param { 1375 | num_output: 2 1376 | pad: 1 1377 | kernel_size: 3 1378 | stride: 1 1379 | engine: CUDNN 1380 | } 1381 | } 1382 | layer { 1383 | name: "scaled_flow_D1_L5" 1384 | type: "Eltwise" 1385 | bottom: "scaled_flow_R_L6to5" 1386 | bottom: "scaled_flow_D1_res_L5" 1387 | top: "scaled_flow_D1_L5" 1388 | eltwise_param { operation: SUM } 1389 | } 1390 | 1391 | # Sub-Pixel Refinement: L5 1392 | layer { 1393 | name: "FlowUnscaling_L5_D2" 1394 | type: "Eltwise" 1395 | bottom: "scaled_flow_D1_L5" 1396 | top: "flow_D1_L5" 1397 | eltwise_param { 1398 | operation: SUM 1399 | coeff: 1.25 1400 | } 1401 | } 1402 | layer { 1403 | name: "coords_D1_L5" 1404 | type: "Eltwise" 1405 | bottom: "flow_D1_L5" 1406 | bottom: "gxy_L5" 1407 | top: "coords_D1_L5" 1408 | eltwise_param { coeff: 1 coeff: 1 } 1409 | } 1410 | layer { 1411 | name: "warped_D1_F1_L5" 1412 | type: "Warp" 1413 | bottom: "F1_L5" 1414 | bottom: "coords_D1_L5" 1415 | top: "warped_D1_F1_L5" 1416 | } 1417 | layer { 1418 | name: "F_D2_L5" 1419 | bottom: "F0_L5" 1420 | bottom: "warped_D1_F1_L5" 1421 | bottom: "scaled_flow_D1_L5" 1422 | top: "F_D2_L5" 1423 | type: "Concat" 1424 | concat_param { axis: 1 } 1425 | } 1426 | layer { 1427 | name: "conv1_D2_L5" 1428 | type: "Convolution" 1429 | bottom: "F_D2_L5" 1430 | top: "conv1_D2_L5" 1431 | param { lr_mult: 1 decay_mult: 1 } 1432 | param { lr_mult: 1 decay_mult: 0 } 1433 | convolution_param { 1434 | num_output: 128 1435 | pad: 1 1436 | kernel_size: 3 1437 | stride: 1 1438 | engine: CUDNN 1439 | } 1440 | } 1441 | layer { 1442 | name: "ReLU1_D2_L5" 1443 | type: "ReLU" 1444 | bottom: "conv1_D2_L5" 1445 | top: "conv1_D2_L5" 1446 | relu_param { negative_slope: 0.1 } 1447 | } 1448 | layer { 1449 | name: "conv2_D2_L5" 1450 | type: "Convolution" 1451 | bottom: "conv1_D2_L5" 1452 | top: "conv2_D2_L5" 1453 | param { lr_mult: 1 decay_mult: 1 } 1454 | param { lr_mult: 1 decay_mult: 0 } 1455 | convolution_param { 1456 | num_output: 128 1457 | pad: 1 1458 | kernel_size: 3 1459 | stride: 1 1460 | engine: CUDNN 1461 | } 1462 | } 1463 | layer { 1464 | name: "ReLU2_D2_L5" 1465 | type: "ReLU" 1466 | bottom: "conv2_D2_L5" 1467 | top: "conv2_D2_L5" 1468 | relu_param { negative_slope: 0.1 } 1469 | } 1470 | layer { 1471 | name: "conv3_D2_L5" 1472 | type: "Convolution" 1473 | bottom: "conv2_D2_L5" 1474 | top: "conv3_D2_L5" 1475 | param { lr_mult: 1 decay_mult: 1 } 1476 | param { lr_mult: 1 decay_mult: 0 } 1477 | convolution_param { 1478 | num_output: 96 1479 | pad: 1 1480 | kernel_size: 3 1481 | stride: 1 1482 | engine: CUDNN 1483 | } 1484 | } 1485 | layer { 1486 | name: "ReLU3_D2_L5" 1487 | type: "ReLU" 1488 | bottom: "conv3_D2_L5" 1489 | top: "conv3_D2_L5" 1490 | relu_param { negative_slope: 0.1 } 1491 | } 1492 | layer { 1493 | name: "conv4_D2_L5" 1494 | type: "Convolution" 1495 | bottom: "conv3_D2_L5" 1496 | top: "conv4_D2_L5" 1497 | param { lr_mult: 1 decay_mult: 1 } 1498 | param { lr_mult: 1 decay_mult: 0 } 1499 | convolution_param { 1500 | num_output: 64 1501 | pad: 1 1502 | kernel_size: 3 1503 | stride: 1 1504 | engine: CUDNN 1505 | } 1506 | } 1507 | layer { 1508 | name: "ReLU4_D2_L5" 1509 | type: "ReLU" 1510 | bottom: "conv4_D2_L5" 1511 | top: "conv4_D2_L5" 1512 | relu_param { negative_slope: 0.1 } 1513 | } 1514 | layer { 1515 | name: "conv5_D2_L5" 1516 | type: "Convolution" 1517 | bottom: "conv4_D2_L5" 1518 | top: "conv5_D2_L5" 1519 | param { lr_mult: 1 decay_mult: 1 } 1520 | param { lr_mult: 1 decay_mult: 0 } 1521 | convolution_param { 1522 | num_output: 32 1523 | pad: 1 1524 | kernel_size: 3 1525 | stride: 1 1526 | engine: CUDNN 1527 | } 1528 | } 1529 | layer { 1530 | name: "ReLU5_D2_L5" 1531 | type: "ReLU" 1532 | bottom: "conv5_D2_L5" 1533 | top: "conv5_D2_L5" 1534 | relu_param { negative_slope: 0.1 } 1535 | } 1536 | layer { 1537 | name: "scaled_flow_D2_res_L5" 1538 | type: "Convolution" 1539 | bottom: "conv5_D2_L5" 1540 | top: "scaled_flow_D2_res_L5" 1541 | param { lr_mult: 1 decay_mult: 1 } 1542 | param { lr_mult: 1 decay_mult: 0 } 1543 | convolution_param { 1544 | num_output: 2 1545 | pad: 1 1546 | kernel_size: 3 1547 | stride: 1 1548 | engine: CUDNN 1549 | } 1550 | } 1551 | layer { 1552 | name: "scaled_flow_D2_L5" 1553 | type: "Eltwise" 1554 | bottom: "scaled_flow_D1_L5" 1555 | bottom: "scaled_flow_D2_res_L5" 1556 | top: "scaled_flow_D2_L5" 1557 | eltwise_param { operation: SUM } 1558 | } 1559 | 1560 | # Flow Regularization: L5 1561 | layer { 1562 | name: "slice_scaled_flow_D_L5" 1563 | type: "Slice" 1564 | bottom: "scaled_flow_D2_L5" 1565 | top: "scaled_flow_D_L5_x" 1566 | top: "scaled_flow_D_L5_y" 1567 | slice_param { axis: 1 slice_point: 1 } 1568 | } 1569 | layer { 1570 | name: "reshaped_scaled_flow_D_L5_x" 1571 | type: "Im2col" 1572 | bottom: "scaled_flow_D_L5_x" 1573 | top: "reshaped_scaled_flow_D_L5_x" 1574 | convolution_param { pad: 1 kernel_size: 3 stride: 1 } 1575 | } 1576 | layer { 1577 | name: "reshaped_scaled_flow_D_L5_y" 1578 | type: "Im2col" 1579 | bottom: "scaled_flow_D_L5_y" 1580 | top: "reshaped_scaled_flow_D_L5_y" 1581 | convolution_param { pad: 1 kernel_size: 3 stride: 1 } 1582 | } 1583 | layer { 1584 | name: "mean_scaled_flow_D_L5_x" 1585 | type: "Reduction" 1586 | bottom: "scaled_flow_D_L5_x" 1587 | top: "mean_scaled_flow_D_L5_x" 1588 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 1589 | } 1590 | layer { 1591 | name: "scaled_flow_D_nomean_L5_x" 1592 | type: "Bias" 1593 | bottom: "scaled_flow_D_L5_x" 1594 | bottom: "mean_scaled_flow_D_L5_x" 1595 | top: "scaled_flow_D_nomean_L5_x" 1596 | bias_param { axis: 0 } 1597 | } 1598 | layer { 1599 | name: "mean_scaled_flow_D_L5_y" 1600 | type: "Reduction" 1601 | bottom: "scaled_flow_D_L5_y" 1602 | top: "mean_scaled_flow_D_L5_y" 1603 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 1604 | } 1605 | layer { 1606 | name: "scaled_flow_D_nomean_L5_y" 1607 | type: "Bias" 1608 | bottom: "scaled_flow_D_L5_y" 1609 | bottom: "mean_scaled_flow_D_L5_y" 1610 | top: "scaled_flow_D_nomean_L5_y" 1611 | bias_param { axis: 0 } 1612 | } 1613 | layer { 1614 | name: "FlowUnscaling_L5_R" 1615 | type: "Eltwise" 1616 | bottom: "scaled_flow_D2_L5" 1617 | top: "flow_D2_L5" 1618 | eltwise_param { 1619 | operation: SUM 1620 | coeff: 1.25 1621 | } 1622 | } 1623 | layer { 1624 | name: "coords_R_L5" 1625 | type: "Eltwise" 1626 | bottom: "flow_D2_L5" 1627 | bottom: "gxy_L5" 1628 | top: "coords_R_L5" 1629 | eltwise_param { coeff: 1 coeff: 1 } 1630 | } 1631 | layer { 1632 | name: "warped_img1_aug_L5" 1633 | type: "Warp" 1634 | bottom: "img1_aug_L5" 1635 | bottom: "coords_R_L5" 1636 | top: "warped_img1_aug_L5" 1637 | } 1638 | layer { 1639 | name: "img_diff_L5" 1640 | type: "Eltwise" 1641 | bottom: "img0_aug_L5" 1642 | bottom: "warped_img1_aug_L5" 1643 | top: "img_diff_L5" 1644 | eltwise_param { 1645 | operation: SUM 1646 | coeff: 1.0 1647 | coeff: -1.0 1648 | } 1649 | } 1650 | layer { 1651 | name: "channelNorm_L5" 1652 | type: "ChannelNorm" 1653 | bottom: "img_diff_L5" 1654 | top: "channelNorm_L5" 1655 | } 1656 | layer { 1657 | name: "concat_F0_R_L5" 1658 | type: "Concat" 1659 | bottom: "channelNorm_L5" 1660 | bottom: "scaled_flow_D_nomean_L5_x" 1661 | bottom: "scaled_flow_D_nomean_L5_y" 1662 | bottom: "F0_L5" 1663 | top: "concat_F0_R_L5" 1664 | concat_param { axis: 1 } 1665 | } 1666 | layer { 1667 | name: "conv1_R_L5" 1668 | type: "Convolution" 1669 | bottom: "concat_F0_R_L5" 1670 | top: "conv1_R_L5" 1671 | param { lr_mult: 1 decay_mult: 1 } 1672 | param { lr_mult: 1 decay_mult: 0 } 1673 | convolution_param { 1674 | num_output: 128 1675 | pad: 1 1676 | kernel_size: 3 1677 | stride: 1 1678 | engine: CUDNN 1679 | } 1680 | } 1681 | layer { 1682 | name: "relu_conv1_R_L5" 1683 | type: "ReLU" 1684 | bottom: "conv1_R_L5" 1685 | top: "conv1_R_L5" 1686 | relu_param { negative_slope: 0.1 } 1687 | } 1688 | layer { 1689 | name: "conv2_R_L5" 1690 | type: "Convolution" 1691 | bottom: "conv1_R_L5" 1692 | top: "conv2_R_L5" 1693 | param { lr_mult: 1 decay_mult: 1 } 1694 | param { lr_mult: 1 decay_mult: 0 } 1695 | convolution_param { 1696 | num_output: 128 1697 | pad: 1 1698 | kernel_size: 3 1699 | stride: 1 1700 | engine: CUDNN 1701 | } 1702 | } 1703 | layer { 1704 | name: "relu_conv2_R_L5" 1705 | type: "ReLU" 1706 | bottom: "conv2_R_L5" 1707 | top: "conv2_R_L5" 1708 | relu_param { negative_slope: 0.1 } 1709 | } 1710 | layer { 1711 | name: "conv3_R_L5" 1712 | type: "Convolution" 1713 | bottom: "conv2_R_L5" 1714 | top: "conv3_R_L5" 1715 | param { lr_mult: 1 decay_mult: 1 } 1716 | param { lr_mult: 1 decay_mult: 0 } 1717 | convolution_param { 1718 | num_output: 64 1719 | pad: 1 1720 | kernel_size: 3 1721 | stride: 1 1722 | engine: CUDNN 1723 | } 1724 | } 1725 | layer { 1726 | name: "relu_conv3_R_L5" 1727 | type: "ReLU" 1728 | bottom: "conv3_R_L5" 1729 | top: "conv3_R_L5" 1730 | relu_param { negative_slope: 0.1 } 1731 | } 1732 | layer { 1733 | name: "conv4_R_L5" 1734 | type: "Convolution" 1735 | bottom: "conv3_R_L5" 1736 | top: "conv4_R_L5" 1737 | param { lr_mult: 1 decay_mult: 1 } 1738 | param { lr_mult: 1 decay_mult: 0 } 1739 | convolution_param { 1740 | num_output: 64 1741 | pad: 1 1742 | kernel_size: 3 1743 | stride: 1 1744 | engine: CUDNN 1745 | } 1746 | } 1747 | layer { 1748 | name: "relu_conv4_R_L5" 1749 | type: "ReLU" 1750 | bottom: "conv4_R_L5" 1751 | top: "conv4_R_L5" 1752 | relu_param { negative_slope: 0.1 } 1753 | } 1754 | layer { 1755 | name: "conv5_R_L5" 1756 | type: "Convolution" 1757 | bottom: "conv4_R_L5" 1758 | top: "conv5_R_L5" 1759 | param { lr_mult: 1 decay_mult: 1 } 1760 | param { lr_mult: 1 decay_mult: 0 } 1761 | convolution_param { 1762 | num_output: 32 1763 | pad: 1 1764 | kernel_size: 3 1765 | stride: 1 1766 | engine: CUDNN 1767 | } 1768 | } 1769 | layer { 1770 | name: "relu_conv5_R_L5" 1771 | type: "ReLU" 1772 | bottom: "conv5_R_L5" 1773 | top: "conv5_R_L5" 1774 | relu_param { negative_slope: 0.1 } 1775 | } 1776 | layer { 1777 | name: "conv6_R_L5" 1778 | type: "Convolution" 1779 | bottom: "conv5_R_L5" 1780 | top: "conv6_R_L5" 1781 | param { lr_mult: 1 decay_mult: 1 } 1782 | param { lr_mult: 1 decay_mult: 0 } 1783 | convolution_param { 1784 | num_output: 32 1785 | pad: 1 1786 | kernel_size: 3 1787 | stride: 1 1788 | engine: CUDNN 1789 | } 1790 | } 1791 | layer { 1792 | name: "relu_conv6_R_L5" 1793 | type: "ReLU" 1794 | bottom: "conv6_R_L5" 1795 | top: "conv6_R_L5" 1796 | relu_param { negative_slope: 0.1 } 1797 | } 1798 | layer { 1799 | name: "dist_R_L5" 1800 | type: "Convolution" 1801 | bottom: "conv6_R_L5" 1802 | top: "dist_R_L5" 1803 | param { lr_mult: 1 decay_mult: 1 } 1804 | param { lr_mult: 0 } 1805 | convolution_param { 1806 | num_output: 9 1807 | pad: 1 1808 | kernel_size: 3 1809 | stride: 1 1810 | engine: CUDNN 1811 | } 1812 | } 1813 | layer { 1814 | name: "neg_sq_dist_R_L5" 1815 | type: "NegSquare" 1816 | bottom: "dist_R_L5" 1817 | top: "neg_sq_dist_R_L5" 1818 | } 1819 | layer { 1820 | name: "exp_kernel_R_L5" 1821 | type: "ExpMax" 1822 | bottom: "neg_sq_dist_R_L5" 1823 | top: "exp_kernel_R_L5" 1824 | } 1825 | layer { 1826 | name: "sum_exp_kernel_R_L5" 1827 | type: "Convolution" 1828 | bottom: "exp_kernel_R_L5" 1829 | top: "sum_exp_kernel_R_L5" 1830 | param { lr_mult: 0 } 1831 | param { lr_mult: 0 } 1832 | convolution_param { 1833 | num_output: 1 1834 | kernel_size: 1 1835 | weight_filler { type: "constant" value: 1 } 1836 | } 1837 | } 1838 | layer { 1839 | name: "divisor_R_L5" 1840 | bottom: "sum_exp_kernel_R_L5" 1841 | top: "divisor_R_L5" 1842 | type: "Power" 1843 | power_param { power: -1 scale: 1 shift: 0 } 1844 | } 1845 | layer { 1846 | name: "f-lconv_L5_x" 1847 | type: "Eltwise" 1848 | bottom: "reshaped_scaled_flow_D_L5_x" 1849 | bottom: "exp_kernel_R_L5" 1850 | top: "f-lconv_L5_x" 1851 | eltwise_param { operation: PROD } 1852 | } 1853 | layer { 1854 | name: "scaled_flow0_R_L5_x" 1855 | type: "Convolution" 1856 | bottom: "f-lconv_L5_x" 1857 | top: "scaled_flow0_R_L5_x" 1858 | param { lr_mult: 0 } 1859 | param { lr_mult: 0 } 1860 | convolution_param { 1861 | num_output: 1 1862 | kernel_size: 1 1863 | weight_filler { type: "constant" value: 1 } 1864 | } 1865 | } 1866 | layer { 1867 | name: "scaled_flow1_R_L5_x" 1868 | type: "Eltwise" 1869 | bottom: "scaled_flow0_R_L5_x" 1870 | bottom: "divisor_R_L5" 1871 | top: "scaled_flow1_R_L5_x" 1872 | eltwise_param { 1873 | operation: PROD 1874 | } 1875 | } 1876 | layer { 1877 | name: "f-lconv_L5_y" 1878 | type: "Eltwise" 1879 | bottom: "reshaped_scaled_flow_D_L5_y" 1880 | bottom: "exp_kernel_R_L5" 1881 | top: "f-lconv_L5_y" 1882 | eltwise_param { operation: PROD } 1883 | } 1884 | layer { 1885 | name: "scaled_flow0_R_L5_y" 1886 | type: "Convolution" 1887 | bottom: "f-lconv_L5_y" 1888 | top: "scaled_flow0_R_L5_y" 1889 | param { lr_mult: 0 } 1890 | param { lr_mult: 0 } 1891 | convolution_param { 1892 | num_output: 1 1893 | kernel_size: 1 1894 | weight_filler { type: "constant" value: 1 } 1895 | } 1896 | } 1897 | layer { 1898 | name: "scaled_flow1_R_L5_y" 1899 | type: "Eltwise" 1900 | bottom: "scaled_flow0_R_L5_y" 1901 | bottom: "divisor_R_L5" 1902 | top: "scaled_flow1_R_L5_y" 1903 | eltwise_param { 1904 | operation: PROD 1905 | } 1906 | } 1907 | layer { 1908 | name: "scaled_flow_R_L5" 1909 | bottom: "scaled_flow1_R_L5_x" 1910 | bottom: "scaled_flow1_R_L5_y" 1911 | top: "scaled_flow_R_L5" 1912 | type: "Concat" 1913 | concat_param { axis: 1 } 1914 | } 1915 | 1916 | # Confidence Prediction: L5 1917 | layer { 1918 | name: "conf0_R_L5" 1919 | type: "Convolution" 1920 | bottom: "conv6_R_L5" 1921 | top: "conf0_R_L5" 1922 | param { lr_mult: 1 decay_mult: 1 } 1923 | param { lr_mult: 1 decay_mult: 0 } 1924 | convolution_param { 1925 | num_output: 1 1926 | pad: 1 1927 | kernel_size: 3 1928 | stride: 1 1929 | engine: CUDNN 1930 | } 1931 | } 1932 | layer { 1933 | name: "conf_R_L5" 1934 | type: "Sigmoid" 1935 | bottom: "conf0_R_L5" 1936 | top: "conf_R_L5" 1937 | } 1938 | 1939 | # Flow Field Deformation: L4 1940 | layer { 1941 | name: "conf_R_L5to4" 1942 | type: "Deconvolution" 1943 | bottom: "conf_R_L5" 1944 | top: "conf_R_L5to4" 1945 | param { lr_mult: 0 decay_mult: 0 } 1946 | convolution_param { 1947 | num_output: 1 1948 | pad: 1 1949 | kernel_size: 4 1950 | stride: 2 1951 | weight_filler { type: "bilinear" } 1952 | bias_term: false 1953 | engine: CUDNN 1954 | } 1955 | } 1956 | layer { 1957 | name: "scaled_flow_R_L5to4" 1958 | type: "Deconvolution" 1959 | bottom: "scaled_flow_R_L5" 1960 | top: "scaled_flow_R_L5to4" 1961 | param { lr_mult: 0 decay_mult: 0 } 1962 | convolution_param { 1963 | num_output: 2 1964 | group: 2 1965 | pad: 1 1966 | kernel_size: 4 1967 | stride: 2 1968 | weight_filler { type: "bilinear" } 1969 | bias_term: false 1970 | engine: CUDNN 1971 | } 1972 | } 1973 | layer { 1974 | name: "acorr_F0_L4" 1975 | type: "Correlation" 1976 | bottom: "F0_L4" 1977 | bottom: "F0_L4" 1978 | top: "acorr_F0_L4" 1979 | correlation_param { 1980 | pad: 6 1981 | kernel_size: 1 1982 | max_displacement: 6 1983 | stride_1: 1 1984 | stride_2: 2 1985 | } 1986 | } 1987 | layer { 1988 | name: "ReLU1_acorr_F0_L4" 1989 | type: "ReLU" 1990 | bottom: "acorr_F0_L4" 1991 | top: "acorr_F0_L4" 1992 | relu_param { negative_slope: 0.1 } 1993 | } 1994 | layer { 1995 | name: "F_FD_L4" 1996 | type: "Concat" 1997 | bottom: "acorr_F0_L4" 1998 | bottom: "conf_R_L5to4" 1999 | top: "F_FD_L4" 2000 | concat_param { axis: 1 } 2001 | } 2002 | layer { 2003 | name: "conv1_FD_L4" 2004 | type: "Convolution" 2005 | bottom: "F_FD_L4" 2006 | top: "conv1_FD_L4" 2007 | param { lr_mult: 1 decay_mult: 1 } 2008 | param { lr_mult: 1 decay_mult: 0 } 2009 | convolution_param { 2010 | num_output: 128 2011 | pad: 1 2012 | kernel_size: 3 2013 | stride: 1 2014 | engine: CUDNN 2015 | } 2016 | } 2017 | layer { 2018 | name: "ReLU1_FD_L4" 2019 | type: "ReLU" 2020 | bottom: "conv1_FD_L4" 2021 | top: "conv1_FD_L4" 2022 | relu_param { negative_slope: 0.1 } 2023 | } 2024 | layer { 2025 | name: "conv2_FD_L4" 2026 | type: "Convolution" 2027 | bottom: "conv1_FD_L4" 2028 | top: "conv2_FD_L4" 2029 | param { lr_mult: 1 decay_mult: 1 } 2030 | param { lr_mult: 1 decay_mult: 0 } 2031 | convolution_param { 2032 | num_output: 64 2033 | pad: 1 2034 | kernel_size: 3 2035 | stride: 1 2036 | engine: CUDNN 2037 | } 2038 | } 2039 | layer { 2040 | name: "ReLU2_FD_L4" 2041 | type: "ReLU" 2042 | bottom: "conv2_FD_L4" 2043 | top: "conv2_FD_L4" 2044 | relu_param { negative_slope: 0.1 } 2045 | } 2046 | layer { 2047 | name: "conv3_FD_L4" 2048 | type: "Convolution" 2049 | bottom: "conv2_FD_L4" 2050 | top: "conv3_FD_L4" 2051 | param { lr_mult: 1 decay_mult: 1 } 2052 | param { lr_mult: 1 decay_mult: 0 } 2053 | convolution_param { 2054 | num_output: 32 2055 | pad: 1 2056 | kernel_size: 3 2057 | stride: 1 2058 | engine: CUDNN 2059 | } 2060 | } 2061 | layer { 2062 | name: "ReLU3_FD_L4" 2063 | type: "ReLU" 2064 | bottom: "conv3_FD_L4" 2065 | top: "conv3_FD_L4" 2066 | relu_param { negative_slope: 0.1 } 2067 | } 2068 | layer { 2069 | name: "disp_FD_L4" 2070 | type: "Convolution" 2071 | bottom: "conv3_FD_L4" 2072 | top: "disp_FD_L4" 2073 | param { lr_mult: 1 decay_mult: 1 } 2074 | param { lr_mult: 1 decay_mult: 0 } 2075 | convolution_param { 2076 | num_output: 2 2077 | pad: 2 2078 | kernel_size: 5 2079 | stride: 1 2080 | engine: CUDNN 2081 | } 2082 | } 2083 | layer { 2084 | name: "gxy_L4" 2085 | type: "Grid" 2086 | top: "gxy_L4" 2087 | bottom: "disp_FD_L4" 2088 | propagate_down: false 2089 | } 2090 | layer { 2091 | name: "coords_disp_FD_L4" 2092 | type: "Eltwise" 2093 | bottom: "disp_FD_L4" 2094 | bottom: "gxy_L4" 2095 | top: "coords_disp_FD_L4" 2096 | eltwise_param { coeff: 1 coeff: 1 } 2097 | } 2098 | layer { 2099 | name: "scaled_flow_FD_L4" 2100 | type: "Warp" 2101 | bottom: "scaled_flow_R_L5to4" 2102 | bottom: "coords_disp_FD_L4" 2103 | top: "scaled_flow_FD_L4" 2104 | } 2105 | 2106 | # Confidence Prediction: L4 2107 | layer { 2108 | name: "conf0_FD_L4" 2109 | type: "Convolution" 2110 | bottom: "conv3_FD_L4" 2111 | top: "conf0_FD_L4" 2112 | param { lr_mult: 1 decay_mult: 1 } 2113 | param { lr_mult: 1 decay_mult: 0 } 2114 | convolution_param { 2115 | num_output: 1 2116 | pad: 2 2117 | kernel_size: 5 2118 | stride: 1 2119 | engine: CUDNN 2120 | } 2121 | } 2122 | layer { 2123 | name: "conf_FD_L4" 2124 | type: "Sigmoid" 2125 | bottom: "conf0_FD_L4" 2126 | top: "conf_FD_L4" 2127 | } 2128 | 2129 | # Cost Volume Modulation: L4 2130 | layer { 2131 | name: "FlowUnscaling_L4_D1" 2132 | type: "Eltwise" 2133 | bottom: "scaled_flow_FD_L4" 2134 | top: "flow_FD_L4" 2135 | eltwise_param { 2136 | operation: SUM 2137 | coeff: 2.5 2138 | } 2139 | } 2140 | layer { 2141 | name: "coords_L4" 2142 | type: "Eltwise" 2143 | bottom: "flow_FD_L4" 2144 | bottom: "gxy_L4" 2145 | top: "coords_L4" 2146 | eltwise_param { coeff: 1 coeff: 1 } 2147 | } 2148 | layer { 2149 | name: "warped_F1_L4" 2150 | type: "Warp" 2151 | bottom: "F1_L4" 2152 | bottom: "coords_L4" 2153 | top: "warped_F1_L4" 2154 | } 2155 | layer { 2156 | name: "corr_L4" 2157 | type: "Correlation" 2158 | bottom: "F0_L4" 2159 | bottom: "warped_F1_L4" 2160 | top: "corr_L4" 2161 | correlation_param { 2162 | pad: 4 2163 | kernel_size: 1 2164 | max_displacement: 4 2165 | stride_1: 1 2166 | stride_2: 1 2167 | } 2168 | } 2169 | layer { 2170 | name: "ReLU_corr_L4" 2171 | type: "ReLU" 2172 | bottom: "corr_L4" 2173 | top: "corr_L4" 2174 | relu_param { 2175 | negative_slope: 0.1 2176 | } 2177 | } 2178 | layer { 2179 | name: "F_VM_L4" 2180 | bottom: "F0_L4" 2181 | bottom: "corr_L4" 2182 | bottom: "conf_FD_L4" 2183 | top: "F_VM_L4" 2184 | type: "Concat" 2185 | concat_param { axis: 1 } 2186 | } 2187 | layer { 2188 | name: "conv1_VM_L4" 2189 | type: "Convolution" 2190 | bottom: "F_VM_L4" 2191 | top: "conv1_VM_L4" 2192 | param { lr_mult: 1 decay_mult: 1 } 2193 | param { lr_mult: 1 decay_mult: 0 } 2194 | convolution_param { 2195 | num_output: 128 2196 | pad: 1 2197 | kernel_size: 3 2198 | stride: 1 2199 | engine: CUDNN 2200 | } 2201 | } 2202 | layer { 2203 | name: "ReLU1_VM_L4" 2204 | type: "ReLU" 2205 | bottom: "conv1_VM_L4" 2206 | top: "conv1_VM_L4" 2207 | relu_param { negative_slope: 0.1 } 2208 | } 2209 | layer { 2210 | name: "conv2_VM_L4" 2211 | type: "Convolution" 2212 | bottom: "conv1_VM_L4" 2213 | top: "conv2_VM_L4" 2214 | param { lr_mult: 1 decay_mult: 1 } 2215 | param { lr_mult: 1 decay_mult: 0 } 2216 | convolution_param { 2217 | num_output: 64 2218 | pad: 1 2219 | kernel_size: 3 2220 | stride: 1 2221 | engine: CUDNN 2222 | } 2223 | } 2224 | layer { 2225 | name: "ReLU2_VM_L4" 2226 | type: "ReLU" 2227 | bottom: "conv2_VM_L4" 2228 | top: "conv2_VM_L4" 2229 | relu_param { negative_slope: 0.1 } 2230 | } 2231 | layer { 2232 | name: "conv3a_VM_L4" 2233 | type: "Convolution" 2234 | bottom: "conv2_VM_L4" 2235 | top: "conv3a_VM_L4" 2236 | param { lr_mult: 1 decay_mult: 1 } 2237 | param { lr_mult: 1 decay_mult: 0 } 2238 | convolution_param { 2239 | num_output: 32 2240 | pad: 1 2241 | kernel_size: 3 2242 | stride: 1 2243 | engine: CUDNN 2244 | } 2245 | } 2246 | layer { 2247 | name: "ReLU3a_VM_L4" 2248 | type: "ReLU" 2249 | bottom: "conv3a_VM_L4" 2250 | top: "conv3a_VM_L4" 2251 | relu_param { negative_slope: 0.1 } 2252 | } 2253 | layer { 2254 | name: "scalar_corr_L4" 2255 | type: "Convolution" 2256 | bottom: "conv3a_VM_L4" 2257 | top: "scalar_corr_L4" 2258 | param { lr_mult: 1 decay_mult: 1 } 2259 | param { lr_mult: 1 decay_mult: 0 } 2260 | convolution_param { 2261 | num_output: 81 2262 | pad: 0 2263 | kernel_size: 1 2264 | stride: 1 2265 | engine: CUDNN 2266 | } 2267 | } 2268 | layer { 2269 | name: "conv3b_VM_L4" 2270 | type: "Convolution" 2271 | bottom: "conv2_VM_L4" 2272 | top: "conv3b_VM_L4" 2273 | param { lr_mult: 1 decay_mult: 1 } 2274 | param { lr_mult: 1 decay_mult: 0 } 2275 | convolution_param { 2276 | num_output: 32 2277 | pad: 1 2278 | kernel_size: 3 2279 | stride: 1 2280 | engine: CUDNN 2281 | } 2282 | } 2283 | layer { 2284 | name: "ReLU3b_VM_L4" 2285 | type: "ReLU" 2286 | bottom: "conv3b_VM_L4" 2287 | top: "conv3b_VM_L4" 2288 | relu_param { negative_slope: 0.1 } 2289 | } 2290 | layer { 2291 | name: "offset_corr_L4" 2292 | type: "Convolution" 2293 | bottom: "conv3b_VM_L4" 2294 | top: "offset_corr_L4" 2295 | param { lr_mult: 1 decay_mult: 1 } 2296 | param { lr_mult: 1 decay_mult: 0 } 2297 | convolution_param { 2298 | num_output: 81 2299 | pad: 0 2300 | kernel_size: 1 2301 | stride: 1 2302 | engine: CUDNN 2303 | } 2304 | } 2305 | layer { 2306 | name: "scaled_corr_L4" 2307 | type: "Eltwise" 2308 | bottom: "corr_L4" 2309 | bottom: "scalar_corr_L4" 2310 | top: "scaled_corr_L4" 2311 | eltwise_param { operation: PROD } 2312 | } 2313 | layer { 2314 | name: "adj_corr_L4" 2315 | type: "Eltwise" 2316 | bottom: "scaled_corr_L4" 2317 | bottom: "offset_corr_L4" 2318 | top: "adj_corr_L4" 2319 | eltwise_param { operation: SUM } 2320 | } 2321 | 2322 | # Feature Matching: L4 2323 | layer { 2324 | name: "conv1_D1_L4" 2325 | type: "Convolution" 2326 | bottom: "adj_corr_L4" 2327 | top: "conv1_D1_L4" 2328 | param { lr_mult: 1 decay_mult: 1 } 2329 | param { lr_mult: 1 decay_mult: 0 } 2330 | convolution_param { 2331 | num_output: 128 2332 | pad: 1 2333 | kernel_size: 3 2334 | stride: 1 2335 | engine: CUDNN 2336 | } 2337 | } 2338 | layer { 2339 | name: "ReLU1_D1_L4" 2340 | type: "ReLU" 2341 | bottom: "conv1_D1_L4" 2342 | top: "conv1_D1_L4" 2343 | relu_param { negative_slope: 0.1 } 2344 | } 2345 | layer { 2346 | name: "conv2_D1_L4" 2347 | type: "Convolution" 2348 | bottom: "conv1_D1_L4" 2349 | top: "conv2_D1_L4" 2350 | param { lr_mult: 1 decay_mult: 1 } 2351 | param { lr_mult: 1 decay_mult: 0 } 2352 | convolution_param { 2353 | num_output: 128 2354 | pad: 1 2355 | kernel_size: 3 2356 | stride: 1 2357 | engine: CUDNN 2358 | } 2359 | } 2360 | layer { 2361 | name: "ReLU2_D1_L4" 2362 | type: "ReLU" 2363 | bottom: "conv2_D1_L4" 2364 | top: "conv2_D1_L4" 2365 | relu_param { negative_slope: 0.1 } 2366 | } 2367 | layer { 2368 | name: "conv3_D1_L4" 2369 | type: "Convolution" 2370 | bottom: "conv2_D1_L4" 2371 | top: "conv3_D1_L4" 2372 | param { lr_mult: 1 decay_mult: 1 } 2373 | param { lr_mult: 1 decay_mult: 0 } 2374 | convolution_param { 2375 | num_output: 96 2376 | pad: 1 2377 | kernel_size: 3 2378 | stride: 1 2379 | engine: CUDNN 2380 | } 2381 | } 2382 | layer { 2383 | name: "ReLU3_D1_L4" 2384 | type: "ReLU" 2385 | bottom: "conv3_D1_L4" 2386 | top: "conv3_D1_L4" 2387 | relu_param { negative_slope: 0.1 } 2388 | } 2389 | layer { 2390 | name: "conv4_D1_L4" 2391 | type: "Convolution" 2392 | bottom: "conv3_D1_L4" 2393 | top: "conv4_D1_L4" 2394 | param { lr_mult: 1 decay_mult: 1 } 2395 | param { lr_mult: 1 decay_mult: 0 } 2396 | convolution_param { 2397 | num_output: 64 2398 | pad: 1 2399 | kernel_size: 3 2400 | stride: 1 2401 | engine: CUDNN 2402 | } 2403 | } 2404 | layer { 2405 | name: "ReLU4_D1_L4" 2406 | type: "ReLU" 2407 | bottom: "conv4_D1_L4" 2408 | top: "conv4_D1_L4" 2409 | relu_param { negative_slope: 0.1 } 2410 | } 2411 | layer { 2412 | name: "conv5_D1_L4" 2413 | type: "Convolution" 2414 | bottom: "conv4_D1_L4" 2415 | top: "conv5_D1_L4" 2416 | param { lr_mult: 1 decay_mult: 1 } 2417 | param { lr_mult: 1 decay_mult: 0 } 2418 | convolution_param { 2419 | num_output: 32 2420 | pad: 1 2421 | kernel_size: 3 2422 | stride: 1 2423 | engine: CUDNN 2424 | } 2425 | } 2426 | layer { 2427 | name: "ReLU5_D1_L4" 2428 | type: "ReLU" 2429 | bottom: "conv5_D1_L4" 2430 | top: "conv5_D1_L4" 2431 | relu_param { negative_slope: 0.1 } 2432 | } 2433 | layer { 2434 | name: "scaled_flow_D1_res_L4" 2435 | type: "Convolution" 2436 | bottom: "conv5_D1_L4" 2437 | top: "scaled_flow_D1_res_L4" 2438 | param { lr_mult: 1 decay_mult: 1 } 2439 | param { lr_mult: 1 decay_mult: 0 } 2440 | convolution_param { 2441 | num_output: 2 2442 | pad: 2 2443 | kernel_size: 5 2444 | stride: 1 2445 | engine: CUDNN 2446 | } 2447 | } 2448 | layer { 2449 | name: "scaled_flow_D1_L4" 2450 | type: "Eltwise" 2451 | bottom: "scaled_flow_FD_L4" 2452 | bottom: "scaled_flow_D1_res_L4" 2453 | top: "scaled_flow_D1_L4" 2454 | eltwise_param { operation: SUM } 2455 | } 2456 | 2457 | # Sub-Pixel Refinement: L4 2458 | layer { 2459 | name: "FlowUnscaling_L4_D2" 2460 | type: "Eltwise" 2461 | bottom: "scaled_flow_D1_L4" 2462 | top: "flow_D1_L4" 2463 | eltwise_param { 2464 | operation: SUM 2465 | coeff: 2.5 2466 | } 2467 | } 2468 | layer { 2469 | name: "coords_D1_L4" 2470 | type: "Eltwise" 2471 | bottom: "flow_D1_L4" 2472 | bottom: "gxy_L4" 2473 | top: "coords_D1_L4" 2474 | eltwise_param { coeff: 1 coeff: 1 } 2475 | } 2476 | layer { 2477 | name: "warped_F1_D1_L4" 2478 | type: "Warp" 2479 | bottom: "F1_L4" 2480 | bottom: "coords_D1_L4" 2481 | top: "warped_F1_D1_L4" 2482 | } 2483 | layer { 2484 | name: "F_D2_L4" 2485 | bottom: "F0_L4" 2486 | bottom: "warped_F1_D1_L4" 2487 | bottom: "scaled_flow_D1_L4" 2488 | top: "F_D2_L4" 2489 | type: "Concat" 2490 | concat_param { axis: 1 } 2491 | } 2492 | layer { 2493 | name: "conv1_D2_L4" 2494 | type: "Convolution" 2495 | bottom: "F_D2_L4" 2496 | top: "conv1_D2_L4" 2497 | param { lr_mult: 1 decay_mult: 1 } 2498 | param { lr_mult: 1 decay_mult: 0 } 2499 | convolution_param { 2500 | num_output: 128 2501 | pad: 1 2502 | kernel_size: 3 2503 | stride: 1 2504 | engine: CUDNN 2505 | } 2506 | } 2507 | layer { 2508 | name: "ReLU1_D2_L4" 2509 | type: "ReLU" 2510 | bottom: "conv1_D2_L4" 2511 | top: "conv1_D2_L4" 2512 | relu_param { negative_slope: 0.1 } 2513 | } 2514 | layer { 2515 | name: "conv2_D2_L4" 2516 | type: "Convolution" 2517 | bottom: "conv1_D2_L4" 2518 | top: "conv2_D2_L4" 2519 | param { lr_mult: 1 decay_mult: 1 } 2520 | param { lr_mult: 1 decay_mult: 0 } 2521 | convolution_param { 2522 | num_output: 128 2523 | pad: 1 2524 | kernel_size: 3 2525 | stride: 1 2526 | engine: CUDNN 2527 | } 2528 | } 2529 | layer { 2530 | name: "ReLU2_D2_L4" 2531 | type: "ReLU" 2532 | bottom: "conv2_D2_L4" 2533 | top: "conv2_D2_L4" 2534 | relu_param { negative_slope: 0.1 } 2535 | } 2536 | layer { 2537 | name: "conv3_D2_L4" 2538 | type: "Convolution" 2539 | bottom: "conv2_D2_L4" 2540 | top: "conv3_D2_L4" 2541 | param { lr_mult: 1 decay_mult: 1 } 2542 | param { lr_mult: 1 decay_mult: 0 } 2543 | convolution_param { 2544 | num_output: 96 2545 | pad: 1 2546 | kernel_size: 3 2547 | stride: 1 2548 | engine: CUDNN 2549 | } 2550 | } 2551 | layer { 2552 | name: "ReLU3_D2_L4" 2553 | type: "ReLU" 2554 | bottom: "conv3_D2_L4" 2555 | top: "conv3_D2_L4" 2556 | relu_param { negative_slope: 0.1 } 2557 | } 2558 | layer { 2559 | name: "conv4_D2_L4" 2560 | type: "Convolution" 2561 | bottom: "conv3_D2_L4" 2562 | top: "conv4_D2_L4" 2563 | param { lr_mult: 1 decay_mult: 1 } 2564 | param { lr_mult: 1 decay_mult: 0 } 2565 | convolution_param { 2566 | num_output: 64 2567 | pad: 1 2568 | kernel_size: 3 2569 | stride: 1 2570 | engine: CUDNN 2571 | } 2572 | } 2573 | layer { 2574 | name: "ReLU4_D2_L4" 2575 | type: "ReLU" 2576 | bottom: "conv4_D2_L4" 2577 | top: "conv4_D2_L4" 2578 | relu_param { negative_slope: 0.1 } 2579 | } 2580 | layer { 2581 | name: "conv5_D2_L4" 2582 | type: "Convolution" 2583 | bottom: "conv4_D2_L4" 2584 | top: "conv5_D2_L4" 2585 | param { lr_mult: 1 decay_mult: 1 } 2586 | param { lr_mult: 1 decay_mult: 0 } 2587 | convolution_param { 2588 | num_output: 32 2589 | pad: 1 2590 | kernel_size: 3 2591 | stride: 1 2592 | engine: CUDNN 2593 | } 2594 | } 2595 | layer { 2596 | name: "ReLU5_D2_L4" 2597 | type: "ReLU" 2598 | bottom: "conv5_D2_L4" 2599 | top: "conv5_D2_L4" 2600 | relu_param { negative_slope: 0.1 } 2601 | } 2602 | layer { 2603 | name: "scaled_flow_D2_res_L4" 2604 | type: "Convolution" 2605 | bottom: "conv5_D2_L4" 2606 | top: "scaled_flow_D2_res_L4" 2607 | param { lr_mult: 1 decay_mult: 1 } 2608 | param { lr_mult: 1 decay_mult: 0 } 2609 | convolution_param { 2610 | num_output: 2 2611 | pad: 2 2612 | kernel_size: 5 2613 | stride: 1 2614 | engine: CUDNN 2615 | } 2616 | } 2617 | layer { 2618 | name: "scaled_flow_D2_L4" 2619 | type: "Eltwise" 2620 | bottom: "scaled_flow_D1_L4" 2621 | bottom: "scaled_flow_D2_res_L4" 2622 | top: "scaled_flow_D2_L4" 2623 | eltwise_param { operation: SUM } 2624 | } 2625 | 2626 | # Flow Regularization: L4 2627 | layer { 2628 | name: "slice_scaled_flow_D_L4" 2629 | type: "Slice" 2630 | bottom: "scaled_flow_D2_L4" 2631 | top: "scaled_flow_D_L4_x" 2632 | top: "scaled_flow_D_L4_y" 2633 | slice_param { axis: 1 slice_point: 1 } 2634 | } 2635 | layer { 2636 | name: "reshaped_scaled_flow_D_L4_x" 2637 | type: "Im2col" 2638 | bottom: "scaled_flow_D_L4_x" 2639 | top: "reshaped_scaled_flow_D_L4_x" 2640 | convolution_param { pad: 2 kernel_size: 5 stride: 1 } 2641 | } 2642 | layer { 2643 | name: "reshaped_scaled_flow_D_L4_y" 2644 | type: "Im2col" 2645 | bottom: "scaled_flow_D_L4_y" 2646 | top: "reshaped_scaled_flow_D_L4_y" 2647 | convolution_param { pad: 2 kernel_size: 5 stride: 1 } 2648 | } 2649 | layer { 2650 | name: "mean_scaled_flow_D_L4_x" 2651 | type: "Reduction" 2652 | bottom: "scaled_flow_D_L4_x" 2653 | top: "mean_scaled_flow_D_L4_x" 2654 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 2655 | } 2656 | layer { 2657 | name: "scaled_flow_D_nomean_L4_x" 2658 | type: "Bias" 2659 | bottom: "scaled_flow_D_L4_x" 2660 | bottom: "mean_scaled_flow_D_L4_x" 2661 | top: "scaled_flow_D_nomean_L4_x" 2662 | bias_param { axis: 0 } 2663 | } 2664 | layer { 2665 | name: "mean_scaled_flow_D_L4_y" 2666 | type: "Reduction" 2667 | bottom: "scaled_flow_D_L4_y" 2668 | top: "mean_scaled_flow_D_L4_y" 2669 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 2670 | } 2671 | layer { 2672 | name: "scaled_flow_D_nomean_L4_y" 2673 | type: "Bias" 2674 | bottom: "scaled_flow_D_L4_y" 2675 | bottom: "mean_scaled_flow_D_L4_y" 2676 | top: "scaled_flow_D_nomean_L4_y" 2677 | bias_param { axis: 0 } 2678 | } 2679 | layer { 2680 | name: "FlowUnscaling_L4_R" 2681 | type: "Eltwise" 2682 | bottom: "scaled_flow_D2_L4" 2683 | top: "flow_D2_L4" 2684 | eltwise_param { 2685 | operation: SUM 2686 | coeff: 2.5 2687 | } 2688 | } 2689 | layer { 2690 | name: "coords_D2_L4" 2691 | type: "Eltwise" 2692 | bottom: "flow_D2_L4" 2693 | bottom: "gxy_L4" 2694 | top: "coords_D2_L4" 2695 | eltwise_param { coeff: 1 coeff: 1 } 2696 | } 2697 | layer { 2698 | name: "warped_img1_aug_L4" 2699 | type: "Warp" 2700 | bottom: "img1_aug_L4" 2701 | bottom: "coords_D2_L4" 2702 | top: "warped_img1_aug_L4" 2703 | } 2704 | layer { 2705 | name: "img_diff_L4" 2706 | type: "Eltwise" 2707 | bottom: "img0_aug_L4" 2708 | bottom: "warped_img1_aug_L4" 2709 | top: "img_diff_L4" 2710 | eltwise_param { 2711 | operation: SUM 2712 | coeff: 1.0 2713 | coeff: -1.0 2714 | } 2715 | } 2716 | layer { 2717 | name: "channelNorm_L4" 2718 | type: "ChannelNorm" 2719 | bottom: "img_diff_L4" 2720 | top: "channelNorm_L4" 2721 | } 2722 | layer { 2723 | name: "F0_128_L4" 2724 | type: "Convolution" 2725 | bottom: "F0_L4" 2726 | top: "F0_128_L4" 2727 | param { lr_mult: 1 decay_mult: 1 } 2728 | param { lr_mult: 1 decay_mult: 0 } 2729 | convolution_param { 2730 | num_output: 128 2731 | pad: 0 2732 | kernel_size: 1 2733 | stride: 1 2734 | engine: CUDNN 2735 | } 2736 | } 2737 | layer { 2738 | name: "relu_F0_128_L4" 2739 | type: "ReLU" 2740 | bottom: "F0_128_L4" 2741 | top: "F0_128_L4" 2742 | relu_param { negative_slope: 0.1 } 2743 | } 2744 | layer { 2745 | name: "concat_F0_R_L4" 2746 | type: "Concat" 2747 | bottom: "channelNorm_L4" 2748 | bottom: "scaled_flow_D_nomean_L4_x" 2749 | bottom: "scaled_flow_D_nomean_L4_y" 2750 | bottom: "F0_128_L4" 2751 | top: "concat_F0_R_L4" 2752 | concat_param { axis: 1 } 2753 | } 2754 | layer { 2755 | name: "conv1_R_L4" 2756 | type: "Convolution" 2757 | bottom: "concat_F0_R_L4" 2758 | top: "conv1_R_L4" 2759 | param { lr_mult: 1 decay_mult: 1 } 2760 | param { lr_mult: 1 decay_mult: 0 } 2761 | convolution_param { 2762 | num_output: 128 2763 | pad: 1 2764 | kernel_size: 3 2765 | stride: 1 2766 | engine: CUDNN 2767 | } 2768 | } 2769 | layer { 2770 | name: "relu_conv1_R_L4" 2771 | type: "ReLU" 2772 | bottom: "conv1_R_L4" 2773 | top: "conv1_R_L4" 2774 | relu_param { negative_slope: 0.1 } 2775 | } 2776 | layer { 2777 | name: "conv2_R_L4" 2778 | type: "Convolution" 2779 | bottom: "conv1_R_L4" 2780 | top: "conv2_R_L4" 2781 | param { lr_mult: 1 decay_mult: 1 } 2782 | param { lr_mult: 1 decay_mult: 0 } 2783 | convolution_param { 2784 | num_output: 128 2785 | pad: 1 2786 | kernel_size: 3 2787 | stride: 1 2788 | engine: CUDNN 2789 | } 2790 | } 2791 | layer { 2792 | name: "relu_conv2_R_L4" 2793 | type: "ReLU" 2794 | bottom: "conv2_R_L4" 2795 | top: "conv2_R_L4" 2796 | relu_param { negative_slope: 0.1 } 2797 | } 2798 | layer { 2799 | name: "conv3_R_L4" 2800 | type: "Convolution" 2801 | bottom: "conv2_R_L4" 2802 | top: "conv3_R_L4" 2803 | param { lr_mult: 1 decay_mult: 1 } 2804 | param { lr_mult: 1 decay_mult: 0 } 2805 | convolution_param { 2806 | num_output: 64 2807 | pad: 1 2808 | kernel_size: 3 2809 | stride: 1 2810 | engine: CUDNN 2811 | } 2812 | } 2813 | layer { 2814 | name: "relu_conv3_R_L4" 2815 | type: "ReLU" 2816 | bottom: "conv3_R_L4" 2817 | top: "conv3_R_L4" 2818 | relu_param { negative_slope: 0.1 } 2819 | } 2820 | layer { 2821 | name: "conv4_R_L4" 2822 | type: "Convolution" 2823 | bottom: "conv3_R_L4" 2824 | top: "conv4_R_L4" 2825 | param { lr_mult: 1 decay_mult: 1 } 2826 | param { lr_mult: 1 decay_mult: 0 } 2827 | convolution_param { 2828 | num_output: 64 2829 | pad: 1 2830 | kernel_size: 3 2831 | stride: 1 2832 | engine: CUDNN 2833 | } 2834 | } 2835 | layer { 2836 | name: "relu_conv4_R_L4" 2837 | type: "ReLU" 2838 | bottom: "conv4_R_L4" 2839 | top: "conv4_R_L4" 2840 | relu_param { negative_slope: 0.1 } 2841 | } 2842 | layer { 2843 | name: "conv5_R_L4" 2844 | type: "Convolution" 2845 | bottom: "conv4_R_L4" 2846 | top: "conv5_R_L4" 2847 | param { lr_mult: 1 decay_mult: 1 } 2848 | param { lr_mult: 1 decay_mult: 0 } 2849 | convolution_param { 2850 | num_output: 32 2851 | pad: 1 2852 | kernel_size: 3 2853 | stride: 1 2854 | engine: CUDNN 2855 | } 2856 | } 2857 | layer { 2858 | name: "relu_conv5_R_L4" 2859 | type: "ReLU" 2860 | bottom: "conv5_R_L4" 2861 | top: "conv5_R_L4" 2862 | relu_param { negative_slope: 0.1 } 2863 | } 2864 | layer { 2865 | name: "conv6_R_L4" 2866 | type: "Convolution" 2867 | bottom: "conv5_R_L4" 2868 | top: "conv6_R_L4" 2869 | param { lr_mult: 1 decay_mult: 1 } 2870 | param { lr_mult: 1 decay_mult: 0 } 2871 | convolution_param { 2872 | num_output: 32 2873 | pad: 1 2874 | kernel_size: 3 2875 | stride: 1 2876 | engine: CUDNN 2877 | } 2878 | } 2879 | layer { 2880 | name: "relu_conv6_R_L4" 2881 | type: "ReLU" 2882 | bottom: "conv6_R_L4" 2883 | top: "conv6_R_L4" 2884 | relu_param { negative_slope: 0.1 } 2885 | } 2886 | layer { 2887 | name: "distH_R_L4" 2888 | type: "Convolution" 2889 | bottom: "conv6_R_L4" 2890 | top: "distH_R_L4" 2891 | param { lr_mult: 1 decay_mult: 1 } 2892 | param { lr_mult: 0 } 2893 | convolution_param { 2894 | num_output: 25 2895 | pad_h: 2 2896 | kernel_h: 5 2897 | stride_h: 1 2898 | pad_w: 0 2899 | kernel_w: 1 2900 | stride_w: 1 2901 | engine: CUDNN 2902 | } 2903 | } 2904 | layer { 2905 | name: "distW_R_L4" 2906 | type: "Convolution" 2907 | bottom: "distH_R_L4" 2908 | top: "distW_R_L4" 2909 | param { lr_mult: 1 decay_mult: 1 } 2910 | param { lr_mult: 0 } 2911 | convolution_param { 2912 | num_output: 25 2913 | pad_h: 0 2914 | kernel_h: 1 2915 | stride_h: 1 2916 | pad_w: 2 2917 | kernel_w: 5 2918 | stride_w: 1 2919 | engine: CUDNN 2920 | } 2921 | } 2922 | layer { 2923 | name: "neg_sq_dist_R_L4" 2924 | type: "NegSquare" 2925 | bottom: "distW_R_L4" 2926 | top: "neg_sq_dist_R_L4" 2927 | } 2928 | layer { 2929 | name: "exp_kernel_R_L4" 2930 | type: "ExpMax" 2931 | bottom: "neg_sq_dist_R_L4" 2932 | top: "exp_kernel_R_L4" 2933 | } 2934 | layer { 2935 | name: "sum_exp_kernel_R_L4" 2936 | type: "Convolution" 2937 | bottom: "exp_kernel_R_L4" 2938 | top: "sum_exp_kernel_R_L4" 2939 | param { lr_mult: 0 } 2940 | param { lr_mult: 0 } 2941 | convolution_param { 2942 | num_output: 1 2943 | kernel_size: 1 2944 | weight_filler { type: "constant" value: 1 } 2945 | } 2946 | } 2947 | layer { 2948 | name: "divisor_R_L4" 2949 | bottom: "sum_exp_kernel_R_L4" 2950 | top: "divisor_R_L4" 2951 | type: "Power" 2952 | power_param { power: -1 scale: 1 shift: 0 } 2953 | } 2954 | layer { 2955 | name: "f-lconv_L4_x" 2956 | type: "Eltwise" 2957 | bottom: "reshaped_scaled_flow_D_L4_x" 2958 | bottom: "exp_kernel_R_L4" 2959 | top: "f-lconv_L4_x" 2960 | eltwise_param { operation: PROD } 2961 | } 2962 | layer { 2963 | name: "scaled_flow0_R_L4_x" 2964 | type: "Convolution" 2965 | bottom: "f-lconv_L4_x" 2966 | top: "scaled_flow0_R_L4_x" 2967 | param { lr_mult: 0 } 2968 | param { lr_mult: 0 } 2969 | convolution_param { 2970 | num_output: 1 2971 | kernel_size: 1 2972 | weight_filler { type: "constant" value: 1 } 2973 | } 2974 | } 2975 | layer { 2976 | name: "scaled_flow1_R_L4_x" 2977 | type: "Eltwise" 2978 | bottom: "scaled_flow0_R_L4_x" 2979 | bottom: "divisor_R_L4" 2980 | top: "scaled_flow1_R_L4_x" 2981 | eltwise_param { 2982 | operation: PROD 2983 | } 2984 | } 2985 | layer { 2986 | name: "f-lconv_L4_y" 2987 | type: "Eltwise" 2988 | bottom: "reshaped_scaled_flow_D_L4_y" 2989 | bottom: "exp_kernel_R_L4" 2990 | top: "f-lconv_L4_y" 2991 | eltwise_param { operation: PROD } 2992 | } 2993 | layer { 2994 | name: "scaled_flow0_R_L4_y" 2995 | type: "Convolution" 2996 | bottom: "f-lconv_L4_y" 2997 | top: "scaled_flow0_R_L4_y" 2998 | param { lr_mult: 0 } 2999 | param { lr_mult: 0 } 3000 | convolution_param { 3001 | num_output: 1 3002 | kernel_size: 1 3003 | weight_filler { type: "constant" value: 1 } 3004 | } 3005 | } 3006 | layer { 3007 | name: "scaled_flow1_R_L4_y" 3008 | type: "Eltwise" 3009 | bottom: "scaled_flow0_R_L4_y" 3010 | bottom: "divisor_R_L4" 3011 | top: "scaled_flow1_R_L4_y" 3012 | eltwise_param { 3013 | operation: PROD 3014 | } 3015 | } 3016 | layer { 3017 | name: "scaled_flow_R_L4" 3018 | bottom: "scaled_flow1_R_L4_x" 3019 | bottom: "scaled_flow1_R_L4_y" 3020 | top: "scaled_flow_R_L4" 3021 | type: "Concat" 3022 | concat_param { axis: 1 } 3023 | } 3024 | 3025 | # Confidence Prediction: L4 3026 | layer { 3027 | name: "conf0_R_L4" 3028 | type: "Convolution" 3029 | bottom: "conv6_R_L4" 3030 | top: "conf0_R_L4" 3031 | param { lr_mult: 1 decay_mult: 1 } 3032 | param { lr_mult: 1 decay_mult: 0 } 3033 | convolution_param { 3034 | num_output: 1 3035 | pad: 2 3036 | kernel_size: 5 3037 | stride: 1 3038 | engine: CUDNN 3039 | } 3040 | } 3041 | layer { 3042 | name: "conf_R_L4" 3043 | type: "Sigmoid" 3044 | bottom: "conf0_R_L4" 3045 | top: "conf_R_L4" 3046 | } 3047 | 3048 | # Flow Field Deformation: L3 3049 | layer { 3050 | name: "conf_R_L4to3" 3051 | type: "Deconvolution" 3052 | bottom: "conf_R_L4" 3053 | top: "conf_R_L4to3" 3054 | param { lr_mult: 0 decay_mult: 0 } 3055 | convolution_param { 3056 | num_output: 1 3057 | pad: 1 3058 | kernel_size: 4 3059 | stride: 2 3060 | weight_filler { type: "bilinear" } 3061 | bias_term: false 3062 | engine: CUDNN 3063 | } 3064 | } 3065 | layer { 3066 | name: "scaled_flow_R_L4to3" 3067 | type: "Deconvolution" 3068 | bottom: "scaled_flow_R_L4" 3069 | top: "scaled_flow_R_L4to3" 3070 | param { lr_mult: 0 decay_mult: 0 } 3071 | convolution_param { 3072 | num_output: 2 3073 | group: 2 3074 | pad: 1 3075 | kernel_size: 4 3076 | stride: 2 3077 | weight_filler { type: "bilinear" } 3078 | bias_term: false 3079 | engine: CUDNN 3080 | } 3081 | } 3082 | layer { 3083 | name: "acorr_F0_L3" 3084 | type: "Correlation" 3085 | bottom: "F0_L3" 3086 | bottom: "F0_L3" 3087 | top: "acorr_F0_L3" 3088 | correlation_param { 3089 | pad: 8 3090 | kernel_size: 1 3091 | max_displacement: 8 3092 | stride_1: 1 3093 | stride_2: 2 3094 | } 3095 | } 3096 | layer { 3097 | name: "ReLU1_acorr_F0_L3" 3098 | type: "ReLU" 3099 | bottom: "acorr_F0_L3" 3100 | top: "acorr_F0_L3" 3101 | relu_param { negative_slope: 0.1 } 3102 | } 3103 | layer { 3104 | name: "F_FD_L3" 3105 | type: "Concat" 3106 | bottom: "acorr_F0_L3" 3107 | bottom: "conf_R_L4to3" 3108 | top: "F_FD_L3" 3109 | concat_param { axis: 1 } 3110 | } 3111 | layer { 3112 | name: "conv1_FD_L3" 3113 | type: "Convolution" 3114 | bottom: "F_FD_L3" 3115 | top: "conv1_FD_L3" 3116 | param { lr_mult: 1 decay_mult: 1 } 3117 | param { lr_mult: 1 decay_mult: 0 } 3118 | convolution_param { 3119 | num_output: 128 3120 | pad: 1 3121 | kernel_size: 3 3122 | stride: 1 3123 | engine: CUDNN 3124 | } 3125 | } 3126 | layer { 3127 | name: "ReLU1_FD_L3" 3128 | type: "ReLU" 3129 | bottom: "conv1_FD_L3" 3130 | top: "conv1_FD_L3" 3131 | relu_param { negative_slope: 0.1 } 3132 | } 3133 | layer { 3134 | name: "conv2_FD_L3" 3135 | type: "Convolution" 3136 | bottom: "conv1_FD_L3" 3137 | top: "conv2_FD_L3" 3138 | param { lr_mult: 1 decay_mult: 1 } 3139 | param { lr_mult: 1 decay_mult: 0 } 3140 | convolution_param { 3141 | num_output: 64 3142 | pad: 1 3143 | kernel_size: 3 3144 | stride: 1 3145 | engine: CUDNN 3146 | } 3147 | } 3148 | layer { 3149 | name: "ReLU2_FD_L3" 3150 | type: "ReLU" 3151 | bottom: "conv2_FD_L3" 3152 | top: "conv2_FD_L3" 3153 | relu_param { negative_slope: 0.1 } 3154 | } 3155 | layer { 3156 | name: "conv3_FD_L3" 3157 | type: "Convolution" 3158 | bottom: "conv2_FD_L3" 3159 | top: "conv3_FD_L3" 3160 | param { lr_mult: 1 decay_mult: 1 } 3161 | param { lr_mult: 1 decay_mult: 0 } 3162 | convolution_param { 3163 | num_output: 32 3164 | pad: 1 3165 | kernel_size: 3 3166 | stride: 1 3167 | engine: CUDNN 3168 | } 3169 | } 3170 | layer { 3171 | name: "ReLU3_FD_L3" 3172 | type: "ReLU" 3173 | bottom: "conv3_FD_L3" 3174 | top: "conv3_FD_L3" 3175 | relu_param { negative_slope: 0.1 } 3176 | } 3177 | layer { 3178 | name: "disp_FD_L3" 3179 | type: "Convolution" 3180 | bottom: "conv3_FD_L3" 3181 | top: "disp_FD_L3" 3182 | param { lr_mult: 1 decay_mult: 1 } 3183 | param { lr_mult: 1 decay_mult: 0 } 3184 | convolution_param { 3185 | num_output: 2 3186 | pad: 2 3187 | kernel_size: 5 3188 | stride: 1 3189 | engine: CUDNN 3190 | } 3191 | } 3192 | layer { 3193 | name: "gxy_L3" 3194 | type: "Grid" 3195 | top: "gxy_L3" 3196 | bottom: "disp_FD_L3" 3197 | propagate_down: false 3198 | } 3199 | layer { 3200 | name: "coords_disp_FD_L3" 3201 | type: "Eltwise" 3202 | bottom: "disp_FD_L3" 3203 | bottom: "gxy_L3" 3204 | top: "coords_disp_FD_L3" 3205 | eltwise_param { coeff: 1 coeff: 1 } 3206 | } 3207 | layer { 3208 | name: "scaled_flow_FD_L3" 3209 | type: "Warp" 3210 | bottom: "scaled_flow_R_L4to3" 3211 | bottom: "coords_disp_FD_L3" 3212 | top: "scaled_flow_FD_L3" 3213 | } 3214 | 3215 | # Confidence Prediction: L3 3216 | layer { 3217 | name: "conf0_FD_L3" 3218 | type: "Convolution" 3219 | bottom: "conv3_FD_L3" 3220 | top: "conf0_FD_L3" 3221 | param { lr_mult: 1 decay_mult: 1 } 3222 | param { lr_mult: 1 decay_mult: 0 } 3223 | convolution_param { 3224 | num_output: 1 3225 | pad: 2 3226 | kernel_size: 5 3227 | stride: 1 3228 | engine: CUDNN 3229 | } 3230 | } 3231 | layer { 3232 | name: "conf_FD_L3" 3233 | type: "Sigmoid" 3234 | bottom: "conf0_FD_L3" 3235 | top: "conf_FD_L3" 3236 | } 3237 | 3238 | # Cost Volume Modulation: L3 3239 | layer { 3240 | name: "FlowUnscaling_L3_D1" 3241 | type: "Eltwise" 3242 | bottom: "scaled_flow_FD_L3" 3243 | top: "flow_FD_L3" 3244 | eltwise_param { 3245 | operation: SUM 3246 | coeff: 5.0 3247 | } 3248 | } 3249 | layer { 3250 | name: "coords_L3" 3251 | type: "Eltwise" 3252 | bottom: "flow_FD_L3" 3253 | bottom: "gxy_L3" 3254 | top: "coords_L3" 3255 | eltwise_param { coeff: 1 coeff: 1 } 3256 | } 3257 | layer { 3258 | name: "warped_F1_L3" 3259 | type: "Warp" 3260 | bottom: "F1_L3" 3261 | bottom: "coords_L3" 3262 | top: "warped_F1_L3" 3263 | } 3264 | layer { 3265 | name: "corr_L3" 3266 | type: "Correlation" 3267 | bottom: "F0_L3" 3268 | bottom: "warped_F1_L3" 3269 | top: "corr_L3" 3270 | correlation_param { 3271 | pad: 4 3272 | kernel_size: 1 3273 | max_displacement: 4 3274 | stride_1: 1 3275 | stride_2: 1 3276 | } 3277 | } 3278 | layer { 3279 | name: "ReLU_corr_L3" 3280 | type: "ReLU" 3281 | bottom: "corr_L3" 3282 | top: "corr_L3" 3283 | relu_param { 3284 | negative_slope: 0.1 3285 | } 3286 | } 3287 | layer { 3288 | name: "F_VM_L3" 3289 | bottom: "F0_L3" 3290 | bottom: "corr_L3" 3291 | bottom: "conf_FD_L3" 3292 | top: "F_VM_L3" 3293 | type: "Concat" 3294 | concat_param { axis: 1 } 3295 | } 3296 | layer { 3297 | name: "conv1_VM_L3" 3298 | type: "Convolution" 3299 | bottom: "F_VM_L3" 3300 | top: "conv1_VM_L3" 3301 | param { lr_mult: 1 decay_mult: 1 } 3302 | param { lr_mult: 1 decay_mult: 0 } 3303 | convolution_param { 3304 | num_output: 128 3305 | pad: 1 3306 | kernel_size: 3 3307 | stride: 1 3308 | engine: CUDNN 3309 | } 3310 | } 3311 | layer { 3312 | name: "ReLU1_VM_L3" 3313 | type: "ReLU" 3314 | bottom: "conv1_VM_L3" 3315 | top: "conv1_VM_L3" 3316 | relu_param { negative_slope: 0.1 } 3317 | } 3318 | layer { 3319 | name: "conv2_VM_L3" 3320 | type: "Convolution" 3321 | bottom: "conv1_VM_L3" 3322 | top: "conv2_VM_L3" 3323 | param { lr_mult: 1 decay_mult: 1 } 3324 | param { lr_mult: 1 decay_mult: 0 } 3325 | convolution_param { 3326 | num_output: 64 3327 | pad: 1 3328 | kernel_size: 3 3329 | stride: 1 3330 | engine: CUDNN 3331 | } 3332 | } 3333 | layer { 3334 | name: "ReLU2_VM_L3" 3335 | type: "ReLU" 3336 | bottom: "conv2_VM_L3" 3337 | top: "conv2_VM_L3" 3338 | relu_param { negative_slope: 0.1 } 3339 | } 3340 | layer { 3341 | name: "conv3a_VM_L3" 3342 | type: "Convolution" 3343 | bottom: "conv2_VM_L3" 3344 | top: "conv3a_VM_L3" 3345 | param { lr_mult: 1 decay_mult: 1 } 3346 | param { lr_mult: 1 decay_mult: 0 } 3347 | convolution_param { 3348 | num_output: 32 3349 | pad: 1 3350 | kernel_size: 3 3351 | stride: 1 3352 | engine: CUDNN 3353 | } 3354 | } 3355 | layer { 3356 | name: "ReLU3a_VM_L3" 3357 | type: "ReLU" 3358 | bottom: "conv3a_VM_L3" 3359 | top: "conv3a_VM_L3" 3360 | relu_param { negative_slope: 0.1 } 3361 | } 3362 | layer { 3363 | name: "scalar_corr_L3" 3364 | type: "Convolution" 3365 | bottom: "conv3a_VM_L3" 3366 | top: "scalar_corr_L3" 3367 | param { lr_mult: 1 decay_mult: 1 } 3368 | param { lr_mult: 1 decay_mult: 0 } 3369 | convolution_param { 3370 | num_output: 81 3371 | pad: 0 3372 | kernel_size: 1 3373 | stride: 1 3374 | engine: CUDNN 3375 | } 3376 | } 3377 | layer { 3378 | name: "conv3b_VM_L3" 3379 | type: "Convolution" 3380 | bottom: "conv2_VM_L3" 3381 | top: "conv3b_VM_L3" 3382 | param { lr_mult: 1 decay_mult: 1 } 3383 | param { lr_mult: 1 decay_mult: 0 } 3384 | convolution_param { 3385 | num_output: 32 3386 | pad: 1 3387 | kernel_size: 3 3388 | stride: 1 3389 | engine: CUDNN 3390 | } 3391 | } 3392 | layer { 3393 | name: "ReLU3b_VM_L3" 3394 | type: "ReLU" 3395 | bottom: "conv3b_VM_L3" 3396 | top: "conv3b_VM_L3" 3397 | relu_param { negative_slope: 0.1 } 3398 | } 3399 | layer { 3400 | name: "offset_corr_L3" 3401 | type: "Convolution" 3402 | bottom: "conv3b_VM_L3" 3403 | top: "offset_corr_L3" 3404 | param { lr_mult: 1 decay_mult: 1 } 3405 | param { lr_mult: 1 decay_mult: 0 } 3406 | convolution_param { 3407 | num_output: 81 3408 | pad: 0 3409 | kernel_size: 1 3410 | stride: 1 3411 | engine: CUDNN 3412 | } 3413 | } 3414 | layer { 3415 | name: "scaled_corr_L3" 3416 | type: "Eltwise" 3417 | bottom: "corr_L3" 3418 | bottom: "scalar_corr_L3" 3419 | top: "scaled_corr_L3" 3420 | eltwise_param { operation: PROD } 3421 | } 3422 | layer { 3423 | name: "adj_corr_L3" 3424 | type: "Eltwise" 3425 | bottom: "scaled_corr_L3" 3426 | bottom: "offset_corr_L3" 3427 | top: "adj_corr_L3" 3428 | eltwise_param { operation: SUM } 3429 | } 3430 | 3431 | # Feature Matching: L3 3432 | layer { 3433 | name: "conv1_D1_L3" 3434 | type: "Convolution" 3435 | bottom: "adj_corr_L3" 3436 | top: "conv1_D1_L3" 3437 | param { lr_mult: 1 decay_mult: 1 } 3438 | param { lr_mult: 1 decay_mult: 0 } 3439 | convolution_param { 3440 | num_output: 128 3441 | pad: 1 3442 | kernel_size: 3 3443 | stride: 1 3444 | engine: CUDNN 3445 | } 3446 | } 3447 | layer { 3448 | name: "ReLU1_D1_L3" 3449 | type: "ReLU" 3450 | bottom: "conv1_D1_L3" 3451 | top: "conv1_D1_L3" 3452 | relu_param { negative_slope: 0.1 } 3453 | } 3454 | layer { 3455 | name: "conv2_D1_L3" 3456 | type: "Convolution" 3457 | bottom: "conv1_D1_L3" 3458 | top: "conv2_D1_L3" 3459 | param { lr_mult: 1 decay_mult: 1 } 3460 | param { lr_mult: 1 decay_mult: 0 } 3461 | convolution_param { 3462 | num_output: 128 3463 | pad: 1 3464 | kernel_size: 3 3465 | stride: 1 3466 | engine: CUDNN 3467 | } 3468 | } 3469 | layer { 3470 | name: "ReLU2_D1_L3" 3471 | type: "ReLU" 3472 | bottom: "conv2_D1_L3" 3473 | top: "conv2_D1_L3" 3474 | relu_param { negative_slope: 0.1 } 3475 | } 3476 | layer { 3477 | name: "conv3_D1_L3" 3478 | type: "Convolution" 3479 | bottom: "conv2_D1_L3" 3480 | top: "conv3_D1_L3" 3481 | param { lr_mult: 1 decay_mult: 1 } 3482 | param { lr_mult: 1 decay_mult: 0 } 3483 | convolution_param { 3484 | num_output: 96 3485 | pad: 1 3486 | kernel_size: 3 3487 | stride: 1 3488 | engine: CUDNN 3489 | } 3490 | } 3491 | layer { 3492 | name: "ReLU3_D1_L3" 3493 | type: "ReLU" 3494 | bottom: "conv3_D1_L3" 3495 | top: "conv3_D1_L3" 3496 | relu_param { negative_slope: 0.1 } 3497 | } 3498 | layer { 3499 | name: "conv4_D1_L3" 3500 | type: "Convolution" 3501 | bottom: "conv3_D1_L3" 3502 | top: "conv4_D1_L3" 3503 | param { lr_mult: 1 decay_mult: 1 } 3504 | param { lr_mult: 1 decay_mult: 0 } 3505 | convolution_param { 3506 | num_output: 64 3507 | pad: 1 3508 | kernel_size: 3 3509 | stride: 1 3510 | engine: CUDNN 3511 | } 3512 | } 3513 | layer { 3514 | name: "ReLU4_D1_L3" 3515 | type: "ReLU" 3516 | bottom: "conv4_D1_L3" 3517 | top: "conv4_D1_L3" 3518 | relu_param { negative_slope: 0.1 } 3519 | } 3520 | layer { 3521 | name: "conv5_D1_L3" 3522 | type: "Convolution" 3523 | bottom: "conv4_D1_L3" 3524 | top: "conv5_D1_L3" 3525 | param { lr_mult: 1 decay_mult: 1 } 3526 | param { lr_mult: 1 decay_mult: 0 } 3527 | convolution_param { 3528 | num_output: 32 3529 | pad: 1 3530 | kernel_size: 3 3531 | stride: 1 3532 | engine: CUDNN 3533 | } 3534 | } 3535 | layer { 3536 | name: "ReLU5_D1_L3" 3537 | type: "ReLU" 3538 | bottom: "conv5_D1_L3" 3539 | top: "conv5_D1_L3" 3540 | relu_param { negative_slope: 0.1 } 3541 | } 3542 | layer { 3543 | name: "scaled_flow_D1_res_L3" 3544 | type: "Convolution" 3545 | bottom: "conv5_D1_L3" 3546 | top: "scaled_flow_D1_res_L3" 3547 | param { lr_mult: 1 decay_mult: 1 } 3548 | param { lr_mult: 1 decay_mult: 0 } 3549 | convolution_param { 3550 | num_output: 2 3551 | pad: 2 3552 | kernel_size: 5 3553 | stride: 1 3554 | engine: CUDNN 3555 | } 3556 | } 3557 | layer { 3558 | name: "scaled_flow_D1_L3" 3559 | type: "Eltwise" 3560 | bottom: "scaled_flow_FD_L3" 3561 | bottom: "scaled_flow_D1_res_L3" 3562 | top: "scaled_flow_D1_L3" 3563 | eltwise_param { operation: SUM } 3564 | } 3565 | 3566 | # Sub-Pixel Refinement: L3 3567 | layer { 3568 | name: "FlowUnscaling_L3_D2" 3569 | type: "Eltwise" 3570 | bottom: "scaled_flow_D1_L3" 3571 | top: "flow_D1_L3" 3572 | eltwise_param { 3573 | operation: SUM 3574 | coeff: 5.0 3575 | } 3576 | } 3577 | layer { 3578 | name: "coords_D1_L3" 3579 | type: "Eltwise" 3580 | bottom: "flow_D1_L3" 3581 | bottom: "gxy_L3" 3582 | top: "coords_D1_L3" 3583 | eltwise_param { coeff: 1 coeff: 1 } 3584 | } 3585 | layer { 3586 | name: "warped_F1_D1_L3" 3587 | type: "Warp" 3588 | bottom: "F1_L3" 3589 | bottom: "coords_D1_L3" 3590 | top: "warped_F1_D1_L3" 3591 | } 3592 | layer { 3593 | name: "F_D2_L3" 3594 | bottom: "F0_L3" 3595 | bottom: "warped_F1_D1_L3" 3596 | bottom: "scaled_flow_D1_L3" 3597 | top: "F_D2_L3" 3598 | type: "Concat" 3599 | concat_param { axis: 1 } 3600 | } 3601 | layer { 3602 | name: "conv1_D2_L3" 3603 | type: "Convolution" 3604 | bottom: "F_D2_L3" 3605 | top: "conv1_D2_L3" 3606 | param { lr_mult: 1 decay_mult: 1 } 3607 | param { lr_mult: 1 decay_mult: 0 } 3608 | convolution_param { 3609 | num_output: 128 3610 | pad: 1 3611 | kernel_size: 3 3612 | stride: 1 3613 | engine: CUDNN 3614 | } 3615 | } 3616 | layer { 3617 | name: "ReLU1_D2_L3" 3618 | type: "ReLU" 3619 | bottom: "conv1_D2_L3" 3620 | top: "conv1_D2_L3" 3621 | relu_param { negative_slope: 0.1 } 3622 | } 3623 | layer { 3624 | name: "conv2_D2_L3" 3625 | type: "Convolution" 3626 | bottom: "conv1_D2_L3" 3627 | top: "conv2_D2_L3" 3628 | param { lr_mult: 1 decay_mult: 1 } 3629 | param { lr_mult: 1 decay_mult: 0 } 3630 | convolution_param { 3631 | num_output: 128 3632 | pad: 1 3633 | kernel_size: 3 3634 | stride: 1 3635 | engine: CUDNN 3636 | } 3637 | } 3638 | layer { 3639 | name: "ReLU2_D2_L3" 3640 | type: "ReLU" 3641 | bottom: "conv2_D2_L3" 3642 | top: "conv2_D2_L3" 3643 | relu_param { negative_slope: 0.1 } 3644 | } 3645 | layer { 3646 | name: "conv3_D2_L3" 3647 | type: "Convolution" 3648 | bottom: "conv2_D2_L3" 3649 | top: "conv3_D2_L3" 3650 | param { lr_mult: 1 decay_mult: 1 } 3651 | param { lr_mult: 1 decay_mult: 0 } 3652 | convolution_param { 3653 | num_output: 96 3654 | pad: 1 3655 | kernel_size: 3 3656 | stride: 1 3657 | engine: CUDNN 3658 | } 3659 | } 3660 | layer { 3661 | name: "ReLU3_D2_L3" 3662 | type: "ReLU" 3663 | bottom: "conv3_D2_L3" 3664 | top: "conv3_D2_L3" 3665 | relu_param { negative_slope: 0.1 } 3666 | } 3667 | layer { 3668 | name: "conv4_D2_L3" 3669 | type: "Convolution" 3670 | bottom: "conv3_D2_L3" 3671 | top: "conv4_D2_L3" 3672 | param { lr_mult: 1 decay_mult: 1 } 3673 | param { lr_mult: 1 decay_mult: 0 } 3674 | convolution_param { 3675 | num_output: 64 3676 | pad: 1 3677 | kernel_size: 3 3678 | stride: 1 3679 | engine: CUDNN 3680 | } 3681 | } 3682 | layer { 3683 | name: "ReLU4_D2_L3" 3684 | type: "ReLU" 3685 | bottom: "conv4_D2_L3" 3686 | top: "conv4_D2_L3" 3687 | relu_param { negative_slope: 0.1 } 3688 | } 3689 | layer { 3690 | name: "conv5_D2_L3" 3691 | type: "Convolution" 3692 | bottom: "conv4_D2_L3" 3693 | top: "conv5_D2_L3" 3694 | param { lr_mult: 1 decay_mult: 1 } 3695 | param { lr_mult: 1 decay_mult: 0 } 3696 | convolution_param { 3697 | num_output: 32 3698 | pad: 1 3699 | kernel_size: 3 3700 | stride: 1 3701 | engine: CUDNN 3702 | } 3703 | } 3704 | layer { 3705 | name: "ReLU5_D2_L3" 3706 | type: "ReLU" 3707 | bottom: "conv5_D2_L3" 3708 | top: "conv5_D2_L3" 3709 | relu_param { negative_slope: 0.1 } 3710 | } 3711 | layer { 3712 | name: "scaled_flow_D2_res_L3" 3713 | type: "Convolution" 3714 | bottom: "conv5_D2_L3" 3715 | top: "scaled_flow_D2_res_L3" 3716 | param { lr_mult: 1 decay_mult: 1 } 3717 | param { lr_mult: 1 decay_mult: 0 } 3718 | convolution_param { 3719 | num_output: 2 3720 | pad: 2 3721 | kernel_size: 5 3722 | stride: 1 3723 | engine: CUDNN 3724 | } 3725 | } 3726 | layer { 3727 | name: "scaled_flow_D2_L3" 3728 | type: "Eltwise" 3729 | bottom: "scaled_flow_D1_L3" 3730 | bottom: "scaled_flow_D2_res_L3" 3731 | top: "scaled_flow_D2_L3" 3732 | eltwise_param { operation: SUM } 3733 | } 3734 | 3735 | # Flow Regularization: L3 3736 | layer { 3737 | name: "slice_scaled_flow_D_L3" 3738 | type: "Slice" 3739 | bottom: "scaled_flow_D2_L3" 3740 | top: "scaled_flow_D_L3_x" 3741 | top: "scaled_flow_D_L3_y" 3742 | slice_param { axis: 1 slice_point: 1 } 3743 | } 3744 | layer { 3745 | name: "reshaped_scaled_flow_D_L3_x" 3746 | type: "Im2col" 3747 | bottom: "scaled_flow_D_L3_x" 3748 | top: "reshaped_scaled_flow_D_L3_x" 3749 | convolution_param { pad: 2 kernel_size: 5 stride: 1 } 3750 | } 3751 | layer { 3752 | name: "reshaped_scaled_flow_D_L3_y" 3753 | type: "Im2col" 3754 | bottom: "scaled_flow_D_L3_y" 3755 | top: "reshaped_scaled_flow_D_L3_y" 3756 | convolution_param { pad: 2 kernel_size: 5 stride: 1 } 3757 | } 3758 | layer { 3759 | name: "mean_scaled_flow_D_L3_x" 3760 | type: "Reduction" 3761 | bottom: "scaled_flow_D_L3_x" 3762 | top: "mean_scaled_flow_D_L3_x" 3763 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 3764 | } 3765 | layer { 3766 | name: "scaled_flow_D_nomean_L3_x" 3767 | type: "Bias" 3768 | bottom: "scaled_flow_D_L3_x" 3769 | bottom: "mean_scaled_flow_D_L3_x" 3770 | top: "scaled_flow_D_nomean_L3_x" 3771 | bias_param { axis: 0 } 3772 | } 3773 | layer { 3774 | name: "mean_scaled_flow_D_L3_y" 3775 | type: "Reduction" 3776 | bottom: "scaled_flow_D_L3_y" 3777 | top: "mean_scaled_flow_D_L3_y" 3778 | reduction_param { operation: MEAN axis: 1 coeff: -1 } 3779 | } 3780 | layer { 3781 | name: "scaled_flow_D_nomean_L3_y" 3782 | type: "Bias" 3783 | bottom: "scaled_flow_D_L3_y" 3784 | bottom: "mean_scaled_flow_D_L3_y" 3785 | top: "scaled_flow_D_nomean_L3_y" 3786 | bias_param { axis: 0 } 3787 | } 3788 | layer { 3789 | name: "FlowUnscaling_L3_R" 3790 | type: "Eltwise" 3791 | bottom: "scaled_flow_D2_L3" 3792 | top: "flow_D2_L3" 3793 | eltwise_param { 3794 | operation: SUM 3795 | coeff: 5.0 3796 | } 3797 | } 3798 | layer { 3799 | name: "coords_D2_L3" 3800 | type: "Eltwise" 3801 | bottom: "flow_D2_L3" 3802 | bottom: "gxy_L3" 3803 | top: "coords_D2_L3" 3804 | eltwise_param { coeff: 1 coeff: 1 } 3805 | } 3806 | layer { 3807 | name: "warped_img1_aug_L3" 3808 | type: "Warp" 3809 | bottom: "img1_aug_L3" 3810 | bottom: "coords_D2_L3" 3811 | top: "warped_img1_aug_L3" 3812 | } 3813 | layer { 3814 | name: "img_diff_L3" 3815 | type: "Eltwise" 3816 | bottom: "img0_aug_L3" 3817 | bottom: "warped_img1_aug_L3" 3818 | top: "img_diff_L3" 3819 | eltwise_param { 3820 | operation: SUM 3821 | coeff: 1.0 3822 | coeff: -1.0 3823 | } 3824 | } 3825 | layer { 3826 | name: "channelNorm_L3" 3827 | type: "ChannelNorm" 3828 | bottom: "img_diff_L3" 3829 | top: "channelNorm_L3" 3830 | } 3831 | layer { 3832 | name: "F0_128_L3" 3833 | type: "Convolution" 3834 | bottom: "F0_L3" 3835 | top: "F0_128_L3" 3836 | param { lr_mult: 1 decay_mult: 1 } 3837 | param { lr_mult: 1 decay_mult: 0 } 3838 | convolution_param { 3839 | num_output: 128 3840 | pad: 0 3841 | kernel_size: 1 3842 | stride: 1 3843 | engine: CUDNN 3844 | } 3845 | } 3846 | layer { 3847 | name: "relu_F0_128_L3" 3848 | type: "ReLU" 3849 | bottom: "F0_128_L3" 3850 | top: "F0_128_L3" 3851 | relu_param { negative_slope: 0.1 } 3852 | } 3853 | layer { 3854 | name: "concat_F0_R_L3" 3855 | type: "Concat" 3856 | bottom: "channelNorm_L3" 3857 | bottom: "scaled_flow_D_nomean_L3_x" 3858 | bottom: "scaled_flow_D_nomean_L3_y" 3859 | bottom: "F0_128_L3" 3860 | top: "concat_F0_R_L3" 3861 | concat_param { axis: 1 } 3862 | } 3863 | layer { 3864 | name: "conv1_R_L3" 3865 | type: "Convolution" 3866 | bottom: "concat_F0_R_L3" 3867 | top: "conv1_R_L3" 3868 | param { lr_mult: 1 decay_mult: 1 } 3869 | param { lr_mult: 1 decay_mult: 0 } 3870 | convolution_param { 3871 | num_output: 128 3872 | pad: 1 3873 | kernel_size: 3 3874 | stride: 1 3875 | engine: CUDNN 3876 | } 3877 | } 3878 | layer { 3879 | name: "relu_conv1_R_L3" 3880 | type: "ReLU" 3881 | bottom: "conv1_R_L3" 3882 | top: "conv1_R_L3" 3883 | relu_param { negative_slope: 0.1 } 3884 | } 3885 | layer { 3886 | name: "conv2_R_L3" 3887 | type: "Convolution" 3888 | bottom: "conv1_R_L3" 3889 | top: "conv2_R_L3" 3890 | param { lr_mult: 1 decay_mult: 1 } 3891 | param { lr_mult: 1 decay_mult: 0 } 3892 | convolution_param { 3893 | num_output: 128 3894 | pad: 1 3895 | kernel_size: 3 3896 | stride: 1 3897 | engine: CUDNN 3898 | } 3899 | } 3900 | layer { 3901 | name: "relu_conv2_R_L3" 3902 | type: "ReLU" 3903 | bottom: "conv2_R_L3" 3904 | top: "conv2_R_L3" 3905 | relu_param { negative_slope: 0.1 } 3906 | } 3907 | layer { 3908 | name: "conv3_R_L3" 3909 | type: "Convolution" 3910 | bottom: "conv2_R_L3" 3911 | top: "conv3_R_L3" 3912 | param { lr_mult: 1 decay_mult: 1 } 3913 | param { lr_mult: 1 decay_mult: 0 } 3914 | convolution_param { 3915 | num_output: 64 3916 | pad: 1 3917 | kernel_size: 3 3918 | stride: 1 3919 | engine: CUDNN 3920 | } 3921 | } 3922 | layer { 3923 | name: "relu_conv3_R_L3" 3924 | type: "ReLU" 3925 | bottom: "conv3_R_L3" 3926 | top: "conv3_R_L3" 3927 | relu_param { negative_slope: 0.1 } 3928 | } 3929 | layer { 3930 | name: "conv4_R_L3" 3931 | type: "Convolution" 3932 | bottom: "conv3_R_L3" 3933 | top: "conv4_R_L3" 3934 | param { lr_mult: 1 decay_mult: 1 } 3935 | param { lr_mult: 1 decay_mult: 0 } 3936 | convolution_param { 3937 | num_output: 64 3938 | pad: 1 3939 | kernel_size: 3 3940 | stride: 1 3941 | engine: CUDNN 3942 | } 3943 | } 3944 | layer { 3945 | name: "relu_conv4_R_L3" 3946 | type: "ReLU" 3947 | bottom: "conv4_R_L3" 3948 | top: "conv4_R_L3" 3949 | relu_param { negative_slope: 0.1 } 3950 | } 3951 | layer { 3952 | name: "conv5_R_L3" 3953 | type: "Convolution" 3954 | bottom: "conv4_R_L3" 3955 | top: "conv5_R_L3" 3956 | param { lr_mult: 1 decay_mult: 1 } 3957 | param { lr_mult: 1 decay_mult: 0 } 3958 | convolution_param { 3959 | num_output: 32 3960 | pad: 1 3961 | kernel_size: 3 3962 | stride: 1 3963 | engine: CUDNN 3964 | } 3965 | } 3966 | layer { 3967 | name: "relu_conv5_R_L3" 3968 | type: "ReLU" 3969 | bottom: "conv5_R_L3" 3970 | top: "conv5_R_L3" 3971 | relu_param { negative_slope: 0.1 } 3972 | } 3973 | layer { 3974 | name: "conv6_R_L3" 3975 | type: "Convolution" 3976 | bottom: "conv5_R_L3" 3977 | top: "conv6_R_L3" 3978 | param { lr_mult: 1 decay_mult: 1 } 3979 | param { lr_mult: 1 decay_mult: 0 } 3980 | convolution_param { 3981 | num_output: 32 3982 | pad: 1 3983 | kernel_size: 3 3984 | stride: 1 3985 | engine: CUDNN 3986 | } 3987 | } 3988 | layer { 3989 | name: "relu_conv6_R_L3" 3990 | type: "ReLU" 3991 | bottom: "conv6_R_L3" 3992 | top: "conv6_R_L3" 3993 | relu_param { negative_slope: 0.1 } 3994 | } 3995 | layer { 3996 | name: "distH_R_L3" 3997 | type: "Convolution" 3998 | bottom: "conv6_R_L3" 3999 | top: "distH_R_L3" 4000 | param { lr_mult: 1 decay_mult: 1 } 4001 | param { lr_mult: 0 } 4002 | convolution_param { 4003 | num_output: 25 4004 | pad_h: 2 4005 | kernel_h: 5 4006 | stride_h: 1 4007 | pad_w: 0 4008 | kernel_w: 1 4009 | stride_w: 1 4010 | engine: CUDNN 4011 | } 4012 | } 4013 | layer { 4014 | name: "distW_R_L3" 4015 | type: "Convolution" 4016 | bottom: "distH_R_L3" 4017 | top: "distW_R_L3" 4018 | param { lr_mult: 1 decay_mult: 1 } 4019 | param { lr_mult: 0 } 4020 | convolution_param { 4021 | num_output: 25 4022 | pad_h: 0 4023 | kernel_h: 1 4024 | stride_h: 1 4025 | pad_w: 2 4026 | kernel_w: 5 4027 | stride_w: 1 4028 | engine: CUDNN 4029 | } 4030 | } 4031 | layer { 4032 | name: "neg_sq_dist_R_L3" 4033 | type: "NegSquare" 4034 | bottom: "distW_R_L3" 4035 | top: "neg_sq_dist_R_L3" 4036 | } 4037 | layer { 4038 | name: "exp_kernel_R_L3" 4039 | type: "ExpMax" 4040 | bottom: "neg_sq_dist_R_L3" 4041 | top: "exp_kernel_R_L3" 4042 | } 4043 | layer { 4044 | name: "sum_exp_kernel_R_L3" 4045 | type: "Convolution" 4046 | bottom: "exp_kernel_R_L3" 4047 | top: "sum_exp_kernel_R_L3" 4048 | param { lr_mult: 0 } 4049 | param { lr_mult: 0 } 4050 | convolution_param { 4051 | num_output: 1 4052 | kernel_size: 1 4053 | weight_filler { type: "constant" value: 1 } 4054 | } 4055 | } 4056 | layer { 4057 | name: "divisor_R_L3" 4058 | bottom: "sum_exp_kernel_R_L3" 4059 | top: "divisor_R_L3" 4060 | type: "Power" 4061 | power_param { power: -1 scale: 1 shift: 0 } 4062 | } 4063 | layer { 4064 | name: "f-lconv_L3_x" 4065 | type: "Eltwise" 4066 | bottom: "reshaped_scaled_flow_D_L3_x" 4067 | bottom: "exp_kernel_R_L3" 4068 | top: "f-lconv_L3_x" 4069 | eltwise_param { operation: PROD } 4070 | } 4071 | layer { 4072 | name: "scaled_flow0_R_L3_x" 4073 | type: "Convolution" 4074 | bottom: "f-lconv_L3_x" 4075 | top: "scaled_flow0_R_L3_x" 4076 | param { lr_mult: 0 } 4077 | param { lr_mult: 0 } 4078 | convolution_param { 4079 | num_output: 1 4080 | kernel_size: 1 4081 | weight_filler { type: "constant" value: 1 } 4082 | } 4083 | } 4084 | layer { 4085 | name: "scaled_flow1_R_L3_x" 4086 | type: "Eltwise" 4087 | bottom: "scaled_flow0_R_L3_x" 4088 | bottom: "divisor_R_L3" 4089 | top: "scaled_flow1_R_L3_x" 4090 | eltwise_param { 4091 | operation: PROD 4092 | } 4093 | } 4094 | layer { 4095 | name: "f-lconv_L3_y" 4096 | type: "Eltwise" 4097 | bottom: "reshaped_scaled_flow_D_L3_y" 4098 | bottom: "exp_kernel_R_L3" 4099 | top: "f-lconv_L3_y" 4100 | eltwise_param { operation: PROD } 4101 | } 4102 | layer { 4103 | name: "scaled_flow0_R_L3_y" 4104 | type: "Convolution" 4105 | bottom: "f-lconv_L3_y" 4106 | top: "scaled_flow0_R_L3_y" 4107 | param { lr_mult: 0 } 4108 | param { lr_mult: 0 } 4109 | convolution_param { 4110 | num_output: 1 4111 | kernel_size: 1 4112 | weight_filler { type: "constant" value: 1 } 4113 | } 4114 | } 4115 | layer { 4116 | name: "scaled_flow1_R_L3_y" 4117 | type: "Eltwise" 4118 | bottom: "scaled_flow0_R_L3_y" 4119 | bottom: "divisor_R_L3" 4120 | top: "scaled_flow1_R_L3_y" 4121 | eltwise_param { 4122 | operation: PROD 4123 | } 4124 | } 4125 | layer { 4126 | name: "scaled_flow_R_L3" 4127 | bottom: "scaled_flow1_R_L3_x" 4128 | bottom: "scaled_flow1_R_L3_y" 4129 | top: "scaled_flow_R_L3" 4130 | type: "Concat" 4131 | concat_param { axis: 1 } 4132 | } 4133 | 4134 | ####################################### 4135 | layer { 4136 | name: "unresized_flow" 4137 | type: "Eltwise" 4138 | bottom: "scaled_flow_R_L3" 4139 | top: "unresized_flow" 4140 | eltwise_param { operation: SUM coeff: 20.0 } 4141 | propagate_down: false 4142 | } 4143 | layer { 4144 | name: "Resample_predict_flow" 4145 | type: "Resample" 4146 | bottom: "unresized_flow" 4147 | top: "predict_flow_resize" 4148 | resample_param { 4149 | width: $TARGET_WIDTH 4150 | height: $TARGET_HEIGHT 4151 | type: LINEAR 4152 | antialias: true 4153 | } 4154 | } 4155 | layer { 4156 | name: "ScaleMag_predict_flow" 4157 | type: "Convolution" 4158 | bottom: "predict_flow_resize" 4159 | top: "predict_flow_final" 4160 | convolution_param { 4161 | num_output: 2 4162 | pad: 0 4163 | kernel_size: 1 4164 | stride: 1 4165 | weight_filler { 4166 | type: "diagonal" 4167 | diag_val: $SCALE_WIDTH 4168 | diag_val: $SCALE_HEIGHT 4169 | } 4170 | bias_filler { 4171 | type: "constant" 4172 | } 4173 | } 4174 | } 4175 | layer { 4176 | name: "FLOWriter" 4177 | type: "FLOWriter" 4178 | bottom: "predict_flow_final" 4179 | writer_param { 4180 | folder: $OUTFOLDER 4181 | prefix: $CNN 4182 | suffix: "" 4183 | scale: 1.0 4184 | } 4185 | } 4186 | -------------------------------------------------------------------------------- /models/testing/test_batch.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os, sys 3 | import subprocess 4 | from math import ceil 5 | 6 | caffe_bin = 'bin/caffe.bin' 7 | img_size_bin = 'bin/get_image_size' 8 | 9 | template = './deploy_MODEL.prototxt' # MODEL = LiteFlowNet3-ft-sintel or LiteFlowNet3-S-ft-sintel or LiteFlowNet3-ft-kitti or LiteFlowNet3-S-ft-kitti 10 | cnn_model = 'MODEL' 11 | 12 | # ========================================================= 13 | def get_image_size(filename): 14 | global img_size_bin 15 | dim_list = [int(dimstr) for dimstr in str(subprocess.check_output([img_size_bin, filename])).split(',')] 16 | if not len(dim_list) == 2: 17 | print('Could not determine size of image %s' % filename) 18 | sys.exit(1) 19 | return dim_list 20 | 21 | 22 | def sizes_equal(size1, size2): 23 | return size1[0] == size2[0] and size1[1] == size2[1] 24 | 25 | 26 | def check_image_lists(lists): 27 | images = [[], []] 28 | 29 | with open(lists[0], 'r') as f: 30 | images[0] = [line.strip() for line in f.readlines() if len(line.strip()) > 0] 31 | with open(lists[1], 'r') as f: 32 | images[1] = [line.strip() for line in f.readlines() if len(line.strip()) > 0] 33 | 34 | if len(images[0]) != len(images[1]): 35 | print("Unequal amount of images in the given lists (%d vs. %d)" % (len(images[0]), len(images[1]))) 36 | sys.exit(1) 37 | 38 | if not os.path.isfile(images[0][0]): 39 | print('Image %s not found' % images[0][0]) 40 | sys.exit(1) 41 | 42 | base_size = get_image_size(images[0][0]) 43 | 44 | return base_size[0], base_size[1], len(images[0]) 45 | 46 | 47 | my_dir = os.path.dirname(os.path.realpath(__file__)) 48 | os.chdir(my_dir) 49 | 50 | if not (os.path.isfile(caffe_bin) and os.path.isfile(img_size_bin)): 51 | print('Caffe tool binaries not found. Did you compile caffe with tools (make all tools)?') 52 | sys.exit(1) 53 | 54 | img_files = sys.argv[1:] 55 | using_lists = False 56 | list_length = 1 57 | 58 | if img_files[0][-4:].lower() == '.txt': 59 | print("Checking the images in your lists...") 60 | (width, height, list_length) = check_image_lists(img_files) 61 | using_lists = True 62 | print("Done.") 63 | else: 64 | print("Image files: " + str(img_files)) 65 | 66 | # Check images 67 | 68 | for img_file in img_files: 69 | if not os.path.isfile(img_file): 70 | print('Image %s not found' % img_file) 71 | sys.exit(1) 72 | 73 | 74 | # Get image sizes and check 75 | img_sizes = [get_image_size(img_file) for img_file in img_files] 76 | 77 | print("Image sizes: " + str(img_sizes)) 78 | 79 | if not sizes_equal(img_sizes[0], img_sizes[1]): 80 | print('Images do not have the same size.') 81 | sys.exit(1) 82 | 83 | width = img_sizes[0][0] 84 | height = img_sizes[0][1] 85 | 86 | # Prepare prototxt 87 | subprocess.call('mkdir -p tmp', shell=True) 88 | 89 | if not using_lists: 90 | with open('tmp/img1.txt', "w") as tfile: 91 | tfile.write("%s\n" % img_files[0]) 92 | 93 | with open('tmp/img2.txt', "w") as tfile: 94 | tfile.write("%s\n" % img_files[1]) 95 | else: 96 | subprocess.call(['cp', img_files[0], 'tmp/img1.txt']) 97 | subprocess.call(['cp', img_files[1], 'tmp/img2.txt']) 98 | 99 | divisor = 32. 100 | adapted_width = ceil(width/divisor) * divisor 101 | adapted_height = ceil(height/divisor) * divisor 102 | rescale_coeff_x = width / adapted_width 103 | rescale_coeff_y = height / adapted_height 104 | 105 | replacement_list = { 106 | '$ADAPTED_WIDTH': ('%d' % adapted_width), 107 | '$ADAPTED_HEIGHT': ('%d' % adapted_height), 108 | '$TARGET_WIDTH': ('%d' % width), 109 | '$TARGET_HEIGHT': ('%d' % height), 110 | '$SCALE_WIDTH': ('%.8f' % rescale_coeff_x), 111 | '$SCALE_HEIGHT': ('%.8f' % rescale_coeff_y), 112 | '$OUTFOLDER': ('%s' % '"' + img_files[2] + '"'), 113 | '$CNN': ('%s' % '"' + cnn_model + '-"') 114 | } 115 | 116 | proto = '' 117 | with open(template, "r") as tfile: 118 | proto = tfile.read() 119 | 120 | for r in replacement_list: 121 | proto = proto.replace(r, replacement_list[r]) 122 | 123 | with open('tmp/deploy.prototxt', "w") as tfile: 124 | tfile.write(proto) 125 | 126 | # Run caffe 127 | 128 | args = [caffe_bin, 'test', '-model', 'tmp/deploy.prototxt', 129 | '-weights', '../trained/' + cnn_model + '.caffemodel', 130 | '-iterations', str(list_length), 131 | '-gpu', '0'] 132 | 133 | cmd = str.join(' ', args) 134 | print('Executing %s' % cmd) 135 | 136 | subprocess.call(args) 137 | 138 | print('\nThe resulting FLOW is stored in CNN-NNNNNNN.flo') 139 | -------------------------------------------------------------------------------- /models/testing/test_iter.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os, sys 3 | import subprocess 4 | from math import ceil 5 | 6 | caffe_bin = 'bin/caffe.bin' 7 | img_size_bin = 'bin/get_image_size' 8 | 9 | template = './deploy_MODEL.prototxt' # MODEL = LiteFlowNet3-ft-sintel or LiteFlowNet3-S-ft-sintel or LiteFlowNet3-ft-kitti or LiteFlowNet3-S-ft-kitti 10 | cnn_model = 'MODEL' 11 | 12 | # ========================================================= 13 | def get_image_size(filename): 14 | global img_size_bin 15 | dim_list = [int(dimstr) for dimstr in str(subprocess.check_output([img_size_bin, filename])).split(',')] 16 | if not len(dim_list) == 2: 17 | print('Could not determine size of image %s' % filename) 18 | sys.exit(1) 19 | return dim_list 20 | 21 | 22 | def sizes_equal(size1, size2): 23 | return size1[0] == size2[0] and size1[1] == size2[1] 24 | 25 | 26 | def check_image_lists(lists): 27 | images = [[], []] 28 | 29 | with open(lists[0], 'r') as f: 30 | images[0] = [line.strip() for line in f.readlines() if len(line.strip()) > 0] 31 | with open(lists[1], 'r') as f: 32 | images[1] = [line.strip() for line in f.readlines() if len(line.strip()) > 0] 33 | 34 | if len(images[0]) != len(images[1]): 35 | print("Unequal amount of images in the given lists (%d vs. %d)" % (len(images[0]), len(images[1]))) 36 | sys.exit(1) 37 | 38 | if not os.path.isfile(images[0][0]): 39 | print('Image %s not found' % images[0][0]) 40 | sys.exit(1) 41 | 42 | base_size = get_image_size(images[0][0]) 43 | 44 | for idx in range(len(images[0])): 45 | print("Checking image pair %d of %d" % (idx+1, len(images[0]))) 46 | img1 = images[0][idx] 47 | img2 = images[1][idx] 48 | 49 | if not os.path.isfile(img1): 50 | print('Image %s not found' % img1) 51 | sys.exit(1) 52 | 53 | if not os.path.isfile(img2): 54 | print('Image %s not found' % img2) 55 | sys.exit(1) 56 | 57 | img1_size = get_image_size(img1) 58 | img2_size = get_image_size(img2) 59 | 60 | if not (sizes_equal(base_size, img1_size) and sizes_equal(base_size, img2_size)): 61 | print('The images do not all have the same size. (Images: %s or %s vs. %s)\n Please use the pair-mode.' % (img1, img2, images[0][idx])) 62 | sys.exit(1) 63 | 64 | return base_size[0], base_size[1], len(images[0]) 65 | 66 | my_dir = os.path.dirname(os.path.realpath(__file__)) 67 | os.chdir(my_dir) 68 | 69 | if not (os.path.isfile(caffe_bin) and os.path.isfile(img_size_bin)): 70 | print('Caffe tool binaries not found. Did you compile caffe with tools (make all tools)?') 71 | sys.exit(1) 72 | 73 | img_files = sys.argv[1:] 74 | print("Image files: " + str(img_files)) 75 | 76 | 77 | # Frame-by-frame processing 78 | images = [[], []] 79 | 80 | with open(img_files[0], 'r') as f: 81 | images[0] = [line.strip() for line in f.readlines() if len(line.strip()) > 0] 82 | with open(img_files[1], 'r') as f: 83 | images[1] = [line.strip() for line in f.readlines() if len(line.strip()) > 0] 84 | 85 | for idx in reversed(range(len(images[0]))): 86 | 87 | img1_size = get_image_size(images[0][idx]) 88 | img2_size = get_image_size(images[1][idx]) 89 | 90 | if not (sizes_equal(img1_size, img2_size)): 91 | print('The images do not have the same size. (Images: %s or %s vs. %s)\n Please use the pair-mode.' % (img1, img2, images[0][idx])) 92 | sys.exit(1) 93 | 94 | width = img1_size[0] 95 | height = img1_size[1] 96 | 97 | # Prepare prototxt 98 | subprocess.call('mkdir -p tmp', shell=True) 99 | 100 | with open('tmp/img1.txt', "w") as tfile: 101 | tfile.write("%s\n" % images[0][idx]) 102 | 103 | with open('tmp/img2.txt', "w") as tfile: 104 | tfile.write("%s\n" % images[1][idx]) 105 | 106 | 107 | divisor = 32. 108 | adapted_width = ceil(width/divisor) * divisor 109 | adapted_height = ceil(height/divisor) * divisor 110 | rescale_coeff_x = width / adapted_width 111 | rescale_coeff_y = height / adapted_height 112 | 113 | replacement_list = { 114 | '$ADAPTED_WIDTH': ('%d' % adapted_width), 115 | '$ADAPTED_HEIGHT': ('%d' % adapted_height), 116 | '$TARGET_WIDTH': ('%d' % width), 117 | '$TARGET_HEIGHT': ('%d' % height), 118 | '$SCALE_WIDTH': ('%.8f' % rescale_coeff_x), 119 | '$SCALE_HEIGHT': ('%.8f' % rescale_coeff_y), 120 | '$OUTFOLDER': ('%s' % '"' + img_files[2] + '"'), 121 | '$CNN': ('%s' % '"' + cnn_model + '-"'), 122 | } 123 | 124 | proto = '' 125 | with open(template, "r") as tfile: 126 | proto = tfile.read() 127 | 128 | for r in replacement_list: 129 | proto = proto.replace(r, replacement_list[r]) 130 | 131 | with open('tmp/deploy.prototxt', "w") as tfile: 132 | tfile.write(proto) 133 | 134 | # Run caffe 135 | args = [caffe_bin, 'test', '-model', 'tmp/deploy.prototxt', 136 | '-weights', '../trained' + cnn_model + '.caffemodel', 137 | '-iterations', str(1), 138 | '-gpu', '0'] 139 | 140 | cmd = str.join(' ', args) 141 | print('Executing %s' % cmd) 142 | subprocess.call(args) 143 | 144 | if idx > 0: 145 | os.rename(img_files[2] + '/' + cnn_model + '-0000000.flo', img_files[2] + '/' + cnn_model +'-' + '{0:07d}'.format(idx) + '.flo') 146 | 147 | 148 | print('\nThe resulting FLOW is stored in CNN-NNNNNNN.flo') 149 | --------------------------------------------------------------------------------