├── .gitignore ├── LICENSE ├── README.md ├── assets ├── freiburg_matches.gif ├── freiburg_sequence │ ├── 1341847980.722988.png │ ├── 1341847981.726650.png │ ├── 1341847982.730674.png │ ├── 1341847983.738736.png │ ├── 1341847984.743352.png │ ├── 1341847985.746954.png │ ├── 1341847986.762616.png │ ├── 1341847987.758741.png │ ├── 1341847988.769740.png │ ├── 1341847989.802890.png │ ├── 1341847990.810771.png │ ├── 1341847991.814748.png │ ├── 1341847992.818723.png │ ├── 1341847993.826735.png │ ├── 1341847994.866828.png │ ├── 1341847995.870641.png │ └── 1341847996.874766.png ├── indoor_evaluation.png ├── indoor_matches.png ├── magicleap.png ├── megadepth_train_scenes.txt ├── megadepth_validation_scenes.txt ├── outdoor_matches.png ├── phototourism_sample_images │ ├── london_bridge_19481797_2295892421.jpg │ ├── london_bridge_49190386_5209386933.jpg │ ├── london_bridge_78916675_4568141288.jpg │ ├── london_bridge_94185272_3874562886.jpg │ ├── piazza_san_marco_06795901_3725050516.jpg │ ├── piazza_san_marco_15148634_5228701572.jpg │ ├── piazza_san_marco_18627786_5929294590.jpg │ ├── piazza_san_marco_43351518_2659980686.jpg │ ├── piazza_san_marco_58751010_4849458397.jpg │ ├── st_pauls_cathedral_30776973_2635313996.jpg │ ├── st_pauls_cathedral_37347628_10902811376.jpg │ ├── united_states_capitol_26757027_6717084061.jpg │ └── united_states_capitol_98169888_3347710852.jpg ├── phototourism_sample_pairs.txt ├── phototourism_test_pairs.txt ├── phototourism_test_pairs_original.txt ├── scannet_sample_images │ ├── scene0711_00_frame-001680.jpg │ ├── scene0711_00_frame-001995.jpg │ ├── scene0713_00_frame-001320.jpg │ ├── scene0713_00_frame-002025.jpg │ ├── scene0721_00_frame-000375.jpg │ ├── scene0721_00_frame-002745.jpg │ ├── scene0722_00_frame-000045.jpg │ ├── scene0722_00_frame-000735.jpg │ ├── scene0726_00_frame-000135.jpg │ ├── scene0726_00_frame-000210.jpg │ ├── scene0737_00_frame-000930.jpg │ ├── scene0737_00_frame-001095.jpg │ ├── scene0738_00_frame-000885.jpg │ ├── scene0738_00_frame-001065.jpg │ ├── scene0743_00_frame-000000.jpg │ ├── scene0743_00_frame-001275.jpg │ ├── scene0744_00_frame-000585.jpg │ ├── scene0744_00_frame-002310.jpg │ ├── scene0747_00_frame-000000.jpg │ ├── scene0747_00_frame-001530.jpg │ ├── scene0752_00_frame-000075.jpg │ ├── scene0752_00_frame-001440.jpg │ ├── scene0755_00_frame-000120.jpg │ ├── scene0755_00_frame-002055.jpg │ ├── scene0758_00_frame-000165.jpg │ ├── scene0758_00_frame-000510.jpg │ ├── scene0768_00_frame-001095.jpg │ ├── scene0768_00_frame-003435.jpg │ ├── scene0806_00_frame-000225.jpg │ └── scene0806_00_frame-001095.jpg ├── scannet_sample_pairs_with_gt.txt ├── scannet_test_pairs_with_gt.txt ├── teaser.png ├── yfcc_test_pairs_with_gt.txt └── yfcc_test_pairs_with_gt_original.txt ├── demo_superglue.py ├── match_pairs.py ├── models ├── __init__.py ├── matching.py ├── superglue.py ├── superpoint.py ├── utils.py └── weights │ ├── superglue_indoor.pth │ ├── superglue_outdoor.pth │ └── superpoint_v1.pth └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.DS_Store 3 | *.swp 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | SUPERGLUE: LEARNING FEATURE MATCHING WITH GRAPH NEURAL NETWORKS 2 | SOFTWARE LICENSE AGREEMENT 3 | ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY 4 | 5 | BY USING OR DOWNLOADING THE SOFTWARE, YOU ARE AGREEING TO THE TERMS OF THIS LICENSE AGREEMENT. IF YOU DO NOT AGREE WITH THESE TERMS, YOU MAY NOT USE OR DOWNLOAD THE SOFTWARE. 6 | 7 | This is a license agreement ("Agreement") between your academic institution or non-profit organization or self (called "Licensee" or "You" in this Agreement) and Magic Leap, Inc. (called "Licensor" in this Agreement). All rights not specifically granted to you in this Agreement are reserved for Licensor. 8 | 9 | RESERVATION OF OWNERSHIP AND GRANT OF LICENSE: 10 | Licensor retains exclusive ownership of any copy of the Software (as defined below) licensed under this Agreement and hereby grants to Licensee a personal, non-exclusive, non-transferable license to use the Software for noncommercial research purposes, without the right to sublicense, pursuant to the terms and conditions of this Agreement. As used in this Agreement, the term "Software" means (i) the actual copy of all or any portion of code for program routines made accessible to Licensee by Licensor pursuant to this Agreement, inclusive of backups, updates, and/or merged copies permitted hereunder or subsequently supplied by Licensor, including all or any file structures, programming instructions, user interfaces and screen formats and sequences as well as any and all documentation and instructions related to it, and (ii) all or any derivatives and/or modifications created or made by You to any of the items specified in (i). 11 | 12 | CONFIDENTIALITY: Licensee acknowledges that the Software is proprietary to Licensor, and as such, Licensee agrees to receive all such materials in confidence and use the Software only in accordance with the terms of this Agreement. Licensee agrees to use reasonable effort to protect the Software from unauthorized use, reproduction, distribution, or publication. 13 | 14 | COPYRIGHT: The Software is owned by Licensor and is protected by United States copyright laws and applicable international treaties and/or conventions. 15 | 16 | PERMITTED USES: The Software may be used for your own noncommercial internal research purposes. You understand and agree that Licensor is not obligated to implement any suggestions and/or feedback you might provide regarding the Software, but to the extent Licensor does so, you are not entitled to any compensation related thereto. 17 | 18 | DERIVATIVES: You may create derivatives of or make modifications to the Software, however, You agree that all and any such derivatives and modifications will be owned by Licensor and become a part of the Software licensed to You under this Agreement. You may only use such derivatives and modifications for your own noncommercial internal research purposes, and you may not otherwise use, distribute or copy such derivatives and modifications in violation of this Agreement. 19 | 20 | BACKUPS: If Licensee is an organization, it may make that number of copies of the Software necessary for internal noncommercial use at a single site within its organization provided that all information appearing in or on the original labels, including the copyright and trademark notices are copied onto the labels of the copies. 21 | 22 | USES NOT PERMITTED: You may not distribute, copy or use the Software except as explicitly permitted herein. Licensee has not been granted any trademark license as part of this Agreement and may not use the name or mark "Magic Leap" or any renditions thereof without the prior written permission of Licensor. 23 | 24 | You may not sell, rent, lease, sublicense, lend, time-share or transfer, in whole or in part, or provide third parties access to prior or present versions (or any parts thereof) of the Software. 25 | 26 | ASSIGNMENT: You may not assign this Agreement or your rights hereunder without the prior written consent of Licensor. Any attempted assignment without such consent shall be null and void. 27 | 28 | TERM: The term of the license granted by this Agreement is from Licensee's acceptance of this Agreement by downloading the Software or by using the Software until terminated as provided below. 29 | 30 | The Agreement automatically terminates without notice if you fail to comply with any provision of this Agreement. Licensee may terminate this Agreement by ceasing using the Software. Upon any termination of this Agreement, Licensee will delete any and all copies of the Software. You agree that all provisions which operate to protect the proprietary rights of Licensor shall remain in force should breach occur and that the obligation of confidentiality described in this Agreement is binding in perpetuity and, as such, survives the term of the Agreement. 31 | 32 | FEE: Provided Licensee abides completely by the terms and conditions of this Agreement, there is no fee due to Licensor for Licensee's use of the Software in accordance with this Agreement. 33 | 34 | DISCLAIMER OF WARRANTIES: THE SOFTWARE IS PROVIDED "AS-IS" WITHOUT WARRANTY OF ANY KIND INCLUDING ANY WARRANTIES OF PERFORMANCE OR MERCHANTABILITY OR FITNESS FOR A PARTICULAR USE OR PURPOSE OR OF NON-INFRINGEMENT. LICENSEE BEARS ALL RISK RELATING TO QUALITY AND PERFORMANCE OF THE SOFTWARE AND RELATED MATERIALS. 35 | 36 | SUPPORT AND MAINTENANCE: No Software support or training by the Licensor is provided as part of this Agreement. 37 | 38 | EXCLUSIVE REMEDY AND LIMITATION OF LIABILITY: To the maximum extent permitted under applicable law, Licensor shall not be liable for direct, indirect, special, incidental, or consequential damages or lost profits related to Licensee's use of and/or inability to use the Software, even if Licensor is advised of the possibility of such damage. 39 | 40 | EXPORT REGULATION: Licensee agrees to comply with any and all applicable U.S. export control laws, regulations, and/or other laws related to embargoes and sanction programs administered by the Office of Foreign Assets Control. 41 | 42 | SEVERABILITY: If any provision(s) of this Agreement shall be held to be invalid, illegal, or unenforceable by a court or other tribunal of competent jurisdiction, the validity, legality and enforceability of the remaining provisions shall not in any way be affected or impaired thereby. 43 | 44 | NO IMPLIED WAIVERS: No failure or delay by Licensor in enforcing any right or remedy under this Agreement shall be construed as a waiver of any future or other exercise of such right or remedy by Licensor. 45 | 46 | GOVERNING LAW: This Agreement shall be construed and enforced in accordance with the laws of the State of Florida without reference to conflict of laws principles. You consent to the personal jurisdiction of the courts of this County and waive their rights to venue outside of Broward County, Florida. 47 | 48 | ENTIRE AGREEMENT AND AMENDMENTS: This Agreement constitutes the sole and entire agreement between Licensee and Licensor as to the matter set forth herein and supersedes any previous agreements, understandings, and arrangements between the parties relating hereto. 49 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ### Research @ Magic Leap (CVPR 2020, Oral) 4 | 5 | # SuperGlue Inference and Evaluation Demo Script 6 | 7 | ## Introduction 8 | SuperGlue is a CVPR 2020 research project done at Magic Leap. The SuperGlue network is a Graph Neural Network combined with an Optimal Matching layer that is trained to perform matching on two sets of sparse image features. This repo includes PyTorch code and pretrained weights for running the SuperGlue matching network on top of [SuperPoint](https://arxiv.org/abs/1712.07629) keypoints and descriptors. Given a pair of images, you can use this repo to extract matching features across the image pair. 9 | 10 |

11 | 12 |

13 | 14 | SuperGlue operates as a "middle-end," performing context aggregation, matching, and filtering in a single end-to-end architecture. For more details, please see: 15 | 16 | * Full paper PDF: [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://arxiv.org/abs/1911.11763). 17 | 18 | * Authors: *Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich* 19 | 20 | * Website: [psarlin.com/superglue](https://psarlin.com/superglue) for videos, slides, recent updates, and more visualizations. 21 | 22 | * `hloc`: a new toolbox for visual localization and SfM with SuperGlue, available at [cvg/Hierarchical-Localization](https://github.com/cvg/Hierarchical-Localization/). Winner of 3 CVPR 2020 competitions on localization and image matching! 23 | 24 | We provide two pre-trained weights files: an indoor model trained on ScanNet data, and an outdoor model trained on MegaDepth data. Both models are inside the [weights directory](./models/weights). By default, the demo will run the **indoor** model. 25 | 26 | ## Dependencies 27 | * Python 3 >= 3.5 28 | * PyTorch >= 1.1 29 | * OpenCV >= 3.4 (4.1.2.30 recommended for best GUI keyboard interaction, see this [note](#additional-notes)) 30 | * Matplotlib >= 3.1 31 | * NumPy >= 1.18 32 | 33 | Simply run the following command: `pip3 install numpy opencv-python torch matplotlib` 34 | 35 | ## Contents 36 | There are two main top-level scripts in this repo: 37 | 38 | 1. `demo_superglue.py` : runs a live demo on a webcam, IP camera, image directory or movie file 39 | 2. `match_pairs.py`: reads image pairs from files and dumps matches to disk (also runs evaluation if ground truth relative poses are provided) 40 | 41 | ## Live Matching Demo Script (`demo_superglue.py`) 42 | This demo runs SuperPoint + SuperGlue feature matching on an anchor image and live image. You can update the anchor image by pressing the `n` key. The demo can read image streams from a USB or IP camera, a directory containing images, or a video file. You can pass all of these inputs using the `--input` flag. 43 | 44 | ### Run the demo on a live webcam 45 | 46 | Run the demo on the default USB webcam (ID #0), running on a CUDA GPU if one is found: 47 | 48 | ```sh 49 | ./demo_superglue.py 50 | ``` 51 | 52 | Keyboard control: 53 | 54 | * `n`: select the current frame as the anchor 55 | * `e`/`r`: increase/decrease the keypoint confidence threshold 56 | * `d`/`f`: increase/decrease the match filtering threshold 57 | * `k`: toggle the visualization of keypoints 58 | * `q`: quit 59 | 60 | Run the demo on 320x240 images running on the CPU: 61 | 62 | ```sh 63 | ./demo_superglue.py --resize 320 240 --force_cpu 64 | ``` 65 | 66 | The `--resize` flag can be used to resize the input image in three ways: 67 | 68 | 1. `--resize` `width` `height` : will resize to exact `width` x `height` dimensions 69 | 2. `--resize` `max_dimension` : will resize largest input image dimension to `max_dimension` 70 | 3. `--resize` `-1` : will not resize (i.e. use original image dimensions) 71 | 72 | The default will resize images to `640x480`. 73 | 74 | ### Run the demo on a directory of images 75 | 76 | The `--input` flag also accepts a path to a directory. We provide a directory of sample images from a sequence. To run the demo on the directory of images in `freiburg_sequence/` on a headless server (will not display to the screen) and write the output visualization images to `dump_demo_sequence/`: 77 | 78 | ```sh 79 | ./demo_superglue.py --input assets/freiburg_sequence/ --output_dir dump_demo_sequence --resize 320 240 --no_display 80 | ``` 81 | 82 | You should see this output on the sample Freiburg-TUM RGBD sequence: 83 | 84 | 85 | 86 | The matches are colored by their predicted confidence in a jet colormap (Red: more confident, Blue: less confident). 87 | 88 | ### Additional useful command line parameters 89 | * Use `--image_glob` to change the image file extension (default: `*.png`, `*.jpg`, `*.jpeg`). 90 | * Use `--skip` to skip intermediate frames (default: `1`). 91 | * Use `--max_length` to cap the total number of frames processed (default: `1000000`). 92 | * Use `--show_keypoints` to visualize the detected keypoints (default: `False`). 93 | 94 | ## Run Matching+Evaluation (`match_pairs.py`) 95 | 96 | This repo also contains a script `match_pairs.py` that runs the matching from a list of image pairs. With this script, you can: 97 | 98 | * Run the matcher on a set of image pairs (no ground truth needed) 99 | * Visualize the keypoints and matches, based on their confidence 100 | * Evaluate and visualize the match correctness, if the ground truth relative poses and intrinsics are provided 101 | * Save the keypoints, matches, and evaluation results for further processing 102 | * Collate evaluation results over many pairs and generate result tables 103 | 104 | ### Matches only mode 105 | 106 | The simplest usage of this script will process the image pairs listed in a given text file and dump the keypoints and matches to compressed numpy `npz` files. We provide the challenging ScanNet pairs from the main paper in `assets/example_indoor_pairs/`. Running the following will run SuperPoint + SuperGlue on each image pair, and dump the results to `dump_match_pairs/`: 107 | 108 | ```sh 109 | ./match_pairs.py 110 | ``` 111 | 112 | The resulting `.npz` files can be read from Python as follows: 113 | 114 | ```python 115 | >>> import numpy as np 116 | >>> path = 'dump_match_pairs/scene0711_00_frame-001680_scene0711_00_frame-001995_matches.npz' 117 | >>> npz = np.load(path) 118 | >>> npz.files 119 | ['keypoints0', 'keypoints1', 'matches', 'match_confidence'] 120 | >>> npz['keypoints0'].shape 121 | (382, 2) 122 | >>> npz['keypoints1'].shape 123 | (391, 2) 124 | >>> npz['matches'].shape 125 | (382,) 126 | >>> np.sum(npz['matches']>-1) 127 | 115 128 | >>> npz['match_confidence'].shape 129 | (382,) 130 | ``` 131 | 132 | For each keypoint in `keypoints0`, the `matches` array indicates the index of the matching keypoint in `keypoints1`, or `-1` if the keypoint is unmatched. 133 | 134 | ### Visualization mode 135 | 136 | You can add the flag `--viz` to dump image outputs which visualize the matches: 137 | 138 | ```sh 139 | ./match_pairs.py --viz 140 | ``` 141 | 142 | You should see images like this inside of `dump_match_pairs/` (or something very close to it, see this [note](#a-note-on-reproducibility)): 143 | 144 | 145 | 146 | The matches are colored by their predicted confidence in a jet colormap (Red: more confident, Blue: less confident). 147 | 148 | ### Evaluation mode 149 | 150 | You can also estimate the pose using RANSAC + Essential Matrix decomposition and evaluate it if the ground truth relative poses and intrinsics are provided in the input `.txt` files. Each `.txt` file contains three key ground truth matrices: a 3x3 intrinsics matrix of image0: `K0`, a 3x3 intrinsics matrix of image1: `K1` , and a 4x4 matrix of the relative pose extrinsics `T_0to1`. 151 | 152 | To run the evaluation on the sample set of images (by default reading `assets/scannet_sample_pairs_with_gt.txt`), you can run: 153 | 154 | ```sh 155 | ./match_pairs.py --eval 156 | ``` 157 | 158 | 159 | Since you enabled `--eval`, you should see collated results printed to the terminal. For the example images provided, you should get the following numbers (or something very close to it, see this [note](#a-note-on-reproducibility)): 160 | 161 | ```txt 162 | Evaluation Results (mean over 15 pairs): 163 | AUC@5 AUC@10 AUC@20 Prec MScore 164 | 26.99 48.40 64.47 73.52 19.60 165 | ``` 166 | 167 | The resulting `.npz` files in `dump_match_pairs/` will now contain scalar values related to the evaluation, computed on the sample images provided. Here is what you should find in one of the generated evaluation files: 168 | 169 | ```python 170 | >>> import numpy as np 171 | >>> path = 'dump_match_pairs/scene0711_00_frame-001680_scene0711_00_frame-001995_evaluation.npz' 172 | >>> npz = np.load(path) 173 | >>> print(npz.files) 174 | ['error_t', 'error_R', 'precision', 'matching_score', 'num_correct', 'epipolar_errors'] 175 | ``` 176 | 177 | You can also visualize the evaluation metrics by running the following command: 178 | 179 | ```sh 180 | ./match_pairs.py --eval --viz 181 | ``` 182 | 183 | You should also now see additional images in `dump_match_pairs/` which visualize the evaluation numbers (or something very close to it, see this [note](#a-note-on-reproducibility)): 184 | 185 | 186 | 187 | The top left corner of the image shows the pose error and number of inliers, while the lines are colored by their epipolar error computed with the ground truth relative pose (red: higher error, green: lower error). 188 | 189 | ### Running on sample outdoor pairs 190 | 191 |
192 | [Click to expand] 193 | 194 | In this repo, we also provide a few challenging Phototourism pairs, so that you can re-create some of the figures from the paper. Run this script to run matching and visualization (no ground truth is provided, see this [note](#reproducing-outdoor-evaluation-final-table)) on the provided pairs: 195 | 196 | ```sh 197 | ./match_pairs.py --resize 1600 --superglue outdoor --max_keypoints 2048 --nms_radius 3 --resize_float --input_dir assets/phototourism_sample_images/ --input_pairs assets/phototourism_sample_pairs.txt --output_dir dump_match_pairs_outdoor --viz 198 | ``` 199 | 200 | You should now image pairs such as these in `dump_match_pairs_outdoor/` (or something very close to it, see this [note](#a-note-on-reproducibility)): 201 | 202 | 203 | 204 |
205 | 206 | ### Recommended settings for indoor / outdoor 207 | 208 |
209 | [Click to expand] 210 | 211 | For **indoor** images, we recommend the following settings (these are the defaults): 212 | 213 | ```sh 214 | ./match_pairs.py --resize 640 --superglue indoor --max_keypoints 1024 --nms_radius 4 215 | ``` 216 | 217 | For **outdoor** images, we recommend the following settings: 218 | 219 | ```sh 220 | ./match_pairs.py --resize 1600 --superglue outdoor --max_keypoints 2048 --nms_radius 3 --resize_float 221 | ``` 222 | 223 | You can provide your own list of pairs `--input_pairs` for images contained in `--input_dir`. Images can be resized before network inference with `--resize`. If you are re-running the same evaluation many times, you can use the `--cache` flag to reuse old computation. 224 |
225 | 226 | ### Test set pair file format explained 227 | 228 |
229 | [Click to expand] 230 | 231 | We provide the list of ScanNet test pairs in `assets/scannet_test_pairs_with_gt.txt` (with ground truth) and Phototourism test pairs `assets/phototourism_test_pairs.txt` (without ground truth) used to evaluate the matching from the paper. Each line corresponds to one pair and is structured as follows: 232 | 233 | ``` 234 | path_image_A path_image_B exif_rotationA exif_rotationB [KA_0 ... KA_8] [KB_0 ... KB_8] [T_AB_0 ... T_AB_15] 235 | ``` 236 | 237 | The `path_image_A` and `path_image_B` entries are paths to image A and B, respectively. The `exif_rotation` is an integer in the range [0, 3] that comes from the original EXIF metadata associated with the image, where, 0: no rotation, 1: 90 degree clockwise, 2: 180 degree clockwise, 3: 270 degree clockwise. If the EXIF data is not known, you can just provide a zero here and no rotation will be performed. `KA` and `KB` are the flattened `3x3` matrices of image A and image B intrinsics. `T_AB` is a flattened `4x4` matrix of the extrinsics between the pair. 238 |
239 | 240 | ### Reproducing the indoor evaluation on ScanNet 241 | 242 |
243 | [Click to expand] 244 | 245 | We provide the groundtruth for ScanNet in our format in the file `assets/scannet_test_pairs_with_gt.txt` for convenience. In order to reproduce similar tables to what was in the paper, you will need to download the dataset (we do not provide the raw test images). To download the ScanNet dataset, do the following: 246 | 247 | 1. Head to the [ScanNet](https://github.com/ScanNet/ScanNet) github repo to download the ScanNet test set (100 scenes). 248 | 2. You will need to extract the raw sensor data from the 100 `.sens` files in each scene in the test set using the [SensReader](https://github.com/ScanNet/ScanNet/tree/master/SensReader) tool. 249 | 250 | Once the ScanNet dataset is downloaded in `~/data/scannet`, you can run the following: 251 | 252 | ```sh 253 | ./match_pairs.py --input_dir ~/data/scannet --input_pairs assets/scannet_test_pairs_with_gt.txt --output_dir dump_scannet_test_results --eval 254 | ``` 255 | 256 | You should get the following table for ScanNet (or something very close to it, see this [note](#a-note-on-reproducibility)): 257 | 258 | ```txt 259 | Evaluation Results (mean over 1500 pairs): 260 | AUC@5 AUC@10 AUC@20 Prec MScore 261 | 16.12 33.76 51.79 84.37 31.14 262 | ``` 263 | 264 |
265 | 266 | ### Reproducing the outdoor evaluation on YFCC 267 | 268 |
269 | [Click to expand] 270 | 271 | We provide the groundtruth for YFCC in our format in the file `assets/yfcc_test_pairs_with_gt.txt` for convenience. In order to reproduce similar tables to what was in the paper, you will need to download the dataset (we do not provide the raw test images). To download the YFCC dataset, you can use the [OANet](https://github.com/zjhthu/OANet) repo: 272 | 273 | ```sh 274 | git clone https://github.com/zjhthu/OANet 275 | cd OANet 276 | bash download_data.sh raw_data raw_data_yfcc.tar.gz 0 8 277 | tar -xvf raw_data_yfcc.tar.gz 278 | mv raw_data/yfcc100m ~/data 279 | ``` 280 | 281 | Once the YFCC dataset is downloaded in `~/data/yfcc100m`, you can run the following: 282 | 283 | ```sh 284 | ./match_pairs.py --input_dir ~/data/yfcc100m --input_pairs assets/yfcc_test_pairs_with_gt.txt --output_dir dump_yfcc_test_results --eval --resize 1600 --superglue outdoor --max_keypoints 2048 --nms_radius 3 --resize_float 285 | ``` 286 | 287 | You should get the following table for YFCC (or something very close to it, see this [note](#a-note-on-reproducibility)): 288 | 289 | ```txt 290 | Evaluation Results (mean over 4000 pairs): 291 | AUC@5 AUC@10 AUC@20 Prec MScore 292 | 39.02 59.51 75.72 98.72 23.61 293 | ``` 294 | 295 |
296 | 297 | ### Reproducing outdoor evaluation on Phototourism 298 | 299 |
300 | [Click to expand] 301 | 302 | The Phototourism results shown in the paper were produced using similar data as the test set from the [Image Matching Challenge 2020](https://vision.uvic.ca/image-matching-challenge/), which holds the ground truth data private for the test set. We list the pairs we used in `assets/phototourism_test_pairs.txt`. To reproduce similar numbers on this test set, please submit to the challenge benchmark. While the challenge is still live, we cannot share the test set publically since we want to help maintain the integrity of the challenge. 303 | 304 |
305 | 306 | ### Correcting EXIF rotation data in YFCC and Phototourism 307 | 308 |
309 | [Click to expand] 310 | 311 | In this repo, we provide manually corrected the EXIF rotation data for the outdoor evaluations on YFCC and Phototourism. For the YFCC dataset we found 7 images with incorrect EXIF rotation flags, resulting in 148 pairs out of 4000 being corrected. For Phototourism, we found 36 images with incorrect EXIF rotation flags, resulting in 212 out of 2200 pairs being corrected. 312 | 313 | The SuperGlue paper reports the results of SuperGlue **without** the corrected rotations, while the numbers in this README are reported **with** the corrected rotations. We found that our final conclusions from the evaluation still hold with or without the corrected rotations. For backwards compatability, we included the original, uncorrected EXIF rotation data in `assets/phototourism_test_pairs_original.txt` and `assets/yfcc_test_pairs_with_gt_original.txt` respectively. 314 | 315 |
316 | 317 | ### Outdoor training / validation scene splits of MegaDepth 318 | 319 |
320 | [Click to expand] 321 | 322 | For training and validation of the outdoor model, we used scenes from the [MegaDepth dataset](http://www.cs.cornell.edu/projects/megadepth/). We provide the list of scenes used to train the outdoor model in the `assets/` directory: 323 | 324 | * Training set: `assets/megadepth_train_scenes.txt` 325 | * Validation set: `assets/megadepth_validation_scenes.txt` 326 | 327 |
328 | 329 | ### A note on reproducibility 330 | 331 |
332 | [Click to expand] 333 | 334 | After simplifying the model code and evaluation code and preparing it for release, we made some improvements and tweaks that result in slightly different numbers than what was reported in the paper. The numbers and figures reported in the README were done using Ubuntu 16.04, OpenCV 3.4.5, and PyTorch 1.1.0. Even with matching the library versions, we observed some slight differences across Mac and Ubuntu, which we believe are due to differences in OpenCV's image resize function implementation and randomization of RANSAC. 335 |
336 | 337 | ### Creating high-quality PDF visualizations and faster visualization with --fast_viz 338 | 339 |
340 | [Click to expand] 341 | 342 | When generating output images with `match_pairs.py`, the default `--viz` flag uses a Matplotlib renderer which allows for the generation of camera-ready PDF visualizations if you additionally use `--viz_extension pdf` instead of the default png extension. 343 | 344 | ``` 345 | ./match_pairs.py --viz --viz_extension pdf 346 | ``` 347 | 348 | Alternatively, you might want to save visualization images but have the generation be much faster. You can use the `--fast_viz` flag to use an OpenCV-based image renderer as follows: 349 | 350 | ``` 351 | ./match_pairs.py --viz --fast_viz 352 | ``` 353 | 354 | If you would also like an OpenCV display window to preview the results (you must use non-pdf output and use fast_fiz), simply run: 355 | 356 | ``` 357 | ./match_pairs.py --viz --fast_viz --opencv_display 358 | ``` 359 | 360 |
361 | 362 | 363 | ## BibTeX Citation 364 | If you use any ideas from the paper or code from this repo, please consider citing: 365 | 366 | ```txt 367 | @inproceedings{sarlin20superglue, 368 | author = {Paul-Edouard Sarlin and 369 | Daniel DeTone and 370 | Tomasz Malisiewicz and 371 | Andrew Rabinovich}, 372 | title = {{SuperGlue}: Learning Feature Matching with Graph Neural Networks}, 373 | booktitle = {CVPR}, 374 | year = {2020}, 375 | url = {https://arxiv.org/abs/1911.11763} 376 | } 377 | ``` 378 | 379 | ## Additional Notes 380 | * For the demo, we found that the keyboard interaction works well with OpenCV 4.1.2.30, older versions were less responsive and the newest version had a [OpenCV bug on Mac](https://stackoverflow.com/questions/60032540/opencv-cv2-imshow-is-not-working-because-of-the-qt) 381 | * We generally do not recommend to run SuperPoint+SuperGlue below 160x120 resolution (QQVGA) and above 2000x1500 382 | * We do not intend to release the SuperGlue training code. 383 | * We do not intend to release the SIFT-based or homography SuperGlue models. 384 | 385 | ## Legal Disclaimer 386 | Magic Leap is proud to provide its latest samples, toolkits, and research projects on Github to foster development and gather feedback from the spatial computing community. Use of the resources within this repo is subject to (a) the license(s) included herein, or (b) if no license is included, Magic Leap's [Developer Agreement](https://id.magicleap.com/terms/developer), which is available on our [Developer Portal](https://developer.magicleap.com/). 387 | If you need more, just ask on the [forums](https://forum.magicleap.com/hc/en-us/community/topics)! 388 | We're thrilled to be part of a well-meaning, friendly and welcoming community of millions. 389 | -------------------------------------------------------------------------------- /assets/freiburg_matches.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_matches.gif -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847980.722988.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847980.722988.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847981.726650.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847981.726650.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847982.730674.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847982.730674.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847983.738736.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847983.738736.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847984.743352.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847984.743352.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847985.746954.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847985.746954.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847986.762616.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847986.762616.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847987.758741.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847987.758741.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847988.769740.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847988.769740.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847989.802890.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847989.802890.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847990.810771.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847990.810771.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847991.814748.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847991.814748.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847992.818723.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847992.818723.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847993.826735.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847993.826735.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847994.866828.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847994.866828.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847995.870641.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847995.870641.png -------------------------------------------------------------------------------- /assets/freiburg_sequence/1341847996.874766.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/freiburg_sequence/1341847996.874766.png -------------------------------------------------------------------------------- /assets/indoor_evaluation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/indoor_evaluation.png -------------------------------------------------------------------------------- /assets/indoor_matches.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/indoor_matches.png -------------------------------------------------------------------------------- /assets/magicleap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/magicleap.png -------------------------------------------------------------------------------- /assets/megadepth_train_scenes.txt: -------------------------------------------------------------------------------- 1 | 0000 2 | 0001 3 | 0002 4 | 0003 5 | 0004 6 | 0005 7 | 0007 8 | 0008 9 | 0011 10 | 0012 11 | 0013 12 | 0015 13 | 0017 14 | 0020 15 | 0022 16 | 0023 17 | 0026 18 | 0027 19 | 0032 20 | 0033 21 | 0034 22 | 0035 23 | 0036 24 | 0037 25 | 0039 26 | 0041 27 | 0042 28 | 0043 29 | 0044 30 | 0046 31 | 0048 32 | 0049 33 | 0050 34 | 0056 35 | 0057 36 | 0060 37 | 0061 38 | 0062 39 | 0063 40 | 0065 41 | 0067 42 | 0070 43 | 0071 44 | 0076 45 | 0080 46 | 0083 47 | 0086 48 | 0087 49 | 0090 50 | 0092 51 | 0094 52 | 0095 53 | 0098 54 | 0099 55 | 0100 56 | 0101 57 | 0102 58 | 0103 59 | 0104 60 | 0105 61 | 0107 62 | 0115 63 | 0117 64 | 0122 65 | 0130 66 | 0137 67 | 0141 68 | 0143 69 | 0147 70 | 0148 71 | 0149 72 | 0150 73 | 0151 74 | 0156 75 | 0160 76 | 0162 77 | 0176 78 | 0177 79 | 0183 80 | 0189 81 | 0190 82 | 0197 83 | 0200 84 | 0209 85 | 0214 86 | 0224 87 | 0231 88 | 0235 89 | 0237 90 | 0238 91 | 0240 92 | 0243 93 | 0252 94 | 0257 95 | 0258 96 | 0265 97 | 0269 98 | 0275 99 | 0277 100 | 0281 101 | 0290 102 | 0299 103 | 0306 104 | 0307 105 | 0312 106 | 0323 107 | 0326 108 | 0327 109 | 0331 110 | 0335 111 | 0341 112 | 0348 113 | 0360 114 | 0366 115 | 0377 116 | 0380 117 | 0389 118 | 0394 119 | 0402 120 | 0406 121 | 0407 122 | 0411 123 | 0430 124 | 0446 125 | 0455 126 | 0472 127 | 0474 128 | 0476 129 | 0478 130 | 0493 131 | 0494 132 | 0496 133 | 0505 134 | 0559 135 | 0733 136 | 0860 137 | 1017 138 | 4541 139 | 5000 140 | 5001 141 | 5002 142 | 5003 143 | 5004 144 | 5005 145 | 5006 146 | 5007 147 | 5008 148 | 5009 149 | 5010 150 | 5011 151 | 5012 152 | 5013 153 | 5017 154 | -------------------------------------------------------------------------------- /assets/megadepth_validation_scenes.txt: -------------------------------------------------------------------------------- 1 | 0016 2 | 0047 3 | 0058 4 | 0064 5 | 0121 6 | 0129 7 | 0133 8 | 0168 9 | 0175 10 | 0178 11 | 0181 12 | 0185 13 | 0186 14 | 0204 15 | 0205 16 | 0212 17 | 0217 18 | 0223 19 | 0229 20 | 0271 21 | 0285 22 | 0286 23 | 0294 24 | 0303 25 | 0349 26 | 0387 27 | 0412 28 | 0443 29 | 0482 30 | 0768 31 | 1001 32 | 3346 33 | 5014 34 | 5015 35 | 5016 36 | 5018 37 | -------------------------------------------------------------------------------- /assets/outdoor_matches.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/outdoor_matches.png -------------------------------------------------------------------------------- /assets/phototourism_sample_images/london_bridge_19481797_2295892421.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/london_bridge_19481797_2295892421.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/london_bridge_49190386_5209386933.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/london_bridge_49190386_5209386933.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/london_bridge_78916675_4568141288.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/london_bridge_78916675_4568141288.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/london_bridge_94185272_3874562886.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/london_bridge_94185272_3874562886.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/piazza_san_marco_06795901_3725050516.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/piazza_san_marco_06795901_3725050516.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/piazza_san_marco_15148634_5228701572.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/piazza_san_marco_15148634_5228701572.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/piazza_san_marco_18627786_5929294590.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/piazza_san_marco_18627786_5929294590.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/piazza_san_marco_43351518_2659980686.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/piazza_san_marco_43351518_2659980686.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/piazza_san_marco_58751010_4849458397.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/piazza_san_marco_58751010_4849458397.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/st_pauls_cathedral_30776973_2635313996.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/st_pauls_cathedral_30776973_2635313996.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/st_pauls_cathedral_37347628_10902811376.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/st_pauls_cathedral_37347628_10902811376.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/united_states_capitol_26757027_6717084061.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/united_states_capitol_26757027_6717084061.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_images/united_states_capitol_98169888_3347710852.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/phototourism_sample_images/united_states_capitol_98169888_3347710852.jpg -------------------------------------------------------------------------------- /assets/phototourism_sample_pairs.txt: -------------------------------------------------------------------------------- 1 | london_bridge_78916675_4568141288.jpg london_bridge_19481797_2295892421.jpg 2 | london_bridge_94185272_3874562886.jpg london_bridge_49190386_5209386933.jpg 3 | piazza_san_marco_15148634_5228701572.jpg piazza_san_marco_06795901_3725050516.jpg 4 | piazza_san_marco_43351518_2659980686.jpg piazza_san_marco_06795901_3725050516.jpg 5 | piazza_san_marco_58751010_4849458397.jpg piazza_san_marco_18627786_5929294590.jpg 6 | st_pauls_cathedral_37347628_10902811376.jpg st_pauls_cathedral_30776973_2635313996.jpg 7 | united_states_capitol_98169888_3347710852.jpg united_states_capitol_98169888_3347710852.jpg 8 | -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0711_00_frame-001680.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0711_00_frame-001680.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0711_00_frame-001995.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0711_00_frame-001995.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0713_00_frame-001320.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0713_00_frame-001320.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0713_00_frame-002025.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0713_00_frame-002025.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0721_00_frame-000375.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0721_00_frame-000375.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0721_00_frame-002745.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0721_00_frame-002745.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0722_00_frame-000045.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0722_00_frame-000045.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0722_00_frame-000735.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0722_00_frame-000735.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0726_00_frame-000135.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0726_00_frame-000135.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0726_00_frame-000210.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0726_00_frame-000210.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0737_00_frame-000930.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0737_00_frame-000930.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0737_00_frame-001095.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0737_00_frame-001095.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0738_00_frame-000885.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0738_00_frame-000885.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0738_00_frame-001065.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0738_00_frame-001065.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0743_00_frame-000000.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0743_00_frame-000000.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0743_00_frame-001275.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0743_00_frame-001275.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0744_00_frame-000585.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0744_00_frame-000585.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0744_00_frame-002310.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0744_00_frame-002310.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0747_00_frame-000000.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0747_00_frame-000000.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0747_00_frame-001530.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0747_00_frame-001530.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0752_00_frame-000075.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0752_00_frame-000075.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0752_00_frame-001440.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0752_00_frame-001440.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0755_00_frame-000120.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0755_00_frame-000120.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0755_00_frame-002055.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0755_00_frame-002055.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0758_00_frame-000165.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0758_00_frame-000165.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0758_00_frame-000510.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0758_00_frame-000510.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0768_00_frame-001095.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0768_00_frame-001095.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0768_00_frame-003435.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0768_00_frame-003435.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0806_00_frame-000225.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0806_00_frame-000225.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_images/scene0806_00_frame-001095.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/scannet_sample_images/scene0806_00_frame-001095.jpg -------------------------------------------------------------------------------- /assets/scannet_sample_pairs_with_gt.txt: -------------------------------------------------------------------------------- 1 | scene0711_00_frame-001680.jpg scene0711_00_frame-001995.jpg 0 0 1163.45 0. 653.626 0. 1164.79 481.6 0. 0. 1. 1163.45 0. 653.626 0. 1164.79 481.6 0. 0. 1. 0.78593 -0.35128 0.50884 -1.51061 0.39215 0.91944 0.02904 -0.05367 -0.47805 0.17672 0.86037 0.056 0. 0. 0. 1. 2 | scene0713_00_frame-001320.jpg scene0713_00_frame-002025.jpg 0 0 1161.75 0. 658.042 0. 1159.11 486.467 0. 0. 1. 1161.75 0. 658.042 0. 1159.11 486.467 0. 0. 1. 0.40375 0.5041 -0.76346 1.01594 -0.56758 0.79251 0.22312 -0.24205 0.71752 0.34323 0.60609 0.05395 0. 0. 0. 1. 3 | scene0721_00_frame-000375.jpg scene0721_00_frame-002745.jpg 0 0 1165.37 0. 650.862 0. 1166.11 488.595 0. 0. 1. 1165.37 0. 650.862 0. 1166.11 488.595 0. 0. 1. 0.57186 0.27762 -0.77194 5.55343 -0.66662 0.70569 -0.24004 2.77116 0.47811 0.65186 0.58863 -1.65749 0. 0. 0. 1. 4 | scene0722_00_frame-000045.jpg scene0722_00_frame-000735.jpg 0 0 1161.04 0. 648.21 0. 1161.72 485.785 0. 0. 1. 1161.04 0. 648.21 0. 1161.72 485.785 0. 0. 1. 0.65066 0.11299 -0.75092 0.78349 -0.27579 0.95651 -0.09504 0.06852 0.70752 0.26894 0.65352 -0.4044 0. 0. 0. 1. 5 | scene0726_00_frame-000135.jpg scene0726_00_frame-000210.jpg 0 0 1165.37 0. 650.862 0. 1166.11 488.595 0. 0. 1. 1165.37 0. 650.862 0. 1166.11 488.595 0. 0. 1. 0.62595 0.64423 -0.43949 0.40691 -0.72171 0.69207 -0.01342 -0.01081 0.29551 0.32558 0.89815 -0.10968 0. 0. 0. 1. 6 | scene0737_00_frame-000930.jpg scene0737_00_frame-001095.jpg 0 0 1163.45 0. 653.626 0. 1164.79 481.6 0. 0. 1. 1163.45 0. 653.626 0. 1164.79 481.6 0. 0. 1. 0.97106 0.20815 -0.11712 0.25128 -0.12658 0.86436 0.48669 -0.8737 0.20253 -0.45778 0.86569 0.21029 0. 0. 0. 1. 7 | scene0738_00_frame-000885.jpg scene0738_00_frame-001065.jpg 0 0 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 0.64207 -0.45589 0.61637 -0.98316 0.50535 0.85627 0.10691 -0.6276 -0.57652 0.24284 0.78016 0.73601 0. 0. 0. 1. 8 | scene0743_00_frame-000000.jpg scene0743_00_frame-001275.jpg 0 0 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. -0.98671 0.01989 -0.16126 0.33601 -0.15707 0.13734 0.97799 -1.55255 0.0416 0.99033 -0.13239 1.55932 0. 0. 0. 1. 9 | scene0744_00_frame-000585.jpg scene0744_00_frame-002310.jpg 0 0 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 0.16458 0.52969 -0.83207 2.99964 -0.49288 0.77487 0.39579 -0.14204 0.85439 0.34497 0.3886 0.43063 0. 0. 0. 1. 10 | scene0747_00_frame-000000.jpg scene0747_00_frame-001530.jpg 0 0 1161.75 0. 658.042 0. 1159.11 486.467 0. 0. 1. 1161.75 0. 658.042 0. 1159.11 486.467 0. 0. 1. 0.47608 -0.36152 0.80165 -3.08857 0.73195 0.66818 -0.13337 0.33645 -0.48743 0.65027 0.58272 -0.16841 0. 0. 0. 1. 11 | scene0752_00_frame-000075.jpg scene0752_00_frame-001440.jpg 0 0 1161.75 0. 658.042 0. 1159.11 486.467 0. 0. 1. 1161.75 0. 658.042 0. 1159.11 486.467 0. 0. 1. 0.30504 0.62126 -0.72179 1.17972 -0.5394 0.73733 0.40668 -0.18845 0.78485 0.26528 0.56002 -0.03684 0. 0. 0. 1. 12 | scene0755_00_frame-000120.jpg scene0755_00_frame-002055.jpg 0 0 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 0.10156 0.41946 -0.90207 1.60366 -0.17977 0.89957 0.39806 -0.32295 0.97845 0.12173 0.16677 2.06474 0. 0. 0. 1. 13 | scene0758_00_frame-000165.jpg scene0758_00_frame-000510.jpg 0 0 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 1165.72 0. 649.095 0. 1165.74 484.765 0. 0. 1. 0.6495 0.22728 -0.7256 1.0867 -0.29881 0.9538 0.03129 -0.03638 0.69919 0.19649 0.68741 -0.00462 0. 0. 0. 1. 14 | scene0768_00_frame-001095.jpg scene0768_00_frame-003435.jpg 0 0 1165.48 0. 654.942 0. 1164.54 477.277 0. 0. 1. 1165.48 0. 654.942 0. 1164.54 477.277 0. 0. 1. 0.61456 0.61121 -0.49875 1.44296 -0.46515 0.79138 0.39667 -0.32226 0.63715 -0.01178 0.77065 0.31639 0. 0. 0. 1. 15 | scene0806_00_frame-000225.jpg scene0806_00_frame-001095.jpg 0 0 1165.37 0. 650.862 0. 1166.11 488.595 0. 0. 1. 1165.37 0. 650.862 0. 1166.11 488.595 0. 0. 1. 0.15102 0.52115 -0.84 1.95984 -0.41332 0.80519 0.42525 -1.02578 0.89798 0.28297 0.337 1.24882 0. 0. 0. 1. 16 | -------------------------------------------------------------------------------- /assets/teaser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/assets/teaser.png -------------------------------------------------------------------------------- /demo_superglue.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # 3 | # %BANNER_BEGIN% 4 | # --------------------------------------------------------------------- 5 | # %COPYRIGHT_BEGIN% 6 | # 7 | # Magic Leap, Inc. ("COMPANY") CONFIDENTIAL 8 | # 9 | # Unpublished Copyright (c) 2020 10 | # Magic Leap, Inc., All Rights Reserved. 11 | # 12 | # NOTICE: All information contained herein is, and remains the property 13 | # of COMPANY. The intellectual and technical concepts contained herein 14 | # are proprietary to COMPANY and may be covered by U.S. and Foreign 15 | # Patents, patents in process, and are protected by trade secret or 16 | # copyright law. Dissemination of this information or reproduction of 17 | # this material is strictly forbidden unless prior written permission is 18 | # obtained from COMPANY. Access to the source code contained herein is 19 | # hereby forbidden to anyone except current COMPANY employees, managers 20 | # or contractors who have executed Confidentiality and Non-disclosure 21 | # agreements explicitly covering such access. 22 | # 23 | # The copyright notice above does not evidence any actual or intended 24 | # publication or disclosure of this source code, which includes 25 | # information that is confidential and/or proprietary, and is a trade 26 | # secret, of COMPANY. ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, 27 | # PUBLIC PERFORMANCE, OR PUBLIC DISPLAY OF OR THROUGH USE OF THIS 28 | # SOURCE CODE WITHOUT THE EXPRESS WRITTEN CONSENT OF COMPANY IS 29 | # STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE LAWS AND 30 | # INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE 31 | # CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS 32 | # TO REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, 33 | # USE, OR SELL ANYTHING THAT IT MAY DESCRIBE, IN WHOLE OR IN PART. 34 | # 35 | # %COPYRIGHT_END% 36 | # ---------------------------------------------------------------------- 37 | # %AUTHORS_BEGIN% 38 | # 39 | # Originating Authors: Paul-Edouard Sarlin 40 | # Daniel DeTone 41 | # Tomasz Malisiewicz 42 | # 43 | # %AUTHORS_END% 44 | # --------------------------------------------------------------------*/ 45 | # %BANNER_END% 46 | 47 | from pathlib import Path 48 | import argparse 49 | import cv2 50 | import matplotlib.cm as cm 51 | import torch 52 | 53 | from models.matching import Matching 54 | from models.utils import (AverageTimer, VideoStreamer, 55 | make_matching_plot_fast, frame2tensor) 56 | 57 | torch.set_grad_enabled(False) 58 | 59 | 60 | if __name__ == '__main__': 61 | parser = argparse.ArgumentParser( 62 | description='SuperGlue demo', 63 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 64 | parser.add_argument( 65 | '--input', type=str, default='0', 66 | help='ID of a USB webcam, URL of an IP camera, ' 67 | 'or path to an image directory or movie file') 68 | parser.add_argument( 69 | '--output_dir', type=str, default=None, 70 | help='Directory where to write output frames (If None, no output)') 71 | 72 | parser.add_argument( 73 | '--image_glob', type=str, nargs='+', default=['*.png', '*.jpg', '*.jpeg'], 74 | help='Glob if a directory of images is specified') 75 | parser.add_argument( 76 | '--skip', type=int, default=1, 77 | help='Images to skip if input is a movie or directory') 78 | parser.add_argument( 79 | '--max_length', type=int, default=1000000, 80 | help='Maximum length if input is a movie or directory') 81 | parser.add_argument( 82 | '--resize', type=int, nargs='+', default=[640, 480], 83 | help='Resize the input image before running inference. If two numbers, ' 84 | 'resize to the exact dimensions, if one number, resize the max ' 85 | 'dimension, if -1, do not resize') 86 | 87 | parser.add_argument( 88 | '--superglue', choices={'indoor', 'outdoor'}, default='indoor', 89 | help='SuperGlue weights') 90 | parser.add_argument( 91 | '--max_keypoints', type=int, default=-1, 92 | help='Maximum number of keypoints detected by Superpoint' 93 | ' (\'-1\' keeps all keypoints)') 94 | parser.add_argument( 95 | '--keypoint_threshold', type=float, default=0.005, 96 | help='SuperPoint keypoint detector confidence threshold') 97 | parser.add_argument( 98 | '--nms_radius', type=int, default=4, 99 | help='SuperPoint Non Maximum Suppression (NMS) radius' 100 | ' (Must be positive)') 101 | parser.add_argument( 102 | '--sinkhorn_iterations', type=int, default=20, 103 | help='Number of Sinkhorn iterations performed by SuperGlue') 104 | parser.add_argument( 105 | '--match_threshold', type=float, default=0.2, 106 | help='SuperGlue match threshold') 107 | 108 | parser.add_argument( 109 | '--show_keypoints', action='store_true', 110 | help='Show the detected keypoints') 111 | parser.add_argument( 112 | '--no_display', action='store_true', 113 | help='Do not display images to screen. Useful if running remotely') 114 | parser.add_argument( 115 | '--force_cpu', action='store_true', 116 | help='Force pytorch to run in CPU mode.') 117 | 118 | opt = parser.parse_args() 119 | print(opt) 120 | 121 | if len(opt.resize) == 2 and opt.resize[1] == -1: 122 | opt.resize = opt.resize[0:1] 123 | if len(opt.resize) == 2: 124 | print('Will resize to {}x{} (WxH)'.format( 125 | opt.resize[0], opt.resize[1])) 126 | elif len(opt.resize) == 1 and opt.resize[0] > 0: 127 | print('Will resize max dimension to {}'.format(opt.resize[0])) 128 | elif len(opt.resize) == 1: 129 | print('Will not resize images') 130 | else: 131 | raise ValueError('Cannot specify more than two integers for --resize') 132 | 133 | device = 'cuda' if torch.cuda.is_available() and not opt.force_cpu else 'cpu' 134 | print('Running inference on device \"{}\"'.format(device)) 135 | config = { 136 | 'superpoint': { 137 | 'nms_radius': opt.nms_radius, 138 | 'keypoint_threshold': opt.keypoint_threshold, 139 | 'max_keypoints': opt.max_keypoints 140 | }, 141 | 'superglue': { 142 | 'weights': opt.superglue, 143 | 'sinkhorn_iterations': opt.sinkhorn_iterations, 144 | 'match_threshold': opt.match_threshold, 145 | } 146 | } 147 | matching = Matching(config).eval().to(device) 148 | keys = ['keypoints', 'scores', 'descriptors'] 149 | 150 | vs = VideoStreamer(opt.input, opt.resize, opt.skip, 151 | opt.image_glob, opt.max_length) 152 | frame, ret = vs.next_frame() 153 | assert ret, 'Error when reading the first frame (try different --input?)' 154 | 155 | frame_tensor = frame2tensor(frame, device) 156 | last_data = matching.superpoint({'image': frame_tensor}) 157 | last_data = {k+'0': last_data[k] for k in keys} 158 | last_data['image0'] = frame_tensor 159 | last_frame = frame 160 | last_image_id = 0 161 | 162 | if opt.output_dir is not None: 163 | print('==> Will write outputs to {}'.format(opt.output_dir)) 164 | Path(opt.output_dir).mkdir(exist_ok=True) 165 | 166 | # Create a window to display the demo. 167 | if not opt.no_display: 168 | cv2.namedWindow('SuperGlue matches', cv2.WINDOW_NORMAL) 169 | cv2.resizeWindow('SuperGlue matches', 640*2, 480) 170 | else: 171 | print('Skipping visualization, will not show a GUI.') 172 | 173 | # Print the keyboard help menu. 174 | print('==> Keyboard control:\n' 175 | '\tn: select the current frame as the anchor\n' 176 | '\te/r: increase/decrease the keypoint confidence threshold\n' 177 | '\td/f: increase/decrease the match filtering threshold\n' 178 | '\tk: toggle the visualization of keypoints\n' 179 | '\tq: quit') 180 | 181 | timer = AverageTimer() 182 | 183 | while True: 184 | frame, ret = vs.next_frame() 185 | if not ret: 186 | print('Finished demo_superglue.py') 187 | break 188 | timer.update('data') 189 | stem0, stem1 = last_image_id, vs.i - 1 190 | 191 | frame_tensor = frame2tensor(frame, device) 192 | pred = matching({**last_data, 'image1': frame_tensor}) 193 | kpts0 = last_data['keypoints0'][0].cpu().numpy() 194 | kpts1 = pred['keypoints1'][0].cpu().numpy() 195 | matches = pred['matches0'][0].cpu().numpy() 196 | confidence = pred['matching_scores0'][0].cpu().numpy() 197 | timer.update('forward') 198 | 199 | valid = matches > -1 200 | mkpts0 = kpts0[valid] 201 | mkpts1 = kpts1[matches[valid]] 202 | color = cm.jet(confidence[valid]) 203 | text = [ 204 | 'SuperGlue', 205 | 'Keypoints: {}:{}'.format(len(kpts0), len(kpts1)), 206 | 'Matches: {}'.format(len(mkpts0)) 207 | ] 208 | k_thresh = matching.superpoint.config['keypoint_threshold'] 209 | m_thresh = matching.superglue.config['match_threshold'] 210 | small_text = [ 211 | 'Keypoint Threshold: {:.4f}'.format(k_thresh), 212 | 'Match Threshold: {:.2f}'.format(m_thresh), 213 | 'Image Pair: {:06}:{:06}'.format(stem0, stem1), 214 | ] 215 | out = make_matching_plot_fast( 216 | last_frame, frame, kpts0, kpts1, mkpts0, mkpts1, color, text, 217 | path=None, show_keypoints=opt.show_keypoints, small_text=small_text) 218 | 219 | if not opt.no_display: 220 | cv2.imshow('SuperGlue matches', out) 221 | key = chr(cv2.waitKey(1) & 0xFF) 222 | if key == 'q': 223 | vs.cleanup() 224 | print('Exiting (via q) demo_superglue.py') 225 | break 226 | elif key == 'n': # set the current frame as anchor 227 | last_data = {k+'0': pred[k+'1'] for k in keys} 228 | last_data['image0'] = frame_tensor 229 | last_frame = frame 230 | last_image_id = (vs.i - 1) 231 | elif key in ['e', 'r']: 232 | # Increase/decrease keypoint threshold by 10% each keypress. 233 | d = 0.1 * (-1 if key == 'e' else 1) 234 | matching.superpoint.config['keypoint_threshold'] = min(max( 235 | 0.0001, matching.superpoint.config['keypoint_threshold']*(1+d)), 1) 236 | print('\nChanged the keypoint threshold to {:.4f}'.format( 237 | matching.superpoint.config['keypoint_threshold'])) 238 | elif key in ['d', 'f']: 239 | # Increase/decrease match threshold by 0.05 each keypress. 240 | d = 0.05 * (-1 if key == 'd' else 1) 241 | matching.superglue.config['match_threshold'] = min(max( 242 | 0.05, matching.superglue.config['match_threshold']+d), .95) 243 | print('\nChanged the match threshold to {:.2f}'.format( 244 | matching.superglue.config['match_threshold'])) 245 | elif key == 'k': 246 | opt.show_keypoints = not opt.show_keypoints 247 | 248 | timer.update('viz') 249 | timer.print() 250 | 251 | if opt.output_dir is not None: 252 | #stem = 'matches_{:06}_{:06}'.format(last_image_id, vs.i-1) 253 | stem = 'matches_{:06}_{:06}'.format(stem0, stem1) 254 | out_file = str(Path(opt.output_dir, stem + '.png')) 255 | print('\nWriting image to {}'.format(out_file)) 256 | cv2.imwrite(out_file, out) 257 | 258 | cv2.destroyAllWindows() 259 | vs.cleanup() 260 | -------------------------------------------------------------------------------- /match_pairs.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # 3 | # %BANNER_BEGIN% 4 | # --------------------------------------------------------------------- 5 | # %COPYRIGHT_BEGIN% 6 | # 7 | # Magic Leap, Inc. ("COMPANY") CONFIDENTIAL 8 | # 9 | # Unpublished Copyright (c) 2020 10 | # Magic Leap, Inc., All Rights Reserved. 11 | # 12 | # NOTICE: All information contained herein is, and remains the property 13 | # of COMPANY. The intellectual and technical concepts contained herein 14 | # are proprietary to COMPANY and may be covered by U.S. and Foreign 15 | # Patents, patents in process, and are protected by trade secret or 16 | # copyright law. Dissemination of this information or reproduction of 17 | # this material is strictly forbidden unless prior written permission is 18 | # obtained from COMPANY. Access to the source code contained herein is 19 | # hereby forbidden to anyone except current COMPANY employees, managers 20 | # or contractors who have executed Confidentiality and Non-disclosure 21 | # agreements explicitly covering such access. 22 | # 23 | # The copyright notice above does not evidence any actual or intended 24 | # publication or disclosure of this source code, which includes 25 | # information that is confidential and/or proprietary, and is a trade 26 | # secret, of COMPANY. ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, 27 | # PUBLIC PERFORMANCE, OR PUBLIC DISPLAY OF OR THROUGH USE OF THIS 28 | # SOURCE CODE WITHOUT THE EXPRESS WRITTEN CONSENT OF COMPANY IS 29 | # STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE LAWS AND 30 | # INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE 31 | # CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS 32 | # TO REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, 33 | # USE, OR SELL ANYTHING THAT IT MAY DESCRIBE, IN WHOLE OR IN PART. 34 | # 35 | # %COPYRIGHT_END% 36 | # ---------------------------------------------------------------------- 37 | # %AUTHORS_BEGIN% 38 | # 39 | # Originating Authors: Paul-Edouard Sarlin 40 | # Daniel DeTone 41 | # Tomasz Malisiewicz 42 | # 43 | # %AUTHORS_END% 44 | # --------------------------------------------------------------------*/ 45 | # %BANNER_END% 46 | 47 | from pathlib import Path 48 | import argparse 49 | import random 50 | import numpy as np 51 | import matplotlib.cm as cm 52 | import torch 53 | 54 | 55 | from models.matching import Matching 56 | from models.utils import (compute_pose_error, compute_epipolar_error, 57 | estimate_pose, make_matching_plot, 58 | error_colormap, AverageTimer, pose_auc, read_image, 59 | rotate_intrinsics, rotate_pose_inplane, 60 | scale_intrinsics) 61 | 62 | torch.set_grad_enabled(False) 63 | 64 | 65 | if __name__ == '__main__': 66 | parser = argparse.ArgumentParser( 67 | description='Image pair matching and pose evaluation with SuperGlue', 68 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 69 | 70 | parser.add_argument( 71 | '--input_pairs', type=str, default='assets/scannet_sample_pairs_with_gt.txt', 72 | help='Path to the list of image pairs') 73 | parser.add_argument( 74 | '--input_dir', type=str, default='assets/scannet_sample_images/', 75 | help='Path to the directory that contains the images') 76 | parser.add_argument( 77 | '--output_dir', type=str, default='dump_match_pairs/', 78 | help='Path to the directory in which the .npz results and optionally,' 79 | 'the visualization images are written') 80 | 81 | parser.add_argument( 82 | '--max_length', type=int, default=-1, 83 | help='Maximum number of pairs to evaluate') 84 | parser.add_argument( 85 | '--resize', type=int, nargs='+', default=[640, 480], 86 | help='Resize the input image before running inference. If two numbers, ' 87 | 'resize to the exact dimensions, if one number, resize the max ' 88 | 'dimension, if -1, do not resize') 89 | parser.add_argument( 90 | '--resize_float', action='store_true', 91 | help='Resize the image after casting uint8 to float') 92 | 93 | parser.add_argument( 94 | '--superglue', choices={'indoor', 'outdoor'}, default='indoor', 95 | help='SuperGlue weights') 96 | parser.add_argument( 97 | '--max_keypoints', type=int, default=1024, 98 | help='Maximum number of keypoints detected by Superpoint' 99 | ' (\'-1\' keeps all keypoints)') 100 | parser.add_argument( 101 | '--keypoint_threshold', type=float, default=0.005, 102 | help='SuperPoint keypoint detector confidence threshold') 103 | parser.add_argument( 104 | '--nms_radius', type=int, default=4, 105 | help='SuperPoint Non Maximum Suppression (NMS) radius' 106 | ' (Must be positive)') 107 | parser.add_argument( 108 | '--sinkhorn_iterations', type=int, default=20, 109 | help='Number of Sinkhorn iterations performed by SuperGlue') 110 | parser.add_argument( 111 | '--match_threshold', type=float, default=0.2, 112 | help='SuperGlue match threshold') 113 | 114 | parser.add_argument( 115 | '--viz', action='store_true', 116 | help='Visualize the matches and dump the plots') 117 | parser.add_argument( 118 | '--eval', action='store_true', 119 | help='Perform the evaluation' 120 | ' (requires ground truth pose and intrinsics)') 121 | parser.add_argument( 122 | '--fast_viz', action='store_true', 123 | help='Use faster image visualization with OpenCV instead of Matplotlib') 124 | parser.add_argument( 125 | '--cache', action='store_true', 126 | help='Skip the pair if output .npz files are already found') 127 | parser.add_argument( 128 | '--show_keypoints', action='store_true', 129 | help='Plot the keypoints in addition to the matches') 130 | parser.add_argument( 131 | '--viz_extension', type=str, default='png', choices=['png', 'pdf'], 132 | help='Visualization file extension. Use pdf for highest-quality.') 133 | parser.add_argument( 134 | '--opencv_display', action='store_true', 135 | help='Visualize via OpenCV before saving output images') 136 | parser.add_argument( 137 | '--shuffle', action='store_true', 138 | help='Shuffle ordering of pairs before processing') 139 | parser.add_argument( 140 | '--force_cpu', action='store_true', 141 | help='Force pytorch to run in CPU mode.') 142 | 143 | opt = parser.parse_args() 144 | print(opt) 145 | 146 | assert not (opt.opencv_display and not opt.viz), 'Must use --viz with --opencv_display' 147 | assert not (opt.opencv_display and not opt.fast_viz), 'Cannot use --opencv_display without --fast_viz' 148 | assert not (opt.fast_viz and not opt.viz), 'Must use --viz with --fast_viz' 149 | assert not (opt.fast_viz and opt.viz_extension == 'pdf'), 'Cannot use pdf extension with --fast_viz' 150 | 151 | if len(opt.resize) == 2 and opt.resize[1] == -1: 152 | opt.resize = opt.resize[0:1] 153 | if len(opt.resize) == 2: 154 | print('Will resize to {}x{} (WxH)'.format( 155 | opt.resize[0], opt.resize[1])) 156 | elif len(opt.resize) == 1 and opt.resize[0] > 0: 157 | print('Will resize max dimension to {}'.format(opt.resize[0])) 158 | elif len(opt.resize) == 1: 159 | print('Will not resize images') 160 | else: 161 | raise ValueError('Cannot specify more than two integers for --resize') 162 | 163 | with open(opt.input_pairs, 'r') as f: 164 | pairs = [l.split() for l in f.readlines()] 165 | 166 | if opt.max_length > -1: 167 | pairs = pairs[0:np.min([len(pairs), opt.max_length])] 168 | 169 | if opt.shuffle: 170 | random.Random(0).shuffle(pairs) 171 | 172 | if opt.eval: 173 | if not all([len(p) == 38 for p in pairs]): 174 | raise ValueError( 175 | 'All pairs should have ground truth info for evaluation.' 176 | 'File \"{}\" needs 38 valid entries per row'.format(opt.input_pairs)) 177 | 178 | # Load the SuperPoint and SuperGlue models. 179 | device = 'cuda' if torch.cuda.is_available() and not opt.force_cpu else 'cpu' 180 | print('Running inference on device \"{}\"'.format(device)) 181 | config = { 182 | 'superpoint': { 183 | 'nms_radius': opt.nms_radius, 184 | 'keypoint_threshold': opt.keypoint_threshold, 185 | 'max_keypoints': opt.max_keypoints 186 | }, 187 | 'superglue': { 188 | 'weights': opt.superglue, 189 | 'sinkhorn_iterations': opt.sinkhorn_iterations, 190 | 'match_threshold': opt.match_threshold, 191 | } 192 | } 193 | matching = Matching(config).eval().to(device) 194 | 195 | # Create the output directories if they do not exist already. 196 | input_dir = Path(opt.input_dir) 197 | print('Looking for data in directory \"{}\"'.format(input_dir)) 198 | output_dir = Path(opt.output_dir) 199 | output_dir.mkdir(exist_ok=True, parents=True) 200 | print('Will write matches to directory \"{}\"'.format(output_dir)) 201 | if opt.eval: 202 | print('Will write evaluation results', 203 | 'to directory \"{}\"'.format(output_dir)) 204 | if opt.viz: 205 | print('Will write visualization images to', 206 | 'directory \"{}\"'.format(output_dir)) 207 | 208 | timer = AverageTimer(newline=True) 209 | for i, pair in enumerate(pairs): 210 | name0, name1 = pair[:2] 211 | stem0, stem1 = Path(name0).stem, Path(name1).stem 212 | matches_path = output_dir / '{}_{}_matches.npz'.format(stem0, stem1) 213 | eval_path = output_dir / '{}_{}_evaluation.npz'.format(stem0, stem1) 214 | viz_path = output_dir / '{}_{}_matches.{}'.format(stem0, stem1, opt.viz_extension) 215 | viz_eval_path = output_dir / \ 216 | '{}_{}_evaluation.{}'.format(stem0, stem1, opt.viz_extension) 217 | 218 | # Handle --cache logic. 219 | do_match = True 220 | do_eval = opt.eval 221 | do_viz = opt.viz 222 | do_viz_eval = opt.eval and opt.viz 223 | if opt.cache: 224 | if matches_path.exists(): 225 | try: 226 | results = np.load(matches_path) 227 | except: 228 | raise IOError('Cannot load matches .npz file: %s' % 229 | matches_path) 230 | 231 | kpts0, kpts1 = results['keypoints0'], results['keypoints1'] 232 | matches, conf = results['matches'], results['match_confidence'] 233 | do_match = False 234 | if opt.eval and eval_path.exists(): 235 | try: 236 | results = np.load(eval_path) 237 | except: 238 | raise IOError('Cannot load eval .npz file: %s' % eval_path) 239 | err_R, err_t = results['error_R'], results['error_t'] 240 | precision = results['precision'] 241 | matching_score = results['matching_score'] 242 | num_correct = results['num_correct'] 243 | epi_errs = results['epipolar_errors'] 244 | do_eval = False 245 | if opt.viz and viz_path.exists(): 246 | do_viz = False 247 | if opt.viz and opt.eval and viz_eval_path.exists(): 248 | do_viz_eval = False 249 | timer.update('load_cache') 250 | 251 | if not (do_match or do_eval or do_viz or do_viz_eval): 252 | timer.print('Finished pair {:5} of {:5}'.format(i, len(pairs))) 253 | continue 254 | 255 | # If a rotation integer is provided (e.g. from EXIF data), use it: 256 | if len(pair) >= 5: 257 | rot0, rot1 = int(pair[2]), int(pair[3]) 258 | else: 259 | rot0, rot1 = 0, 0 260 | 261 | # Load the image pair. 262 | image0, inp0, scales0 = read_image( 263 | input_dir / name0, device, opt.resize, rot0, opt.resize_float) 264 | image1, inp1, scales1 = read_image( 265 | input_dir / name1, device, opt.resize, rot1, opt.resize_float) 266 | if image0 is None or image1 is None: 267 | print('Problem reading image pair: {} {}'.format( 268 | input_dir/name0, input_dir/name1)) 269 | exit(1) 270 | timer.update('load_image') 271 | 272 | if do_match: 273 | # Perform the matching. 274 | pred = matching({'image0': inp0, 'image1': inp1}) 275 | pred = {k: v[0].cpu().numpy() for k, v in pred.items()} 276 | kpts0, kpts1 = pred['keypoints0'], pred['keypoints1'] 277 | matches, conf = pred['matches0'], pred['matching_scores0'] 278 | timer.update('matcher') 279 | 280 | # Write the matches to disk. 281 | out_matches = {'keypoints0': kpts0, 'keypoints1': kpts1, 282 | 'matches': matches, 'match_confidence': conf} 283 | np.savez(str(matches_path), **out_matches) 284 | 285 | # Keep the matching keypoints. 286 | valid = matches > -1 287 | mkpts0 = kpts0[valid] 288 | mkpts1 = kpts1[matches[valid]] 289 | mconf = conf[valid] 290 | 291 | if do_eval: 292 | # Estimate the pose and compute the pose error. 293 | assert len(pair) == 38, 'Pair does not have ground truth info' 294 | K0 = np.array(pair[4:13]).astype(float).reshape(3, 3) 295 | K1 = np.array(pair[13:22]).astype(float).reshape(3, 3) 296 | T_0to1 = np.array(pair[22:]).astype(float).reshape(4, 4) 297 | 298 | # Scale the intrinsics to resized image. 299 | K0 = scale_intrinsics(K0, scales0) 300 | K1 = scale_intrinsics(K1, scales1) 301 | 302 | # Update the intrinsics + extrinsics if EXIF rotation was found. 303 | if rot0 != 0 or rot1 != 0: 304 | cam0_T_w = np.eye(4) 305 | cam1_T_w = T_0to1 306 | if rot0 != 0: 307 | K0 = rotate_intrinsics(K0, image0.shape, rot0) 308 | cam0_T_w = rotate_pose_inplane(cam0_T_w, rot0) 309 | if rot1 != 0: 310 | K1 = rotate_intrinsics(K1, image1.shape, rot1) 311 | cam1_T_w = rotate_pose_inplane(cam1_T_w, rot1) 312 | cam1_T_cam0 = cam1_T_w @ np.linalg.inv(cam0_T_w) 313 | T_0to1 = cam1_T_cam0 314 | 315 | epi_errs = compute_epipolar_error(mkpts0, mkpts1, T_0to1, K0, K1) 316 | correct = epi_errs < 5e-4 317 | num_correct = np.sum(correct) 318 | precision = np.mean(correct) if len(correct) > 0 else 0 319 | matching_score = num_correct / len(kpts0) if len(kpts0) > 0 else 0 320 | 321 | thresh = 1. # In pixels relative to resized image size. 322 | ret = estimate_pose(mkpts0, mkpts1, K0, K1, thresh) 323 | if ret is None: 324 | err_t, err_R = np.inf, np.inf 325 | else: 326 | R, t, inliers = ret 327 | err_t, err_R = compute_pose_error(T_0to1, R, t) 328 | 329 | # Write the evaluation results to disk. 330 | out_eval = {'error_t': err_t, 331 | 'error_R': err_R, 332 | 'precision': precision, 333 | 'matching_score': matching_score, 334 | 'num_correct': num_correct, 335 | 'epipolar_errors': epi_errs} 336 | np.savez(str(eval_path), **out_eval) 337 | timer.update('eval') 338 | 339 | if do_viz: 340 | # Visualize the matches. 341 | color = cm.jet(mconf) 342 | text = [ 343 | 'SuperGlue', 344 | 'Keypoints: {}:{}'.format(len(kpts0), len(kpts1)), 345 | 'Matches: {}'.format(len(mkpts0)), 346 | ] 347 | if rot0 != 0 or rot1 != 0: 348 | text.append('Rotation: {}:{}'.format(rot0, rot1)) 349 | 350 | # Display extra parameter info. 351 | k_thresh = matching.superpoint.config['keypoint_threshold'] 352 | m_thresh = matching.superglue.config['match_threshold'] 353 | small_text = [ 354 | 'Keypoint Threshold: {:.4f}'.format(k_thresh), 355 | 'Match Threshold: {:.2f}'.format(m_thresh), 356 | 'Image Pair: {}:{}'.format(stem0, stem1), 357 | ] 358 | 359 | make_matching_plot( 360 | image0, image1, kpts0, kpts1, mkpts0, mkpts1, color, 361 | text, viz_path, opt.show_keypoints, 362 | opt.fast_viz, opt.opencv_display, 'Matches', small_text) 363 | 364 | timer.update('viz_match') 365 | 366 | if do_viz_eval: 367 | # Visualize the evaluation results for the image pair. 368 | color = np.clip((epi_errs - 0) / (1e-3 - 0), 0, 1) 369 | color = error_colormap(1 - color) 370 | deg, delta = ' deg', 'Delta ' 371 | if not opt.fast_viz: 372 | deg, delta = '°', '$\\Delta$' 373 | e_t = 'FAIL' if np.isinf(err_t) else '{:.1f}{}'.format(err_t, deg) 374 | e_R = 'FAIL' if np.isinf(err_R) else '{:.1f}{}'.format(err_R, deg) 375 | text = [ 376 | 'SuperGlue', 377 | '{}R: {}'.format(delta, e_R), '{}t: {}'.format(delta, e_t), 378 | 'inliers: {}/{}'.format(num_correct, (matches > -1).sum()), 379 | ] 380 | if rot0 != 0 or rot1 != 0: 381 | text.append('Rotation: {}:{}'.format(rot0, rot1)) 382 | 383 | # Display extra parameter info (only works with --fast_viz). 384 | k_thresh = matching.superpoint.config['keypoint_threshold'] 385 | m_thresh = matching.superglue.config['match_threshold'] 386 | small_text = [ 387 | 'Keypoint Threshold: {:.4f}'.format(k_thresh), 388 | 'Match Threshold: {:.2f}'.format(m_thresh), 389 | 'Image Pair: {}:{}'.format(stem0, stem1), 390 | ] 391 | 392 | make_matching_plot( 393 | image0, image1, kpts0, kpts1, mkpts0, 394 | mkpts1, color, text, viz_eval_path, 395 | opt.show_keypoints, opt.fast_viz, 396 | opt.opencv_display, 'Relative Pose', small_text) 397 | 398 | timer.update('viz_eval') 399 | 400 | timer.print('Finished pair {:5} of {:5}'.format(i, len(pairs))) 401 | 402 | if opt.eval: 403 | # Collate the results into a final table and print to terminal. 404 | pose_errors = [] 405 | precisions = [] 406 | matching_scores = [] 407 | for pair in pairs: 408 | name0, name1 = pair[:2] 409 | stem0, stem1 = Path(name0).stem, Path(name1).stem 410 | eval_path = output_dir / \ 411 | '{}_{}_evaluation.npz'.format(stem0, stem1) 412 | results = np.load(eval_path) 413 | pose_error = np.maximum(results['error_t'], results['error_R']) 414 | pose_errors.append(pose_error) 415 | precisions.append(results['precision']) 416 | matching_scores.append(results['matching_score']) 417 | thresholds = [5, 10, 20] 418 | aucs = pose_auc(pose_errors, thresholds) 419 | aucs = [100.*yy for yy in aucs] 420 | prec = 100.*np.mean(precisions) 421 | ms = 100.*np.mean(matching_scores) 422 | print('Evaluation Results (mean over {} pairs):'.format(len(pairs))) 423 | print('AUC@5\t AUC@10\t AUC@20\t Prec\t MScore\t') 424 | print('{:.2f}\t {:.2f}\t {:.2f}\t {:.2f}\t {:.2f}\t'.format( 425 | aucs[0], aucs[1], aucs[2], prec, ms)) 426 | -------------------------------------------------------------------------------- /models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/models/__init__.py -------------------------------------------------------------------------------- /models/matching.py: -------------------------------------------------------------------------------- 1 | # %BANNER_BEGIN% 2 | # --------------------------------------------------------------------- 3 | # %COPYRIGHT_BEGIN% 4 | # 5 | # Magic Leap, Inc. ("COMPANY") CONFIDENTIAL 6 | # 7 | # Unpublished Copyright (c) 2020 8 | # Magic Leap, Inc., All Rights Reserved. 9 | # 10 | # NOTICE: All information contained herein is, and remains the property 11 | # of COMPANY. The intellectual and technical concepts contained herein 12 | # are proprietary to COMPANY and may be covered by U.S. and Foreign 13 | # Patents, patents in process, and are protected by trade secret or 14 | # copyright law. Dissemination of this information or reproduction of 15 | # this material is strictly forbidden unless prior written permission is 16 | # obtained from COMPANY. Access to the source code contained herein is 17 | # hereby forbidden to anyone except current COMPANY employees, managers 18 | # or contractors who have executed Confidentiality and Non-disclosure 19 | # agreements explicitly covering such access. 20 | # 21 | # The copyright notice above does not evidence any actual or intended 22 | # publication or disclosure of this source code, which includes 23 | # information that is confidential and/or proprietary, and is a trade 24 | # secret, of COMPANY. ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, 25 | # PUBLIC PERFORMANCE, OR PUBLIC DISPLAY OF OR THROUGH USE OF THIS 26 | # SOURCE CODE WITHOUT THE EXPRESS WRITTEN CONSENT OF COMPANY IS 27 | # STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE LAWS AND 28 | # INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE 29 | # CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS 30 | # TO REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, 31 | # USE, OR SELL ANYTHING THAT IT MAY DESCRIBE, IN WHOLE OR IN PART. 32 | # 33 | # %COPYRIGHT_END% 34 | # ---------------------------------------------------------------------- 35 | # %AUTHORS_BEGIN% 36 | # 37 | # Originating Authors: Paul-Edouard Sarlin 38 | # 39 | # %AUTHORS_END% 40 | # --------------------------------------------------------------------*/ 41 | # %BANNER_END% 42 | 43 | import torch 44 | 45 | from .superpoint import SuperPoint 46 | from .superglue import SuperGlue 47 | 48 | 49 | class Matching(torch.nn.Module): 50 | """ Image Matching Frontend (SuperPoint + SuperGlue) """ 51 | def __init__(self, config={}): 52 | super().__init__() 53 | self.superpoint = SuperPoint(config.get('superpoint', {})) 54 | self.superglue = SuperGlue(config.get('superglue', {})) 55 | 56 | def forward(self, data): 57 | """ Run SuperPoint (optionally) and SuperGlue 58 | SuperPoint is skipped if ['keypoints0', 'keypoints1'] exist in input 59 | Args: 60 | data: dictionary with minimal keys: ['image0', 'image1'] 61 | """ 62 | pred = {} 63 | 64 | # Extract SuperPoint (keypoints, scores, descriptors) if not provided 65 | if 'keypoints0' not in data: 66 | pred0 = self.superpoint({'image': data['image0']}) 67 | pred = {**pred, **{k+'0': v for k, v in pred0.items()}} 68 | if 'keypoints1' not in data: 69 | pred1 = self.superpoint({'image': data['image1']}) 70 | pred = {**pred, **{k+'1': v for k, v in pred1.items()}} 71 | 72 | # Batch all features 73 | # We should either have i) one image per batch, or 74 | # ii) the same number of local features for all images in the batch. 75 | data = {**data, **pred} 76 | 77 | for k in data: 78 | if isinstance(data[k], (list, tuple)): 79 | data[k] = torch.stack(data[k]) 80 | 81 | # Perform the matching 82 | pred = {**pred, **self.superglue(data)} 83 | 84 | return pred 85 | -------------------------------------------------------------------------------- /models/superglue.py: -------------------------------------------------------------------------------- 1 | # %BANNER_BEGIN% 2 | # --------------------------------------------------------------------- 3 | # %COPYRIGHT_BEGIN% 4 | # 5 | # Magic Leap, Inc. ("COMPANY") CONFIDENTIAL 6 | # 7 | # Unpublished Copyright (c) 2020 8 | # Magic Leap, Inc., All Rights Reserved. 9 | # 10 | # NOTICE: All information contained herein is, and remains the property 11 | # of COMPANY. The intellectual and technical concepts contained herein 12 | # are proprietary to COMPANY and may be covered by U.S. and Foreign 13 | # Patents, patents in process, and are protected by trade secret or 14 | # copyright law. Dissemination of this information or reproduction of 15 | # this material is strictly forbidden unless prior written permission is 16 | # obtained from COMPANY. Access to the source code contained herein is 17 | # hereby forbidden to anyone except current COMPANY employees, managers 18 | # or contractors who have executed Confidentiality and Non-disclosure 19 | # agreements explicitly covering such access. 20 | # 21 | # The copyright notice above does not evidence any actual or intended 22 | # publication or disclosure of this source code, which includes 23 | # information that is confidential and/or proprietary, and is a trade 24 | # secret, of COMPANY. ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, 25 | # PUBLIC PERFORMANCE, OR PUBLIC DISPLAY OF OR THROUGH USE OF THIS 26 | # SOURCE CODE WITHOUT THE EXPRESS WRITTEN CONSENT OF COMPANY IS 27 | # STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE LAWS AND 28 | # INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE 29 | # CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS 30 | # TO REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, 31 | # USE, OR SELL ANYTHING THAT IT MAY DESCRIBE, IN WHOLE OR IN PART. 32 | # 33 | # %COPYRIGHT_END% 34 | # ---------------------------------------------------------------------- 35 | # %AUTHORS_BEGIN% 36 | # 37 | # Originating Authors: Paul-Edouard Sarlin 38 | # 39 | # %AUTHORS_END% 40 | # --------------------------------------------------------------------*/ 41 | # %BANNER_END% 42 | 43 | from copy import deepcopy 44 | from pathlib import Path 45 | from typing import List, Tuple 46 | 47 | import torch 48 | from torch import nn 49 | 50 | 51 | def MLP(channels: List[int], do_bn: bool = True) -> nn.Module: 52 | """ Multi-layer perceptron """ 53 | n = len(channels) 54 | layers = [] 55 | for i in range(1, n): 56 | layers.append( 57 | nn.Conv1d(channels[i - 1], channels[i], kernel_size=1, bias=True)) 58 | if i < (n-1): 59 | if do_bn: 60 | layers.append(nn.BatchNorm1d(channels[i])) 61 | layers.append(nn.ReLU()) 62 | return nn.Sequential(*layers) 63 | 64 | 65 | def normalize_keypoints(kpts, image_shape): 66 | """ Normalize keypoints locations based on image image_shape""" 67 | _, _, height, width = image_shape 68 | one = kpts.new_tensor(1) 69 | size = torch.stack([one*width, one*height])[None] 70 | center = size / 2 71 | scaling = size.max(1, keepdim=True).values * 0.7 72 | return (kpts - center[:, None, :]) / scaling[:, None, :] 73 | 74 | 75 | class KeypointEncoder(nn.Module): 76 | """ Joint encoding of visual appearance and location using MLPs""" 77 | def __init__(self, feature_dim: int, layers: List[int]) -> None: 78 | super().__init__() 79 | self.encoder = MLP([3] + layers + [feature_dim]) 80 | nn.init.constant_(self.encoder[-1].bias, 0.0) 81 | 82 | def forward(self, kpts, scores): 83 | inputs = [kpts.transpose(1, 2), scores.unsqueeze(1)] 84 | return self.encoder(torch.cat(inputs, dim=1)) 85 | 86 | 87 | def attention(query: torch.Tensor, key: torch.Tensor, value: torch.Tensor) -> Tuple[torch.Tensor,torch.Tensor]: 88 | dim = query.shape[1] 89 | scores = torch.einsum('bdhn,bdhm->bhnm', query, key) / dim**.5 90 | prob = torch.nn.functional.softmax(scores, dim=-1) 91 | return torch.einsum('bhnm,bdhm->bdhn', prob, value), prob 92 | 93 | 94 | class MultiHeadedAttention(nn.Module): 95 | """ Multi-head attention to increase model expressivitiy """ 96 | def __init__(self, num_heads: int, d_model: int): 97 | super().__init__() 98 | assert d_model % num_heads == 0 99 | self.dim = d_model // num_heads 100 | self.num_heads = num_heads 101 | self.merge = nn.Conv1d(d_model, d_model, kernel_size=1) 102 | self.proj = nn.ModuleList([deepcopy(self.merge) for _ in range(3)]) 103 | 104 | def forward(self, query: torch.Tensor, key: torch.Tensor, value: torch.Tensor) -> torch.Tensor: 105 | batch_dim = query.size(0) 106 | query, key, value = [l(x).view(batch_dim, self.dim, self.num_heads, -1) 107 | for l, x in zip(self.proj, (query, key, value))] 108 | x, _ = attention(query, key, value) 109 | return self.merge(x.contiguous().view(batch_dim, self.dim*self.num_heads, -1)) 110 | 111 | 112 | class AttentionalPropagation(nn.Module): 113 | def __init__(self, feature_dim: int, num_heads: int): 114 | super().__init__() 115 | self.attn = MultiHeadedAttention(num_heads, feature_dim) 116 | self.mlp = MLP([feature_dim*2, feature_dim*2, feature_dim]) 117 | nn.init.constant_(self.mlp[-1].bias, 0.0) 118 | 119 | def forward(self, x: torch.Tensor, source: torch.Tensor) -> torch.Tensor: 120 | message = self.attn(x, source, source) 121 | return self.mlp(torch.cat([x, message], dim=1)) 122 | 123 | 124 | class AttentionalGNN(nn.Module): 125 | def __init__(self, feature_dim: int, layer_names: List[str]) -> None: 126 | super().__init__() 127 | self.layers = nn.ModuleList([ 128 | AttentionalPropagation(feature_dim, 4) 129 | for _ in range(len(layer_names))]) 130 | self.names = layer_names 131 | 132 | def forward(self, desc0: torch.Tensor, desc1: torch.Tensor) -> Tuple[torch.Tensor,torch.Tensor]: 133 | for layer, name in zip(self.layers, self.names): 134 | if name == 'cross': 135 | src0, src1 = desc1, desc0 136 | else: # if name == 'self': 137 | src0, src1 = desc0, desc1 138 | delta0, delta1 = layer(desc0, src0), layer(desc1, src1) 139 | desc0, desc1 = (desc0 + delta0), (desc1 + delta1) 140 | return desc0, desc1 141 | 142 | 143 | def log_sinkhorn_iterations(Z: torch.Tensor, log_mu: torch.Tensor, log_nu: torch.Tensor, iters: int) -> torch.Tensor: 144 | """ Perform Sinkhorn Normalization in Log-space for stability""" 145 | u, v = torch.zeros_like(log_mu), torch.zeros_like(log_nu) 146 | for _ in range(iters): 147 | u = log_mu - torch.logsumexp(Z + v.unsqueeze(1), dim=2) 148 | v = log_nu - torch.logsumexp(Z + u.unsqueeze(2), dim=1) 149 | return Z + u.unsqueeze(2) + v.unsqueeze(1) 150 | 151 | 152 | def log_optimal_transport(scores: torch.Tensor, alpha: torch.Tensor, iters: int) -> torch.Tensor: 153 | """ Perform Differentiable Optimal Transport in Log-space for stability""" 154 | b, m, n = scores.shape 155 | one = scores.new_tensor(1) 156 | ms, ns = (m*one).to(scores), (n*one).to(scores) 157 | 158 | bins0 = alpha.expand(b, m, 1) 159 | bins1 = alpha.expand(b, 1, n) 160 | alpha = alpha.expand(b, 1, 1) 161 | 162 | couplings = torch.cat([torch.cat([scores, bins0], -1), 163 | torch.cat([bins1, alpha], -1)], 1) 164 | 165 | norm = - (ms + ns).log() 166 | log_mu = torch.cat([norm.expand(m), ns.log()[None] + norm]) 167 | log_nu = torch.cat([norm.expand(n), ms.log()[None] + norm]) 168 | log_mu, log_nu = log_mu[None].expand(b, -1), log_nu[None].expand(b, -1) 169 | 170 | Z = log_sinkhorn_iterations(couplings, log_mu, log_nu, iters) 171 | Z = Z - norm # multiply probabilities by M+N 172 | return Z 173 | 174 | 175 | def arange_like(x, dim: int): 176 | return x.new_ones(x.shape[dim]).cumsum(0) - 1 # traceable in 1.1 177 | 178 | 179 | class SuperGlue(nn.Module): 180 | """SuperGlue feature matching middle-end 181 | 182 | Given two sets of keypoints and locations, we determine the 183 | correspondences by: 184 | 1. Keypoint Encoding (normalization + visual feature and location fusion) 185 | 2. Graph Neural Network with multiple self and cross-attention layers 186 | 3. Final projection layer 187 | 4. Optimal Transport Layer (a differentiable Hungarian matching algorithm) 188 | 5. Thresholding matrix based on mutual exclusivity and a match_threshold 189 | 190 | The correspondence ids use -1 to indicate non-matching points. 191 | 192 | Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew 193 | Rabinovich. SuperGlue: Learning Feature Matching with Graph Neural 194 | Networks. In CVPR, 2020. https://arxiv.org/abs/1911.11763 195 | 196 | """ 197 | default_config = { 198 | 'descriptor_dim': 256, 199 | 'weights': 'indoor', 200 | 'keypoint_encoder': [32, 64, 128, 256], 201 | 'GNN_layers': ['self', 'cross'] * 9, 202 | 'sinkhorn_iterations': 100, 203 | 'match_threshold': 0.2, 204 | } 205 | 206 | def __init__(self, config): 207 | super().__init__() 208 | self.config = {**self.default_config, **config} 209 | 210 | self.kenc = KeypointEncoder( 211 | self.config['descriptor_dim'], self.config['keypoint_encoder']) 212 | 213 | self.gnn = AttentionalGNN( 214 | feature_dim=self.config['descriptor_dim'], layer_names=self.config['GNN_layers']) 215 | 216 | self.final_proj = nn.Conv1d( 217 | self.config['descriptor_dim'], self.config['descriptor_dim'], 218 | kernel_size=1, bias=True) 219 | 220 | bin_score = torch.nn.Parameter(torch.tensor(1.)) 221 | self.register_parameter('bin_score', bin_score) 222 | 223 | assert self.config['weights'] in ['indoor', 'outdoor'] 224 | path = Path(__file__).parent 225 | path = path / 'weights/superglue_{}.pth'.format(self.config['weights']) 226 | self.load_state_dict(torch.load(str(path))) 227 | print('Loaded SuperGlue model (\"{}\" weights)'.format( 228 | self.config['weights'])) 229 | 230 | def forward(self, data): 231 | """Run SuperGlue on a pair of keypoints and descriptors""" 232 | desc0, desc1 = data['descriptors0'], data['descriptors1'] 233 | kpts0, kpts1 = data['keypoints0'], data['keypoints1'] 234 | 235 | if kpts0.shape[1] == 0 or kpts1.shape[1] == 0: # no keypoints 236 | shape0, shape1 = kpts0.shape[:-1], kpts1.shape[:-1] 237 | return { 238 | 'matches0': kpts0.new_full(shape0, -1, dtype=torch.int), 239 | 'matches1': kpts1.new_full(shape1, -1, dtype=torch.int), 240 | 'matching_scores0': kpts0.new_zeros(shape0), 241 | 'matching_scores1': kpts1.new_zeros(shape1), 242 | } 243 | 244 | # Keypoint normalization. 245 | kpts0 = normalize_keypoints(kpts0, data['image0'].shape) 246 | kpts1 = normalize_keypoints(kpts1, data['image1'].shape) 247 | 248 | # Keypoint MLP encoder. 249 | desc0 = desc0 + self.kenc(kpts0, data['scores0']) 250 | desc1 = desc1 + self.kenc(kpts1, data['scores1']) 251 | 252 | # Multi-layer Transformer network. 253 | desc0, desc1 = self.gnn(desc0, desc1) 254 | 255 | # Final MLP projection. 256 | mdesc0, mdesc1 = self.final_proj(desc0), self.final_proj(desc1) 257 | 258 | # Compute matching descriptor distance. 259 | scores = torch.einsum('bdn,bdm->bnm', mdesc0, mdesc1) 260 | scores = scores / self.config['descriptor_dim']**.5 261 | 262 | # Run the optimal transport. 263 | scores = log_optimal_transport( 264 | scores, self.bin_score, 265 | iters=self.config['sinkhorn_iterations']) 266 | 267 | # Get the matches with score above "match_threshold". 268 | max0, max1 = scores[:, :-1, :-1].max(2), scores[:, :-1, :-1].max(1) 269 | indices0, indices1 = max0.indices, max1.indices 270 | mutual0 = arange_like(indices0, 1)[None] == indices1.gather(1, indices0) 271 | mutual1 = arange_like(indices1, 1)[None] == indices0.gather(1, indices1) 272 | zero = scores.new_tensor(0) 273 | mscores0 = torch.where(mutual0, max0.values.exp(), zero) 274 | mscores1 = torch.where(mutual1, mscores0.gather(1, indices1), zero) 275 | valid0 = mutual0 & (mscores0 > self.config['match_threshold']) 276 | valid1 = mutual1 & valid0.gather(1, indices1) 277 | indices0 = torch.where(valid0, indices0, indices0.new_tensor(-1)) 278 | indices1 = torch.where(valid1, indices1, indices1.new_tensor(-1)) 279 | 280 | return { 281 | 'matches0': indices0, # use -1 for invalid match 282 | 'matches1': indices1, # use -1 for invalid match 283 | 'matching_scores0': mscores0, 284 | 'matching_scores1': mscores1, 285 | } 286 | -------------------------------------------------------------------------------- /models/superpoint.py: -------------------------------------------------------------------------------- 1 | # %BANNER_BEGIN% 2 | # --------------------------------------------------------------------- 3 | # %COPYRIGHT_BEGIN% 4 | # 5 | # Magic Leap, Inc. ("COMPANY") CONFIDENTIAL 6 | # 7 | # Unpublished Copyright (c) 2020 8 | # Magic Leap, Inc., All Rights Reserved. 9 | # 10 | # NOTICE: All information contained herein is, and remains the property 11 | # of COMPANY. The intellectual and technical concepts contained herein 12 | # are proprietary to COMPANY and may be covered by U.S. and Foreign 13 | # Patents, patents in process, and are protected by trade secret or 14 | # copyright law. Dissemination of this information or reproduction of 15 | # this material is strictly forbidden unless prior written permission is 16 | # obtained from COMPANY. Access to the source code contained herein is 17 | # hereby forbidden to anyone except current COMPANY employees, managers 18 | # or contractors who have executed Confidentiality and Non-disclosure 19 | # agreements explicitly covering such access. 20 | # 21 | # The copyright notice above does not evidence any actual or intended 22 | # publication or disclosure of this source code, which includes 23 | # information that is confidential and/or proprietary, and is a trade 24 | # secret, of COMPANY. ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, 25 | # PUBLIC PERFORMANCE, OR PUBLIC DISPLAY OF OR THROUGH USE OF THIS 26 | # SOURCE CODE WITHOUT THE EXPRESS WRITTEN CONSENT OF COMPANY IS 27 | # STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE LAWS AND 28 | # INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE 29 | # CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS 30 | # TO REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, 31 | # USE, OR SELL ANYTHING THAT IT MAY DESCRIBE, IN WHOLE OR IN PART. 32 | # 33 | # %COPYRIGHT_END% 34 | # ---------------------------------------------------------------------- 35 | # %AUTHORS_BEGIN% 36 | # 37 | # Originating Authors: Paul-Edouard Sarlin 38 | # 39 | # %AUTHORS_END% 40 | # --------------------------------------------------------------------*/ 41 | # %BANNER_END% 42 | 43 | from pathlib import Path 44 | import torch 45 | from torch import nn 46 | 47 | def simple_nms(scores, nms_radius: int): 48 | """ Fast Non-maximum suppression to remove nearby points """ 49 | assert(nms_radius >= 0) 50 | 51 | def max_pool(x): 52 | return torch.nn.functional.max_pool2d( 53 | x, kernel_size=nms_radius*2+1, stride=1, padding=nms_radius) 54 | 55 | zeros = torch.zeros_like(scores) 56 | max_mask = scores == max_pool(scores) 57 | for _ in range(2): 58 | supp_mask = max_pool(max_mask.float()) > 0 59 | supp_scores = torch.where(supp_mask, zeros, scores) 60 | new_max_mask = supp_scores == max_pool(supp_scores) 61 | max_mask = max_mask | (new_max_mask & (~supp_mask)) 62 | return torch.where(max_mask, scores, zeros) 63 | 64 | 65 | def remove_borders(keypoints, scores, border: int, height: int, width: int): 66 | """ Removes keypoints too close to the border """ 67 | mask_h = (keypoints[:, 0] >= border) & (keypoints[:, 0] < (height - border)) 68 | mask_w = (keypoints[:, 1] >= border) & (keypoints[:, 1] < (width - border)) 69 | mask = mask_h & mask_w 70 | return keypoints[mask], scores[mask] 71 | 72 | 73 | def top_k_keypoints(keypoints, scores, k: int): 74 | if k >= len(keypoints): 75 | return keypoints, scores 76 | scores, indices = torch.topk(scores, k, dim=0) 77 | return keypoints[indices], scores 78 | 79 | 80 | def sample_descriptors(keypoints, descriptors, s: int = 8): 81 | """ Interpolate descriptors at keypoint locations """ 82 | b, c, h, w = descriptors.shape 83 | keypoints = keypoints - s / 2 + 0.5 84 | keypoints /= torch.tensor([(w*s - s/2 - 0.5), (h*s - s/2 - 0.5)], 85 | ).to(keypoints)[None] 86 | keypoints = keypoints*2 - 1 # normalize to (-1, 1) 87 | args = {'align_corners': True} if torch.__version__ >= '1.3' else {} 88 | descriptors = torch.nn.functional.grid_sample( 89 | descriptors, keypoints.view(b, 1, -1, 2), mode='bilinear', **args) 90 | descriptors = torch.nn.functional.normalize( 91 | descriptors.reshape(b, c, -1), p=2, dim=1) 92 | return descriptors 93 | 94 | 95 | class SuperPoint(nn.Module): 96 | """SuperPoint Convolutional Detector and Descriptor 97 | 98 | SuperPoint: Self-Supervised Interest Point Detection and 99 | Description. Daniel DeTone, Tomasz Malisiewicz, and Andrew 100 | Rabinovich. In CVPRW, 2019. https://arxiv.org/abs/1712.07629 101 | 102 | """ 103 | default_config = { 104 | 'descriptor_dim': 256, 105 | 'nms_radius': 4, 106 | 'keypoint_threshold': 0.005, 107 | 'max_keypoints': -1, 108 | 'remove_borders': 4, 109 | } 110 | 111 | def __init__(self, config): 112 | super().__init__() 113 | self.config = {**self.default_config, **config} 114 | 115 | self.relu = nn.ReLU(inplace=True) 116 | self.pool = nn.MaxPool2d(kernel_size=2, stride=2) 117 | c1, c2, c3, c4, c5 = 64, 64, 128, 128, 256 118 | 119 | self.conv1a = nn.Conv2d(1, c1, kernel_size=3, stride=1, padding=1) 120 | self.conv1b = nn.Conv2d(c1, c1, kernel_size=3, stride=1, padding=1) 121 | self.conv2a = nn.Conv2d(c1, c2, kernel_size=3, stride=1, padding=1) 122 | self.conv2b = nn.Conv2d(c2, c2, kernel_size=3, stride=1, padding=1) 123 | self.conv3a = nn.Conv2d(c2, c3, kernel_size=3, stride=1, padding=1) 124 | self.conv3b = nn.Conv2d(c3, c3, kernel_size=3, stride=1, padding=1) 125 | self.conv4a = nn.Conv2d(c3, c4, kernel_size=3, stride=1, padding=1) 126 | self.conv4b = nn.Conv2d(c4, c4, kernel_size=3, stride=1, padding=1) 127 | 128 | self.convPa = nn.Conv2d(c4, c5, kernel_size=3, stride=1, padding=1) 129 | self.convPb = nn.Conv2d(c5, 65, kernel_size=1, stride=1, padding=0) 130 | 131 | self.convDa = nn.Conv2d(c4, c5, kernel_size=3, stride=1, padding=1) 132 | self.convDb = nn.Conv2d( 133 | c5, self.config['descriptor_dim'], 134 | kernel_size=1, stride=1, padding=0) 135 | 136 | path = Path(__file__).parent / 'weights/superpoint_v1.pth' 137 | self.load_state_dict(torch.load(str(path))) 138 | 139 | mk = self.config['max_keypoints'] 140 | if mk == 0 or mk < -1: 141 | raise ValueError('\"max_keypoints\" must be positive or \"-1\"') 142 | 143 | print('Loaded SuperPoint model') 144 | 145 | def forward(self, data): 146 | """ Compute keypoints, scores, descriptors for image """ 147 | # Shared Encoder 148 | x = self.relu(self.conv1a(data['image'])) 149 | x = self.relu(self.conv1b(x)) 150 | x = self.pool(x) 151 | x = self.relu(self.conv2a(x)) 152 | x = self.relu(self.conv2b(x)) 153 | x = self.pool(x) 154 | x = self.relu(self.conv3a(x)) 155 | x = self.relu(self.conv3b(x)) 156 | x = self.pool(x) 157 | x = self.relu(self.conv4a(x)) 158 | x = self.relu(self.conv4b(x)) 159 | 160 | # Compute the dense keypoint scores 161 | cPa = self.relu(self.convPa(x)) 162 | scores = self.convPb(cPa) 163 | scores = torch.nn.functional.softmax(scores, 1)[:, :-1] 164 | b, _, h, w = scores.shape 165 | scores = scores.permute(0, 2, 3, 1).reshape(b, h, w, 8, 8) 166 | scores = scores.permute(0, 1, 3, 2, 4).reshape(b, h*8, w*8) 167 | scores = simple_nms(scores, self.config['nms_radius']) 168 | 169 | # Extract keypoints 170 | keypoints = [ 171 | torch.nonzero(s > self.config['keypoint_threshold']) 172 | for s in scores] 173 | scores = [s[tuple(k.t())] for s, k in zip(scores, keypoints)] 174 | 175 | # Discard keypoints near the image borders 176 | keypoints, scores = list(zip(*[ 177 | remove_borders(k, s, self.config['remove_borders'], h*8, w*8) 178 | for k, s in zip(keypoints, scores)])) 179 | 180 | # Keep the k keypoints with highest score 181 | if self.config['max_keypoints'] >= 0: 182 | keypoints, scores = list(zip(*[ 183 | top_k_keypoints(k, s, self.config['max_keypoints']) 184 | for k, s in zip(keypoints, scores)])) 185 | 186 | # Convert (h, w) to (x, y) 187 | keypoints = [torch.flip(k, [1]).float() for k in keypoints] 188 | 189 | # Compute the dense descriptors 190 | cDa = self.relu(self.convDa(x)) 191 | descriptors = self.convDb(cDa) 192 | descriptors = torch.nn.functional.normalize(descriptors, p=2, dim=1) 193 | 194 | # Extract descriptors 195 | descriptors = [sample_descriptors(k[None], d[None], 8)[0] 196 | for k, d in zip(keypoints, descriptors)] 197 | 198 | return { 199 | 'keypoints': keypoints, 200 | 'scores': scores, 201 | 'descriptors': descriptors, 202 | } 203 | -------------------------------------------------------------------------------- /models/utils.py: -------------------------------------------------------------------------------- 1 | # %BANNER_BEGIN% 2 | # --------------------------------------------------------------------- 3 | # %COPYRIGHT_BEGIN% 4 | # 5 | # Magic Leap, Inc. ("COMPANY") CONFIDENTIAL 6 | # 7 | # Unpublished Copyright (c) 2020 8 | # Magic Leap, Inc., All Rights Reserved. 9 | # 10 | # NOTICE: All information contained herein is, and remains the property 11 | # of COMPANY. The intellectual and technical concepts contained herein 12 | # are proprietary to COMPANY and may be covered by U.S. and Foreign 13 | # Patents, patents in process, and are protected by trade secret or 14 | # copyright law. Dissemination of this information or reproduction of 15 | # this material is strictly forbidden unless prior written permission is 16 | # obtained from COMPANY. Access to the source code contained herein is 17 | # hereby forbidden to anyone except current COMPANY employees, managers 18 | # or contractors who have executed Confidentiality and Non-disclosure 19 | # agreements explicitly covering such access. 20 | # 21 | # The copyright notice above does not evidence any actual or intended 22 | # publication or disclosure of this source code, which includes 23 | # information that is confidential and/or proprietary, and is a trade 24 | # secret, of COMPANY. ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, 25 | # PUBLIC PERFORMANCE, OR PUBLIC DISPLAY OF OR THROUGH USE OF THIS 26 | # SOURCE CODE WITHOUT THE EXPRESS WRITTEN CONSENT OF COMPANY IS 27 | # STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE LAWS AND 28 | # INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE 29 | # CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS 30 | # TO REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, 31 | # USE, OR SELL ANYTHING THAT IT MAY DESCRIBE, IN WHOLE OR IN PART. 32 | # 33 | # %COPYRIGHT_END% 34 | # ---------------------------------------------------------------------- 35 | # %AUTHORS_BEGIN% 36 | # 37 | # Originating Authors: Paul-Edouard Sarlin 38 | # Daniel DeTone 39 | # Tomasz Malisiewicz 40 | # 41 | # %AUTHORS_END% 42 | # --------------------------------------------------------------------*/ 43 | # %BANNER_END% 44 | 45 | from pathlib import Path 46 | import time 47 | from collections import OrderedDict 48 | from threading import Thread 49 | import numpy as np 50 | import cv2 51 | import torch 52 | import matplotlib.pyplot as plt 53 | import matplotlib 54 | matplotlib.use('Agg') 55 | 56 | 57 | class AverageTimer: 58 | """ Class to help manage printing simple timing of code execution. """ 59 | 60 | def __init__(self, smoothing=0.3, newline=False): 61 | self.smoothing = smoothing 62 | self.newline = newline 63 | self.times = OrderedDict() 64 | self.will_print = OrderedDict() 65 | self.reset() 66 | 67 | def reset(self): 68 | now = time.time() 69 | self.start = now 70 | self.last_time = now 71 | for name in self.will_print: 72 | self.will_print[name] = False 73 | 74 | def update(self, name='default'): 75 | now = time.time() 76 | dt = now - self.last_time 77 | if name in self.times: 78 | dt = self.smoothing * dt + (1 - self.smoothing) * self.times[name] 79 | self.times[name] = dt 80 | self.will_print[name] = True 81 | self.last_time = now 82 | 83 | def print(self, text='Timer'): 84 | total = 0. 85 | print('[{}]'.format(text), end=' ') 86 | for key in self.times: 87 | val = self.times[key] 88 | if self.will_print[key]: 89 | print('%s=%.3f' % (key, val), end=' ') 90 | total += val 91 | print('total=%.3f sec {%.1f FPS}' % (total, 1./total), end=' ') 92 | if self.newline: 93 | print(flush=True) 94 | else: 95 | print(end='\r', flush=True) 96 | self.reset() 97 | 98 | 99 | class VideoStreamer: 100 | """ Class to help process image streams. Four types of possible inputs:" 101 | 1.) USB Webcam. 102 | 2.) An IP camera 103 | 3.) A directory of images (files in directory matching 'image_glob'). 104 | 4.) A video file, such as an .mp4 or .avi file. 105 | """ 106 | def __init__(self, basedir, resize, skip, image_glob, max_length=1000000): 107 | self._ip_grabbed = False 108 | self._ip_running = False 109 | self._ip_camera = False 110 | self._ip_image = None 111 | self._ip_index = 0 112 | self.cap = [] 113 | self.camera = True 114 | self.video_file = False 115 | self.listing = [] 116 | self.resize = resize 117 | self.interp = cv2.INTER_AREA 118 | self.i = 0 119 | self.skip = skip 120 | self.max_length = max_length 121 | if isinstance(basedir, int) or basedir.isdigit(): 122 | print('==> Processing USB webcam input: {}'.format(basedir)) 123 | self.cap = cv2.VideoCapture(int(basedir)) 124 | self.listing = range(0, self.max_length) 125 | elif basedir.startswith(('http', 'rtsp')): 126 | print('==> Processing IP camera input: {}'.format(basedir)) 127 | self.cap = cv2.VideoCapture(basedir) 128 | self.start_ip_camera_thread() 129 | self._ip_camera = True 130 | self.listing = range(0, self.max_length) 131 | elif Path(basedir).is_dir(): 132 | print('==> Processing image directory input: {}'.format(basedir)) 133 | self.listing = list(Path(basedir).glob(image_glob[0])) 134 | for j in range(1, len(image_glob)): 135 | image_path = list(Path(basedir).glob(image_glob[j])) 136 | self.listing = self.listing + image_path 137 | self.listing.sort() 138 | self.listing = self.listing[::self.skip] 139 | self.max_length = np.min([self.max_length, len(self.listing)]) 140 | if self.max_length == 0: 141 | raise IOError('No images found (maybe bad \'image_glob\' ?)') 142 | self.listing = self.listing[:self.max_length] 143 | self.camera = False 144 | elif Path(basedir).exists(): 145 | print('==> Processing video input: {}'.format(basedir)) 146 | self.cap = cv2.VideoCapture(basedir) 147 | self.cap.set(cv2.CAP_PROP_BUFFERSIZE, 1) 148 | num_frames = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT)) 149 | self.listing = range(0, num_frames) 150 | self.listing = self.listing[::self.skip] 151 | self.video_file = True 152 | self.max_length = np.min([self.max_length, len(self.listing)]) 153 | self.listing = self.listing[:self.max_length] 154 | else: 155 | raise ValueError('VideoStreamer input \"{}\" not recognized.'.format(basedir)) 156 | if self.camera and not self.cap.isOpened(): 157 | raise IOError('Could not read camera') 158 | 159 | def load_image(self, impath): 160 | """ Read image as grayscale and resize to img_size. 161 | Inputs 162 | impath: Path to input image. 163 | Returns 164 | grayim: uint8 numpy array sized H x W. 165 | """ 166 | grayim = cv2.imread(impath, 0) 167 | if grayim is None: 168 | raise Exception('Error reading image %s' % impath) 169 | w, h = grayim.shape[1], grayim.shape[0] 170 | w_new, h_new = process_resize(w, h, self.resize) 171 | grayim = cv2.resize( 172 | grayim, (w_new, h_new), interpolation=self.interp) 173 | return grayim 174 | 175 | def next_frame(self): 176 | """ Return the next frame, and increment internal counter. 177 | Returns 178 | image: Next H x W image. 179 | status: True or False depending whether image was loaded. 180 | """ 181 | 182 | if self.i == self.max_length: 183 | return (None, False) 184 | if self.camera: 185 | 186 | if self._ip_camera: 187 | #Wait for first image, making sure we haven't exited 188 | while self._ip_grabbed is False and self._ip_exited is False: 189 | time.sleep(.001) 190 | 191 | ret, image = self._ip_grabbed, self._ip_image.copy() 192 | if ret is False: 193 | self._ip_running = False 194 | else: 195 | ret, image = self.cap.read() 196 | if ret is False: 197 | print('VideoStreamer: Cannot get image from camera') 198 | return (None, False) 199 | w, h = image.shape[1], image.shape[0] 200 | if self.video_file: 201 | self.cap.set(cv2.CAP_PROP_POS_FRAMES, self.listing[self.i]) 202 | 203 | w_new, h_new = process_resize(w, h, self.resize) 204 | image = cv2.resize(image, (w_new, h_new), 205 | interpolation=self.interp) 206 | image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) 207 | else: 208 | image_file = str(self.listing[self.i]) 209 | image = self.load_image(image_file) 210 | self.i = self.i + 1 211 | return (image, True) 212 | 213 | def start_ip_camera_thread(self): 214 | self._ip_thread = Thread(target=self.update_ip_camera, args=()) 215 | self._ip_running = True 216 | self._ip_thread.start() 217 | self._ip_exited = False 218 | return self 219 | 220 | def update_ip_camera(self): 221 | while self._ip_running: 222 | ret, img = self.cap.read() 223 | if ret is False: 224 | self._ip_running = False 225 | self._ip_exited = True 226 | self._ip_grabbed = False 227 | return 228 | 229 | self._ip_image = img 230 | self._ip_grabbed = ret 231 | self._ip_index += 1 232 | #print('IPCAMERA THREAD got frame {}'.format(self._ip_index)) 233 | 234 | 235 | def cleanup(self): 236 | self._ip_running = False 237 | 238 | # --- PREPROCESSING --- 239 | 240 | def process_resize(w, h, resize): 241 | assert(len(resize) > 0 and len(resize) <= 2) 242 | if len(resize) == 1 and resize[0] > -1: 243 | scale = resize[0] / max(h, w) 244 | w_new, h_new = int(round(w*scale)), int(round(h*scale)) 245 | elif len(resize) == 1 and resize[0] == -1: 246 | w_new, h_new = w, h 247 | else: # len(resize) == 2: 248 | w_new, h_new = resize[0], resize[1] 249 | 250 | # Issue warning if resolution is too small or too large. 251 | if max(w_new, h_new) < 160: 252 | print('Warning: input resolution is very small, results may vary') 253 | elif max(w_new, h_new) > 2000: 254 | print('Warning: input resolution is very large, results may vary') 255 | 256 | return w_new, h_new 257 | 258 | 259 | def frame2tensor(frame, device): 260 | return torch.from_numpy(frame/255.).float()[None, None].to(device) 261 | 262 | 263 | def read_image(path, device, resize, rotation, resize_float): 264 | image = cv2.imread(str(path), cv2.IMREAD_GRAYSCALE) 265 | if image is None: 266 | return None, None, None 267 | w, h = image.shape[1], image.shape[0] 268 | w_new, h_new = process_resize(w, h, resize) 269 | scales = (float(w) / float(w_new), float(h) / float(h_new)) 270 | 271 | if resize_float: 272 | image = cv2.resize(image.astype('float32'), (w_new, h_new)) 273 | else: 274 | image = cv2.resize(image, (w_new, h_new)).astype('float32') 275 | 276 | if rotation != 0: 277 | image = np.rot90(image, k=rotation) 278 | if rotation % 2: 279 | scales = scales[::-1] 280 | 281 | inp = frame2tensor(image, device) 282 | return image, inp, scales 283 | 284 | 285 | # --- GEOMETRY --- 286 | 287 | 288 | def estimate_pose(kpts0, kpts1, K0, K1, thresh, conf=0.99999): 289 | if len(kpts0) < 5: 290 | return None 291 | 292 | f_mean = np.mean([K0[0, 0], K1[1, 1], K0[0, 0], K1[1, 1]]) 293 | norm_thresh = thresh / f_mean 294 | 295 | kpts0 = (kpts0 - K0[[0, 1], [2, 2]][None]) / K0[[0, 1], [0, 1]][None] 296 | kpts1 = (kpts1 - K1[[0, 1], [2, 2]][None]) / K1[[0, 1], [0, 1]][None] 297 | 298 | E, mask = cv2.findEssentialMat( 299 | kpts0, kpts1, np.eye(3), threshold=norm_thresh, prob=conf, 300 | method=cv2.RANSAC) 301 | 302 | assert E is not None 303 | 304 | best_num_inliers = 0 305 | ret = None 306 | for _E in np.split(E, len(E) / 3): 307 | n, R, t, _ = cv2.recoverPose( 308 | _E, kpts0, kpts1, np.eye(3), 1e9, mask=mask) 309 | if n > best_num_inliers: 310 | best_num_inliers = n 311 | ret = (R, t[:, 0], mask.ravel() > 0) 312 | return ret 313 | 314 | 315 | def rotate_intrinsics(K, image_shape, rot): 316 | """image_shape is the shape of the image after rotation""" 317 | assert rot <= 3 318 | h, w = image_shape[:2][::-1 if (rot % 2) else 1] 319 | fx, fy, cx, cy = K[0, 0], K[1, 1], K[0, 2], K[1, 2] 320 | rot = rot % 4 321 | if rot == 1: 322 | return np.array([[fy, 0., cy], 323 | [0., fx, w-1-cx], 324 | [0., 0., 1.]], dtype=K.dtype) 325 | elif rot == 2: 326 | return np.array([[fx, 0., w-1-cx], 327 | [0., fy, h-1-cy], 328 | [0., 0., 1.]], dtype=K.dtype) 329 | else: # if rot == 3: 330 | return np.array([[fy, 0., h-1-cy], 331 | [0., fx, cx], 332 | [0., 0., 1.]], dtype=K.dtype) 333 | 334 | 335 | def rotate_pose_inplane(i_T_w, rot): 336 | rotation_matrices = [ 337 | np.array([[np.cos(r), -np.sin(r), 0., 0.], 338 | [np.sin(r), np.cos(r), 0., 0.], 339 | [0., 0., 1., 0.], 340 | [0., 0., 0., 1.]], dtype=np.float32) 341 | for r in [np.deg2rad(d) for d in (0, 270, 180, 90)] 342 | ] 343 | return np.dot(rotation_matrices[rot], i_T_w) 344 | 345 | 346 | def scale_intrinsics(K, scales): 347 | scales = np.diag([1./scales[0], 1./scales[1], 1.]) 348 | return np.dot(scales, K) 349 | 350 | 351 | def to_homogeneous(points): 352 | return np.concatenate([points, np.ones_like(points[:, :1])], axis=-1) 353 | 354 | 355 | def compute_epipolar_error(kpts0, kpts1, T_0to1, K0, K1): 356 | kpts0 = (kpts0 - K0[[0, 1], [2, 2]][None]) / K0[[0, 1], [0, 1]][None] 357 | kpts1 = (kpts1 - K1[[0, 1], [2, 2]][None]) / K1[[0, 1], [0, 1]][None] 358 | kpts0 = to_homogeneous(kpts0) 359 | kpts1 = to_homogeneous(kpts1) 360 | 361 | t0, t1, t2 = T_0to1[:3, 3] 362 | t_skew = np.array([ 363 | [0, -t2, t1], 364 | [t2, 0, -t0], 365 | [-t1, t0, 0] 366 | ]) 367 | E = t_skew @ T_0to1[:3, :3] 368 | 369 | Ep0 = kpts0 @ E.T # N x 3 370 | p1Ep0 = np.sum(kpts1 * Ep0, -1) # N 371 | Etp1 = kpts1 @ E # N x 3 372 | d = p1Ep0**2 * (1.0 / (Ep0[:, 0]**2 + Ep0[:, 1]**2) 373 | + 1.0 / (Etp1[:, 0]**2 + Etp1[:, 1]**2)) 374 | return d 375 | 376 | 377 | def angle_error_mat(R1, R2): 378 | cos = (np.trace(np.dot(R1.T, R2)) - 1) / 2 379 | cos = np.clip(cos, -1., 1.) # numercial errors can make it out of bounds 380 | return np.rad2deg(np.abs(np.arccos(cos))) 381 | 382 | 383 | def angle_error_vec(v1, v2): 384 | n = np.linalg.norm(v1) * np.linalg.norm(v2) 385 | return np.rad2deg(np.arccos(np.clip(np.dot(v1, v2) / n, -1.0, 1.0))) 386 | 387 | 388 | def compute_pose_error(T_0to1, R, t): 389 | R_gt = T_0to1[:3, :3] 390 | t_gt = T_0to1[:3, 3] 391 | error_t = angle_error_vec(t, t_gt) 392 | error_t = np.minimum(error_t, 180 - error_t) # ambiguity of E estimation 393 | error_R = angle_error_mat(R, R_gt) 394 | return error_t, error_R 395 | 396 | 397 | def pose_auc(errors, thresholds): 398 | sort_idx = np.argsort(errors) 399 | errors = np.array(errors.copy())[sort_idx] 400 | recall = (np.arange(len(errors)) + 1) / len(errors) 401 | errors = np.r_[0., errors] 402 | recall = np.r_[0., recall] 403 | aucs = [] 404 | for t in thresholds: 405 | last_index = np.searchsorted(errors, t) 406 | r = np.r_[recall[:last_index], recall[last_index-1]] 407 | e = np.r_[errors[:last_index], t] 408 | aucs.append(np.trapz(r, x=e)/t) 409 | return aucs 410 | 411 | 412 | # --- VISUALIZATION --- 413 | 414 | 415 | def plot_image_pair(imgs, dpi=100, size=6, pad=.5): 416 | n = len(imgs) 417 | assert n == 2, 'number of images must be two' 418 | figsize = (size*n, size*3/4) if size is not None else None 419 | _, ax = plt.subplots(1, n, figsize=figsize, dpi=dpi) 420 | for i in range(n): 421 | ax[i].imshow(imgs[i], cmap=plt.get_cmap('gray'), vmin=0, vmax=255) 422 | ax[i].get_yaxis().set_ticks([]) 423 | ax[i].get_xaxis().set_ticks([]) 424 | for spine in ax[i].spines.values(): # remove frame 425 | spine.set_visible(False) 426 | plt.tight_layout(pad=pad) 427 | 428 | 429 | def plot_keypoints(kpts0, kpts1, color='w', ps=2): 430 | ax = plt.gcf().axes 431 | ax[0].scatter(kpts0[:, 0], kpts0[:, 1], c=color, s=ps) 432 | ax[1].scatter(kpts1[:, 0], kpts1[:, 1], c=color, s=ps) 433 | 434 | 435 | def plot_matches(kpts0, kpts1, color, lw=1.5, ps=4): 436 | fig = plt.gcf() 437 | ax = fig.axes 438 | fig.canvas.draw() 439 | 440 | transFigure = fig.transFigure.inverted() 441 | fkpts0 = transFigure.transform(ax[0].transData.transform(kpts0)) 442 | fkpts1 = transFigure.transform(ax[1].transData.transform(kpts1)) 443 | 444 | fig.lines = [matplotlib.lines.Line2D( 445 | (fkpts0[i, 0], fkpts1[i, 0]), (fkpts0[i, 1], fkpts1[i, 1]), zorder=1, 446 | transform=fig.transFigure, c=color[i], linewidth=lw) 447 | for i in range(len(kpts0))] 448 | ax[0].scatter(kpts0[:, 0], kpts0[:, 1], c=color, s=ps) 449 | ax[1].scatter(kpts1[:, 0], kpts1[:, 1], c=color, s=ps) 450 | 451 | 452 | def make_matching_plot(image0, image1, kpts0, kpts1, mkpts0, mkpts1, 453 | color, text, path, show_keypoints=False, 454 | fast_viz=False, opencv_display=False, 455 | opencv_title='matches', small_text=[]): 456 | 457 | if fast_viz: 458 | make_matching_plot_fast(image0, image1, kpts0, kpts1, mkpts0, mkpts1, 459 | color, text, path, show_keypoints, 10, 460 | opencv_display, opencv_title, small_text) 461 | return 462 | 463 | plot_image_pair([image0, image1]) 464 | if show_keypoints: 465 | plot_keypoints(kpts0, kpts1, color='k', ps=4) 466 | plot_keypoints(kpts0, kpts1, color='w', ps=2) 467 | plot_matches(mkpts0, mkpts1, color) 468 | 469 | fig = plt.gcf() 470 | txt_color = 'k' if image0[:100, :150].mean() > 200 else 'w' 471 | fig.text( 472 | 0.01, 0.99, '\n'.join(text), transform=fig.axes[0].transAxes, 473 | fontsize=15, va='top', ha='left', color=txt_color) 474 | 475 | txt_color = 'k' if image0[-100:, :150].mean() > 200 else 'w' 476 | fig.text( 477 | 0.01, 0.01, '\n'.join(small_text), transform=fig.axes[0].transAxes, 478 | fontsize=5, va='bottom', ha='left', color=txt_color) 479 | 480 | plt.savefig(str(path), bbox_inches='tight', pad_inches=0) 481 | plt.close() 482 | 483 | 484 | def make_matching_plot_fast(image0, image1, kpts0, kpts1, mkpts0, 485 | mkpts1, color, text, path=None, 486 | show_keypoints=False, margin=10, 487 | opencv_display=False, opencv_title='', 488 | small_text=[]): 489 | H0, W0 = image0.shape 490 | H1, W1 = image1.shape 491 | H, W = max(H0, H1), W0 + W1 + margin 492 | 493 | out = 255*np.ones((H, W), np.uint8) 494 | out[:H0, :W0] = image0 495 | out[:H1, W0+margin:] = image1 496 | out = np.stack([out]*3, -1) 497 | 498 | if show_keypoints: 499 | kpts0, kpts1 = np.round(kpts0).astype(int), np.round(kpts1).astype(int) 500 | white = (255, 255, 255) 501 | black = (0, 0, 0) 502 | for x, y in kpts0: 503 | cv2.circle(out, (x, y), 2, black, -1, lineType=cv2.LINE_AA) 504 | cv2.circle(out, (x, y), 1, white, -1, lineType=cv2.LINE_AA) 505 | for x, y in kpts1: 506 | cv2.circle(out, (x + margin + W0, y), 2, black, -1, 507 | lineType=cv2.LINE_AA) 508 | cv2.circle(out, (x + margin + W0, y), 1, white, -1, 509 | lineType=cv2.LINE_AA) 510 | 511 | mkpts0, mkpts1 = np.round(mkpts0).astype(int), np.round(mkpts1).astype(int) 512 | color = (np.array(color[:, :3])*255).astype(int)[:, ::-1] 513 | for (x0, y0), (x1, y1), c in zip(mkpts0, mkpts1, color): 514 | c = c.tolist() 515 | cv2.line(out, (x0, y0), (x1 + margin + W0, y1), 516 | color=c, thickness=1, lineType=cv2.LINE_AA) 517 | # display line end-points as circles 518 | cv2.circle(out, (x0, y0), 2, c, -1, lineType=cv2.LINE_AA) 519 | cv2.circle(out, (x1 + margin + W0, y1), 2, c, -1, 520 | lineType=cv2.LINE_AA) 521 | 522 | # Scale factor for consistent visualization across scales. 523 | sc = min(H / 640., 2.0) 524 | 525 | # Big text. 526 | Ht = int(30 * sc) # text height 527 | txt_color_fg = (255, 255, 255) 528 | txt_color_bg = (0, 0, 0) 529 | for i, t in enumerate(text): 530 | cv2.putText(out, t, (int(8*sc), Ht*(i+1)), cv2.FONT_HERSHEY_DUPLEX, 531 | 1.0*sc, txt_color_bg, 2, cv2.LINE_AA) 532 | cv2.putText(out, t, (int(8*sc), Ht*(i+1)), cv2.FONT_HERSHEY_DUPLEX, 533 | 1.0*sc, txt_color_fg, 1, cv2.LINE_AA) 534 | 535 | # Small text. 536 | Ht = int(18 * sc) # text height 537 | for i, t in enumerate(reversed(small_text)): 538 | cv2.putText(out, t, (int(8*sc), int(H-Ht*(i+.6))), cv2.FONT_HERSHEY_DUPLEX, 539 | 0.5*sc, txt_color_bg, 2, cv2.LINE_AA) 540 | cv2.putText(out, t, (int(8*sc), int(H-Ht*(i+.6))), cv2.FONT_HERSHEY_DUPLEX, 541 | 0.5*sc, txt_color_fg, 1, cv2.LINE_AA) 542 | 543 | if path is not None: 544 | cv2.imwrite(str(path), out) 545 | 546 | if opencv_display: 547 | cv2.imshow(opencv_title, out) 548 | cv2.waitKey(1) 549 | 550 | return out 551 | 552 | 553 | def error_colormap(x): 554 | return np.clip( 555 | np.stack([2-x*2, x*2, np.zeros_like(x), np.ones_like(x)], -1), 0, 1) 556 | -------------------------------------------------------------------------------- /models/weights/superglue_indoor.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/models/weights/superglue_indoor.pth -------------------------------------------------------------------------------- /models/weights/superglue_outdoor.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/models/weights/superglue_outdoor.pth -------------------------------------------------------------------------------- /models/weights/superpoint_v1.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/models/weights/superpoint_v1.pth -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | matplotlib>=3.1.3 2 | torch>=1.1.0 3 | opencv-python==4.1.2.30 4 | numpy>=1.18.1 5 | --------------------------------------------------------------------------------