├── README.md └── local_feature_evaluation ├── README.md ├── camera.py ├── data └── aachen-day-night │ ├── database.db │ ├── database_v1_1.db │ ├── image_pairs_to_match.txt │ └── image_pairs_to_match_v1_1.txt ├── matchers.py ├── modify_database_with_custom_features_and_matches.py ├── reconstruction_pipeline.py ├── reconstruction_pipeline_aachen_v1_1.py └── utils.py /README.md: -------------------------------------------------------------------------------- 1 | # Long-Term Visual Localization Benchmark 2 | This is the Github repository for the long-term localization benchmark hosted at http://visuallocalization.net/. 3 | The main purpose of the repository is to allow easy discussion and the reporting of issues. 4 | 5 | We also use this repository to point to publicly available implementations of visual localization methods: 6 | * Active Search: https://www.graphics.rwth-aachen.de/software/image-localization 7 | * DenseVLAD: http://www.ok.ctrl.titech.ac.jp/~torii/project/247/ 8 | * NetVLAD: https://www.di.ens.fr/willow/research/netvlad/ 9 | * DenseSFM: http://www.ok.sc.e.titech.ac.jp/res/DenseSfM/index.html 10 | * InLoc: http://www.ok.sc.e.titech.ac.jp/INLOC/ 11 | * Hierarchical Localization Toolbox: https://github.com/cvg/Hierarchical-Localization 12 | 13 | You can find information on how to use custom features inside a retrieval + pose estimation pipeline in the `local_feature_evaluation/` subdirectory. More specifically, the section `Using COLMAP for Visual Localization with Custom Features` in that directory's README file provides the information necessary to implement such a pipeline using COLMAP and an external retrieval approach. 14 | -------------------------------------------------------------------------------- /local_feature_evaluation/README.md: -------------------------------------------------------------------------------- 1 | # Visual Localization Benchmark - Local Features 2 | 3 | This repository provides a code for evaluating local features in the context of long-term visual localization as part of the [Long-term Visual Localization Benchmark](https://visuallocalization.net/). Given features extracted for images, the code provides functionality to match them. The matches are then used to first triangulate the 3D structure of the scene from a set of database images. The matches between a set of query images and the database images then provide a set of 2D-3D matches that are used to estimate the camera poses of the query images. The output is a submission file that can be uploaded directly to the benchmark's website. 4 | 5 | The steps of the reconstruction pipeline are the following: 6 | * generate an empty reconstruction with the parameters of database images, 7 | * import the features into the database, 8 | * match the features and import the matches into the database, 9 | * geometrically verify the matches, 10 | * triangulate the database observations in the 3D model at **fixed intrinsics**, 11 | * register the query images at **fixed intrinsics**. 12 | 13 | We provide a mutual nearest neighbors matcher implemented in PyTorch as an example - you can replace it by your favorite matching method (e.g. one way nearest neighbors + ratio test). 14 | 15 | **Apart from the matcher, please keep the rest of the pipeline intact in order to have as fair a comparison as possible.** 16 | 17 | ## Usage 18 | ### Prerequisites 19 | The provided code requires that [COLMAP](https://colmap.github.io/), a state-of-the-art Structure-from-Motion pipeline, is available on the system. 20 | 21 | The code was tested on Python 3.7; it should work without issues on Python 3+. [Conda](https://docs.conda.io/en/latest/) can be used to install the missing packages: 22 | 23 | ```bash 24 | conda install numpy tqdm 25 | conda install pytorch cudatoolkit=9.0 -c pytorch # required for the provided mutual NN matcher 26 | ``` 27 | 28 | ### Dataset Preparation 29 | This code currently supports **only the Aachen Day-Night** dataset. Further datasets might be supported in the future. 30 | For the dataset, we provide two files ``database.db`` and ``image_pairs_to_match.txt``, which are in the ``data/aachen-day-night/`` sub-directory of this repository. You will need to move them to directory where you are storing the Aachen Day-Night dataset. In order for the script to function properly, the directory should have the following structure: 31 | 32 | ``` 33 | . 34 | ├── database.db 35 | ├── image_pairs_to_match.txt 36 | ├── images 37 | │ └── images_upright 38 | ├── 3D-models 39 | │ ├── database_intrinsics.txt 40 | │ └── aachen_cvpr2018_db.nvm 41 | └── queries/night_time_queries_with_intrinsics.txt 42 | ``` 43 | Here, 44 | - `database.db` is an COLMAP database containing all images and intrinsics (this file is **provided in this repository**). 45 | 46 | - `image_pairs_to_match.txt` is a list of images to be matched (one pair per line) (this file is **provided in this repository**). 47 | 48 | - `images` contains both the database and query images. For Aachen Day-Night, it should contain one sub-folder `images_upright`, which in turn has two subfolders - `db` and `query`. This data is provided by the dataset. 49 | 50 | - `3D-models` contains the reference database model in `nvm` format and a list of database images with their COLMAP intrinsics. This data is provided by the dataset. 51 | 52 | - `night_time_queries_with_intrinsics.txt` contains the list of query images with their COLMAP intrinsics (the intrinsics are not used since they are supposed to be part of the database already - a list of query images should suffice). **Only the night-time images are currently used**. This data is provided by the dataset. 53 | 54 | In order to run the script, you will first need to extract your local features (see below). Then call the script, e.g., via 55 | ``` 56 | python reconstruction_pipeline.py 57 | --dataset_path /local/aachen 58 | --colmap_path /local/colmap/build/src/exe 59 | --method_name d2-net 60 | ``` 61 | 62 | ### Local Features Format 63 | 64 | Our script tries to load local features per image from files. It is your responsibility to create these files. 65 | 66 | The local features should be stored in the [`npz`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html) format with two fields: 67 | 68 | - `keypoints` - `N x 2` matrix with `x, y` coordinates of each keypoint in COLMAP format (the `X` axis points to the right, the `Y` axis to the bottom) 69 | 70 | - `descriptors` - `N x D` matrix with the descriptors (L2 normalized if you plan on using the provided mutual nearest neighbors matcher) 71 | 72 | Moreover, they are supposed to be saved alongside their corresponding images with an extension corresponding to the `method_name` (e.g. if `method_name = d2-net` the features for the image `/local/aachen/images/images_upright/db/1000.jpg` should be in the file `/local/aachen/images/images_upright/db/1000.jpg.d2-net`). 73 | 74 | **Important information**: In order to work, our script requires that the local features are extracted at the **original image resolutions**. If you downscale the images before feature extraction, you will need to scale the keypoint positions to the original resolutions. **Otherwise, the camera pose estimation stage will fail**. 75 | 76 | ### Local Feature Evaluation on Aachen Day-Night v1.1 77 | Make sure that you downloaded both the original Aachen Day-Night dataset and the Aachen Day-Night v1.1 dataset (both are available at [visuallocalization.net](https://www.visuallocalization.net/datasets/)). Follow the instructions in the README file of the Aachen Day-Night v1.1 dataset when extracting the data from the .zip file. For the v1.1 version of the dataset, we provide two files ``database_v1_1.db`` and ``image_pairs_to_match_v1_1.txt``, which are in the ``data/aachen-day-night/`` sub-directory of this repository. You will need to move them to directory where you are storing the Aachen Day-Night dataset. In order for the script to function properly, the directory should have the following structure: 78 | 79 | ``` 80 | . 81 | ├── database_v1_1.db 82 | ├── image_pairs_to_match_v1_1.txt 83 | ├── images 84 | │ └── images_upright 85 | ├── 3D-models 86 | │ ├── aachen_v_1_1/ 87 | │ ├── aachen_v_1_1.nvm 88 | │ └── database_intrinsics_v1_1.txt 89 | └── queries/night_time_queries_with_intrinsics.txt 90 | ``` 91 | 92 | As for the original Aachen Day-Night dataset, you can run pose estimation for your local features through the script 93 | ``` 94 | python reconstruction_pipeline_aachen_v1_1.py 95 | --dataset_path /local/aachen 96 | --colmap_path /local/colmap/build/src/exe 97 | --method_name d2-net 98 | ``` 99 | Please see above for details. 100 | 101 | ### Citing 102 | 103 | If you use this code in your project, please cite the following paper: 104 | 105 | ``` 106 | @InProceedings{Dusmanu2019CVPR, 107 | author = {Dusmanu, Mihai and Rocco, Ignacio and Pajdla, Tomas and Pollefeys, Marc and Sivic, Josef and Torii, Akihiko and Sattler, Torsten}, 108 | title = {{D2-Net: A Trainable CNN for Joint Detection and Description of Local Features}}, 109 | booktitle = {Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 110 | year = {2019}, 111 | } 112 | ``` 113 | 114 | # Using COLMAP with Custom Features 115 | For convenience, and separate from the functionality of the local features benchmark, we also provide functionality to simply import custom local features and matches into any COLMAP database. This allows to use custom features inside COLMAP for arbitrary datasets. 116 | 117 | The provided approach assumes that there is a directory ```data_directory/``` that contains the images, an existing COLMAP database (for example ```db.db```), and text file containing pairs of image names to be matched (for example ```match_list.txt```). The database can be created by running COLMAP's feature detection on the images in the database (it cannot be empty). In order to use custom local features, ```.npz``` files containing these features (in the format from above) need to be saved alongside their corresponding images with an extension corresponding to the ```method_name```. 118 | 119 | Given these pre-requisites, ```modify_database_with_custom_features_and_matches.py``` can be used to import the features into the database, match images, and run geometric verification. The resulting database can then be used with COLMAP to reconstruct the scene. 120 | 121 | An example call to ```modify_database_with_custom_features_and_matches.py```, using D2-Net features, is 122 | ``` 123 | python modify_database_with_custom_features_and_matches.py 124 | --dataset_path data_directory/ 125 | --colmap_path /local/colmap/build/src/exe 126 | --method_name d2-net 127 | --database_name db.db 128 | --image_path images/ 129 | --match_list match_list.txt 130 | ``` 131 | The call assumes that there is a directory ```data_directory/``` that contains an existing COLMAP database ```db.db```, an folder ```images/``` that contains all images (for each image with name ```XX.jpg```, there is assume to be a file ```XX.jpg.d2-net``` containing the D2-Net features for that image), and a text file ```match_list.txt``` that contains the image pairs to be matched. The call will create a new database file ```d2-net.db``` in the ```data_directory/``` directory. 132 | 133 | **Important information:** 134 | * The provided approach can currently not be used to extend an existing database where new images were added as it deletes all features and matches in the database. A way around this would be to first export and then re-import the existing matches in the database. 135 | * The call will abort with an error if there is already a database with the name ```d2-net.db``` (or in general ```your_method_name.db```). 136 | * We are currently not providing support for this functionality. It is provided simply for convenience for experienced COLMAP users. 137 | * If the existing database contains feature matches, especially verified matches, then this will cause problems. Make sure that the database does not contain such matches, e.g., by clearing them inside COLMAP's gui. 138 | 139 | # Using COLMAP for Visual Localization with Custom Features 140 | The above pipeline can be used to implement the following visual localization pipeline in COLMAP using custom features: 141 | 1. For each query image, find the top-k most similar database images. 142 | 2. Match custom features between each query and its top-k retrieved images. 143 | 3. Estimate the poses of the query images with respect to a 3D model based on the matches from step 2. 144 | 145 | The following describes how to implement steps 2 and 3 in COLMAP. 146 | 147 | ### Prerequisites 148 | * We assume that image retrieval is implemented elsewhere, e.g., using DenseVLAD or NetVLAD, and that text file containing pairs of query file names and database file names (one pair per line, see COLMAP's "Custom Matching" format for image pairs [here](https://colmap.github.io/tutorial.html#feature-matching-and-geometric-verification)) is available. In the following, we will refer to this file as `retrieval_list.txt`. 149 | * A COLMAP database (e.g., `db.db`) containing the database and query images as well as their intrinsics. This database can be constructed by extracting SIFT features. **Important**: Make sure that the correct camera intrinsics that should be used for pose estimation for the query images are stored inside the database. 150 | * A COLMAP sparse reconstruction consistent with the database, i.e., the IMAGE_IDs stored in `images.bin` / `images.txt` are consistent with the IMAGE_IDs stored in the database. The sparse model can be either a complete reconstruction, e.g., a 3D model obtained from SIFT features, or an "empty" reconstruction, i.e., a model that only contains camera poses and camera intrinsics but no 3D points. 151 | * A text file containing pairs of database images that should be matched. For example, this could be an exhaustive list of pairs or the pairs of database images you matched when building a 3D model (see above). The format is the same as for `retrieval_list.txt`. In the following, we will refer to this list as `database_pairs_list.txt`. 152 | 153 | ### Localization 154 | 1. *Extract custom features*: Run your feature extractor and store the features in the local feature format described above. You should extract features for both database and query features. 155 | 2. *Import all features and match database images*: This is done using the `modify_database_with_custom_features_and_matches.py` tool provided in this repository (please see above for details) by calling (as an example): 156 | ``` 157 | python modify_database_with_custom_features_and_matches.py 158 | --dataset_path data_directory/ 159 | --colmap_path /local/colmap/build/src/exe 160 | --method_name your_feature_name 161 | --database_name db.db 162 | --image_path images/ 163 | --match_list database_pairs_list.txt 164 | ``` 165 | Here, `data_directory/` is the directory containing all the data and `db.db` is the name of the existing database (i.e., `data_directory/db.db` is the path of this file). `your_feature_name` is the name of your features (see above) and the directory `data_directory/images/` contains the query and database images (potentially in subdirectories). It is assumed that `database_pairs_list.txt` is located in `data_directory/`, i.e., that the file is `data_directory/database_pairs_list.txt`. This call will create a new database `data_directory/your_feature_name.db` from `db.db` (without altering `db.db`), will imported the custom features to `your_feature_name.db`, match the pairs of database images specified in `database_pairs_list.txt`, perform spatial verification, and store the matches in `your_feature_name.db`. This step can take a long time. **Important**: Make sure that `data_directory/your_feature_name.db` does not exist prior to the call. 166 | 167 | 3. *Build a database 3D model*: In the next step, we build the 3D model from the custom features and the matches that will be used for localization. This is done by calling 168 | ``` 169 | colmap point_triangulator --database_path data_directory/your_feature_name.db --image_path data_directory/images/ --input_path data_directory/existing_model/ --output_path data_directory/model_your_feature_name/ --clear_points 1 170 | ``` 171 | Here, `data_directory/existing_model/` is the existing 3D model mentioned in the prerequisites above. The resulting 3D model will be stored in `data_directory/model_your_feature_name/`. Make sure that this directory exists before the call. Note that this call does not run Structure-from-Motion from scratch but rather only triangulates the 3D structure of the scene from known poses for the database images. The poses and intrinsics of the database images are not changed. 172 | 173 | 4. *Perform feature matching between the query images and the retrieved database images*: This can be done by calling 174 | ``` 175 | python modify_database_with_custom_features_and_matches.py 176 | --dataset_path data_directory/ 177 | --colmap_path /local/colmap/build/src/exe 178 | --method_name your_feature_name 179 | --database_name db.db 180 | --image_path images/ 181 | --match_list retrieval_list.txt 182 | --matching_only True 183 | 184 | ``` 185 | This will perform feature matching between the query images and the top-k retrieved database images (as specified by the pairs in `retrieval_list.txt`) and import the results after geometric verification into `data_directory/your_feature_name.db`. **Important**: The actual custom feature descriptors are not stored in the COLMAP database. For matching, the features are loaded from the files generated in step 1, so make sure that you did not delete them before this step. After this step, you can delete them. 186 | 187 | 5. *Estimate the camera poses of the query images*: This final step uses the model build in step 3 and the matches from step 4 for camera pose estimation in COLMAP. This is achieved by calling 188 | ``` 189 | colmap image_registrator --database_path data_directory/your_feature_name.db --input_path data_directory/model_your_feature_name/ --output_path data_directory/model_your_feature_name_with_queries/ 190 | ``` 191 | This will register the images into the 3D model without altering it. The result is a 3D model containing the original 3D structure and database poses from `data_directory/model_your_feature_name/` as well as the poses of the query images (and their inlier 2D-3D matches) that could be localized. The result is stored as a COLMAP binary model in `data_directory/model_your_feature_name_with_queries/` (the path has to exist before the call). Afterwards, you can extract the camera poses from `data_directory/model_your_feature_name_with_queries/images.bin` (see [here](https://colmap.github.io/format.html#binary-file-format) for details). You might want to adjust some of COLMAP's parameters to improve performance. For example, for D2-Net on the Extended CMU Seasons dataset, we used the following settings: `--Mapper.min_num_matches 4 --Mapper.init_min_num_inliers 4 --Mapper.abs_pose_min_num_inliers 4 --Mapper.abs_pose_min_inlier_ratio 0.05 --Mapper.ba_refine_focal_length 0 --Mapper.ba_refine_extra_params 0 --Mapper.ba_local_max_num_iterations 50 --Mapper.abs_pose_max_error 20 --Mapper.filter_max_reproj_error 12`. 192 | -------------------------------------------------------------------------------- /local_feature_evaluation/camera.py: -------------------------------------------------------------------------------- 1 | # Simple COLMAP camera class. 2 | class Camera: 3 | def __init__(self): 4 | self.camera_model = None 5 | self.intrinsics = None 6 | self.qvec = None 7 | self.t = None 8 | 9 | def set_intrinsics(self, camera_model, intrinsics): 10 | self.camera_model = camera_model 11 | self.intrinsics = intrinsics 12 | 13 | def set_pose(self, qvec, t): 14 | self.qvec = qvec 15 | self.t = t 16 | -------------------------------------------------------------------------------- /local_feature_evaluation/data/aachen-day-night/database.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tsattler/visuallocalizationbenchmark/1f8e4311e6fc41519e3d9909e14fce9a7f013732/local_feature_evaluation/data/aachen-day-night/database.db -------------------------------------------------------------------------------- /local_feature_evaluation/data/aachen-day-night/database_v1_1.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tsattler/visuallocalizationbenchmark/1f8e4311e6fc41519e3d9909e14fce9a7f013732/local_feature_evaluation/data/aachen-day-night/database_v1_1.db -------------------------------------------------------------------------------- /local_feature_evaluation/matchers.py: -------------------------------------------------------------------------------- 1 | import torch 2 | 3 | 4 | # Mutual nearest neighbors matcher for L2 normalized descriptors. 5 | def mutual_nn_matcher(descriptors1, descriptors2): 6 | device = descriptors1.device 7 | sim = descriptors1 @ descriptors2.t() 8 | nn12 = torch.max(sim, dim=1)[1] 9 | nn21 = torch.max(sim, dim=0)[1] 10 | ids1 = torch.arange(0, sim.shape[0], device=device) 11 | mask = ids1 == nn21[nn12] 12 | matches = torch.stack([ids1[mask], nn12[mask]]).t() 13 | return matches.data.cpu().numpy() 14 | 15 | 16 | # Symmetric Lowe's ratio test matcher for L2 normalized descriptors. 17 | def ratio_matcher(descriptors1, descriptors2, ratio=0.8): 18 | device = descriptors1.device 19 | sim = descriptors1 @ descriptors2.t() 20 | 21 | # Retrieve top 2 nearest neighbors 1->2. 22 | nns_sim, nns = torch.topk(sim, 2, dim=1) 23 | nns_dist = torch.sqrt(2 - 2 * nns_sim) 24 | # Compute Lowe's ratio. 25 | ratios12 = nns_dist[:, 0] / (nns_dist[:, 1] + 1e-8) 26 | # Save first NN. 27 | nn12 = nns[:, 0] 28 | 29 | # Retrieve top 2 nearest neighbors 1->2. 30 | nns_sim, nns = torch.topk(sim.t(), 2, dim=1) 31 | nns_dist = torch.sqrt(2 - 2 * nns_sim) 32 | # Compute Lowe's ratio. 33 | ratios21 = nns_dist[:, 0] / (nns_dist[:, 1] + 1e-8) 34 | # Save first NN. 35 | nn21 = nns[:, 0] 36 | 37 | # Symmetric ratio test. 38 | ids1 = torch.arange(0, sim.shape[0], device=device) 39 | mask = torch.min(ratios12 <= ratio, ratios21[nn12] <= ratio) 40 | 41 | # Final matches. 42 | matches = torch.stack([ids1[mask], nn12[mask]], dim=-1) 43 | 44 | return matches.data.cpu().numpy() 45 | 46 | 47 | # Mutual NN + symmetric Lowe's ratio test matcher for L2 normalized descriptors. 48 | def mutual_nn_ratio_matcher(descriptors1, descriptors2, ratio=0.8): 49 | device = descriptors1.device 50 | sim = descriptors1 @ descriptors2.t() 51 | 52 | # Retrieve top 2 nearest neighbors 1->2. 53 | nns_sim, nns = torch.topk(sim, 2, dim=1) 54 | nns_dist = torch.sqrt(2 - 2 * nns_sim) 55 | # Compute Lowe's ratio. 56 | ratios12 = nns_dist[:, 0] / (nns_dist[:, 1] + 1e-8) 57 | # Save first NN and match similarity. 58 | nn12 = nns[:, 0] 59 | 60 | # Retrieve top 2 nearest neighbors 1->2. 61 | nns_sim, nns = torch.topk(sim.t(), 2, dim=1) 62 | nns_dist = torch.sqrt(2 - 2 * nns_sim) 63 | # Compute Lowe's ratio. 64 | ratios21 = nns_dist[:, 0] / (nns_dist[:, 1] + 1e-8) 65 | # Save first NN. 66 | nn21 = nns[:, 0] 67 | 68 | # Mutual NN + symmetric ratio test. 69 | ids1 = torch.arange(0, sim.shape[0], device=device) 70 | mask = torch.min(ids1 == nn21[nn12], torch.min(ratios12 <= ratio, ratios21[nn12] <= ratio)) 71 | 72 | # Final matches. 73 | matches = torch.stack([ids1[mask], nn12[mask]], dim=-1) 74 | 75 | return matches.data.cpu().numpy() 76 | 77 | -------------------------------------------------------------------------------- /local_feature_evaluation/modify_database_with_custom_features_and_matches.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | import numpy as np 4 | 5 | import os 6 | 7 | import shutil 8 | 9 | import subprocess 10 | 11 | import sqlite3 12 | 13 | import torch 14 | 15 | import types 16 | 17 | from tqdm import tqdm 18 | 19 | from matchers import mutual_nn_matcher 20 | 21 | from camera import Camera 22 | 23 | from utils import quaternion_to_rotation_matrix, camera_center_to_translation 24 | 25 | import sys 26 | IS_PYTHON3 = sys.version_info[0] >= 3 27 | 28 | def array_to_blob(array): 29 | if IS_PYTHON3: 30 | return array.tostring() 31 | else: 32 | return np.getbuffer(array) 33 | 34 | def recover_database_images_and_ids(paths, args): 35 | # Connect to the database. 36 | connection = sqlite3.connect(paths.database_path) 37 | cursor = connection.cursor() 38 | 39 | # Recover database images and ids. 40 | images = {} 41 | cameras = {} 42 | cursor.execute("SELECT name, image_id, camera_id FROM images;") 43 | for row in cursor: 44 | images[row[0]] = row[1] 45 | cameras[row[0]] = row[2] 46 | 47 | # Close the connection to the database. 48 | cursor.close() 49 | connection.close() 50 | 51 | return images, cameras 52 | 53 | 54 | def import_features(images, paths, args): 55 | # Connect to the database. 56 | connection = sqlite3.connect(paths.database_path) 57 | cursor = connection.cursor() 58 | 59 | cursor.execute("DELETE FROM keypoints;") 60 | cursor.execute("DELETE FROM descriptors;") 61 | cursor.execute("DELETE FROM matches;") 62 | connection.commit() 63 | 64 | # Import the features. 65 | print('Importing features...') 66 | 67 | for image_name, image_id in tqdm(images.items(), total=len(images.items())): 68 | features_path = os.path.join(paths.image_path, '%s.%s' % (image_name, args.method_name)) 69 | 70 | keypoints = np.load(features_path)['keypoints'] 71 | n_keypoints = keypoints.shape[0] 72 | 73 | # Keep only x, y coordinates. 74 | keypoints = keypoints[:, : 2] 75 | # Add placeholder scale, orientation. 76 | keypoints = np.concatenate([keypoints, np.ones((n_keypoints, 1)), np.zeros((n_keypoints, 1))], axis=1).astype(np.float32) 77 | 78 | keypoints_str = keypoints.tostring() 79 | cursor.execute("INSERT INTO keypoints(image_id, rows, cols, data) VALUES(?, ?, ?, ?);", 80 | (image_id, keypoints.shape[0], keypoints.shape[1], keypoints_str)) 81 | 82 | connection.commit() 83 | # Close the connection to the database. 84 | cursor.close() 85 | connection.close() 86 | 87 | 88 | def image_ids_to_pair_id(image_id1, image_id2): 89 | if image_id1 > image_id2: 90 | return 2147483647 * image_id2 + image_id1 91 | else: 92 | return 2147483647 * image_id1 + image_id2 93 | 94 | 95 | def match_features(images, paths, args): 96 | # Connect to the database. 97 | connection = sqlite3.connect(paths.database_path) 98 | cursor = connection.cursor() 99 | 100 | # Match the features and insert the matches in the database. 101 | print('Matching...') 102 | 103 | with open(paths.match_list_path, 'r') as f: 104 | raw_pairs = f.readlines() 105 | 106 | image_pair_ids = set() 107 | for raw_pair in tqdm(raw_pairs, total=len(raw_pairs)): 108 | image_name1, image_name2 = raw_pair.strip('\n').split(' ') 109 | 110 | features_path1 = os.path.join(paths.image_path, '%s.%s' % (image_name1, args.method_name)) 111 | features_path2 = os.path.join(paths.image_path, '%s.%s' % (image_name2, args.method_name)) 112 | 113 | #print(features_path1, features_path2) 114 | D1 = np.load(features_path1)['descriptors']; 115 | D2 = np.load(features_path2)['descriptors']; 116 | D1 = D1[:min(D1.shape[0],25000),:]; 117 | D2 = D2[:min(D2.shape[0],25000),:]; 118 | #print(D1.shape, D2.shape) 119 | 120 | #descriptors1 = torch.from_numpy(np.load(features_path1)['descriptors']).to(device) 121 | #descriptors2 = torch.from_numpy(np.load(features_path2)['descriptors']).to(device) 122 | descriptors1 = torch.from_numpy(D1).to(device) 123 | descriptors2 = torch.from_numpy(D2).to(device) 124 | matches = mutual_nn_matcher(descriptors1, descriptors2).astype(np.uint32) 125 | 126 | image_id1, image_id2 = images[image_name1], images[image_name2] 127 | image_pair_id = image_ids_to_pair_id(image_id1, image_id2) 128 | if image_pair_id in image_pair_ids: 129 | continue 130 | image_pair_ids.add(image_pair_id) 131 | 132 | if image_id1 > image_id2: 133 | matches = matches[:, [1, 0]] 134 | 135 | matches_str = matches.tostring() 136 | cursor.execute("INSERT INTO matches(pair_id, rows, cols, data) VALUES(?, ?, ?, ?);", 137 | (image_pair_id, matches.shape[0], matches.shape[1], matches_str)) 138 | 139 | # Close the connection to the database. 140 | connection.commit() 141 | cursor.close() 142 | connection.close() 143 | 144 | 145 | def geometric_verification(paths, args): 146 | print('Running geometric verification...') 147 | 148 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'matches_importer', 149 | '--database_path', paths.database_path, 150 | '--match_list_path', paths.match_list_path, 151 | '--match_type', 'pairs']) 152 | 153 | 154 | 155 | if __name__ == "__main__": 156 | parser = argparse.ArgumentParser() 157 | parser.add_argument('--dataset_path', required=True, help='Path to the dataset') 158 | parser.add_argument('--colmap_path', required=True, help='Path to the COLMAP executable folder') 159 | parser.add_argument('--method_name', required=True, help='Name of the method') 160 | parser.add_argument('--database_name', required=True, help='Name of the COLMAP database relative to dataset_path') 161 | parser.add_argument('--image_path', required=True, help='Name of the image directory relative to dataset_path') 162 | parser.add_argument('--match_list', required=True, help='Name of the text file containing image pairs to be matched, relative to dataset_path') 163 | parser.add_argument('--matching_only', type=bool, default=False, help='Only performs feature matching without creating a new database or importing features') 164 | args = parser.parse_args() 165 | 166 | ## Torch settings for the matcher. 167 | use_cuda = torch.cuda.is_available() 168 | device = torch.device("cuda:0" if use_cuda else "cpu") 169 | 170 | # Create the extra paths. 171 | paths = types.SimpleNamespace() 172 | paths.dummy_database_path = os.path.join(args.dataset_path, args.database_name) 173 | paths.database_path = os.path.join(args.dataset_path, args.method_name + '.db') 174 | paths.image_path = os.path.join(args.dataset_path, args.image_path) 175 | paths.features_path = os.path.join(args.dataset_path, args.method_name) 176 | paths.match_list_path = os.path.join(args.dataset_path, args.match_list) 177 | 178 | if args.matching_only == False: 179 | # Create a copy of the dummy database. 180 | if os.path.exists(paths.database_path): 181 | raise FileExistsError('The database file already exists for method %s.' % args.method_name) 182 | shutil.copyfile(paths.dummy_database_path, paths.database_path) 183 | 184 | # Reconstruction pipeline. 185 | images, cameras = recover_database_images_and_ids(paths, args) 186 | if args.matching_only == False: 187 | import_features(images, paths, args) 188 | 189 | match_features(images, paths, args) 190 | geometric_verification(paths, args) 191 | -------------------------------------------------------------------------------- /local_feature_evaluation/reconstruction_pipeline.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | import numpy as np 4 | 5 | import os 6 | 7 | import shutil 8 | 9 | import subprocess 10 | 11 | import sqlite3 12 | 13 | import torch 14 | 15 | import types 16 | 17 | from tqdm import tqdm 18 | 19 | from matchers import mutual_nn_matcher 20 | 21 | from camera import Camera 22 | 23 | from utils import quaternion_to_rotation_matrix, camera_center_to_translation 24 | 25 | import sys 26 | IS_PYTHON3 = sys.version_info[0] >= 3 27 | 28 | def array_to_blob(array): 29 | if IS_PYTHON3: 30 | return array.tostring() 31 | else: 32 | return np.getbuffer(array) 33 | 34 | def recover_database_images_and_ids(paths, args): 35 | # Connect to the database. 36 | connection = sqlite3.connect(paths.database_path) 37 | cursor = connection.cursor() 38 | 39 | # Recover database images and ids. 40 | images = {} 41 | cameras = {} 42 | cursor.execute("SELECT name, image_id, camera_id FROM images;") 43 | for row in cursor: 44 | images[row[0]] = row[1] 45 | cameras[row[0]] = row[2] 46 | 47 | # Close the connection to the database. 48 | cursor.close() 49 | connection.close() 50 | 51 | return images, cameras 52 | 53 | 54 | def preprocess_reference_model(paths, args): 55 | print('Preprocessing the reference model...') 56 | 57 | # Recover intrinsics. 58 | with open(os.path.join(paths.reference_model_path, 'database_intrinsics.txt')) as f: 59 | raw_intrinsics = f.readlines() 60 | 61 | camera_parameters = {} 62 | 63 | for intrinsics in raw_intrinsics: 64 | intrinsics = intrinsics.strip('\n').split(' ') 65 | 66 | image_name = intrinsics[0] 67 | 68 | camera_model = intrinsics[1] 69 | 70 | intrinsics = [float(param) for param in intrinsics[2 :]] 71 | 72 | camera = Camera() 73 | camera.set_intrinsics(camera_model=camera_model, intrinsics=intrinsics) 74 | 75 | camera_parameters[image_name] = camera 76 | 77 | # Recover poses. 78 | with open(os.path.join(paths.reference_model_path, 'aachen_cvpr2018_db.nvm')) as f: 79 | raw_extrinsics = f.readlines() 80 | 81 | # Skip the header. 82 | n_cameras = int(raw_extrinsics[2]) 83 | raw_extrinsics = raw_extrinsics[3 : 3 + n_cameras] 84 | 85 | for extrinsics in raw_extrinsics: 86 | extrinsics = extrinsics.strip('\n').split(' ') 87 | 88 | image_name = extrinsics[0] 89 | 90 | # Skip the focal length. Skip the distortion and terminal 0. 91 | qw, qx, qy, qz, cx, cy, cz = [float(param) for param in extrinsics[2 : -2]] 92 | 93 | qvec = np.array([qw, qx, qy, qz]) 94 | c = np.array([cx, cy, cz]) 95 | 96 | # NVM -> COLMAP. 97 | t = camera_center_to_translation(c, qvec) 98 | 99 | camera_parameters[image_name].set_pose(qvec=qvec, t=t) 100 | 101 | return camera_parameters 102 | 103 | 104 | def generate_empty_reconstruction(images, cameras, camera_parameters, paths, args): 105 | print('Generating the empty reconstruction...') 106 | 107 | if not os.path.exists(paths.empty_model_path): 108 | os.mkdir(paths.empty_model_path) 109 | 110 | with open(os.path.join(paths.empty_model_path, 'cameras.txt'), 'w') as f: 111 | for image_name in images: 112 | image_id = images[image_name] 113 | camera_id = cameras[image_name] 114 | try: 115 | camera = camera_parameters[image_name] 116 | except: 117 | continue 118 | f.write('%d %s %s\n' % ( 119 | camera_id, 120 | camera.camera_model, 121 | ' '.join(map(str, camera.intrinsics)) 122 | )) 123 | 124 | with open(os.path.join(paths.empty_model_path, 'images.txt'), 'w') as f: 125 | for image_name in images: 126 | image_id = images[image_name] 127 | camera_id = cameras[image_name] 128 | try: 129 | camera = camera_parameters[image_name] 130 | except: 131 | continue 132 | f.write('%d %s %s %d %s\n\n' % ( 133 | image_id, 134 | ' '.join(map(str, camera.qvec)), 135 | ' '.join(map(str, camera.t)), 136 | camera_id, 137 | image_name 138 | )) 139 | 140 | with open(os.path.join(paths.empty_model_path, 'points3D.txt'), 'w') as f: 141 | pass 142 | 143 | 144 | def import_features(images, paths, args): 145 | # Connect to the database. 146 | connection = sqlite3.connect(paths.database_path) 147 | cursor = connection.cursor() 148 | 149 | # Import the features. 150 | print('Importing features...') 151 | 152 | for image_name, image_id in tqdm(images.items(), total=len(images.items())): 153 | features_path = os.path.join(paths.image_path, '%s.%s' % (image_name, args.method_name)) 154 | 155 | keypoints = np.load(features_path)['keypoints'] 156 | n_keypoints = keypoints.shape[0] 157 | 158 | # Keep only x, y coordinates. 159 | keypoints = keypoints[:, : 2] 160 | # Add placeholder scale, orientation. 161 | keypoints = np.concatenate([keypoints, np.ones((n_keypoints, 1)), np.zeros((n_keypoints, 1))], axis=1).astype(np.float32) 162 | 163 | keypoints_str = keypoints.tostring() 164 | cursor.execute("INSERT INTO keypoints(image_id, rows, cols, data) VALUES(?, ?, ?, ?);", 165 | (image_id, keypoints.shape[0], keypoints.shape[1], keypoints_str)) 166 | connection.commit() 167 | 168 | # Close the connection to the database. 169 | cursor.close() 170 | connection.close() 171 | 172 | 173 | def image_ids_to_pair_id(image_id1, image_id2): 174 | if image_id1 > image_id2: 175 | return 2147483647 * image_id2 + image_id1 176 | else: 177 | return 2147483647 * image_id1 + image_id2 178 | 179 | 180 | def match_features(images, paths, args): 181 | # Connect to the database. 182 | connection = sqlite3.connect(paths.database_path) 183 | cursor = connection.cursor() 184 | 185 | # Match the features and insert the matches in the database. 186 | print('Matching...') 187 | 188 | with open(paths.match_list_path, 'r') as f: 189 | raw_pairs = f.readlines() 190 | 191 | image_pair_ids = set() 192 | for raw_pair in tqdm(raw_pairs, total=len(raw_pairs)): 193 | image_name1, image_name2 = raw_pair.strip('\n').split(' ') 194 | 195 | features_path1 = os.path.join(paths.image_path, '%s.%s' % (image_name1, args.method_name)) 196 | features_path2 = os.path.join(paths.image_path, '%s.%s' % (image_name2, args.method_name)) 197 | 198 | descriptors1 = torch.from_numpy(np.load(features_path1)['descriptors']).to(device) 199 | descriptors2 = torch.from_numpy(np.load(features_path2)['descriptors']).to(device) 200 | matches = mutual_nn_matcher(descriptors1, descriptors2).astype(np.uint32) 201 | 202 | image_id1, image_id2 = images[image_name1], images[image_name2] 203 | image_pair_id = image_ids_to_pair_id(image_id1, image_id2) 204 | if image_pair_id in image_pair_ids: 205 | continue 206 | image_pair_ids.add(image_pair_id) 207 | 208 | if image_id1 > image_id2: 209 | matches = matches[:, [1, 0]] 210 | 211 | matches_str = matches.tostring() 212 | cursor.execute("INSERT INTO matches(pair_id, rows, cols, data) VALUES(?, ?, ?, ?);", 213 | (image_pair_id, matches.shape[0], matches.shape[1], matches_str)) 214 | connection.commit() 215 | 216 | # Close the connection to the database. 217 | cursor.close() 218 | connection.close() 219 | 220 | 221 | def geometric_verification(paths, args): 222 | print('Running geometric verification...') 223 | 224 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'matches_importer', 225 | '--database_path', paths.database_path, 226 | '--match_list_path', paths.match_list_path, 227 | '--match_type', 'pairs']) 228 | 229 | 230 | def reconstruct(paths, args): 231 | if not os.path.isdir(paths.database_model_path): 232 | os.mkdir(paths.database_model_path) 233 | 234 | # Reconstruct the database model. 235 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'point_triangulator', 236 | '--database_path', paths.database_path, 237 | '--image_path', paths.image_path, 238 | '--input_path', paths.empty_model_path, 239 | '--output_path', paths.database_model_path, 240 | '--Mapper.ba_refine_focal_length', '0', 241 | '--Mapper.ba_refine_principal_point', '0', 242 | '--Mapper.ba_refine_extra_params', '0']) 243 | 244 | 245 | def register_queries(paths, args): 246 | if not os.path.isdir(paths.final_model_path): 247 | os.mkdir(paths.final_model_path) 248 | 249 | # Register the query images. 250 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'image_registrator', 251 | '--database_path', paths.database_path, 252 | '--input_path', paths.database_model_path, 253 | '--output_path', paths.final_model_path, 254 | '--Mapper.ba_refine_focal_length', '0', 255 | '--Mapper.ba_refine_principal_point', '0', 256 | '--Mapper.ba_refine_extra_params', '0']) 257 | 258 | 259 | def recover_query_poses(paths, args): 260 | print('Recovering query poses...') 261 | 262 | if not os.path.isdir(paths.final_txt_model_path): 263 | os.mkdir(paths.final_txt_model_path) 264 | 265 | # Convert the model to TXT. 266 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'model_converter', 267 | '--input_path', paths.final_model_path, 268 | '--output_path', paths.final_txt_model_path, 269 | '--output_type', 'TXT']) 270 | 271 | # Recover query names. 272 | query_image_list_path = os.path.join(args.dataset_path, 'queries/night_time_queries_with_intrinsics.txt') 273 | 274 | with open(query_image_list_path) as f: 275 | raw_queries = f.readlines() 276 | 277 | query_names = set() 278 | for raw_query in raw_queries: 279 | raw_query = raw_query.strip('\n').split(' ') 280 | query_name = raw_query[0] 281 | query_names.add(query_name) 282 | 283 | with open(os.path.join(paths.final_txt_model_path, 'images.txt')) as f: 284 | raw_extrinsics = f.readlines() 285 | 286 | f = open(paths.prediction_path, 'w') 287 | 288 | # Skip the header. 289 | for extrinsics in raw_extrinsics[4 :: 2]: 290 | extrinsics = extrinsics.strip('\n').split(' ') 291 | 292 | image_name = extrinsics[-1] 293 | 294 | if image_name in query_names: 295 | # Skip the IMAGE_ID ([0]), CAMERA_ID ([-2]), and IMAGE_NAME ([-1]). 296 | f.write('%s %s\n' % (image_name.split('/')[-1], ' '.join(extrinsics[1 : -2]))) 297 | 298 | f.close() 299 | 300 | 301 | if __name__ == "__main__": 302 | parser = argparse.ArgumentParser() 303 | parser.add_argument('--dataset_path', required=True, help='Path to the dataset') 304 | parser.add_argument('--colmap_path', required=True, help='Path to the COLMAP executable folder') 305 | parser.add_argument('--method_name', required=True, help='Name of the method') 306 | args = parser.parse_args() 307 | 308 | # Torch settings for the matcher. 309 | use_cuda = torch.cuda.is_available() 310 | device = torch.device("cuda:0" if use_cuda else "cpu") 311 | 312 | # Create the extra paths. 313 | paths = types.SimpleNamespace() 314 | paths.dummy_database_path = os.path.join(args.dataset_path, 'database.db') 315 | paths.database_path = os.path.join(args.dataset_path, args.method_name + '.db') 316 | paths.image_path = os.path.join(args.dataset_path, 'images', 'images_upright') 317 | paths.features_path = os.path.join(args.dataset_path, args.method_name) 318 | paths.reference_model_path = os.path.join(args.dataset_path, '3D-models') 319 | paths.match_list_path = os.path.join(args.dataset_path, 'image_pairs_to_match.txt') 320 | paths.empty_model_path = os.path.join(args.dataset_path, 'sparse-%s-empty' % args.method_name) 321 | paths.database_model_path = os.path.join(args.dataset_path, 'sparse-%s-database' % args.method_name) 322 | paths.final_model_path = os.path.join(args.dataset_path, 'sparse-%s-final' % args.method_name) 323 | paths.final_txt_model_path = os.path.join(args.dataset_path, 'sparse-%s-final-txt' % args.method_name) 324 | paths.prediction_path = os.path.join(args.dataset_path, 'Aachen_eval_[%s].txt' % args.method_name) 325 | 326 | # Create a copy of the dummy database. 327 | if os.path.exists(paths.database_path): 328 | raise FileExistsError('The database file already exists for method %s.' % args.method_name) 329 | shutil.copyfile(paths.dummy_database_path, paths.database_path) 330 | 331 | # Reconstruction pipeline. 332 | camera_parameters = preprocess_reference_model(paths, args) 333 | images, cameras = recover_database_images_and_ids(paths, args) 334 | generate_empty_reconstruction(images, cameras, camera_parameters, paths, args) 335 | import_features(images, paths, args) 336 | match_features(images, paths, args) 337 | geometric_verification(paths, args) 338 | reconstruct(paths, args) 339 | register_queries(paths, args) 340 | recover_query_poses(paths, args) 341 | -------------------------------------------------------------------------------- /local_feature_evaluation/reconstruction_pipeline_aachen_v1_1.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | import numpy as np 4 | 5 | import os 6 | 7 | import shutil 8 | 9 | import subprocess 10 | 11 | import sqlite3 12 | 13 | import torch 14 | 15 | import types 16 | 17 | from tqdm import tqdm 18 | 19 | from matchers import mutual_nn_matcher 20 | 21 | from camera import Camera 22 | 23 | from utils import quaternion_to_rotation_matrix, camera_center_to_translation 24 | 25 | import sys 26 | IS_PYTHON3 = sys.version_info[0] >= 3 27 | 28 | def array_to_blob(array): 29 | if IS_PYTHON3: 30 | return array.tostring() 31 | else: 32 | return np.getbuffer(array) 33 | 34 | def recover_database_images_and_ids(paths, args): 35 | # Connect to the database. 36 | connection = sqlite3.connect(paths.database_path) 37 | cursor = connection.cursor() 38 | 39 | # Recover database images and ids. 40 | images = {} 41 | cameras = {} 42 | cursor.execute("SELECT name, image_id, camera_id FROM images;") 43 | for row in cursor: 44 | images[row[0]] = row[1] 45 | cameras[row[0]] = row[2] 46 | 47 | # Close the connection to the database. 48 | cursor.close() 49 | connection.close() 50 | 51 | return images, cameras 52 | 53 | 54 | def preprocess_reference_model(paths, args): 55 | print('Preprocessing the reference model...') 56 | 57 | # Recover intrinsics. 58 | with open(os.path.join(paths.reference_model_path, 'aachen_v_1_1/database_intrinsics_v1_1.txt')) as f: 59 | raw_intrinsics = f.readlines() 60 | 61 | camera_parameters = {} 62 | 63 | for intrinsics in raw_intrinsics: 64 | intrinsics = intrinsics.strip('\n').split(' ') 65 | 66 | image_name = intrinsics[0] 67 | 68 | camera_model = intrinsics[1] 69 | 70 | intrinsics = [float(param) for param in intrinsics[2 :]] 71 | 72 | camera = Camera() 73 | camera.set_intrinsics(camera_model=camera_model, intrinsics=intrinsics) 74 | 75 | camera_parameters[image_name] = camera 76 | 77 | # Recover poses. 78 | with open(os.path.join(paths.reference_model_path, 'aachen_v_1_1/aachen_v_1_1.nvm')) as f: 79 | raw_extrinsics = f.readlines() 80 | 81 | # Skip the header. 82 | n_cameras = int(raw_extrinsics[2]) 83 | raw_extrinsics = raw_extrinsics[3 : 3 + n_cameras] 84 | 85 | for extrinsics in raw_extrinsics: 86 | extrinsics = extrinsics.strip('\n').split(' ') 87 | 88 | image_name = extrinsics[0] 89 | 90 | # Skip the focal length. Skip the distortion and terminal 0. 91 | qw, qx, qy, qz, cx, cy, cz = [float(param) for param in extrinsics[2 : -2]] 92 | 93 | qvec = np.array([qw, qx, qy, qz]) 94 | c = np.array([cx, cy, cz]) 95 | 96 | # NVM -> COLMAP. 97 | t = camera_center_to_translation(c, qvec) 98 | 99 | camera_parameters[image_name].set_pose(qvec=qvec, t=t) 100 | 101 | return camera_parameters 102 | 103 | 104 | def generate_empty_reconstruction(images, cameras, camera_parameters, paths, args): 105 | print('Generating the empty reconstruction...') 106 | 107 | if not os.path.exists(paths.empty_model_path): 108 | os.mkdir(paths.empty_model_path) 109 | 110 | with open(os.path.join(paths.empty_model_path, 'cameras.txt'), 'w') as f: 111 | for image_name in images: 112 | image_id = images[image_name] 113 | camera_id = cameras[image_name] 114 | try: 115 | camera = camera_parameters[image_name] 116 | except: 117 | continue 118 | f.write('%d %s %s\n' % ( 119 | camera_id, 120 | camera.camera_model, 121 | ' '.join(map(str, camera.intrinsics)) 122 | )) 123 | 124 | with open(os.path.join(paths.empty_model_path, 'images.txt'), 'w') as f: 125 | for image_name in images: 126 | image_id = images[image_name] 127 | camera_id = cameras[image_name] 128 | try: 129 | camera = camera_parameters[image_name] 130 | except: 131 | continue 132 | f.write('%d %s %s %d %s\n\n' % ( 133 | image_id, 134 | ' '.join(map(str, camera.qvec)), 135 | ' '.join(map(str, camera.t)), 136 | camera_id, 137 | image_name 138 | )) 139 | 140 | with open(os.path.join(paths.empty_model_path, 'points3D.txt'), 'w') as f: 141 | pass 142 | 143 | 144 | def import_features(images, paths, args): 145 | # Connect to the database. 146 | connection = sqlite3.connect(paths.database_path) 147 | cursor = connection.cursor() 148 | 149 | # Import the features. 150 | print('Importing features...') 151 | 152 | for image_name, image_id in tqdm(images.items(), total=len(images.items())): 153 | features_path = os.path.join(paths.image_path, '%s.%s' % (image_name, args.method_name)) 154 | 155 | keypoints = np.load(features_path)['keypoints'] 156 | n_keypoints = keypoints.shape[0] 157 | 158 | # Keep only x, y coordinates. 159 | keypoints = keypoints[:, : 2] 160 | # Add placeholder scale, orientation. 161 | keypoints = np.concatenate([keypoints, np.ones((n_keypoints, 1)), np.zeros((n_keypoints, 1))], axis=1).astype(np.float32) 162 | 163 | keypoints_str = keypoints.tostring() 164 | cursor.execute("INSERT INTO keypoints(image_id, rows, cols, data) VALUES(?, ?, ?, ?);", 165 | (image_id, keypoints.shape[0], keypoints.shape[1], keypoints_str)) 166 | connection.commit() 167 | 168 | # Close the connection to the database. 169 | cursor.close() 170 | connection.close() 171 | 172 | 173 | def image_ids_to_pair_id(image_id1, image_id2): 174 | if image_id1 > image_id2: 175 | return 2147483647 * image_id2 + image_id1 176 | else: 177 | return 2147483647 * image_id1 + image_id2 178 | 179 | 180 | def match_features(images, paths, args): 181 | # Connect to the database. 182 | connection = sqlite3.connect(paths.database_path) 183 | cursor = connection.cursor() 184 | 185 | # Match the features and insert the matches in the database. 186 | print('Matching...') 187 | 188 | with open(paths.match_list_path, 'r') as f: 189 | raw_pairs = f.readlines() 190 | 191 | image_pair_ids = set() 192 | for raw_pair in tqdm(raw_pairs, total=len(raw_pairs)): 193 | image_name1, image_name2 = raw_pair.strip('\n').split(' ') 194 | 195 | features_path1 = os.path.join(paths.image_path, '%s.%s' % (image_name1, args.method_name)) 196 | features_path2 = os.path.join(paths.image_path, '%s.%s' % (image_name2, args.method_name)) 197 | 198 | descriptors1 = torch.from_numpy(np.load(features_path1)['descriptors']).to(device) 199 | descriptors2 = torch.from_numpy(np.load(features_path2)['descriptors']).to(device) 200 | matches = mutual_nn_matcher(descriptors1, descriptors2).astype(np.uint32) 201 | 202 | image_id1, image_id2 = images[image_name1], images[image_name2] 203 | image_pair_id = image_ids_to_pair_id(image_id1, image_id2) 204 | if image_pair_id in image_pair_ids: 205 | continue 206 | image_pair_ids.add(image_pair_id) 207 | 208 | if image_id1 > image_id2: 209 | matches = matches[:, [1, 0]] 210 | 211 | matches_str = matches.tostring() 212 | cursor.execute("INSERT INTO matches(pair_id, rows, cols, data) VALUES(?, ?, ?, ?);", 213 | (image_pair_id, matches.shape[0], matches.shape[1], matches_str)) 214 | connection.commit() 215 | 216 | # Close the connection to the database. 217 | cursor.close() 218 | connection.close() 219 | 220 | 221 | def geometric_verification(paths, args): 222 | print('Running geometric verification...') 223 | 224 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'matches_importer', 225 | '--database_path', paths.database_path, 226 | '--match_list_path', paths.match_list_path, 227 | '--match_type', 'pairs']) 228 | 229 | 230 | def reconstruct(paths, args): 231 | if not os.path.isdir(paths.database_model_path): 232 | os.mkdir(paths.database_model_path) 233 | 234 | # Reconstruct the database model. 235 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'point_triangulator', 236 | '--database_path', paths.database_path, 237 | '--image_path', paths.image_path, 238 | '--input_path', paths.empty_model_path, 239 | '--output_path', paths.database_model_path, 240 | '--Mapper.ba_refine_focal_length', '0', 241 | '--Mapper.ba_refine_principal_point', '0', 242 | '--Mapper.ba_refine_extra_params', '0']) 243 | 244 | 245 | def register_queries(paths, args): 246 | if not os.path.isdir(paths.final_model_path): 247 | os.mkdir(paths.final_model_path) 248 | 249 | # Register the query images. 250 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'image_registrator', 251 | '--database_path', paths.database_path, 252 | '--input_path', paths.database_model_path, 253 | '--output_path', paths.final_model_path, 254 | '--Mapper.ba_refine_focal_length', '0', 255 | '--Mapper.ba_refine_principal_point', '0', 256 | '--Mapper.ba_refine_extra_params', '0']) 257 | 258 | 259 | def recover_query_poses(paths, args): 260 | print('Recovering query poses...') 261 | 262 | if not os.path.isdir(paths.final_txt_model_path): 263 | os.mkdir(paths.final_txt_model_path) 264 | 265 | # Convert the model to TXT. 266 | subprocess.call([os.path.join(args.colmap_path, 'colmap'), 'model_converter', 267 | '--input_path', paths.final_model_path, 268 | '--output_path', paths.final_txt_model_path, 269 | '--output_type', 'TXT']) 270 | 271 | # Recover query names. 272 | query_image_list_path = os.path.join(args.dataset_path, 'queries/night_time_queries_with_intrinsics.txt') 273 | 274 | with open(query_image_list_path) as f: 275 | raw_queries = f.readlines() 276 | 277 | query_names = set() 278 | for raw_query in raw_queries: 279 | raw_query = raw_query.strip('\n').split(' ') 280 | query_name = raw_query[0] 281 | query_names.add(query_name) 282 | 283 | with open(os.path.join(paths.final_txt_model_path, 'images.txt')) as f: 284 | raw_extrinsics = f.readlines() 285 | 286 | f = open(paths.prediction_path, 'w') 287 | 288 | # Skip the header. 289 | for extrinsics in raw_extrinsics[4 :: 2]: 290 | extrinsics = extrinsics.strip('\n').split(' ') 291 | 292 | image_name = extrinsics[-1] 293 | 294 | if image_name in query_names: 295 | # Skip the IMAGE_ID ([0]), CAMERA_ID ([-2]), and IMAGE_NAME ([-1]). 296 | f.write('%s %s\n' % (image_name.split('/')[-1], ' '.join(extrinsics[1 : -2]))) 297 | 298 | f.close() 299 | 300 | 301 | if __name__ == "__main__": 302 | parser = argparse.ArgumentParser() 303 | parser.add_argument('--dataset_path', required=True, help='Path to the dataset') 304 | parser.add_argument('--colmap_path', required=True, help='Path to the COLMAP executable folder') 305 | parser.add_argument('--method_name', required=True, help='Name of the method') 306 | args = parser.parse_args() 307 | 308 | # Torch settings for the matcher. 309 | use_cuda = torch.cuda.is_available() 310 | device = torch.device("cuda:0" if use_cuda else "cpu") 311 | 312 | # Create the extra paths. 313 | paths = types.SimpleNamespace() 314 | paths.dummy_database_path = os.path.join(args.dataset_path, 'database_v1_1.db') 315 | paths.database_path = os.path.join(args.dataset_path, args.method_name + '_v1_1.db') 316 | paths.image_path = os.path.join(args.dataset_path, 'images', 'images_upright') 317 | paths.features_path = os.path.join(args.dataset_path, args.method_name) 318 | paths.reference_model_path = os.path.join(args.dataset_path, '3D-models') 319 | paths.match_list_path = os.path.join(args.dataset_path, 'image_pairs_to_match_v1_1.txt') 320 | paths.empty_model_path = os.path.join(args.dataset_path, 'sparse-%s-empty' % args.method_name) 321 | paths.database_model_path = os.path.join(args.dataset_path, 'sparse-%s-database' % args.method_name) 322 | paths.final_model_path = os.path.join(args.dataset_path, 'sparse-%s-final' % args.method_name) 323 | paths.final_txt_model_path = os.path.join(args.dataset_path, 'sparse-%s-final-txt' % args.method_name) 324 | paths.prediction_path = os.path.join(args.dataset_path, 'Aachen_v1_1_eval_[%s].txt' % args.method_name) 325 | 326 | # Create a copy of the dummy database. 327 | if os.path.exists(paths.database_path): 328 | raise FileExistsError('The database file already exists for method %s.' % args.method_name) 329 | shutil.copyfile(paths.dummy_database_path, paths.database_path) 330 | 331 | # Reconstruction pipeline. 332 | camera_parameters = preprocess_reference_model(paths, args) 333 | images, cameras = recover_database_images_and_ids(paths, args) 334 | generate_empty_reconstruction(images, cameras, camera_parameters, paths, args) 335 | import_features(images, paths, args) 336 | match_features(images, paths, args) 337 | geometric_verification(paths, args) 338 | reconstruct(paths, args) 339 | register_queries(paths, args) 340 | recover_query_poses(paths, args) 341 | -------------------------------------------------------------------------------- /local_feature_evaluation/utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | def quaternion_to_rotation_matrix(qvec): 4 | qvec = qvec / np.linalg.norm(qvec) 5 | w, x, y, z = qvec 6 | R = np.array([[1 - 2 * y * y - 2 * z * z, 2 * x * y - 2 * z * w, 2 * x * z + 2 * y * w], 7 | [2 * x * y + 2 * z * w, 1 - 2 * x * x - 2 * z * z, 2 * y * z - 2 * x * w], 8 | [2 * x * z - 2 * y * w, 2 * y * z + 2 * x * w, 1 - 2 * x * x - 2 * y * y]]) 9 | return R 10 | 11 | 12 | def camera_center_to_translation(c, qvec): 13 | R = quaternion_to_rotation_matrix(qvec) 14 | return (-1) * np.matmul(R, c) 15 | 16 | --------------------------------------------------------------------------------