├── .gitignore
├── README.md
├── evaluate.py
├── generating_queries
    ├── generate_inference_sets.py
    ├── generate_test_sets.py
    └── generate_training_tuples_baseline.py
├── inference.py
├── loupe.py
├── models
    └── lpd_FNSF.py
├── network_architecture.png
├── prepare_data.py
├── train_lpdnet.py
└── utils
    ├── loading_pointclouds.py
    ├── tf_util.py
    └── transform_nets.py


/.gitignore:
--------------------------------------------------------------------------------
1 | *.pyc
2 | *.pickle
3 | .idea
4 | log*
5 | results*


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis
  2 | 
  3 | **[LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis](http://openaccess.thecvf.com/content_ICCV_2019/papers/Liu_LPD-Net_3D_Point_Cloud_Learning_for_Large-Scale_Place_Recognition_and_ICCV_2019_paper.pdf)** ICCV 2019, Seoul, Korea
  4 | 
  5 | <b>Zhe Liu<sup>1</sup>, Shunbo Zhou<sup>1</sup>, Chuanzhe Suo<sup>1</sup></b>, Peng Yin<sup>3</sup>, Wen Chen<sup>1</sup>, Hesheng Wang<sup>2</sup>,Haoang Li<sup>1</sup>, Yun-Hui Liu<sup>1</sup>
  6 | 
  7 | <sup>1</sup>The Chinese University of Hong Kong,  <sup>2</sup>Shanghai Jiao Tong University, <sup>3</sup>Carnegie Mellon University
  8 | 
  9 | 
 10 | ![pic-network](network_architecture.png)
 11 | 
 12 | ## Introduction
 13 | Point cloud based Place Recognition is still an open issue due to the difficulty in extracting local features from the raw 3D point cloud and generating the global descriptor,and it’s even harder in the large-scale dynamic environments. We develop a novel deep neural network, named <u><b>LPD-Net (Large-scale Place Description Network)</b></u>, which can extract discriminative and generalizable
 14 | global descriptors from the raw 3D point cloud. The arXiv version of LPD-Net can be found [here](https://arxiv.org/abs/1812.07050).
 15 | ```
 16 | @InProceedings{Liu_2019_ICCV,
 17 | author = {Liu, Zhe and Zhou, Shunbo and Suo, Chuanzhe and Yin, Peng and Chen, Wen and Wang, Hesheng and Li, Haoang and Liu, Yun-Hui},
 18 | title = {LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis},
 19 | booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
 20 | month = {October},
 21 | year = {2019}
 22 | }
 23 | ```
 24 | ## Benchmark Datasets
 25 | The benchmark datasets introdruced in this work can be downloaded [here](https://drive.google.com/open?id=1H9Ep76l8KkUpwILY-13owsEMbVCYTmyx), which created by PointNetVLAD for point cloud based retrieval for place recognition from the open-source [Oxford RobotCar](https://robotcar-dataset.robots.ox.ac.uk/). Details can be found in [PointNetVLAD](https://arxiv.org/abs/1804.03492). 
 26 | * All submaps are in binary file format
 27 | * Ground truth GPS coordinate of the submaps are found in the corresponding csv files for each run
 28 | * Filename of the submaps are their timestamps which is consistent with the timestamps in the csv files
 29 | * Use CSV files to define positive and negative point clouds
 30 | * All submaps are preprocessed with the road removed and downsampled to 4096 points
 31 | 
 32 | ### Oxford Dataset
 33 | * 45 sets in total of full and partial runs
 34 | * Used both full and partial runs for training but only used full runs for testing/inference
 35 | * Training submaps are found in the folder "pointcloud_20m_10overlap/" and its corresponding csv file is "pointcloud_locations_20m_10overlap.csv"
 36 | * Training submaps are not mutually disjoint per run
 37 | * Each training submap ~20m of car trajectory and subsequent submaps are ~10m apart
 38 | * Test/Inference submaps found in the folder "pointcloud_20m/" and its corresponding csv file is "pointcloud_locations_20m.csv"
 39 | * Test/Inference submaps are mutually disjoint
 40 | 
 41 | 
 42 | ## Project Code
 43 | 
 44 | ### Pre-requisites
 45 | * Python
 46 | * CUDA
 47 | * Tensorflow 
 48 | * Scipy
 49 | * Pandas
 50 | * Sklearn
 51 | 
 52 | Code was tested using Python 3 on Tensorflow 1.4.0 with CUDA 8.0 and Tensorflow 1.12.0 with CUDA 9.0
 53 | 
 54 | ```
 55 | sudo apt-get install python3-pip python3-dev python-virtualenv
 56 | virtualenv --system-site-packages -p python3 ~/tensorflow
 57 | source ~/tensorflow/bin/activate
 58 | easy_install -U pip
 59 | pip3 install --upgrade tensorflow-gpu==1.4.0
 60 | pip install scipy, pandas, sklearn
 61 | pip install glog
 62 | ```
 63 | ### Dataset set-up
 64 | Download the zip file of the benchmark datasets found [here](https://drive.google.com/open?id=1H9Ep76l8KkUpwILY-13owsEMbVCYTmyx). Extract the folder on the same directory as the project code. Thus, on that directory you must have two folders: 1) benchmark_datasets/ and 2) LPD_net/
 65 | 
 66 | ### Data pre-processing
 67 | We preprocess the benchmark datasets at first and store the features of point clouds on bin files to save the training time. The files only need to be generated once and used as input of networks. The generation of these files may take a few hours.
 68 | ```
 69 | # For pre-processing dataset to generate pointcloud with features
 70 | python prepare_data.py
 71 | 
 72 | # Parse Arguments: --k_start 20 --k_end 100 --k_step 10 --bin_core_num 10
 73 | # KNN Neighbor size from 20 to 100 with interval 10, parallel process pool core numbers:10
 74 | ```
 75 | 
 76 | ### Generate pickle files
 77 | We store the positive and negative point clouds to each anchor on pickle files that are used in our training and evaluation codes. The files only need to be generated once. The generation of these files may take a few minutes.
 78 | 
 79 | ```
 80 | cd generating_queries/ 
 81 | 
 82 | # For training tuples in LPD-Net
 83 | python generate_training_tuples_baseline.py
 84 | 
 85 | # For network evaluation
 86 | python generate_test_sets.py
 87 | 
 88 | # For network inference
 89 | python generate_inference_sets.py 
 90 | # Need to modify the variables (folders or index_list) to specify the folder
 91 | ```
 92 | 
 93 | ### Model Training and Evaluation
 94 | To train our network, run the following command:
 95 | ```
 96 | python train_lpdnet.py
 97 | # Parse Arguments: --model lpd_FNSF --log_dir log/ --restore
 98 | 
 99 | # For example, Train lpd_FNSF network from scratch
100 | python train_lpdnet.py --model lpd_FNSF
101 | 
102 | # Retrain lpd_FNSF network with pretrained model
103 | python train_lpdnet.py --log_dir log/<model_folder,eg.lpd_FNSF-18-11-16-12-03> --restore
104 | ```
105 | To evaluate the model, run the following command:
106 | ```
107 | python evaluate.py --log_dir log/<model_folder,eg.lpd_FNSF-18-11-16-12-03>
108 | # The resulst.txt will be saved in results/<model_folder>
109 | ```
110 | To infer the model, run the following command to get global descriptors:
111 | ```
112 | python inference.py --log_dir log/<model_folder,eg.lpd_FNSF-18-11-16-12-03>
113 | # The inference_vectors.bin will be saved in LPD_net folder
114 | ```
115 | 
116 | ## Pre-trained Models
117 | The pre-trained models for lpd_FNSF networks can be downloaded [here](https://drive.google.com/open?id=1H9Ep76l8KkUpwILY-13owsEMbVCYTmyx). Put it under the ```/log```folder
118 | 
119 | ## License
120 | This repository is released under MIT License (see LICENSE file for details).
121 | 


--------------------------------------------------------------------------------
/evaluate.py:
--------------------------------------------------------------------------------
  1 | # Author: Chuanzhe Suo (suo_ivy@foxmail.com) 10/26/2018
  2 | # Thanks to Mikaela Angelina Uy, modified from PointNetVLAD
  3 | # Reference: LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis, ICCV 2019
  4 | 
  5 | import argparse
  6 | import os
  7 | import sys
  8 | import importlib
  9 | import tensorflow as tf
 10 | import numpy as np
 11 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 12 | sys.path.append(BASE_DIR)
 13 | sys.path.append(os.path.join(BASE_DIR, 'models'))
 14 | sys.path.append(os.path.join(BASE_DIR, 'utils'))
 15 | from loading_pointclouds import *
 16 | from sklearn.neighbors import KDTree
 17 | 
 18 | #params
 19 | parser = argparse.ArgumentParser()
 20 | parser.add_argument('--gpu', type=int, default=0, help='GPU to use [default: GPU 1]')
 21 | parser.add_argument('--log_dir', default='log/', help='Log dir [default: log]')
 22 | parser.add_argument('--positives_per_query', type=int, default=4, help='Number of potential positives in each training tuple [default: 2]')
 23 | parser.add_argument('--negatives_per_query', type=int, default=10, help='Number of definite negatives in each training tuple [default: 20]')
 24 | parser.add_argument('--batch_num_queries', type=int, default=3, help='Batch Size during training [default: 1]')
 25 | parser.add_argument('--dimension', type=int, default=256)
 26 | parser.add_argument('--decay_step', type=int, default=200000, help='Decay step for lr decay [default: 200000]')
 27 | parser.add_argument('--decay_rate', type=float, default=0.7, help='Decay rate for lr decay [default: 0.8]')
 28 | FLAGS = parser.parse_args()
 29 | 
 30 | #BATCH_SIZE = FLAGS.batch_size
 31 | BATCH_NUM_QUERIES = FLAGS.batch_num_queries
 32 | EVAL_BATCH_SIZE = 1
 33 | NUM_POINTS = 4096
 34 | POSITIVES_PER_QUERY= FLAGS.positives_per_query
 35 | NEGATIVES_PER_QUERY= FLAGS.negatives_per_query
 36 | GPU_INDEX = FLAGS.gpu
 37 | DECAY_STEP = FLAGS.decay_step
 38 | DECAY_RATE = FLAGS.decay_rate
 39 | 
 40 | DATABASE_FILE= 'generating_queries/oxford_evaluation_database.pickle'
 41 | QUERY_FILE= 'generating_queries/oxford_evaluation_query.pickle'
 42 | 
 43 | LOG_DIR = FLAGS.log_dir
 44 | model = LOG_DIR.split('/')[1]
 45 | RESULTS_FOLDER=os.path.join("results/", model)
 46 | model = model.split('-')[0]
 47 | print(LOG_DIR)
 48 | MODEL = importlib.import_module(model)
 49 | if not os.path.exists(RESULTS_FOLDER): os.makedirs(RESULTS_FOLDER)
 50 | output_file= RESULTS_FOLDER +'/results.txt'
 51 | model_file= "model.ckpt"
 52 | 
 53 | DATABASE_SETS= get_sets_dict(DATABASE_FILE)
 54 | QUERY_SETS= get_sets_dict(QUERY_FILE)
 55 | 
 56 | global DATABASE_VECTORS
 57 | DATABASE_VECTORS=[]
 58 | 
 59 | global QUERY_VECTORS
 60 | QUERY_VECTORS=[]
 61 | 
 62 | BN_INIT_DECAY = 0.5
 63 | BN_DECAY_DECAY_RATE = 0.5
 64 | BN_DECAY_DECAY_STEP = float(DECAY_STEP)
 65 | BN_DECAY_CLIP = 0.99
 66 | 
 67 | def get_bn_decay(batch):
 68 |     bn_momentum = tf.train.exponential_decay(
 69 |                       BN_INIT_DECAY,
 70 |                       batch*BATCH_NUM_QUERIES,
 71 |                       BN_DECAY_DECAY_STEP,
 72 |                       BN_DECAY_DECAY_RATE,
 73 |                       staircase=True)
 74 |     bn_decay = tf.minimum(BN_DECAY_CLIP, 1 - bn_momentum)
 75 |     return bn_decay     
 76 | 
 77 | def evaluate():
 78 |     global DATABASE_VECTORS
 79 |     global QUERY_VECTORS
 80 | 
 81 |     with tf.Graph().as_default():
 82 |         with tf.device('/gpu:'+str(GPU_INDEX)):
 83 |             print("In Graph")
 84 |             query= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, 1, NUM_POINTS)
 85 |             positives= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, POSITIVES_PER_QUERY, NUM_POINTS)
 86 |             negatives= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, NEGATIVES_PER_QUERY, NUM_POINTS)
 87 |             eval_queries= MODEL.placeholder_inputs(EVAL_BATCH_SIZE, 1, NUM_POINTS)
 88 | 
 89 |             is_training_pl = tf.placeholder(tf.bool, shape=())
 90 |             print(is_training_pl)
 91 | 
 92 |             batch = tf.Variable(0)
 93 |             bn_decay = get_bn_decay(batch)
 94 | 
 95 |             with tf.variable_scope("query_triplets") as scope:
 96 |                 vecs= tf.concat([query, positives, negatives],1)
 97 |                 print(vecs)                
 98 |                 out_vecs= MODEL.forward(vecs, is_training_pl, bn_decay=bn_decay)
 99 |                 q_vec, pos_vecs, neg_vecs= tf.split(out_vecs, [1,POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY],1)
100 |                 print(q_vec)
101 |                 print(pos_vecs)
102 |                 print(neg_vecs)
103 | 
104 |             saver = tf.train.Saver()
105 | 
106 |         # Create a session
107 |         gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.95)
108 |         config = tf.ConfigProto(gpu_options=gpu_options)
109 |         config.gpu_options.allow_growth = True
110 |         config.allow_soft_placement = True
111 |         config.log_device_placement = False
112 |         sess = tf.Session(config=config)
113 | 
114 | 
115 |         saver.restore(sess, os.path.join(LOG_DIR, model_file))
116 |         print("Model restored:{}".format(os.path.join(LOG_DIR, model_file)))
117 | 
118 |         ops = {'query': query,
119 |                'positives': positives,
120 |                'negatives': negatives,
121 |                'is_training_pl': is_training_pl,
122 |                'eval_queries': eval_queries,
123 |                'q_vec':q_vec,
124 |                'pos_vecs': pos_vecs,
125 |                'neg_vecs': neg_vecs}
126 |         recall= np.zeros(25)
127 |         count=0
128 |         similarity=[]
129 |         one_percent_recall=[]
130 |         for i in range(len(DATABASE_SETS)):
131 |             DATABASE_VECTORS.append(get_latent_vectors(sess, ops, DATABASE_SETS[i]))
132 | 
133 |         for j in range(len(QUERY_SETS)):
134 |             QUERY_VECTORS.append(get_latent_vectors(sess, ops, QUERY_SETS[j]))
135 | 
136 |         for m in range(len(QUERY_SETS)):
137 |             for n in range(len(QUERY_SETS)):
138 |                 if(m==n):
139 |                     continue
140 |                 pair_recall, pair_similarity, pair_opr = get_recall(sess, ops, m, n)
141 |                 recall+=np.array(pair_recall)
142 |                 count+=1
143 |                 one_percent_recall.append(pair_opr)
144 |                 for x in pair_similarity:
145 |                     similarity.append(x)
146 | 
147 |         print()
148 |         ave_recall=recall/count
149 |         print('ave_recallrecall')
150 |         print(ave_recall)
151 | 
152 |         #print('similarity:')
153 |         #print(similarity)
154 |         average_similarity= np.mean(similarity)
155 |         print('average_similarity')
156 |         print(average_similarity)
157 | 
158 |         ave_one_percent_recall= np.mean(one_percent_recall)
159 |         print('ave_one_percent_recall')
160 |         print(ave_one_percent_recall)
161 | 
162 | 
163 |         #filename=RESULTS_FOLDER +'average_recall_oxford_netmax_sg(finetune_conv5).txt'
164 |         with open(output_file, "a") as output:
165 |             output.write(model)
166 |             output.write("\n\n")
167 |             output.write("Average Recall @N:\n")
168 |             output.write(str(ave_recall))
169 |             output.write("\n\n")
170 |             output.write("Average Similarity:\n")
171 |             output.write(str(average_similarity))
172 |             output.write("\n\n")
173 |             output.write("Average Top 1% Recall:\n")
174 |             output.write(str(ave_one_percent_recall))
175 |             output.write("\n\n")
176 | 
177 | 
178 | def get_latent_vectors(sess, ops, dict_to_process):
179 |     is_training=False
180 |     train_file_idxs = np.arange(0, len(dict_to_process.keys()))
181 |     #print(len(train_file_idxs))
182 |     batch_num= BATCH_NUM_QUERIES*(1+POSITIVES_PER_QUERY+NEGATIVES_PER_QUERY)
183 |     q_output = []
184 |     for q_index in range(len(train_file_idxs)//batch_num):
185 |         file_indices=train_file_idxs[q_index*batch_num:(q_index+1)*(batch_num)]
186 |         file_names=[]
187 |         for index in file_indices:
188 |             file_names.append(dict_to_process[index]["query"])
189 |         queries=load_pc_files(file_names)
190 |         # queries= np.expand_dims(queries,axis=1)
191 |         q1=queries[0:BATCH_NUM_QUERIES]
192 |         q1=np.expand_dims(q1,axis=1)
193 |         #print(q1.shape)
194 | 
195 |         q2=queries[BATCH_NUM_QUERIES:BATCH_NUM_QUERIES*(POSITIVES_PER_QUERY+1)]
196 |         q2=np.reshape(q2,(BATCH_NUM_QUERIES,POSITIVES_PER_QUERY,NUM_POINTS,13))
197 | 
198 |         q3=queries[BATCH_NUM_QUERIES*(POSITIVES_PER_QUERY+1):BATCH_NUM_QUERIES*(NEGATIVES_PER_QUERY+POSITIVES_PER_QUERY+1)]
199 |         q3=np.reshape(q3,(BATCH_NUM_QUERIES,NEGATIVES_PER_QUERY,NUM_POINTS,13))
200 |         feed_dict={ops['query']:q1, ops['positives']:q2, ops['negatives']:q3, ops['is_training_pl']:is_training}
201 |         o1, o2, o3=sess.run([ops['q_vec'], ops['pos_vecs'], ops['neg_vecs']], feed_dict=feed_dict)
202 |         
203 |         o1=np.reshape(o1,(-1,o1.shape[-1]))
204 |         o2=np.reshape(o2,(-1,o2.shape[-1]))
205 |         o3=np.reshape(o3,(-1,o3.shape[-1]))
206 | 
207 |         out=np.vstack((o1,o2,o3))
208 |         q_output.append(out)
209 | 
210 |     q_output=np.array(q_output)
211 |     if(len(q_output)!=0):  
212 |         q_output=q_output.reshape(-1,q_output.shape[-1])
213 |     #print(q_output.shape)
214 | 
215 |     #handle edge case
216 |     for q_index in range((len(train_file_idxs)//batch_num*batch_num),len(dict_to_process.keys())):
217 |         index=train_file_idxs[q_index]
218 |         queries=load_pc_files([dict_to_process[index]["query"]])
219 |         queries= np.expand_dims(queries,axis=1)
220 |         #print(query.shape)
221 |         #exit()
222 |         fake_queries=np.zeros((BATCH_NUM_QUERIES-1,1,NUM_POINTS,13))
223 |         fake_pos=np.zeros((BATCH_NUM_QUERIES,POSITIVES_PER_QUERY,NUM_POINTS,13))
224 |         fake_neg=np.zeros((BATCH_NUM_QUERIES,NEGATIVES_PER_QUERY,NUM_POINTS,13))
225 |         q=np.vstack((queries,fake_queries))
226 |         #print(q.shape)
227 |         feed_dict={ops['query']:q, ops['positives']:fake_pos, ops['negatives']:fake_neg, ops['is_training_pl']:is_training}
228 |         output=sess.run(ops['q_vec'], feed_dict=feed_dict)
229 |         #print(output.shape)
230 |         output=output[0]
231 |         output=np.squeeze(output)
232 |         if (q_output.shape[0]!=0):
233 |             q_output=np.vstack((q_output,output))
234 |         else:
235 |             q_output=output
236 | 
237 |     #q_output=np.array(q_output)
238 |     #q_output=q_output.reshape(-1,q_output.shape[-1])
239 |     print(q_output.shape)
240 |     return q_output
241 | 
242 | def get_recall(sess, ops, m, n):
243 |     global DATABASE_VECTORS
244 |     global QUERY_VECTORS
245 | 
246 |     database_output= DATABASE_VECTORS[m]
247 |     queries_output= QUERY_VECTORS[n]
248 | 
249 |     print(len(queries_output))
250 |     database_nbrs = KDTree(database_output)
251 | 
252 |     num_neighbors=25
253 |     recall=[0]*num_neighbors
254 | 
255 |     top1_similarity_score=[]
256 |     one_percent_retrieved=0
257 |     threshold=max(int(round(len(database_output)/100.0)),1)
258 | 
259 |     num_evaluated=0
260 |     for i in range(len(queries_output)):
261 |         true_neighbors= QUERY_SETS[n][i][m]
262 |         if(len(true_neighbors)==0):
263 |             continue
264 |         num_evaluated+=1
265 |         distances, indices = database_nbrs.query(np.array([queries_output[i]]),k=num_neighbors)
266 |         for j in range(len(indices[0])):
267 |             if indices[0][j] in true_neighbors:
268 |                 if(j==0):
269 |                     similarity= np.dot(queries_output[i],database_output[indices[0][j]])
270 |                     top1_similarity_score.append(similarity)
271 |                 recall[j]+=1
272 |                 break
273 |                 
274 |         if len(list(set(indices[0][0:threshold]).intersection(set(true_neighbors))))>0:
275 |             one_percent_retrieved+=1
276 | 
277 |     one_percent_recall=(one_percent_retrieved/float(num_evaluated))*100
278 |     recall=(np.cumsum(recall)/float(num_evaluated))*100
279 |     print('recall')
280 |     print(recall)
281 |     print('top1_simlar_score')
282 |     print(np.mean(top1_similarity_score))
283 |     print('one_percent_recall')
284 |     print(one_percent_recall)
285 |     return recall, top1_similarity_score, one_percent_recall 
286 | 
287 | def get_similarity(sess, ops, m, n):
288 |     global DATABASE_VECTORS
289 |     global QUERY_VECTORS
290 | 
291 |     database_output= DATABASE_VECTORS[m]
292 |     queries_output= QUERY_VECTORS[n]
293 | 
294 |     threshold= len(queries_output)
295 |     print(len(queries_output))
296 |     database_nbrs = KDTree(database_output)
297 | 
298 |     similarity=[]
299 |     for i in range(len(queries_output)):
300 |         distances, indices = database_nbrs.query(np.array([queries_output[i]]),k=1)
301 |         for j in range(len(indices[0])):
302 |             q_sim= np.dot(q_output[i], database_output[indices[0][j]])
303 |             similarity.append(q_sim)
304 |     average_similarity=np.mean(similarity)
305 |     print(average_similarity)
306 |     return average_similarity 
307 | 
308 | 
309 | if __name__ == "__main__":
310 |     evaluate()
311 | 


--------------------------------------------------------------------------------
/generating_queries/generate_inference_sets.py:
--------------------------------------------------------------------------------
  1 | import pandas as pd
  2 | import numpy as np
  3 | import os
  4 | import pandas as pd
  5 | from sklearn.neighbors import KDTree
  6 | import pickle
  7 | import random
  8 | 
  9 | #####For training and test data split#####
 10 | x_width=150
 11 | y_width=150
 12 | 
 13 | #For Oxford
 14 | p1=[5735712.768124,620084.402381]
 15 | p2=[5735611.299219,620540.270327]
 16 | p3=[5735237.358209,620543.094379]
 17 | p4=[5734749.303802,619932.693364]   
 18 | 
 19 | #For University Sector
 20 | p5=[363621.292362,142864.19756]
 21 | p6=[364788.795462,143125.746609]
 22 | p7=[363597.507711,144011.414174]
 23 | 
 24 | #For Residential Area
 25 | p8=[360895.486453,144999.915143]
 26 | p9=[362357.024536,144894.825301]
 27 | p10=[361368.907155,145209.663042]
 28 | 
 29 | p_dict={"oxford":[p1,p2,p3,p4], "university":[p5,p6,p7], "residential": [p8,p9,p10], "business":[]}
 30 | 
 31 | def check_in_test_set(northing, easting, points, x_width, y_width):
 32 | 	in_test_set=False
 33 | 	for point in points:
 34 | 		if(point[0]-x_width<northing and northing< point[0]+x_width and point[1]-y_width<easting and easting<point[1]+y_width):
 35 | 			in_test_set=True
 36 | 			break
 37 | 	return in_test_set
 38 | ##########################################
 39 | 
 40 | def output_to_file(output, filename):
 41 | 	with open(filename, 'wb') as handle:
 42 | 	    pickle.dump(output, handle, protocol=pickle.HIGHEST_PROTOCOL)
 43 | 	print("Done ", filename)
 44 | 
 45 | 
 46 | def construct_query_and_database_sets(base_path, runs_folder, folders, pointcloud_fols, filename, p, output_name):
 47 | 
 48 | 	database_sets=[]
 49 | 	for folder in folders:
 50 | 		database={}
 51 | 		df_locations= pd.read_csv(os.path.join(base_path,runs_folder,folder,filename),sep=',')
 52 | 		df_locations['timestamp']=runs_folder+folder+pointcloud_fols+df_locations['timestamp'].astype(str)+'.bin'
 53 | 		df_locations=df_locations.rename(columns={'timestamp':'file'})
 54 | 		for index,row in df_locations.iterrows():
 55 | 			database[len(database.keys())]={'query':row['file'],'northing':row['northing'],'easting':row['easting']}
 56 | 		database_sets.append(database)
 57 | 
 58 | 	output_to_file(database_sets, output_name+'_inference_database.pickle')
 59 | 
 60 | 
 61 | ###Building database and query files for evaluation
 62 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 63 | base_path= "../../benchmark_datasets/"
 64 | 
 65 | folders=[]
 66 | index_list=[0]    # Need to be modified to find the folder you want (folders for name and index_list for index)
 67 | runs_folder = "oxford/"
 68 | featurecloud_fols = "/featurecloud_20m_10overlap/"
 69 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
 70 | 
 71 | for index in index_list:
 72 | 	folders.append(all_folders[index])
 73 | 
 74 | print(folders)
 75 | construct_query_and_database_sets(base_path, runs_folder, folders, featurecloud_fols, "pointcloud_locations_20m_10overlap.csv", p_dict["oxford"], "oxford")
 76 | 
 77 | '''
 78 | #For Oxford
 79 | folders=[]
 80 | runs_folder = "oxford/"
 81 | featurecloud_fols = "/featurecloud_20m/"
 82 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
 83 | index_list=[5,6,7,9,10,11,12,13,14,15,16,17,18,19,22,24,31,32,33,38,39,43,44]
 84 | print(len(index_list))
 85 | for index in index_list:
 86 | 	folders.append(all_folders[index])
 87 | 
 88 | print(folders)
 89 | construct_query_and_database_sets(base_path, runs_folder, folders, featurecloud_fols, "pointcloud_locations_20m.csv", p_dict["kitti"], "oxford")
 90 | 
 91 | #For University Sector
 92 | folders=[]
 93 | runs_folder = "inhouse_datasets/"
 94 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
 95 | uni_index=range(10,15)
 96 | for index in uni_index:
 97 | 	folders.append(all_folders[index])
 98 | 
 99 | print(folders)
100 | construct_query_and_database_sets(base_path, runs_folder, folders, "/pointcloud_25m_25/", "pointcloud_centroids_25.csv", p_dict["university"], "university")
101 | 
102 | #For Residential Area
103 | folders=[]
104 | runs_folder = "inhouse_datasets/"
105 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
106 | res_index=range(5,10)
107 | for index in res_index:
108 | 	folders.append(all_folders[index])
109 | 
110 | print(folders)
111 | construct_query_and_database_sets(base_path, runs_folder, folders, "/pointcloud_25m_25/", "pointcloud_centroids_25.csv", p_dict["residential"], "residential")
112 | 
113 | #For Business District
114 | folders=[]
115 | runs_folder = "inhouse_datasets/"
116 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
117 | bus_index=range(5)
118 | for index in bus_index:
119 | 	folders.append(all_folders[index])
120 | 
121 | print(folders)
122 | construct_query_and_database_sets(base_path, runs_folder, folders, "/pointcloud_25m_25/", "pointcloud_centroids_25.csv", p_dict["business"], "business")
123 | '''
124 | 


--------------------------------------------------------------------------------
/generating_queries/generate_test_sets.py:
--------------------------------------------------------------------------------
  1 | import pandas as pd
  2 | import numpy as np
  3 | import os
  4 | import pandas as pd
  5 | from sklearn.neighbors import KDTree
  6 | import pickle
  7 | import random
  8 | 
  9 | #####For training and test data split#####
 10 | x_width=150
 11 | y_width=150
 12 | 
 13 | #For Oxford
 14 | p1=[5735712.768124,620084.402381]
 15 | p2=[5735611.299219,620540.270327]
 16 | p3=[5735237.358209,620543.094379]
 17 | p4=[5734749.303802,619932.693364]   
 18 | 
 19 | #For University Sector
 20 | p5=[363621.292362,142864.19756]
 21 | p6=[364788.795462,143125.746609]
 22 | p7=[363597.507711,144011.414174]
 23 | 
 24 | #For Residential Area
 25 | p8=[360895.486453,144999.915143]
 26 | p9=[362357.024536,144894.825301]
 27 | p10=[361368.907155,145209.663042]
 28 | 
 29 | p_dict={"oxford":[p1,p2,p3,p4], "university":[p5,p6,p7], "residential": [p8,p9,p10], "business":[]}
 30 | 
 31 | def check_in_test_set(northing, easting, points, x_width, y_width):
 32 | 	in_test_set=False
 33 | 	for point in points:
 34 | 		if(point[0]-x_width<northing and northing< point[0]+x_width and point[1]-y_width<easting and easting<point[1]+y_width):
 35 | 			in_test_set=True
 36 | 			break
 37 | 	return in_test_set
 38 | ##########################################
 39 | 
 40 | def output_to_file(output, filename):
 41 | 	with open(filename, 'wb') as handle:
 42 | 	    pickle.dump(output, handle, protocol=pickle.HIGHEST_PROTOCOL)
 43 | 	print("Done ", filename)
 44 | 
 45 | 
 46 | def construct_query_and_database_sets(base_path, runs_folder, folders, pointcloud_fols, filename, p, output_name):
 47 | 	database_trees=[]
 48 | 	test_trees=[]
 49 | 	for folder in folders:
 50 | 		print(folder)
 51 | 		df_database= pd.DataFrame(columns=['file','northing','easting'])
 52 | 		df_test= pd.DataFrame(columns=['file','northing','easting'])
 53 | 		
 54 | 		df_locations= pd.read_csv(os.path.join(base_path,runs_folder,folder,filename),sep=',')
 55 | 		# df_locations['timestamp']=runs_folder+folder+pointcloud_fols+df_locations['timestamp'].astype(str)+'.bin'
 56 | 		# df_locations=df_locations.rename(columns={'timestamp':'file'})
 57 | 		for index, row in df_locations.iterrows():
 58 | 			#entire business district is in the test set
 59 | 			if(output_name=="business"):
 60 | 				df_test=df_test.append(row, ignore_index=True)
 61 | 			elif(check_in_test_set(row['northing'], row['easting'], p, x_width, y_width)):
 62 | 				df_test=df_test.append(row, ignore_index=True)
 63 | 			df_database=df_database.append(row, ignore_index=True)
 64 | 
 65 | 		database_tree = KDTree(df_database[['northing','easting']])
 66 | 		test_tree = KDTree(df_test[['northing','easting']])
 67 | 		database_trees.append(database_tree)
 68 | 		test_trees.append(test_tree)
 69 | 
 70 | 	test_sets=[]
 71 | 	database_sets=[]
 72 | 	for folder in folders:
 73 | 		database={}
 74 | 		test={} 
 75 | 		df_locations= pd.read_csv(os.path.join(base_path,runs_folder,folder,filename),sep=',')
 76 | 		df_locations['timestamp']=runs_folder+folder+pointcloud_fols+df_locations['timestamp'].astype(str)+'.bin'
 77 | 		df_locations=df_locations.rename(columns={'timestamp':'file'})
 78 | 		for index,row in df_locations.iterrows():				
 79 | 			#entire business district is in the test set
 80 | 			if(output_name=="business"):
 81 | 				test[len(test.keys())]={'query':row['file'],'northing':row['northing'],'easting':row['easting']}
 82 | 			elif(check_in_test_set(row['northing'], row['easting'], p, x_width, y_width)):
 83 | 				test[len(test.keys())]={'query':row['file'],'northing':row['northing'],'easting':row['easting']}
 84 | 			database[len(database.keys())]={'query':row['file'],'northing':row['northing'],'easting':row['easting']}
 85 | 		database_sets.append(database)
 86 | 		test_sets.append(test)		
 87 | 
 88 | 	for i in range(len(database_sets)):
 89 | 		tree=database_trees[i]
 90 | 		for j in range(len(test_sets)):
 91 | 			if(i==j):
 92 | 				continue
 93 | 			for key in range(len(test_sets[j].keys())):
 94 | 				coor=np.array([[test_sets[j][key]["northing"],test_sets[j][key]["easting"]]])
 95 | 				index = tree.query_radius(coor, r=25)
 96 | 				#indices of the positive matches in database i of each query (key) in test set j
 97 | 				test_sets[j][key][i]=index[0].tolist()
 98 | 
 99 | 	output_to_file(database_sets, output_name+'_evaluation_database.pickle')
100 | 	output_to_file(test_sets, output_name+'_evaluation_query.pickle')
101 | 
102 | ###Building database and query files for evaluation
103 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
104 | base_path= "../../benchmark_datasets/"
105 | 
106 | #For Oxford
107 | folders=[]
108 | runs_folder = "oxford/"
109 | featurecloud_fols = "/featurecloud_20m/"
110 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
111 | index_list=[5,6,7,9,10,11,12,13,14,15,16,17,18,19,22,24,31,32,33,38,39,43,44]
112 | print(len(index_list))
113 | for index in index_list:
114 | 	folders.append(all_folders[index])
115 | 
116 | print(folders)
117 | construct_query_and_database_sets(base_path, runs_folder, folders, featurecloud_fols, "pointcloud_locations_20m.csv", p_dict["oxford"], "oxford")
118 | 
119 | '''
120 | #For University Sector
121 | folders=[]
122 | runs_folder = "inhouse_datasets/"
123 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
124 | uni_index=range(10,15)
125 | for index in uni_index:
126 | 	folders.append(all_folders[index])
127 | 
128 | print(folders)
129 | construct_query_and_database_sets(base_path, runs_folder, folders, "/pointcloud_25m_25/", "pointcloud_centroids_25.csv", p_dict["university"], "university")
130 | 
131 | #For Residential Area
132 | folders=[]
133 | runs_folder = "inhouse_datasets/"
134 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
135 | res_index=range(5,10)
136 | for index in res_index:
137 | 	folders.append(all_folders[index])
138 | 
139 | print(folders)
140 | construct_query_and_database_sets(base_path, runs_folder, folders, "/pointcloud_25m_25/", "pointcloud_centroids_25.csv", p_dict["residential"], "residential")
141 | 
142 | #For Business District
143 | folders=[]
144 | runs_folder = "inhouse_datasets/"
145 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
146 | bus_index=range(5)
147 | for index in bus_index:
148 | 	folders.append(all_folders[index])
149 | 
150 | print(folders)
151 | construct_query_and_database_sets(base_path, runs_folder, folders, "/pointcloud_25m_25/", "pointcloud_centroids_25.csv", p_dict["business"], "business")
152 | '''
153 | 


--------------------------------------------------------------------------------
/generating_queries/generate_training_tuples_baseline.py:
--------------------------------------------------------------------------------
 1 | import pandas as pd
 2 | import numpy as np
 3 | import os
 4 | import pandas as pd
 5 | from sklearn.neighbors import KDTree
 6 | import pickle
 7 | import random
 8 | 
 9 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
10 | base_path = "../../benchmark_datasets/"
11 | 
12 | runs_folder= "oxford/"
13 | filename = "pointcloud_locations_20m_10overlap.csv"
14 | pointcloud_fols="/pointcloud_20m_10overlap/"
15 | featurecloud_fols="/featurecloud_20m_10overlap/"
16 | 
17 | all_folders=sorted(os.listdir(os.path.join(BASE_DIR,base_path,runs_folder)))
18 | 
19 | folders=[]
20 | 
21 | #All runs are used for training (both full and partial)
22 | index_list=range(len(all_folders))
23 | print("Number of runs: "+str(len(index_list)))
24 | for index in index_list:
25 | 	folders.append(all_folders[index])
26 | print(folders)
27 | 
28 | #####For training and test data split#####
29 | x_width=150
30 | y_width=150
31 | p1=[5735712.768124,620084.402381]
32 | p2=[5735611.299219,620540.270327]
33 | p3=[5735237.358209,620543.094379]
34 | p4=[5734749.303802,619932.693364]   
35 | p=[p1,p2,p3,p4]
36 | 
37 | 
38 | def check_in_test_set(northing, easting, points, x_width, y_width):
39 | 	in_test_set=False
40 | 	for point in points:
41 | 		if(point[0]-x_width<northing and northing< point[0]+x_width and point[1]-y_width<easting and easting<point[1]+y_width):
42 | 			in_test_set=True
43 | 			break
44 | 	return in_test_set
45 | ##########################################
46 | 
47 | 
48 | def construct_query_dict(df_centroids, filename):
49 | 	tree = KDTree(df_centroids[['northing','easting']])
50 | 	ind_nn = tree.query_radius(df_centroids[['northing','easting']],r=10)
51 | 	ind_r = tree.query_radius(df_centroids[['northing','easting']], r=50)
52 | 	queries={}
53 | 	for i in range(len(ind_nn)):
54 | 		query=df_centroids.iloc[i]["file"]
55 | 		positives=np.setdiff1d(ind_nn[i],[i]).tolist()
56 | 		negatives=np.setdiff1d(df_centroids.index.values.tolist(),ind_r[i]).tolist()
57 | 		random.shuffle(negatives)
58 | 		queries[i]={"query":query,"positives":positives,"negatives":negatives}
59 | 
60 | 	with open(filename, 'wb') as handle:
61 | 	    pickle.dump(queries, handle, protocol=pickle.HIGHEST_PROTOCOL)
62 | 
63 | 	print("Done ", filename)
64 | 
65 | 
66 | ####Initialize pandas DataFrame
67 | df_train= pd.DataFrame(columns=['file','northing','easting'])
68 | df_test= pd.DataFrame(columns=['file','northing','easting'])
69 | 
70 | for folder in folders:
71 | 	df_locations= pd.read_csv(os.path.join(base_path,runs_folder,folder,filename),sep=',')
72 | 	df_locations['timestamp']=runs_folder+folder+featurecloud_fols+df_locations['timestamp'].astype(str)+'.bin'
73 | 	df_locations=df_locations.rename(columns={'timestamp':'file'})
74 | 	
75 | 	for index, row in df_locations.iterrows():
76 | 		if(check_in_test_set(row['northing'], row['easting'], p, x_width, y_width)):
77 | 			df_test=df_test.append(row, ignore_index=True)
78 | 		else:
79 | 			df_train=df_train.append(row, ignore_index=True)
80 | 
81 | print("Number of training submaps: "+str(len(df_train['file'])))
82 | print("Number of non-disjoint test submaps: "+str(len(df_test['file'])))
83 | construct_query_dict(df_train,"training_queries_baseline.pickle")
84 | construct_query_dict(df_test,"test_queries_baseline.pickle")
85 | 
86 | 


--------------------------------------------------------------------------------
/inference.py:
--------------------------------------------------------------------------------
  1 | # Author: Chuanzhe Suo (suo_ivy@foxmail.com) 10/26/2018
  2 | # Thanks to Mikaela Angelina Uy, modified from PointNetVLAD
  3 | # Reference: LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis, ICCV 2019
  4 | 
  5 | import argparse
  6 | import os
  7 | import sys
  8 | import importlib
  9 | import tensorflow as tf
 10 | import numpy as np
 11 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 12 | sys.path.append(BASE_DIR)
 13 | sys.path.append(os.path.join(BASE_DIR, 'models'))
 14 | sys.path.append(os.path.join(BASE_DIR, 'utils'))
 15 | from loading_pointclouds import *
 16 | from sklearn.neighbors import KDTree
 17 | 
 18 | #params
 19 | parser = argparse.ArgumentParser()
 20 | parser.add_argument('--gpu', type=int, default=0, help='GPU to use [default: GPU 1]')
 21 | parser.add_argument('--log_dir', default='log/', help='Log dir [default: log]')
 22 | parser.add_argument('--positives_per_query', type=int, default=0, help='Number of potential positives in each training tuple [default: 2]')
 23 | parser.add_argument('--negatives_per_query', type=int, default=0, help='Number of definite negatives in each training tuple [default: 20]')
 24 | parser.add_argument('--batch_num_queries', type=int, default=25, help='Batch Size during training [default: 1]')
 25 | parser.add_argument('--dimension', type=int, default=256)
 26 | parser.add_argument('--decay_step', type=int, default=200000, help='Decay step for lr decay [default: 200000]')
 27 | parser.add_argument('--decay_rate', type=float, default=0.7, help='Decay rate for lr decay [default: 0.8]')
 28 | FLAGS = parser.parse_args()
 29 | 
 30 | #BATCH_SIZE = FLAGS.batch_size
 31 | BATCH_NUM_QUERIES = FLAGS.batch_num_queries
 32 | EVAL_BATCH_SIZE = 1
 33 | NUM_POINTS = 4096
 34 | POSITIVES_PER_QUERY= FLAGS.positives_per_query
 35 | NEGATIVES_PER_QUERY= FLAGS.negatives_per_query
 36 | GPU_INDEX = FLAGS.gpu
 37 | DECAY_STEP = FLAGS.decay_step
 38 | DECAY_RATE = FLAGS.decay_rate
 39 | 
 40 | INFERENCE_FILE= 'generating_queries/oxford_inference_database.pickle'
 41 | 
 42 | LOG_DIR = FLAGS.log_dir
 43 | model = LOG_DIR.split('/')[1]
 44 | model = model.split('-')[0]
 45 | print(LOG_DIR)
 46 | MODEL = importlib.import_module(model)
 47 | model_file= "model.ckpt"
 48 | INFERENCE_SETS= get_sets_dict(INFERENCE_FILE)
 49 | 
 50 | global INFERENCE_VECTORS
 51 | INFERENCE_VECTORS=[]
 52 | 
 53 | BN_INIT_DECAY = 0.5
 54 | BN_DECAY_DECAY_RATE = 0.5
 55 | BN_DECAY_DECAY_STEP = float(DECAY_STEP)
 56 | BN_DECAY_CLIP = 0.99
 57 | 
 58 | def get_bn_decay(batch):
 59 |     bn_momentum = tf.train.exponential_decay(
 60 |                       BN_INIT_DECAY,
 61 |                       batch*BATCH_NUM_QUERIES,
 62 |                       BN_DECAY_DECAY_STEP,
 63 |                       BN_DECAY_DECAY_RATE,
 64 |                       staircase=True)
 65 |     bn_decay = tf.minimum(BN_DECAY_CLIP, 1 - bn_momentum)
 66 |     return bn_decay
 67 | 
 68 | def inference():
 69 |     global INFERENCE_VECTORS
 70 | 
 71 |     with tf.Graph().as_default():
 72 |         with tf.device('/gpu:'+str(GPU_INDEX)):
 73 |             print("In Graph")
 74 |             query= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, 1, NUM_POINTS)
 75 |             positives= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, POSITIVES_PER_QUERY, NUM_POINTS)
 76 |             negatives= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, NEGATIVES_PER_QUERY, NUM_POINTS)
 77 |             eval_queries= MODEL.placeholder_inputs(EVAL_BATCH_SIZE, 1, NUM_POINTS)
 78 | 
 79 |             is_training_pl = tf.placeholder(tf.bool, shape=())
 80 |             print(is_training_pl)
 81 | 
 82 |             batch = tf.Variable(0)
 83 |             bn_decay = get_bn_decay(batch)
 84 | 
 85 |             with tf.variable_scope("query_triplets") as scope:
 86 |                 vecs= tf.concat([query, positives, negatives],1)
 87 |                 print(vecs)
 88 |                 out_vecs= MODEL.forward(vecs, is_training_pl, bn_decay=bn_decay)
 89 |                 q_vec, pos_vecs, neg_vecs= tf.split(out_vecs, [1,POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY],1)
 90 |                 print(q_vec)
 91 |                 print(pos_vecs)
 92 |                 print(neg_vecs)
 93 | 
 94 |             saver = tf.train.Saver()
 95 | 
 96 |         # Create a session
 97 |         gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.95)
 98 |         config = tf.ConfigProto(gpu_options=gpu_options)
 99 |         config.gpu_options.allow_growth = True
100 |         config.allow_soft_placement = True
101 |         config.log_device_placement = False
102 |         sess = tf.Session(config=config)
103 | 
104 | 
105 |         saver.restore(sess, os.path.join(LOG_DIR, model_file))
106 |         print("Model restored:{}".format(os.path.join(LOG_DIR, model_file)))
107 | 
108 |         ops = {'query': query,
109 |                'positives': positives,
110 |                'negatives': negatives,
111 |                'is_training_pl': is_training_pl,
112 |                'eval_queries': eval_queries,
113 |                'q_vec':q_vec,
114 |                'pos_vecs': pos_vecs,
115 |                'neg_vecs': neg_vecs}
116 |         recall= np.zeros(25)
117 |         count=0
118 |         similarity=[]
119 |         one_percent_recall=[]
120 |         for i in range(len(INFERENCE_SETS)):
121 |             INFERENCE_VECTORS.append(get_latent_vectors(sess, ops, INFERENCE_SETS[i]))
122 | 
123 |         INFERENCE_VECTORS = np.asarray(INFERENCE_VECTORS)
124 |         INFERENCE_VECTORS.tofile('inference_vectors.bin')
125 |         print("feature saved:inference_vectors.bin")
126 | 
127 | def get_latent_vectors(sess, ops, dict_to_process):
128 |     is_training=False
129 |     train_file_idxs = np.arange(0, len(dict_to_process.keys()))
130 |     #print(len(train_file_idxs))
131 |     batch_num= BATCH_NUM_QUERIES*(1+POSITIVES_PER_QUERY+NEGATIVES_PER_QUERY)
132 |     q_output = []
133 |     for q_index in range(len(train_file_idxs)//batch_num):
134 |         file_indices=train_file_idxs[q_index*batch_num:(q_index+1)*(batch_num)]
135 |         file_names=[]
136 |         for index in file_indices:
137 |             file_names.append(dict_to_process[index]["query"])
138 |         queries=load_pc_files(file_names)
139 |         # queries= np.expand_dims(queries,axis=1)
140 |         q1=queries[0:BATCH_NUM_QUERIES]
141 |         q1=np.expand_dims(q1,axis=1)
142 |         #print(q1.shape)
143 | 
144 |         q2=queries[BATCH_NUM_QUERIES:BATCH_NUM_QUERIES*(POSITIVES_PER_QUERY+1)]
145 |         q2=np.reshape(q2,(BATCH_NUM_QUERIES,POSITIVES_PER_QUERY,NUM_POINTS,13))
146 | 
147 |         q3=queries[BATCH_NUM_QUERIES*(POSITIVES_PER_QUERY+1):BATCH_NUM_QUERIES*(NEGATIVES_PER_QUERY+POSITIVES_PER_QUERY+1)]
148 |         q3=np.reshape(q3,(BATCH_NUM_QUERIES,NEGATIVES_PER_QUERY,NUM_POINTS,13))
149 |         feed_dict={ops['query']:q1, ops['positives']:q2, ops['negatives']:q3, ops['is_training_pl']:is_training}
150 |         o1, o2, o3=sess.run([ops['q_vec'], ops['pos_vecs'], ops['neg_vecs']], feed_dict=feed_dict)
151 |         
152 |         o1=np.reshape(o1,(-1,o1.shape[-1]))
153 |         o2=np.reshape(o2,(-1,o2.shape[-1]))
154 |         o3=np.reshape(o3,(-1,o3.shape[-1]))
155 | 
156 |         out=np.vstack((o1,o2,o3))
157 |         q_output.append(out)
158 | 
159 |     q_output=np.array(q_output)
160 |     if(len(q_output)!=0):  
161 |         q_output=q_output.reshape(-1,q_output.shape[-1])
162 |     #print(q_output.shape)
163 | 
164 |     #handle edge case
165 |     for q_index in range((len(train_file_idxs)//batch_num*batch_num),len(dict_to_process.keys())):
166 |         index=train_file_idxs[q_index]
167 |         queries=load_pc_files([dict_to_process[index]["query"]])
168 |         queries= np.expand_dims(queries,axis=1)
169 |         #print(query.shape)
170 |         #exit()
171 |         fake_queries=np.zeros((BATCH_NUM_QUERIES-1,1,NUM_POINTS,13))
172 |         fake_pos=np.zeros((BATCH_NUM_QUERIES,POSITIVES_PER_QUERY,NUM_POINTS,13))
173 |         fake_neg=np.zeros((BATCH_NUM_QUERIES,NEGATIVES_PER_QUERY,NUM_POINTS,13))
174 |         q=np.vstack((queries,fake_queries))
175 |         #print(q.shape)
176 |         feed_dict={ops['query']:q, ops['positives']:fake_pos, ops['negatives']:fake_neg, ops['is_training_pl']:is_training}
177 |         output=sess.run(ops['q_vec'], feed_dict=feed_dict)
178 |         #print(output.shape)
179 |         output=output[0]
180 |         output=np.squeeze(output)
181 |         if (q_output.shape[0]!=0):
182 |             q_output=np.vstack((q_output,output))
183 |         else:
184 |             q_output=output
185 | 
186 |     #q_output=np.array(q_output)
187 |     #q_output=q_output.reshape(-1,q_output.shape[-1])
188 |     print(q_output.shape)
189 |     return q_output
190 | 
191 | 
192 | if __name__ == "__main__":
193 |     inference()
194 | 


--------------------------------------------------------------------------------
/loupe.py:
--------------------------------------------------------------------------------
  1 | # Copyright 2017 Antoine Miech All Rights Reserved.
  2 | #
  3 | # Licensed under the Apache License, Version 2.0 (the "License");
  4 | # you may not use this file except in compliance with the License.
  5 | # You may obtain a copy of the License at
  6 | #
  7 | #      http://www.apache.org/licenses/LICENSE-2.0
  8 | #
  9 | # Unless required by applicable law or agreed to in writing, software
 10 | # distributed under the License is distributed on an "AS-IS" BASIS,
 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 | # See the License for the specific language governing permissions and
 13 | # limitations under the License.
 14 | 
 15 | """ Learnable mOdUle for Pooling fEatures (LOUPE)
 16 | Contains a collection of models (NetVLAD, NetRVLAD, NetFV and Soft-DBoW)
 17 | which enables pooling of a list of features into a single compact 
 18 | representation.
 19 | 
 20 | Reference:
 21 | 
 22 | Learnable pooling method with Context Gating for video classification
 23 | Antoine Miech, Ivan Laptev, Josef Sivic
 24 | 
 25 | """
 26 | 
 27 | import math
 28 | import tensorflow as tf
 29 | import tensorflow.contrib.slim as slim
 30 | import numpy as np
 31 | 
 32 | class PoolingBaseModel(object):
 33 |     """Inherit from this class when implementing new models."""
 34 | 
 35 |     def __init__(self, feature_size, max_samples, cluster_size, output_dim,
 36 |             gating=True,add_batch_norm=True, is_training=True):
 37 |         """Initialize a NetVLAD block.
 38 | 
 39 |         Args:
 40 |         feature_size: Dimensionality of the input features.
 41 |         max_samples: The maximum number of samples to pool.
 42 |         cluster_size: The number of clusters.
 43 |         output_dim: size of the output space after dimension reduction.
 44 |         add_batch_norm: (bool) if True, adds batch normalization.
 45 |         is_training: (bool) Whether or not the graph is training.
 46 |         """
 47 | 
 48 |         self.feature_size = feature_size
 49 |         self.max_samples = max_samples
 50 |         self.output_dim = output_dim
 51 |         self.is_training = is_training
 52 |         self.gating = gating
 53 |         self.add_batch_norm = add_batch_norm
 54 |         self.cluster_size = cluster_size
 55 | 
 56 |     def forward(self, reshaped_input):
 57 |         raise NotImplementedError("Models should implement the forward pass.")
 58 | 
 59 |     def context_gating(self, input_layer):
 60 |         """Context Gating
 61 | 
 62 |         Args:
 63 |         input_layer: Input layer in the following shape:
 64 |         'batch_size' x 'number_of_activation'
 65 | 
 66 |         Returns:
 67 |         activation: gated layer in the following shape:
 68 |         'batch_size' x 'number_of_activation'
 69 |         """
 70 | 
 71 |         input_dim = input_layer.get_shape().as_list()[1] 
 72 |         
 73 |         gating_weights = tf.get_variable("gating_weights",
 74 |           [input_dim, input_dim],
 75 |           initializer = tf.random_normal_initializer(
 76 |           stddev=1 / math.sqrt(input_dim)))
 77 |         
 78 |         gates = tf.matmul(input_layer, gating_weights)
 79 |  
 80 |         if self.add_batch_norm:
 81 |           gates = slim.batch_norm(
 82 |               gates,
 83 |               center=True,
 84 |               scale=True,
 85 |               is_training=self.is_training,
 86 |               scope="gating_bn")
 87 |         else:
 88 |           gating_biases = tf.get_variable("gating_biases",
 89 |             [input_dim],
 90 |             initializer = tf.random_normal(stddev=1 / math.sqrt(input_dim)))
 91 |           gates += gating_biases
 92 | 
 93 |         gates = tf.sigmoid(gates)
 94 | 
 95 |         activation = tf.multiply(input_layer,gates)
 96 | 
 97 |         return activation
 98 | 
 99 | #Edited based on the original version
100 | class NetVLAD(PoolingBaseModel):
101 |     """Creates a NetVLAD class.
102 |     """
103 |     def __init__(self, feature_size, max_samples, cluster_size, output_dim,
104 |             gating=True,add_batch_norm=True, is_training=True):
105 |         super(self.__class__, self).__init__(
106 |             feature_size=feature_size,
107 |             max_samples=max_samples,
108 |             cluster_size=cluster_size,
109 |             output_dim=output_dim,
110 |             gating=gating,
111 |             add_batch_norm=add_batch_norm,
112 |             is_training=is_training)
113 | 
114 |     def forward(self, reshaped_input):
115 |         """Forward pass of a NetVLAD block.
116 | 
117 |         Args:
118 |         reshaped_input: If your input is in that form:
119 |         'batch_size' x 'max_samples' x 'feature_size'
120 |         It should be reshaped in the following form:
121 |         'batch_size*max_samples' x 'feature_size'
122 |         by performing:
123 |         reshaped_input = tf.reshape(input, [-1, features_size])
124 | 
125 |         Returns:
126 |         vlad: the pooled vector of size: 'batch_size' x 'output_dim'
127 |         """
128 | 
129 | 
130 |         cluster_weights = tf.get_variable("cluster_weights",
131 |               [self.feature_size, self.cluster_size],
132 |               initializer = tf.random_normal_initializer(
133 |               stddev=1 / math.sqrt(self.feature_size)))
134 |        
135 |         activation = tf.matmul(reshaped_input, cluster_weights)
136 | 
137 |         # activation = tf.contrib.layers.batch_norm(activation, 
138 |         #         center=True, scale=True, 
139 |         #         is_training=self.is_training,
140 |         #         scope='cluster_bn')
141 | 
142 |         # activation = slim.batch_norm(
143 |         #       activation,
144 |         #       center=True,
145 |         #       scale=True,
146 |         #       is_training=self.is_training,
147 |         #       scope="cluster_bn")
148 |         
149 |         if self.add_batch_norm:
150 |           activation = slim.batch_norm(
151 |               activation,
152 |               center=True,
153 |               scale=True,
154 |               is_training=self.is_training,
155 |               scope="cluster_bn", fused=False)
156 |         else:
157 |           cluster_biases = tf.get_variable("cluster_biases",
158 |             [self.cluster_size],
159 |             initializer = tf.random_normal_initializer(
160 |             stddev=1 / math.sqrt(self.feature_size)))
161 |           activation += cluster_biases
162 |         
163 |         activation = tf.nn.softmax(activation)
164 | 
165 |         activation = tf.reshape(activation,
166 |                 [-1, self.max_samples, self.cluster_size])
167 | 
168 |         a_sum = tf.reduce_sum(activation,-2,keep_dims=True)
169 | 
170 |         cluster_weights2 = tf.get_variable("cluster_weights2",
171 |             [1,self.feature_size, self.cluster_size],
172 |             initializer = tf.random_normal_initializer(
173 |                 stddev=1 / math.sqrt(self.feature_size)))
174 |         
175 |         a = tf.multiply(a_sum,cluster_weights2)
176 |         
177 |         activation = tf.transpose(activation,perm=[0,2,1])
178 |         
179 |         reshaped_input = tf.reshape(reshaped_input,[-1,
180 |             self.max_samples, self.feature_size])
181 | 
182 |         vlad = tf.matmul(activation,reshaped_input)
183 |         vlad = tf.transpose(vlad,perm=[0,2,1])
184 |         vlad = tf.subtract(vlad,a)
185 |         
186 | 
187 |         vlad = tf.nn.l2_normalize(vlad,1)
188 | 
189 |         vlad = tf.reshape(vlad,[-1, self.cluster_size*self.feature_size])
190 |         vlad = tf.nn.l2_normalize(vlad,1)
191 | 
192 |         hidden1_weights = tf.get_variable("hidden1_weights",
193 |           [self.cluster_size*self.feature_size, self.output_dim],
194 |           initializer=tf.random_normal_initializer(
195 |           stddev=1 / math.sqrt(self.cluster_size)))
196 |         
197 |         ##Tried using dropout
198 |         #vlad=tf.layers.dropout(vlad,rate=0.5,training=self.is_training)
199 | 
200 |         vlad = tf.matmul(vlad, hidden1_weights)
201 | 
202 |         ##Added a batch norm
203 |         vlad = tf.contrib.layers.batch_norm(vlad, 
204 |                                           center=True, scale=True, 
205 |                                           is_training=self.is_training,
206 |                                           scope='bn')
207 | 
208 |         if self.gating:
209 |             vlad = super(self.__class__, self).context_gating(vlad)
210 | 
211 |         return vlad
212 | 


--------------------------------------------------------------------------------
/models/lpd_FNSF.py:
--------------------------------------------------------------------------------
  1 | '''
  2 |     LPD-Net Model: FN-SF-VLAD
  3 |     Feature Network + FN-Parallel structure (P) + Series-FC structure (SF)
  4 |     # Thanks to Mikaela Angelina Uy, modified from PointNetVLAD
  5 |     # Reference: LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis, ICCV 2019
  6 |     author: suo_ivy
  7 |     created: 10/26/18
  8 | '''
  9 | import os
 10 | import sys
 11 | import tensorflow as tf
 12 | 
 13 | #Taken from Charles Qi's pointnet code
 14 | MODELS_DIR = os.path.dirname(__file__)
 15 | sys.path.append(MODELS_DIR)
 16 | sys.path.append(os.path.join(MODELS_DIR, '../utils'))
 17 | import tf_util
 18 | from transform_nets import input_transform_net, feature_transform_net, neural_feature_net
 19 | 
 20 | #Adopted from Antoine Meich
 21 | import loupe as lp
 22 | 
 23 | 
 24 | 
 25 | def placeholder_inputs(batch_num_queries, num_pointclouds_per_query, num_point):
 26 |     pointclouds_pl = tf.placeholder(tf.float32, shape=(batch_num_queries, num_pointclouds_per_query, num_point, 13))
 27 |     return pointclouds_pl
 28 | 
 29 | #Adopted from the original pointnet code
 30 | def forward(point_cloud, is_training, bn_decay=None):
 31 |     """LPD-Net:FNSF,    INPUT is batch_num_queries X num_pointclouds_per_query X num_points_per_pointcloud X 13,
 32 |                         OUTPUT batch_num_queries X num_pointclouds_per_query X output_dim """
 33 |     batch_num_queries = point_cloud.get_shape()[0].value
 34 |     num_pointclouds_per_query = point_cloud.get_shape()[1].value
 35 |     num_points = point_cloud.get_shape()[2].value
 36 |     CLUSTER_SIZE=64
 37 |     OUTPUT_DIM=256
 38 |     k=20
 39 |     point_cloud = tf.reshape(point_cloud, [batch_num_queries*num_pointclouds_per_query, num_points,13])
 40 | 
 41 |     point_cloud, feature_cloud = tf.split(point_cloud, [3,10], 2)
 42 | 
 43 |     with tf.variable_scope('transform_net1') as sc:
 44 |         input_transform = input_transform_net(point_cloud, is_training, bn_decay, K=3)
 45 |     point_cloud_transformed = tf.matmul(point_cloud, input_transform)
 46 | 
 47 |     # Neural Network to learn neighborhood features
 48 |     # feature_cloud = neural_feature_net(point_cloud, is_training, bn_decay, knn_k=20, F=10)
 49 | 
 50 |     point_cloud_input = tf.concat([point_cloud_transformed, feature_cloud], 2)
 51 | 
 52 |     point_cloud_input = tf.expand_dims(point_cloud_input, -1)
 53 | 
 54 |     net = tf_util.conv2d(point_cloud_input, 64, [1, 13],
 55 |                          padding='VALID', stride=[1,1],
 56 |                          is_training=is_training,
 57 |                          scope='conv1', bn_decay=bn_decay)
 58 |     net = tf_util.conv2d(net, 64, [1, 1],
 59 |                          padding='VALID', stride=[1,1],
 60 |                          is_training=is_training,
 61 |                          scope='conv2', bn_decay=bn_decay)
 62 | 
 63 |     with tf.variable_scope('transform_net2') as sc:
 64 |         feature_transform = feature_transform_net(net, is_training, bn_decay, K=64)
 65 |     feature_transform = tf.matmul(tf.squeeze(net, axis=[2]), feature_transform)
 66 | 
 67 |     # Serial structure
 68 |     # Danymic Graph cnn for feature space
 69 |     with tf.variable_scope('DGfeature') as sc:
 70 |         adj_matrix = tf_util.pairwise_distance(feature_transform)
 71 |         nn_idx = tf_util.knn(adj_matrix, k=k)
 72 |         edge_feature = tf_util.get_edge_feature(net, nn_idx=nn_idx, k=k)
 73 | 
 74 |         net = tf_util.conv2d(edge_feature, 64, [1, 1],
 75 |                              padding='VALID', stride=[1, 1],
 76 |                              bn=True, is_training=is_training,
 77 |                              scope='dgmlp1', bn_decay=bn_decay)
 78 |         net = tf_util.conv2d(net, 64, [1, 1],
 79 |                              padding='VALID', stride=[1, 1],
 80 |                              bn=True, is_training=is_training,
 81 |                              scope='dgmlp2', bn_decay=bn_decay)
 82 |         net = tf.reduce_max(net, axis=-2, keep_dims=True)
 83 | 
 84 |     # Spatial Neighborhood fusion for cartesian space
 85 |     with tf.variable_scope('SNfeature') as sc:
 86 |         adj_matrix = tf_util.pairwise_distance(point_cloud)
 87 |         nn_idx = tf_util.knn(adj_matrix, k=k)
 88 | 
 89 |         idx_ = tf.range(batch_num_queries*num_pointclouds_per_query) * num_points
 90 |         idx_ = tf.reshape(idx_, [batch_num_queries*num_pointclouds_per_query, 1, 1])
 91 | 
 92 |         feature_cloud = tf.reshape(net, [-1, 64])
 93 |         edge_feature = tf.gather(feature_cloud, nn_idx+idx_)
 94 | 
 95 |         net = tf_util.conv2d(edge_feature, 64, [1, 1],
 96 |                              padding='VALID', stride=[1, 1],
 97 |                              bn=True, is_training=is_training,
 98 |                              scope='snmlp1', bn_decay=bn_decay)
 99 |         net = tf_util.conv2d(net, 64, [1, 1],
100 |                              padding='VALID', stride=[1, 1],
101 |                              bn=True, is_training=is_training,
102 |                              scope='snmlp2', bn_decay=bn_decay)
103 |         net = tf.reduce_max(net, axis=-2, keep_dims=True)
104 | 
105 |     # MLP for fusion
106 |     net = tf_util.conv2d(net, 64, [1, 1],
107 |                          padding='VALID', stride=[1,1],
108 |                          is_training=is_training,
109 |                          scope='conv3', bn_decay=bn_decay)
110 |     net = tf_util.conv2d(net, 128, [1, 1],
111 |                          padding='VALID', stride=[1,1],
112 |                          is_training=is_training,
113 |                          scope='conv4', bn_decay=bn_decay)
114 |     net = tf_util.conv2d(net, 1024, [1, 1],
115 |                          padding='VALID', stride=[1,1],
116 |                          is_training=is_training,
117 |                          scope='conv5', bn_decay=bn_decay)
118 |     point_wise_feature = net
119 | 
120 |     NetVLAD = lp.NetVLAD(feature_size=1024, max_samples=num_points, cluster_size=CLUSTER_SIZE, 
121 |                     output_dim=OUTPUT_DIM, gating=True, add_batch_norm=True,
122 |                     is_training=is_training)
123 | 
124 |     net= tf.reshape(net,[-1,1024])
125 |     net = tf.nn.l2_normalize(net,1)
126 |     output = NetVLAD.forward(net)
127 |     print(output)
128 | 
129 |     #normalize to have norm 1
130 |     output = tf.nn.l2_normalize(output,1)
131 |     output =  tf.reshape(output,[batch_num_queries,num_pointclouds_per_query,OUTPUT_DIM])
132 | 
133 |     return output
134 | 
135 | 
136 | def best_pos_distance(query, pos_vecs):
137 |     with tf.name_scope('best_pos_distance') as scope:
138 |         #batch = query.get_shape()[0]
139 |         num_pos = pos_vecs.get_shape()[1]
140 |         query_copies = tf.tile(query, [1,int(num_pos),1]) #shape num_pos x output_dim
141 |         best_pos=tf.reduce_min(tf.reduce_sum(tf.squared_difference(pos_vecs,query_copies),2),1)
142 |         #best_pos=tf.reduce_max(tf.reduce_sum(tf.squared_difference(pos_vecs,query_copies),2),1)
143 |         return best_pos
144 | 
145 | 
146 | 
147 | ##########Losses for PointNetVLAD###########
148 | 
149 | #Returns average loss across the query tuples in a batch, loss in each is the average loss of the definite negatives against the best positive
150 | def triplet_loss(q_vec, pos_vecs, neg_vecs, margin):
151 |      # ''', end_points, reg_weight=0.001):
152 |     best_pos=best_pos_distance(q_vec, pos_vecs)
153 |     num_neg = neg_vecs.get_shape()[1]
154 |     batch = q_vec.get_shape()[0]
155 |     query_copies = tf.tile(q_vec, [1, int(num_neg),1])
156 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
157 |     m=tf.fill([int(batch), int(num_neg)],margin)
158 |     triplet_loss=tf.reduce_mean(tf.reduce_sum(tf.maximum(tf.add(m,tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,query_copies),2))), tf.zeros([int(batch), int(num_neg)])),1))
159 |     return triplet_loss
160 | 
161 | #Lazy variant
162 | def lazy_triplet_loss(q_vec, pos_vecs, neg_vecs, margin):
163 |     best_pos=best_pos_distance(q_vec, pos_vecs)
164 |     num_neg = neg_vecs.get_shape()[1]
165 |     batch = q_vec.get_shape()[0]
166 |     query_copies = tf.tile(q_vec, [1, int(num_neg),1])
167 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
168 |     m=tf.fill([int(batch), int(num_neg)],margin)
169 |     triplet_loss=tf.reduce_mean(tf.reduce_max(tf.maximum(tf.add(m,tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,query_copies),2))), tf.zeros([int(batch), int(num_neg)])),1))
170 |     return triplet_loss
171 | 
172 | 
173 | def softmargin_loss(q_vec, pos_vecs, neg_vecs):
174 |     best_pos=best_pos_distance(q_vec, pos_vecs)
175 |     num_neg = neg_vecs.get_shape()[1]
176 |     batch = q_vec.get_shape()[0]
177 |     query_copies = tf.tile(q_vec, [1, int(num_neg),1])
178 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
179 |     ones=tf.fill([int(batch), int(num_neg)],1.0)
180 |     soft_loss=tf.reduce_mean(tf.reduce_sum(tf.log(tf.exp(tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,query_copies),2)))+1.0),1))
181 |     return soft_los
182 | 
183 | def lazy_softmargin_loss(q_vec, pos_vecs, neg_vecs):
184 |     best_pos=best_pos_distance(q_vec, pos_vecs)
185 |     num_neg = neg_vecs.get_shape()[1]
186 |     batch = q_vec.get_shape()[0]
187 |     query_copies = tf.tile(q_vec, [1, int(num_neg),1])
188 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
189 |     ones=tf.fill([int(batch), int(num_neg)],1.0)
190 |     soft_loss=tf.reduce_mean(tf.reduce_max(tf.log(tf.exp(tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,query_copies),2)))+1.0),1))
191 |     return soft_loss
192 | 
193 | def quadruplet_loss_sm(q_vec, pos_vecs, neg_vecs, other_neg, m2):
194 |     soft_loss= softmargin_loss(q_vec, pos_vecs, neg_vecs)
195 |     
196 |     best_pos=best_pos_distance(q_vec, pos_vecs)
197 |     num_neg = neg_vecs.get_shape()[1]
198 |     batch = q_vec.get_shape()[0]
199 | 
200 |     other_neg_copies = tf.tile(other_neg, [1, int(num_neg),1])
201 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
202 |     m2=tf.fill([int(batch), int(num_neg)],m2)
203 | 
204 |     second_loss=tf.reduce_mean(tf.reduce_sum(tf.maximum(tf.add(m2,tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,other_neg_copies),2))), tf.zeros([int(batch), int(num_neg)])),1))
205 | 
206 |     total_loss= soft_loss+second_loss
207 | 
208 |     return total_loss   
209 | 
210 | def lazy_quadruplet_loss_sm(q_vec, pos_vecs, neg_vecs, other_neg, m2):
211 |     soft_loss= lazy_softmargin_loss(q_vec, pos_vecs, neg_vecs)
212 |     
213 |     best_pos=best_pos_distance(q_vec, pos_vecs)
214 |     num_neg = neg_vecs.get_shape()[1]
215 |     batch = q_vec.get_shape()[0]
216 | 
217 |     other_neg_copies = tf.tile(other_neg, [1, int(num_neg),1])
218 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
219 |     m2=tf.fill([int(batch), int(num_neg)],m2)
220 | 
221 |     second_loss=tf.reduce_mean(tf.reduce_max(tf.maximum(tf.add(m2,tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,other_neg_copies),2))), tf.zeros([int(batch), int(num_neg)])),1))
222 | 
223 |     total_loss= soft_loss+second_loss
224 | 
225 |     return total_loss
226 | 
227 | 
228 | def quadruplet_loss(q_vec, pos_vecs, neg_vecs, other_neg, m1, m2):
229 |     trip_loss= triplet_loss(q_vec, pos_vecs, neg_vecs, m1)
230 |     
231 |     best_pos=best_pos_distance(q_vec, pos_vecs)
232 |     num_neg = neg_vecs.get_shape()[1]
233 |     batch = q_vec.get_shape()[0]
234 | 
235 |     other_neg_copies = tf.tile(other_neg, [1, int(num_neg),1])
236 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
237 |     m2=tf.fill([int(batch), int(num_neg)],m2)
238 | 
239 |     second_loss=tf.reduce_mean(tf.reduce_sum(tf.maximum(tf.add(m2,tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,other_neg_copies),2))), tf.zeros([int(batch), int(num_neg)])),1))
240 | 
241 |     total_loss= trip_loss+second_loss
242 | 
243 |     return total_loss 
244 | 
245 | def lazy_quadruplet_loss(q_vec, pos_vecs, neg_vecs, other_neg, m1, m2):
246 |     trip_loss= lazy_triplet_loss(q_vec, pos_vecs, neg_vecs, m1)
247 |     
248 |     best_pos=best_pos_distance(q_vec, pos_vecs)
249 |     num_neg = neg_vecs.get_shape()[1]
250 |     batch = q_vec.get_shape()[0]
251 | 
252 |     other_neg_copies = tf.tile(other_neg, [1, int(num_neg),1])
253 |     best_pos=tf.tile(tf.reshape(best_pos,(-1,1)),[1, int(num_neg)])
254 |     m2=tf.fill([int(batch), int(num_neg)],m2)
255 | 
256 |     second_loss=tf.reduce_mean(tf.reduce_max(tf.maximum(tf.add(m2,tf.subtract(best_pos,tf.reduce_sum(tf.squared_difference(neg_vecs,other_neg_copies),2))), tf.zeros([int(batch), int(num_neg)])),1))
257 | 
258 |     total_loss= trip_loss+second_loss
259 | 
260 |     return total_loss  
261 | 
262 | 
263 | 
264 | 
265 | 
266 | 


--------------------------------------------------------------------------------
/network_architecture.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Suoivy/LPD-net/77ff996b799a919b29eb13a158d722f5730a5362/network_architecture.png


--------------------------------------------------------------------------------
/prepare_data.py:
--------------------------------------------------------------------------------
  1 | '''
  2 | 	Pre-processing: prepare_data in LPD-Net
  3 | 	generate KNN neighborhoods and calculate feature as the feature matrix of point
  4 | 	Reference: LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis, ICCV 2019
  5 | 
  6 | 	author: Chuanzhe Suo(suo_ivy@foxmail.com)
  7 | 	created: 10/26/18
  8 | '''
  9 | 
 10 | import os
 11 | import sys
 12 | import multiprocessing as multiproc
 13 | from copy import deepcopy
 14 | import glog as logger
 15 | import argparse
 16 | import pickle
 17 | import numpy as np
 18 | import pandas as pd
 19 | from sklearn.neighbors import NearestNeighbors, KDTree
 20 | import math
 21 | import errno
 22 | 
 23 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 24 | base_path = "../benchmark_datasets/"
 25 | 
 26 | runs_folder= "oxford/"
 27 | filename = "pointcloud_locations_20m_10overlap.csv"
 28 | pointcloud_fols="pointcloud_20m_10overlap/"
 29 | featurecloud_fols="featurecloud_20m_10overlap/"
 30 | 
 31 | 
 32 | def calculate_features_old(pointcloud, nbrs_index, eigens):
 33 | 	### calculate handcraft feature with eigens and statistics data
 34 | 
 35 | 	# features using eigens
 36 | 	eig3d = eigens['eigens'][:3]
 37 | 	eig2d = eigens['eigens'][3:5]
 38 | 	vetors = eigens['vectors']
 39 | 	# 3d
 40 | 	C_ = eig3d[2]/(eig3d.sum())
 41 | 	O_ = np.power((eig3d.prod()/np.power(eig3d.sum(),3)),1.0/3)
 42 | 	L_ = (eig3d[0]-eig3d[1])/eig3d[0]
 43 | 	E_ = -((eig3d/eig3d.sum())*np.log(eig3d/eig3d.sum())).sum()
 44 | 	D_ = 3*nbrs_index.shape[0]/(4*math.pi*eig3d.prod())
 45 | 	# 2d
 46 | 	S_2 = eig2d.sum()
 47 | 	L_2 = eig2d[1]/eig2d[0]
 48 | 	# features using statistics data
 49 | 	neighborhood = pointcloud[nbrs_index]
 50 | 	nbr_dz = neighborhood[:,2]-neighborhood[:,2].min()
 51 | 	dZ_ = nbr_dz.max()
 52 | 	vZ_ = np.var(nbr_dz)
 53 | 	V_ = vetors[2][2]
 54 | 
 55 | 	features = np.asarray([C_,O_,L_,E_,D_,S_2,L_2,dZ_,vZ_,V_])
 56 | 	return features
 57 | 
 58 | def calculate_features(pointcloud, nbrs_index, eigens_, vectors_):
 59 |     ### calculate handcraft feature with eigens and statistics data
 60 | 
 61 |     # features using eigens
 62 |     eig3d = eigens_[:3]
 63 |     eig2d = eigens_[3:5]
 64 | 
 65 |     # 3d
 66 |     C_ = eig3d[2] / (eig3d.sum())
 67 |     O_ = np.power((eig3d.prod() / np.power(eig3d.sum(), 3)), 1.0 / 3)
 68 |     L_ = (eig3d[0] - eig3d[1]) / eig3d[0]
 69 |     E_ = -((eig3d / eig3d.sum()) * np.log(eig3d / eig3d.sum())).sum()
 70 |     #P_ = (eig3d[1] - eig3d[2]) / eig3d[0]
 71 |     #S_ = eig3d[2] / eig3d[0]
 72 |     #A_ = (eig3d[0] - eig3d[2]) / eig3d[0]
 73 |     #X_ = eig3d.sum()
 74 |     D_ = 3 * nbrs_index.shape[0] / (4 * math.pi * eig3d.prod())
 75 |     # 2d
 76 |     S_2 = eig2d.sum()
 77 |     L_2 = eig2d[1] / eig2d[0]
 78 |     # features using statistics data
 79 |     neighborhood = pointcloud[nbrs_index]
 80 |     nbr_dz = neighborhood[:, 2] - neighborhood[:, 2].min()
 81 |     dZ_ = nbr_dz.max()
 82 |     vZ_ = np.var(nbr_dz)
 83 |     V_ = vectors_[2][2]
 84 | 
 85 |     features = np.asarray([C_, O_, L_, E_,  D_, S_2, L_2, dZ_, vZ_, V_])#([C_,O_,L_,E_,D_,S_2,L_2,dZ_,vZ_,V_])
 86 |     return features
 87 | 
 88 | def calculate_entropy(eigen):
 89 | 	L_ = (eigen[0] - eigen[1]) / eigen[0]
 90 | 	P_ = (eigen[1] - eigen[2]) / eigen[0]
 91 | 	S_ = eigen[2] / eigen[0]
 92 | 	Entropy = -L_*np.log(L_)-P_*np.log(P_)-S_*np.log(S_)
 93 | 	return Entropy
 94 | 
 95 | def calculate_entropy_array(eigen):
 96 |     L_ = (eigen[:,0] - eigen[:,1]) / eigen[:,0]
 97 |     P_ = (eigen[:,1] - eigen[:,2]) / eigen[:,0]
 98 |     S_ = eigen[:,2] / eigen[:,0]
 99 |     Entropy = -L_*np.log(L_)-P_*np.log(P_)-S_*np.log(S_)
100 |     return Entropy
101 | 
102 | def covariation_eigenvalue(neighborhood_index, args):
103 |     ### calculate covariation and eigenvalue of 3D and 2D
104 |     # prepare neighborhood
105 |     neighborhoods = args.pointcloud[neighborhood_index]
106 | 
107 |     # 3D cov and eigen by matrix
108 |     Ex = np.average(neighborhoods, axis=1)
109 |     Ex = np.reshape(np.tile(Ex,[neighborhoods.shape[1]]), neighborhoods.shape)
110 |     P = neighborhoods-Ex
111 |     cov_ = np.matmul(P.transpose((0,2,1)),P)/(neighborhoods.shape[1]-1)
112 |     eigen_, vec_ = np.linalg.eig(cov_)
113 |     indices = np.argsort(eigen_)
114 |     indices = indices[:,::-1]
115 |     pcs_num_ = eigen_.shape[0]
116 |     indx = np.reshape(np.arange(pcs_num_), [-1, 1])
117 |     eig_ind = indices + indx*3
118 |     vec_ind = np.reshape(eig_ind*3, [-1,1]) + np.full((pcs_num_*3,3), [0,1,2])
119 |     vec_ind = np.reshape(vec_ind, [-1,3,3])
120 |     eigen3d_ = np.take(eigen_, eig_ind)
121 |     vectors_ = np.take(vec_, vec_ind)
122 |     entropy_ = calculate_entropy_array(eigen3d_)
123 | 
124 |     # 2D cov and eigen
125 |     cov2d_ = cov_[:,:2,:2]
126 |     eigen2d, vec_2d = np.linalg.eig(cov2d_)
127 |     indices = np.argsort(eigen2d)
128 |     indices = indices[:, ::-1]
129 |     pcs_num_ = eigen2d.shape[0]
130 |     indx = np.reshape(np.arange(pcs_num_), [-1, 1])
131 |     eig_ind = indices + indx * 2
132 |     eigen2d_ = np.take(eigen2d, eig_ind)
133 | 
134 |     eigens_ = np.append(eigen3d_,eigen2d_,axis=1)
135 | 
136 |     return cov_, entropy_, eigens_, vectors_
137 | 
138 | def build_neighbors_NN(k, args):
139 | 	### using KNN NearestNeighbors cluster according k
140 | 	nbrs = NearestNeighbors(n_neighbors=k).fit(args.pointcloud)
141 | 	distances, indices = nbrs.kneighbors(args.pointcloud)
142 | 	covs, entropy, eigens_, vectors_ = covariation_eigenvalue(indices, args)
143 | 	neighbors = {}
144 | 	neighbors['k'] = k
145 | 	neighbors['indices'] = indices
146 | 	neighbors['covs'] = covs
147 | 	neighbors['entropy'] = entropy
148 | 	neighbors['eigens_'] = eigens_
149 | 	neighbors['vectors_'] = vectors_
150 | 	logger.info("KNN:{}".format(k))
151 | 	return neighbors
152 | 
153 | def build_neighbors_KDT(k, args):
154 | 	### using KNN KDTree cluster according k
155 | 	nbrs = KDTree(args.pointcloud)
156 | 	distances, indices = nbrs.query(args.pointcloud, k=k)
157 | 	covs, entropy, eigens_, vectors_ = covariation_eigenvalue(indices, args)
158 | 	neighbors = {}
159 | 	neighbors['k'] = k
160 | 	neighbors['indices'] = indices
161 | 	neighbors['covs'] = covs
162 | 	neighbors['entropy'] = entropy
163 | 	neighbors['eigens_'] = eigens_
164 | 	neighbors['vectors_'] = vectors_
165 | 	logger.info("KNN:{}".format(k))
166 | 	return neighbors
167 | 
168 | 
169 | def prepare_file(pointcloud_file, args):
170 | 	### Parallel process pointcloud files
171 | 	# load pointcloud file
172 | 	pointcloud = np.fromfile(pointcloud_file, dtype=np.float64)
173 | 	pointcloud = np.reshape(pointcloud, (pointcloud.shape[0]//3, 3))
174 | 	args.pointcloud = pointcloud
175 | 
176 | 	# prepare KNN cluster number k
177 | 	cluster_number = []
178 | 	for ind in range(((args.k_end-args.k_start)//args.k_step)+1):
179 | 		cluster_number.append(args.k_start + ind*args.k_step)
180 | 
181 | 	k_nbrs = []
182 | 	for k in cluster_number:
183 | 		k_nbr = build_neighbors_NN(k, args)
184 | 		k_nbrs.append(k_nbr)
185 | 
186 | 	logger.info("Processing pointcloud file:{}".format(pointcloud_file))
187 | 	# multiprocessing pool to parallel cluster pointcloud
188 | 	#pool = multiproc.Pool(len(cluster_number))
189 | 	#build_neighbors_func = partial(build_neighbors, args=deepcopy(args))
190 | 	#k_nbrs = pool.map(build_neighbors_func, cluster_number)
191 | 	#pool.close()
192 | 	#pool.join()
193 | 
194 | 	# get argmin k according E, different points may have different k
195 | 	k_entropys = []
196 | 	for k_nbr in k_nbrs:
197 | 		k_entropys.append(k_nbr['entropy'])
198 | 	argmink_ind = np.argmin(np.asarray(k_entropys), axis=0)
199 | 
200 | 
201 | 	points_feature = []
202 | 	for index in range(pointcloud.shape[0]):
203 | 		### per point
204 | 		neighborhood = k_nbrs[argmink_ind[index]]['indices'][index]
205 | 		eigens_ = k_nbrs[argmink_ind[index]]['eigens_'][index]
206 | 		vectors_ = k_nbrs[argmink_ind[index]]['vectors_'][index]
207 | 
208 | 		# calculate point feature
209 | 		feature = calculate_features(pointcloud, neighborhood, eigens_, vectors_)
210 | 		points_feature.append(feature)
211 | 	points_feature = np.asarray(points_feature)
212 | 
213 | 	# save to point feature folders and bin files
214 | 	feature_cloud = np.append(pointcloud, points_feature, axis=1)
215 | 	pointfile_path, pointfile_name = os.path.split(pointcloud_file)
216 | 	filepath = os.path.join(os.path.split(pointfile_path)[0], args.featurecloud_fols, pointfile_name)
217 | 	feature_cloud.tofile(filepath)
218 | 
219 | 	# build KDTree and store fot the knn query
220 | 	#kdt = KDTree(pointcloud, leaf_size=50)
221 | 	#treepath = os.path.splitext(filepath)[0] + '.pickle'
222 | 	#with open(treepath, 'wb') as handel:
223 | 	#	pickle.dump(kdt, handel)
224 | 
225 | 	logger.info("Feature cloud file saved:{}".format(filepath))
226 | 
227 | 
228 | def prepare_dataset(args):
229 | 	### Parallel process dataset folders
230 | 	# Initialize pandas DataFrame
231 | 	df_train = pd.DataFrame(columns=['file', 'northing', 'easting'])
232 | 	# load folder csv file
233 | 	df_locations = pd.read_csv(os.path.join(args.dataset_path, args.runs_folder, args.pointcloud_folder, args.filename), sep=',')
234 | 	df_locations['timestamp'] = args.base_path + args.runs_folder + args.pointcloud_folder +'/'+ args.pointcloud_fols + df_locations['timestamp'].astype(str) + '.bin'
235 | 	df_locations = df_locations.rename(columns={'timestamp': 'file'})
236 | 	# creat feature_cloud folder
237 | 	featurecloud_path = os.path.join(args.dataset_path, args.runs_folder, args.pointcloud_folder, args.featurecloud_fols)
238 | 	if not os.path.exists(featurecloud_path):
239 | 		try:
240 | 			os.makedirs(featurecloud_path)
241 | 		except OSError as exc:
242 | 			if exc.errno != errno.EEXIST:
243 | 				raise
244 | 			pass
245 | 
246 | 	pointcloud_files = df_locations['file'].tolist()
247 | 	#print(pointcloud_files)
248 | 
249 | 	# multiprocessing pool to parallel process pointcloud_files
250 | 	pool = multiproc.Pool(args.bin_core_num)
251 | 	for file in pointcloud_files:
252 | 		file = os.path.join(args.BASE_DIR, file)
253 | 		pointfile_path, pointfile_name = os.path.split(file)
254 | 		filepath = os.path.join(os.path.split(pointfile_path)[0], args.featurecloud_fols, pointfile_name)
255 | 		if not os.path.exists(filepath):
256 | 			pool.apply_async(prepare_file,(file, deepcopy(args)))
257 | 		else:
258 | 			logger.info("{} exists, skipped".format(file))
259 | 	pool.close()
260 | 	logger.info("Cloud folder processing:{}".format(args.pointcloud_folder))
261 | 	pool.join()
262 | 	logger.info("end folder processing")
263 | 
264 | def run_all_processes(all_p):
265 | 	try:
266 | 		for p in all_p:
267 | 			p.start()
268 | 		for p in all_p:
269 | 			p.join()
270 | 	except KeyboardInterrupt:
271 | 		for p in all_p:
272 | 			if p.is_alive():
273 | 				p.terminate()
274 | 			p.join()
275 | 		exit(-1)
276 | 
277 | 
278 | def main(args):
279 | 	# prepare dataset folders
280 | 	args.BASE_DIR = BASE_DIR
281 | 	args.base_path = base_path
282 | 	args.dataset_path = os.path.join(BASE_DIR, base_path)
283 | 	args.runs_folder = runs_folder
284 | 	args.pointcloud_fols = pointcloud_fols
285 | 	args.featurecloud_fols = featurecloud_fols
286 | 	args.filename = filename
287 | 	all_folders = sorted(os.listdir(os.path.join(BASE_DIR, base_path, runs_folder)))
288 | 	folders = []
289 | 
290 | 	# All runs are used for training (both full and partial)
291 | 	index_list = range(len(all_folders)-1)
292 | 	print("Number of runs: " + str(len(index_list)))
293 | 	print(all_folders)
294 | 	for index in index_list:
295 | 		folders.append(all_folders[index])
296 | 	print(folders)
297 | 
298 | 	# multiprocessing dataset folder
299 | 	all_p = []
300 | 	for folder in folders:
301 | 		args.pointcloud_folder = folder
302 | 		all_p.append(multiproc.Process(target=prepare_dataset, args=(deepcopy(args),)))
303 | 	run_all_processes(all_p)
304 | 
305 | 	logger.info("Dataset preparation Finised")
306 | 
307 | 
308 | if __name__ == '__main__':
309 | 	parse = argparse.ArgumentParser(sys.argv[0])
310 | 
311 | 	parse.add_argument('--k_start', type=int, default=20,
312 | 	                   help="KNN cluster k range start point")
313 | 	parse.add_argument('--k_end', type=int, default=100,
314 | 	                   help="KNN cluster k range end point")
315 | 	parse.add_argument('--k_step', type=int, default=10,
316 | 	                   help="KNN cluster k range step")
317 | 	parse.add_argument('--bin_core_num', type=int, default=10, help="Parallel process file Pool core num")
318 | 
319 | 	args = parse.parse_args(sys.argv[1:])
320 | 	main(args)
321 | 


--------------------------------------------------------------------------------
/train_lpdnet.py:
--------------------------------------------------------------------------------
  1 | # Author: Chuanzhe Suo (suo_ivy@foxmail.com) 10/26/2018
  2 | # Thanks to Mikaela Angelina Uy, modified from PointNetVLAD
  3 | # Reference: LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis, ICCV 2019
  4 | 
  5 | import argparse
  6 | import importlib
  7 | import os
  8 | import sys
  9 | import time
 10 | import tensorflow as tf
 11 | import numpy as np
 12 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 13 | ROOT_DIR = BASE_DIR
 14 | sys.path.append(BASE_DIR)
 15 | sys.path.append(os.path.join(BASE_DIR, 'models'))
 16 | sys.path.append(os.path.join(BASE_DIR, 'utils'))
 17 | from loading_pointclouds import *
 18 | from sklearn.neighbors import KDTree
 19 | 
 20 | #params
 21 | parser = argparse.ArgumentParser()
 22 | parser.add_argument('--gpu', type=int, default=0, help='GPU to use [default: GPU 0]')
 23 | parser.add_argument('--model', default='lpd_FNSF', help='Model name [default: lpd_FNSF]')
 24 | parser.add_argument('--log_dir', default='log/', help='Log dir [default: log]')
 25 | parser.add_argument('--positives_per_query', type=int, default=2, help='Number of potential positives in each training tuple [default: 2]')
 26 | parser.add_argument('--negatives_per_query', type=int, default=14, help='Number of definite negatives in each training tuple [default: 18]')
 27 | parser.add_argument('--max_epoch', type=int, default=20, help='Epoch to run [default: 20]')
 28 | parser.add_argument('--batch_num_queries', type=int, default=1, help='Batch Size during training [default: 2]')
 29 | parser.add_argument('--learning_rate', type=float, default=0.00005, help='Initial learning rate [default: 0.00005]')
 30 | parser.add_argument('--momentum', type=float, default=0.9, help='Initial learning rate [default: 0.9]')
 31 | parser.add_argument('--optimizer', default='adam', help='adam or momentum [default: adam]')
 32 | parser.add_argument('--decay_step', type=int, default=200000, help='Decay step for lr decay [default: 200000]')
 33 | parser.add_argument('--decay_rate', type=float, default=0.7, help='Decay rate for lr decay [default: 0.7]')
 34 | parser.add_argument('--margin_1', type=float, default=0.5, help='Margin for hinge loss [default: 0.5]')
 35 | parser.add_argument('--margin_2', type=float, default=0.2, help='Margin for hinge loss [default: 0.2]')
 36 | parser.add_argument('--restore', type=int, default=0, help='Train with the stored model(0:No 1:Yes) [default: 0]')
 37 | FLAGS = parser.parse_args()
 38 | 
 39 | RESTORE = FLAGS.restore
 40 | if not RESTORE:
 41 |     MODEL = importlib.import_module(FLAGS.model)  # import network module
 42 |     MODEL_FILE = os.path.join(ROOT_DIR, 'models', FLAGS.model + '.py')
 43 |     LOG_DIR = os.path.join(FLAGS.log_dir, FLAGS.model+time.strftime("-%y-%m-%d-%H-%M", time.localtime()))
 44 |     if not os.path.exists(LOG_DIR): os.makedirs(LOG_DIR)
 45 |     os.system('cp %s %s' % (MODEL_FILE, LOG_DIR)) # bkp of model def
 46 |     LOG_FOUT = open(os.path.join(LOG_DIR, 'log_train.txt'), 'w')
 47 | else:
 48 |     LOG_DIR = FLAGS.log_dir
 49 |     model = LOG_DIR.split('/')[1]
 50 |     model = model.split('-')[0]
 51 |     print('model:{}'.format(model))
 52 |     MODEL = importlib.import_module(model)
 53 |     LOG_FOUT = open(os.path.join(LOG_DIR, 'log_train.txt'), 'a')
 54 | 
 55 | LOG_FOUT.write(str(FLAGS)+'\n')
 56 | 
 57 | 
 58 | BATCH_NUM_QUERIES = FLAGS.batch_num_queries
 59 | EVAL_BATCH_SIZE = 1
 60 | NUM_POINTS = 4096
 61 | POSITIVES_PER_QUERY= FLAGS.positives_per_query
 62 | NEGATIVES_PER_QUERY= FLAGS.negatives_per_query
 63 | MAX_EPOCH = FLAGS.max_epoch
 64 | BASE_LEARNING_RATE = FLAGS.learning_rate
 65 | GPU_INDEX = FLAGS.gpu
 66 | MOMENTUM = FLAGS.momentum
 67 | OPTIMIZER = FLAGS.optimizer
 68 | DECAY_STEP = FLAGS.decay_step
 69 | DECAY_RATE = FLAGS.decay_rate
 70 | MARGIN1 = FLAGS.margin_1
 71 | MARGIN2 = FLAGS.margin_2
 72 | 
 73 | # Generated training queies
 74 | TRAIN_FILE = 'generating_queries/training_queries_baseline.pickle'
 75 | TEST_FILE = 'generating_queries/test_queries_baseline.pickle'
 76 | 
 77 | 
 78 | # Load dictionary of training queries
 79 | TRAINING_QUERIES= get_queries_dict(TRAIN_FILE)
 80 | TEST_QUERIES= get_queries_dict(TEST_FILE)
 81 | 
 82 | BN_INIT_DECAY = 0.5
 83 | BN_DECAY_DECAY_RATE = 0.5
 84 | BN_DECAY_DECAY_STEP = float(DECAY_STEP)
 85 | BN_DECAY_CLIP = 0.99
 86 | 
 87 | global HARD_NEGATIVES
 88 | HARD_NEGATIVES={}
 89 | 
 90 | global TRAINING_LATENT_VECTORS
 91 | TRAINING_LATENT_VECTORS=[]
 92 | 
 93 | def get_bn_decay(batch):
 94 |     bn_momentum = tf.train.exponential_decay(
 95 |                       BN_INIT_DECAY,
 96 |                       batch*BATCH_NUM_QUERIES,
 97 |                       BN_DECAY_DECAY_STEP,
 98 |                       BN_DECAY_DECAY_RATE,
 99 |                       staircase=True)
100 |     bn_decay = tf.minimum(BN_DECAY_CLIP, 1 - bn_momentum)
101 |     return bn_decay
102 | 
103 | def log_string(out_str):
104 |     LOG_FOUT.write(out_str+'\n')
105 |     LOG_FOUT.flush()
106 |     print(out_str)
107 | 
108 | #learning rate halfed every 5 epoch
109 | def get_learning_rate(epoch):
110 |     learning_rate = BASE_LEARNING_RATE*((0.9)**(epoch//5))
111 |     learning_rate = tf.maximum(learning_rate, 0.00001) # CLIP THE LEARNING RATE!
112 |     return learning_rate        
113 | 
114 | 
115 | def train():
116 |     global HARD_NEGATIVES
117 |     with tf.Graph().as_default():
118 |         with tf.device('/gpu:'+str(GPU_INDEX)):
119 |             print("In Graph")
120 |             query= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, 1, NUM_POINTS)
121 |             positives= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, POSITIVES_PER_QUERY, NUM_POINTS)
122 |             negatives= MODEL.placeholder_inputs(BATCH_NUM_QUERIES, NEGATIVES_PER_QUERY, NUM_POINTS)
123 |             other_negatives=  MODEL.placeholder_inputs(BATCH_NUM_QUERIES,1, NUM_POINTS)
124 | 
125 |             is_training_pl = tf.placeholder(tf.bool, shape=())
126 |             print(is_training_pl)
127 |             
128 |             batch = tf.Variable(0)
129 |             epoch_num = tf.placeholder(tf.float32, shape=())
130 |             bn_decay = get_bn_decay(batch)
131 |             tf.summary.scalar('bn_decay', bn_decay)
132 | 
133 |             with tf.variable_scope("query_triplets") as scope:
134 |                 vecs= tf.concat([query, positives, negatives, other_negatives],1)
135 |                 print(vecs)                
136 |                 out_vecs= MODEL.forward(vecs, is_training_pl, bn_decay=bn_decay)
137 |                 q_vec, pos_vecs, neg_vecs, other_neg_vec= tf.split(out_vecs, [1,POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY,1],1)
138 |                 print(q_vec)
139 |                 print(pos_vecs)
140 |                 print(neg_vecs)
141 |                 print(other_neg_vec)
142 | 
143 |             #loss = MODEL.lazy_triplet_loss(q_vec, pos_vecs, neg_vecs, MARGIN1)
144 |             #loss = MODEL.softmargin_loss(q_vec, pos_vecs, neg_vecs)
145 |             #loss = MODEL.quadruplet_loss(q_vec, pos_vecs, neg_vecs, other_neg_vec, MARGIN1, MARGIN2)
146 |             loss = MODEL.lazy_quadruplet_loss(q_vec, pos_vecs, neg_vecs, other_neg_vec, MARGIN1, MARGIN2)
147 |             tf.summary.scalar('loss', loss)
148 | 
149 |             # Get training operator
150 |             learning_rate = get_learning_rate(epoch_num)
151 |             tf.summary.scalar('learning_rate', learning_rate)
152 |             if OPTIMIZER == 'momentum':
153 |                 optimizer = tf.train.MomentumOptimizer(learning_rate, momentum=MOMENTUM)
154 |             elif OPTIMIZER == 'adam':
155 |                 optimizer = tf.train.AdamOptimizer(learning_rate)
156 | 
157 |             update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
158 |             with tf.control_dependencies(update_ops):
159 |                 train_op = optimizer.minimize(loss, global_step=batch)
160 |             
161 |             # Add ops to save and restore all the variables.
162 |             saver = tf.train.Saver()
163 |             
164 |         # Create a session
165 |         gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.95)
166 |         config = tf.ConfigProto(gpu_options=gpu_options)
167 |         config.gpu_options.allow_growth = True
168 |         config.allow_soft_placement = True
169 |         config.log_device_placement = False
170 |         
171 |         sess = tf.Session(config=config)
172 | 
173 |         # Add summary writers
174 |         merged = tf.summary.merge_all()
175 |         train_writer = tf.summary.FileWriter(os.path.join(LOG_DIR, 'train'),
176 |                                   sess.graph)
177 |         test_writer = tf.summary.FileWriter(os.path.join(LOG_DIR, 'test'))
178 | 
179 |         # Initialize a new model
180 |         init = tf.global_variables_initializer()
181 |         sess.run(init)
182 |         print("Initialized")
183 | 
184 |         # Restore a model
185 |         if RESTORE:
186 |             saver.restore(sess, os.path.join(LOG_DIR, "model.ckpt"))
187 |             print("Model restored:{}".format(os.path.join(LOG_DIR, "model.ckpt")))
188 | 
189 | 
190 |         ops = {'query': query,
191 |                'positives': positives,
192 |                'negatives': negatives,
193 |                'other_negatives': other_negatives,
194 |                'is_training_pl': is_training_pl,
195 |                'loss': loss,
196 |                'train_op': train_op,
197 |                'merged': merged,
198 |                'step': batch,
199 |                'epoch_num': epoch_num,
200 |                'q_vec':q_vec,
201 |                'pos_vecs': pos_vecs,
202 |                'neg_vecs': neg_vecs,
203 |                'other_neg_vec': other_neg_vec}
204 | 
205 | 
206 |         for epoch in range(MAX_EPOCH):
207 |             print(epoch)
208 |             print()
209 |             log_string('**** EPOCH %03d ****' % (epoch))
210 |             sys.stdout.flush()
211 | 
212 |             train_one_epoch(sess, ops, train_writer, test_writer, epoch, saver)          
213 | 
214 | 
215 | 
216 | def train_one_epoch(sess, ops, train_writer, test_writer, epoch, saver):
217 |     global HARD_NEGATIVES
218 |     global TRAINING_LATENT_VECTORS
219 | 
220 |     is_training = True
221 |     sampled_neg=4000
222 |     #number of hard negatives in the training tuple
223 |     #which are taken from the sampled negatives
224 |     num_to_take=10
225 | 
226 |     # Shuffle train files
227 |     train_file_idxs = np.arange(0, len(TRAINING_QUERIES.keys()))
228 |     np.random.shuffle(train_file_idxs)
229 | 
230 |     for i in range(len(train_file_idxs)//BATCH_NUM_QUERIES):
231 |         batch_keys= train_file_idxs[i*BATCH_NUM_QUERIES:(i+1)*BATCH_NUM_QUERIES]
232 |         q_tuples=[]
233 | 
234 |         faulty_tuple=False
235 |         no_other_neg=False
236 |         for j in range(BATCH_NUM_QUERIES):
237 |             if(len(TRAINING_QUERIES[batch_keys[j]]["positives"])<POSITIVES_PER_QUERY):
238 |                 faulty_tuple=True
239 |                 break
240 | 
241 |             #no cached feature vectors               
242 |             if(len(TRAINING_LATENT_VECTORS)==0):
243 |                 q_tuples.append(get_query_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_neg=[], other_neg=True))
244 |                 # q_tuples.append(get_rotated_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_neg=[], other_neg=True))
245 |                 # q_tuples.append(get_jittered_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_neg=[], other_neg=True))
246 | 
247 |             elif(len(HARD_NEGATIVES.keys())==0):
248 |                 query=get_feature_representation(TRAINING_QUERIES[batch_keys[j]]['query'], sess, ops)
249 |                 random.shuffle(TRAINING_QUERIES[batch_keys[j]]['negatives'])
250 |                 negatives=TRAINING_QUERIES[batch_keys[j]]['negatives'][0:sampled_neg]
251 |                 hard_negs= get_random_hard_negatives(query, negatives, num_to_take)
252 |                 print(hard_negs)
253 |                 q_tuples.append(get_query_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_negs, other_neg=True))
254 |                 # q_tuples.append(get_rotated_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_negs, other_neg=True))
255 |                 # q_tuples.append(get_jittered_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_negs, other_neg=True))
256 |             else:
257 |                 query=get_feature_representation(TRAINING_QUERIES[batch_keys[j]]['query'], sess, ops)
258 |                 random.shuffle(TRAINING_QUERIES[batch_keys[j]]['negatives'])
259 |                 negatives=TRAINING_QUERIES[batch_keys[j]]['negatives'][0:sampled_neg]
260 |                 hard_negs= get_random_hard_negatives(query, negatives, num_to_take)
261 |                 hard_negs= list(set().union(HARD_NEGATIVES[batch_keys[j]], hard_negs))
262 |                 print('hard',hard_negs)
263 |                 q_tuples.append(get_query_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_negs, other_neg=True))           
264 |                 # q_tuples.append(get_rotated_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_negs, other_neg=True))           
265 |                 # q_tuples.append(get_jittered_tuple(TRAINING_QUERIES[batch_keys[j]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TRAINING_QUERIES, hard_negs, other_neg=True))
266 |             
267 |             if(q_tuples[j][3].shape[0]!=NUM_POINTS):
268 |                 no_other_neg= True
269 |                 break
270 | 
271 |         #construct query array
272 |         if(faulty_tuple):
273 |             log_string('----' + str(i) + '-----')
274 |             log_string('----' + 'FAULTY TUPLE' + '-----')
275 |             continue
276 | 
277 |         if(no_other_neg):
278 |             log_string('----' + str(i) + '-----')
279 |             log_string('----' + 'NO OTHER NEG' + '-----')
280 |             continue            
281 | 
282 |         queries=[]
283 |         positives=[]
284 |         negatives=[]
285 |         other_neg=[]
286 |         for k in range(len(q_tuples)):
287 |             queries.append(q_tuples[k][0])
288 |             positives.append(q_tuples[k][1])
289 |             negatives.append(q_tuples[k][2])
290 |             other_neg.append(q_tuples[k][3])
291 | 
292 |         queries= np.array(queries)
293 |         queries= np.expand_dims(queries,axis=1)
294 |         other_neg= np.array(other_neg)
295 |         other_neg= np.expand_dims(other_neg,axis=1)
296 |         positives= np.array(positives)
297 |         negatives= np.array(negatives)
298 |         log_string('----' + str(i) + '-----')
299 |         if(len(queries.shape)!=4):
300 |             log_string('----' + 'FAULTY QUERY' + '-----')
301 |             continue
302 |         
303 |         feed_dict={ops['query']:queries, ops['positives']:positives, ops['negatives']:negatives, ops['other_negatives']:other_neg, ops['is_training_pl']:is_training, ops['epoch_num']:epoch}
304 |         summary, step, train, loss_val = sess.run([ops['merged'], ops['step'],
305 |                 ops['train_op'], ops['loss']], feed_dict=feed_dict)
306 |         train_writer.add_summary(summary, step)
307 |         log_string('batch loss: %f' % loss_val)
308 | 
309 |         if(i%200==7):
310 |             test_file_idxs = np.arange(0,len(TEST_QUERIES.keys()))
311 |             np.random.shuffle(test_file_idxs)
312 | 
313 |             eval_loss=0
314 |             eval_batches=5
315 |             eval_batches_counted=0
316 |             for eval_batch in range(eval_batches):
317 |                 eval_keys= test_file_idxs[eval_batch*BATCH_NUM_QUERIES:(eval_batch+1)*BATCH_NUM_QUERIES]
318 |                 eval_tuples=[]
319 | 
320 |                 faulty_eval_tuple=False
321 |                 no_other_neg= False
322 |                 for e_tup in range(BATCH_NUM_QUERIES):
323 |                     if(len(TEST_QUERIES[eval_keys[e_tup]]["positives"])<POSITIVES_PER_QUERY):
324 |                         faulty_eval_tuple=True
325 |                         break
326 |                     eval_tuples.append(get_query_tuple(TEST_QUERIES[eval_keys[e_tup]],POSITIVES_PER_QUERY,NEGATIVES_PER_QUERY, TEST_QUERIES, hard_neg=[], other_neg=True)) 
327 | 
328 |                     if(eval_tuples[e_tup][3].shape[0]!=NUM_POINTS):
329 |                         no_other_neg= True
330 |                         break
331 | 
332 |                 if(faulty_eval_tuple):
333 |                     log_string('----' + 'FAULTY EVAL TUPLE' + '-----')
334 |                     continue
335 | 
336 |                 if(no_other_neg):
337 |                     log_string('----' + str(i) + '-----')
338 |                     log_string('----' + 'NO OTHER NEG EVAL' + '-----')
339 |                     continue  
340 | 
341 |                 eval_batches_counted+=1
342 |                 eval_queries=[]
343 |                 eval_positives=[]
344 |                 eval_negatives=[]
345 |                 eval_other_neg=[]
346 | 
347 |                 for tup in range(len(eval_tuples)):
348 |                     eval_queries.append(eval_tuples[tup][0])
349 |                     eval_positives.append(eval_tuples[tup][1])
350 |                     eval_negatives.append(eval_tuples[tup][2])
351 |                     eval_other_neg.append(eval_tuples[tup][3])
352 | 
353 |                 eval_queries= np.array(eval_queries)
354 |                 eval_queries= np.expand_dims(eval_queries,axis=1)                
355 |                 eval_other_neg= np.array(eval_other_neg)
356 |                 eval_other_neg= np.expand_dims(eval_other_neg,axis=1)
357 |                 eval_positives= np.array(eval_positives)
358 |                 eval_negatives= np.array(eval_negatives)
359 |                 feed_dict={ops['query']:eval_queries, ops['positives']:eval_positives, ops['negatives']:eval_negatives, ops['other_negatives']:eval_other_neg, ops['is_training_pl']:False, ops['epoch_num']:epoch}
360 |                 e_summary, e_step, e_loss= sess.run([ops['merged'], ops['step'], ops['loss']], feed_dict=feed_dict)
361 |                 eval_loss+=e_loss
362 |                 if(eval_batch==4):
363 |                     test_writer.add_summary(e_summary, e_step)
364 |             average_eval_loss= float(eval_loss)/eval_batches_counted
365 |             log_string('\t\t\tEVAL')
366 |             log_string('\t\t\teval_loss: %f' %average_eval_loss)
367 | 
368 | 
369 |         if(epoch>5 and i%700 ==29):
370 |             #update cached feature vectors
371 |             TRAINING_LATENT_VECTORS=get_latent_vectors(sess, ops, TRAINING_QUERIES)
372 |             print("Updated cached feature vectors")
373 | 
374 |         if(i%1000==101):
375 |             save_path = saver.save(sess, os.path.join(LOG_DIR, "model.ckpt"))
376 |             log_string("Model saved in file: %s" % save_path)
377 | 
378 | 
379 | def get_feature_representation(filename, sess, ops):
380 |     is_training=False
381 |     queries=load_pc_files([filename])
382 |     queries= np.expand_dims(queries,axis=1)
383 |     if(BATCH_NUM_QUERIES-1>0):
384 |         fake_queries=np.zeros((BATCH_NUM_QUERIES-1,1,NUM_POINTS,13))
385 |         q=np.vstack((queries,fake_queries))
386 |     else:
387 |         q=queries
388 |     fake_pos=np.zeros((BATCH_NUM_QUERIES,POSITIVES_PER_QUERY,NUM_POINTS,13))
389 |     fake_neg=np.zeros((BATCH_NUM_QUERIES,NEGATIVES_PER_QUERY,NUM_POINTS,13))
390 |     fake_other_neg=np.zeros((BATCH_NUM_QUERIES,1,NUM_POINTS,13))
391 |     feed_dict={ops['query']:q, ops['positives']:fake_pos, ops['negatives']:fake_neg, ops['other_negatives']: fake_other_neg, ops['is_training_pl']:is_training}
392 |     output=sess.run(ops['q_vec'], feed_dict=feed_dict)
393 |     output=output[0]
394 |     output=np.squeeze(output)
395 |     return output
396 | 
397 | def get_random_hard_negatives(query_vec, random_negs, num_to_take):
398 |     global TRAINING_LATENT_VECTORS
399 | 
400 |     latent_vecs=[]
401 |     for j in range(len(random_negs)):
402 |         latent_vecs.append(TRAINING_LATENT_VECTORS[random_negs[j]])
403 |     
404 |     latent_vecs=np.array(latent_vecs)
405 |     nbrs = KDTree(latent_vecs)
406 |     distances, indices = nbrs.query(np.array([query_vec]),k=num_to_take)
407 |     hard_negs=np.squeeze(np.array(random_negs)[indices[0]])
408 |     hard_negs= hard_negs.tolist()
409 |     return hard_negs
410 | 
411 | def get_latent_vectors(sess, ops, dict_to_process):
412 |     is_training=False
413 |     train_file_idxs = np.arange(0, len(dict_to_process.keys()))
414 | 
415 |     batch_num= BATCH_NUM_QUERIES*(1+POSITIVES_PER_QUERY+NEGATIVES_PER_QUERY+1)
416 |     q_output = []
417 |     for q_index in range(len(train_file_idxs)//batch_num):
418 |         file_indices=train_file_idxs[q_index*batch_num:(q_index+1)*(batch_num)]
419 |         file_names=[]
420 |         for index in file_indices:
421 |             file_names.append(dict_to_process[index]["query"])
422 |         queries=load_pc_files(file_names)
423 | 
424 |         q1=queries[0:BATCH_NUM_QUERIES]
425 |         q1=np.expand_dims(q1,axis=1)
426 | 
427 |         q2=queries[BATCH_NUM_QUERIES:BATCH_NUM_QUERIES*(POSITIVES_PER_QUERY+1)]
428 |         q2=np.reshape(q2,(BATCH_NUM_QUERIES,POSITIVES_PER_QUERY,NUM_POINTS,13))
429 | 
430 |         q3=queries[BATCH_NUM_QUERIES*(POSITIVES_PER_QUERY+1):BATCH_NUM_QUERIES*(NEGATIVES_PER_QUERY+POSITIVES_PER_QUERY+1)]
431 |         q3=np.reshape(q3,(BATCH_NUM_QUERIES,NEGATIVES_PER_QUERY,NUM_POINTS,13))
432 | 
433 |         q4=queries[BATCH_NUM_QUERIES*(NEGATIVES_PER_QUERY+POSITIVES_PER_QUERY+1):BATCH_NUM_QUERIES*(NEGATIVES_PER_QUERY+POSITIVES_PER_QUERY+2)]
434 |         q4=np.expand_dims(q4,axis=1)
435 | 
436 |         feed_dict={ops['query']:q1, ops['positives']:q2, ops['negatives']:q3,ops['other_negatives']:q4, ops['is_training_pl']:is_training}
437 |         o1, o2, o3, o4=sess.run([ops['q_vec'], ops['pos_vecs'], ops['neg_vecs'], ops['other_neg_vec']], feed_dict=feed_dict)
438 |         
439 |         o1=np.reshape(o1,(-1,o1.shape[-1]))
440 |         o2=np.reshape(o2,(-1,o2.shape[-1]))
441 |         o3=np.reshape(o3,(-1,o3.shape[-1]))
442 |         o4=np.reshape(o4,(-1,o4.shape[-1]))        
443 | 
444 |         out=np.vstack((o1,o2,o3,o4))
445 |         q_output.append(out)
446 | 
447 |     q_output=np.array(q_output)
448 |     if(len(q_output)!=0):  
449 |         q_output=q_output.reshape(-1,q_output.shape[-1])
450 | 
451 |     #handle edge case
452 |     for q_index in range((len(train_file_idxs)//batch_num*batch_num),len(dict_to_process.keys())):
453 |         index=train_file_idxs[q_index]
454 |         queries=load_pc_files([dict_to_process[index]["query"]])
455 |         queries= np.expand_dims(queries,axis=1)
456 | 
457 |         if(BATCH_NUM_QUERIES-1>0):
458 |             fake_queries=np.zeros((BATCH_NUM_QUERIES-1,1,NUM_POINTS,13))
459 |             q=np.vstack((queries,fake_queries))
460 |         else:
461 |             q=queries
462 | 
463 |         fake_pos=np.zeros((BATCH_NUM_QUERIES,POSITIVES_PER_QUERY,NUM_POINTS,13))
464 |         fake_neg=np.zeros((BATCH_NUM_QUERIES,NEGATIVES_PER_QUERY,NUM_POINTS,13))
465 |         fake_other_neg=np.zeros((BATCH_NUM_QUERIES,1,NUM_POINTS,13))
466 |         feed_dict={ops['query']:q, ops['positives']:fake_pos, ops['negatives']:fake_neg, ops['other_negatives']:fake_other_neg, ops['is_training_pl']:is_training}
467 |         output=sess.run(ops['q_vec'], feed_dict=feed_dict)
468 |         output=output[0]
469 |         output=np.squeeze(output)
470 |         if (q_output.shape[0]!=0):
471 |             q_output=np.vstack((q_output,output))
472 |         else:
473 |             q_output=output
474 | 
475 |     print(q_output.shape)
476 |     return q_output
477 | 
478 | if __name__ == "__main__":
479 |     train()
480 | 


--------------------------------------------------------------------------------
/utils/loading_pointclouds.py:
--------------------------------------------------------------------------------
  1 | import os
  2 | import pickle
  3 | import numpy as np
  4 | import random
  5 | 
  6 | UTILS_DIR = os.path.dirname(os.path.abspath(__file__))
  7 | DATASET_PATH=os.path.join(UTILS_DIR, '../../benchmark_datasets/')
  8 | print(DATASET_PATH)
  9 | 
 10 | 
 11 | def get_queries_dict(filename):
 12 | 	#key:{'query':file,'positives':[files],'negatives:[files], 'neighbors':[keys]}
 13 | 	with open(filename, 'rb') as handle:
 14 | 		queries = pickle.load(handle)
 15 | 		print("Queries Loaded.")
 16 | 		return queries
 17 | 
 18 | def get_sets_dict(filename):
 19 | 	#[key_dataset:{key_pointcloud:{'query':file,'northing':value,'easting':value}},key_dataset:{key_pointcloud:{'query':file,'northing':value,'easting':value}}, ...}
 20 | 	with open(filename, 'rb') as handle:
 21 | 		trajectories = pickle.load(handle)
 22 | 		print("Trajectories Loaded.")
 23 | 		return trajectories
 24 | 
 25 | def load_pc_file(filename):
 26 | 	#returns Nx13 matrix (3 pose 10 handcraft features)
 27 | 	pc=np.fromfile(os.path.join(DATASET_PATH,filename), dtype=np.float64)
 28 | 
 29 | 	if(pc.shape[0]!= 4096*13):
 30 | 		print("Error in pointcloud shape")
 31 | 		print(pc.shape)
 32 | 		print(filename)
 33 | 		#return np.array([])
 34 | 		return np.zeros([4096,13])
 35 | 
 36 | 	pc=np.reshape(pc,(pc.shape[0]//13,13))
 37 | 
 38 | 	# preprocessing data
 39 | 	# Normalization
 40 | 	pc[:,3:12] = ((pc-pc.min(axis=0))/(pc.max(axis=0)-pc.min(axis=0)))[:,3:12]
 41 | 	pc[np.isnan(pc)] = 0.0
 42 | 	pc[np.isinf(pc)] = 1.0
 43 | 
 44 | 	return pc
 45 | 
 46 | def load_pc_files(filenames):
 47 | 	pcs=[]
 48 | 	for filename in filenames:
 49 | 		#print(filename)
 50 | 		pc=load_pc_file(filename)
 51 | 		if(pc.shape[0]!=4096):
 52 | 			continue
 53 | 		pcs.append(pc)
 54 | 	pcs=np.array(pcs)
 55 | 	return pcs
 56 | 
 57 | def rotate_point_cloud(batch_data):
 58 | 	""" Randomly rotate the point clouds to augument the dataset
 59 |         rotation is per shape based along up direction
 60 |         Input:
 61 |           BxNx3 array, original batch of point clouds
 62 |         Return:
 63 |           BxNx3 array, rotated batch of point clouds
 64 |     """
 65 | 	rotated_data = np.zeros(batch_data.shape, dtype=np.float32)
 66 | 	for k in range(batch_data.shape[0]):
 67 | 		#rotation_angle = np.random.uniform() * 2 * np.pi
 68 |         #-90 to 90
 69 | 		rotation_angle = (np.random.uniform()*np.pi)- np.pi/2.0
 70 | 		cosval = np.cos(rotation_angle)
 71 | 		sinval = np.sin(rotation_angle)
 72 | 		rotation_matrix = np.array([[cosval, -sinval, 0],
 73 |                                     [sinval, cosval, 0],
 74 |                                     [0, 0, 1]])
 75 | 		shape_pc = batch_data[k, ...]
 76 | 		rotated_data[k, ...] = np.dot(shape_pc.reshape((-1, 3)), rotation_matrix)
 77 | 	return rotated_data
 78 | 
 79 | def jitter_point_cloud(batch_data, sigma=0.005, clip=0.05):
 80 |     """ Randomly jitter points. jittering is per point.
 81 |         Input:
 82 |           BxNx3 array, original batch of point clouds
 83 |         Return:
 84 |           BxNx3 array, jittered batch of point clouds
 85 |     """
 86 |     B, N, C = batch_data.shape
 87 |     assert(clip > 0)
 88 |     jittered_data = np.clip(sigma * np.random.randn(B, N, C), -1*clip, clip)
 89 |     jittered_data += batch_data
 90 |     return jittered_data
 91 | 
 92 | def get_query_tuple(dict_value, num_pos, num_neg, QUERY_DICT, hard_neg=[], other_neg=False):
 93 | 	#get query tuple for dictionary entry
 94 | 	#return list [query,positives,negatives]
 95 | 
 96 | 	query=load_pc_file(dict_value["query"]) #Nx13
 97 | 
 98 | 	random.shuffle(dict_value["positives"])
 99 | 	pos_files=[]
100 | 
101 | 	for i in range(num_pos):
102 | 		pos_files.append(QUERY_DICT[dict_value["positives"][i]]["query"])
103 | 	#positives= load_pc_files(dict_value["positives"][0:num_pos])
104 | 	positives=load_pc_files(pos_files)
105 | 
106 | 	neg_files=[]
107 | 	neg_indices=[]
108 | 	if(len(hard_neg)==0):
109 | 		random.shuffle(dict_value["negatives"])	
110 | 		for i in range(num_neg):
111 | 			neg_files.append(QUERY_DICT[dict_value["negatives"][i]]["query"])
112 | 			neg_indices.append(dict_value["negatives"][i])
113 | 
114 | 	else:
115 | 		random.shuffle(dict_value["negatives"])
116 | 		for i in hard_neg:
117 | 			neg_files.append(QUERY_DICT[i]["query"])
118 | 			neg_indices.append(i)
119 | 		j=0
120 | 		while(len(neg_files)<num_neg):
121 | 
122 | 			if not dict_value["negatives"][j] in hard_neg:
123 | 				neg_files.append(QUERY_DICT[dict_value["negatives"][j]]["query"])
124 | 				neg_indices.append(dict_value["negatives"][j])
125 | 			j+=1
126 | 	
127 | 	negatives=load_pc_files(neg_files)
128 | 
129 | 	if(other_neg==False):
130 | 		return [query,positives,negatives]
131 | 	#For Quadruplet Loss
132 | 	else:
133 | 		#get neighbors of negatives and query
134 | 		neighbors=[]
135 | 		for pos in dict_value["positives"]:
136 | 			neighbors.append(pos)
137 | 		for neg in neg_indices:
138 | 			for pos in QUERY_DICT[neg]["positives"]:
139 | 				neighbors.append(pos)
140 | 		possible_negs= list(set(QUERY_DICT.keys())-set(neighbors))
141 | 		random.shuffle(possible_negs)
142 | 
143 | 		if(len(possible_negs)==0):
144 | 			return [query, positives, negatives, np.array([])]		
145 | 
146 | 		neg2= load_pc_file(QUERY_DICT[possible_negs[0]]["query"])
147 | 
148 | 		return [query,positives,negatives,neg2]
149 | 
150 | 
151 | def get_rotated_tuple(dict_value, num_pos, num_neg, QUERY_DICT, hard_neg=[],other_neg=False):
152 | 	query=load_pc_file(dict_value["query"]) #Nx3
153 | 	q_rot= rotate_point_cloud(np.expand_dims(query, axis=0))
154 | 	q_rot= np.squeeze(q_rot)
155 | 
156 | 	random.shuffle(dict_value["positives"])
157 | 	pos_files=[]
158 | 	for i in range(num_pos):
159 | 		pos_files.append(QUERY_DICT[dict_value["positives"][i]]["query"])
160 | 	#positives= load_pc_files(dict_value["positives"][0:num_pos])
161 | 	positives=load_pc_files(pos_files)
162 | 	p_rot= rotate_point_cloud(positives)
163 | 
164 | 	neg_files=[]
165 | 	neg_indices=[]
166 | 	if(len(hard_neg)==0):
167 | 		random.shuffle(dict_value["negatives"])
168 | 		for i in range(num_neg):
169 | 			neg_files.append(QUERY_DICT[dict_value["negatives"][i]]["query"])
170 | 			neg_indices.append(dict_value["negatives"][i])
171 | 	else:
172 | 		random.shuffle(dict_value["negatives"])
173 | 		for i in hard_neg:
174 | 			neg_files.append(QUERY_DICT[i]["query"])
175 | 			neg_indices.append(i)
176 | 		j=0
177 | 		while(len(neg_files)<num_neg):
178 | 			if not dict_value["negatives"][j] in hard_neg:
179 | 				neg_files.append(QUERY_DICT[dict_value["negatives"][j]]["query"])
180 | 				neg_indices.append(dict_value["negatives"][j])
181 | 			j+=1	
182 | 	negatives=load_pc_files(neg_files)
183 | 	n_rot=rotate_point_cloud(negatives)
184 | 
185 | 	if(other_neg==False):
186 | 		return [q_rot,p_rot,n_rot]
187 | 
188 | 	#For Quadruplet Loss
189 | 	else:
190 | 		#get neighbors of negatives and query
191 | 		neighbors=[]
192 | 		for pos in dict_value["positives"]:
193 | 			neighbors.append(pos)
194 | 		for neg in neg_indices:
195 | 			for pos in QUERY_DICT[neg]["positives"]:
196 | 				neighbors.append(pos)
197 | 		possible_negs= list(set(QUERY_DICT.keys())-set(neighbors))
198 | 		random.shuffle(possible_negs)
199 | 
200 | 		if(len(possible_negs)==0):
201 | 			return [q_jit, p_jit, n_jit, np.array([])]
202 | 
203 | 		neg2= load_pc_file(QUERY_DICT[possible_negs[0]]["query"])
204 | 		n2_rot= rotate_point_cloud(np.expand_dims(neg2, axis=0))
205 | 		n2_rot= np.squeeze(n2_rot)
206 | 
207 | 		return [q_rot,p_rot,n_rot,n2_rot]
208 | 
209 | def get_jittered_tuple(dict_value, num_pos, num_neg, QUERY_DICT, hard_neg=[],other_neg=False):
210 | 	query=load_pc_file(dict_value["query"]) #Nx3
211 | 	#q_rot= rotate_point_cloud(np.expand_dims(query, axis=0))
212 | 	q_jit= jitter_point_cloud(np.expand_dims(query, axis=0))
213 | 	q_jit= np.squeeze(q_jit)
214 | 
215 | 	random.shuffle(dict_value["positives"])
216 | 	pos_files=[]
217 | 	for i in range(num_pos):
218 | 		pos_files.append(QUERY_DICT[dict_value["positives"][i]]["query"])
219 | 	#positives= load_pc_files(dict_value["positives"][0:num_pos])
220 | 	positives=load_pc_files(pos_files)
221 | 	p_jit= jitter_point_cloud(positives)
222 | 
223 | 	neg_files=[]
224 | 	neg_indices=[]
225 | 	if(len(hard_neg)==0):
226 | 		random.shuffle(dict_value["negatives"])
227 | 		for i in range(num_neg):
228 | 			neg_files.append(QUERY_DICT[dict_value["negatives"][i]]["query"])
229 | 			neg_indices.append(dict_value["negatives"][i])
230 | 	else:
231 | 		random.shuffle(dict_value["negatives"])
232 | 		for i in hard_neg:
233 | 			neg_files.append(QUERY_DICT[i]["query"])
234 | 			neg_indices.append(i)
235 | 		j=0
236 | 		while(len(neg_files)<num_neg):
237 | 			if not dict_value["negatives"][j] in hard_neg:
238 | 				neg_files.append(QUERY_DICT[dict_value["negatives"][j]]["query"])
239 | 				neg_indices.append(dict_value["negatives"][j])
240 | 			j+=1	
241 | 	negatives=load_pc_files(neg_files)
242 | 	n_jit=jitter_point_cloud(negatives)
243 | 
244 | 	if(other_neg==False):
245 | 		return [q_jit,p_jit,n_jit]
246 | 
247 | 	#For Quadruplet Loss
248 | 	else:
249 | 		#get neighbors of negatives and query
250 | 		neighbors=[]
251 | 		for pos in dict_value["positives"]:
252 | 			neighbors.append(pos)
253 | 		for neg in neg_indices:
254 | 			for pos in QUERY_DICT[neg]["positives"]:
255 | 				neighbors.append(pos)
256 | 		possible_negs= list(set(QUERY_DICT.keys())-set(neighbors))
257 | 		random.shuffle(possible_negs)
258 | 
259 | 		if(len(possible_negs)==0):
260 | 			return [q_jit, p_jit, n_jit, np.array([])]
261 | 
262 | 		neg2= load_pc_file(QUERY_DICT[possible_negs[0]]["query"])
263 | 		n2_jit= jitter_point_cloud(np.expand_dims(neg2, axis=0))
264 | 		n2_jit= np.squeeze(n2_jit)
265 | 
266 | 		return [q_jit,p_jit,n_jit,n2_jit]
267 | 


--------------------------------------------------------------------------------
/utils/tf_util.py:
--------------------------------------------------------------------------------
  1 | """ Wrapper functions for TensorFlow layers.
  2 | 
  3 | Author: Charles R. Qi
  4 | Date: November 2016
  5 | """
  6 | 
  7 | import numpy as np
  8 | import tensorflow as tf
  9 | 
 10 | def _variable_on_cpu(name, shape, initializer, use_fp16=False):
 11 |   """Helper to create a Variable stored on CPU memory.
 12 |   Args:
 13 |     name: name of the variable
 14 |     shape: list of ints
 15 |     initializer: initializer for Variable
 16 |   Returns:
 17 |     Variable Tensor
 18 |   """
 19 |   with tf.device('/cpu:0'):
 20 |     dtype = tf.float16 if use_fp16 else tf.float32
 21 |     var = tf.get_variable(name, shape, initializer=initializer, dtype=dtype)
 22 |   return var
 23 | 
 24 | def _variable_with_weight_decay(name, shape, stddev, wd, use_xavier=True):
 25 |   """Helper to create an initialized Variable with weight decay.
 26 | 
 27 |   Note that the Variable is initialized with a truncated normal distribution.
 28 |   A weight decay is added only if one is specified.
 29 | 
 30 |   Args:
 31 |     name: name of the variable
 32 |     shape: list of ints
 33 |     stddev: standard deviation of a truncated Gaussian
 34 |     wd: add L2Loss weight decay multiplied by this float. If None, weight
 35 |         decay is not added for this Variable.
 36 |     use_xavier: bool, whether to use xavier initializer
 37 | 
 38 |   Returns:
 39 |     Variable Tensor
 40 |   """
 41 |   if use_xavier:
 42 |     initializer = tf.contrib.layers.xavier_initializer()
 43 |   else:
 44 |     initializer = tf.truncated_normal_initializer(stddev=stddev)
 45 |   var = _variable_on_cpu(name, shape, initializer)
 46 |   if wd is not None:
 47 |     weight_decay = tf.multiply(tf.nn.l2_loss(var), wd, name='weight_loss')
 48 |     tf.add_to_collection('losses', weight_decay)
 49 |   return var
 50 | 
 51 | 
 52 | def conv1d(inputs,
 53 |            num_output_channels,
 54 |            kernel_size,
 55 |            scope,
 56 |            stride=1,
 57 |            padding='SAME',
 58 |            use_xavier=True,
 59 |            stddev=1e-3,
 60 |            weight_decay=0.0,
 61 |            activation_fn=tf.nn.relu,
 62 |            bn=False,
 63 |            bn_decay=None,
 64 |            is_training=None):
 65 |   """ 1D convolution with non-linear operation.
 66 | 
 67 |   Args:
 68 |     inputs: 3-D tensor variable BxLxC
 69 |     num_output_channels: int
 70 |     kernel_size: int
 71 |     scope: string
 72 |     stride: int
 73 |     padding: 'SAME' or 'VALID'
 74 |     use_xavier: bool, use xavier_initializer if true
 75 |     stddev: float, stddev for truncated_normal init
 76 |     weight_decay: float
 77 |     activation_fn: function
 78 |     bn: bool, whether to use batch norm
 79 |     bn_decay: float or float tensor variable in [0,1]
 80 |     is_training: bool Tensor variable
 81 | 
 82 |   Returns:
 83 |     Variable tensor
 84 |   """
 85 |   with tf.variable_scope(scope) as sc:
 86 |     num_in_channels = inputs.get_shape()[-1].value
 87 |     kernel_shape = [kernel_size,
 88 |                     num_in_channels, num_output_channels]
 89 |     kernel = _variable_with_weight_decay('weights',
 90 |                                          shape=kernel_shape,
 91 |                                          use_xavier=use_xavier,
 92 |                                          stddev=stddev,
 93 |                                          wd=weight_decay)
 94 |     outputs = tf.nn.conv1d(inputs, kernel,
 95 |                            stride=stride,
 96 |                            padding=padding)
 97 |     biases = _variable_on_cpu('biases', [num_output_channels],
 98 |                               tf.constant_initializer(0.0))
 99 |     outputs = tf.nn.bias_add(outputs, biases)
100 | 
101 |     if bn:
102 |       outputs = batch_norm_for_conv1d(outputs, is_training,
103 |                                       bn_decay=bn_decay, scope='bn')
104 | 
105 |     if activation_fn is not None:
106 |       outputs = activation_fn(outputs)
107 |     return outputs
108 | 
109 | 
110 | 
111 | def conv2d(inputs,
112 |            num_output_channels,
113 |            kernel_size,
114 |            scope,
115 |            stride=[1, 1],
116 |            padding='SAME',
117 |            use_xavier=True,
118 |            stddev=1e-3,
119 |            weight_decay=0.0,
120 |            activation_fn=tf.nn.relu,
121 |            bn=False,
122 |            bn_decay=None,
123 |            is_training=None):
124 |   """ 2D convolution with non-linear operation.
125 | 
126 |   Args:
127 |     inputs: 4-D tensor variable BxHxWxC
128 |     num_output_channels: int
129 |     kernel_size: a list of 2 ints
130 |     scope: string
131 |     stride: a list of 2 ints
132 |     padding: 'SAME' or 'VALID'
133 |     use_xavier: bool, use xavier_initializer if true
134 |     stddev: float, stddev for truncated_normal init
135 |     weight_decay: float
136 |     activation_fn: function
137 |     bn: bool, whether to use batch norm
138 |     bn_decay: float or float tensor variable in [0,1]
139 |     is_training: bool Tensor variable
140 | 
141 |   Returns:
142 |     Variable tensor
143 |   """
144 |   with tf.variable_scope(scope) as sc:
145 |       kernel_h, kernel_w = kernel_size
146 |       num_in_channels = inputs.get_shape()[-1].value
147 |       kernel_shape = [kernel_h, kernel_w,
148 |                       num_in_channels, num_output_channels]
149 |       kernel = _variable_with_weight_decay('weights',
150 |                                            shape=kernel_shape,
151 |                                            use_xavier=use_xavier,
152 |                                            stddev=stddev,
153 |                                            wd=weight_decay)
154 |       stride_h, stride_w = stride
155 |       outputs = tf.nn.conv2d(inputs, kernel,
156 |                              [1, stride_h, stride_w, 1],
157 |                              padding=padding)
158 |       biases = _variable_on_cpu('biases', [num_output_channels],
159 |                                 tf.constant_initializer(0.0))
160 |       outputs = tf.nn.bias_add(outputs, biases)
161 | 
162 |       if bn:
163 |         outputs = batch_norm_for_conv2d(outputs, is_training,
164 |                                         bn_decay=bn_decay, scope='bn')
165 | 
166 |       if activation_fn is not None:
167 |         outputs = activation_fn(outputs)
168 |       return outputs
169 | 
170 | 
171 | def conv2d_transpose(inputs,
172 |                      num_output_channels,
173 |                      kernel_size,
174 |                      scope,
175 |                      stride=[1, 1],
176 |                      padding='SAME',
177 |                      use_xavier=True,
178 |                      stddev=1e-3,
179 |                      weight_decay=0.0,
180 |                      activation_fn=tf.nn.relu,
181 |                      bn=False,
182 |                      bn_decay=None,
183 |                      is_training=None):
184 |   """ 2D convolution transpose with non-linear operation.
185 | 
186 |   Args:
187 |     inputs: 4-D tensor variable BxHxWxC
188 |     num_output_channels: int
189 |     kernel_size: a list of 2 ints
190 |     scope: string
191 |     stride: a list of 2 ints
192 |     padding: 'SAME' or 'VALID'
193 |     use_xavier: bool, use xavier_initializer if true
194 |     stddev: float, stddev for truncated_normal init
195 |     weight_decay: float
196 |     activation_fn: function
197 |     bn: bool, whether to use batch norm
198 |     bn_decay: float or float tensor variable in [0,1]
199 |     is_training: bool Tensor variable
200 | 
201 |   Returns:
202 |     Variable tensor
203 | 
204 |   Note: conv2d(conv2d_transpose(a, num_out, ksize, stride), a.shape[-1], ksize, stride) == a
205 |   """
206 |   with tf.variable_scope(scope) as sc:
207 |       kernel_h, kernel_w = kernel_size
208 |       num_in_channels = inputs.get_shape()[-1].value
209 |       kernel_shape = [kernel_h, kernel_w,
210 |                       num_output_channels, num_in_channels] # reversed to conv2d
211 |       kernel = _variable_with_weight_decay('weights',
212 |                                            shape=kernel_shape,
213 |                                            use_xavier=use_xavier,
214 |                                            stddev=stddev,
215 |                                            wd=weight_decay)
216 |       stride_h, stride_w = stride
217 |       
218 |       # from slim.convolution2d_transpose
219 |       def get_deconv_dim(dim_size, stride_size, kernel_size, padding):
220 |           dim_size *= stride_size
221 | 
222 |           if padding == 'VALID' and dim_size is not None:
223 |             dim_size += max(kernel_size - stride_size, 0)
224 |           return dim_size
225 | 
226 |       # caculate output shape
227 |       batch_size = inputs.get_shape()[0].value
228 |       height = inputs.get_shape()[1].value
229 |       width = inputs.get_shape()[2].value
230 |       out_height = get_deconv_dim(height, stride_h, kernel_h, padding)
231 |       out_width = get_deconv_dim(width, stride_w, kernel_w, padding)
232 |       output_shape = [batch_size, out_height, out_width, num_output_channels]
233 | 
234 |       outputs = tf.nn.conv2d_transpose(inputs, kernel, output_shape,
235 |                              [1, stride_h, stride_w, 1],
236 |                              padding=padding)
237 |       biases = _variable_on_cpu('biases', [num_output_channels],
238 |                                 tf.constant_initializer(0.0))
239 |       outputs = tf.nn.bias_add(outputs, biases)
240 | 
241 |       if bn:
242 |         outputs = batch_norm_for_conv2d(outputs, is_training,
243 |                                         bn_decay=bn_decay, scope='bn')
244 | 
245 |       if activation_fn is not None:
246 |         outputs = activation_fn(outputs)
247 |       return outputs
248 | 
249 |    
250 | 
251 | def conv3d(inputs,
252 |            num_output_channels,
253 |            kernel_size,
254 |            scope,
255 |            stride=[1, 1, 1],
256 |            padding='SAME',
257 |            use_xavier=True,
258 |            stddev=1e-3,
259 |            weight_decay=0.0,
260 |            activation_fn=tf.nn.relu,
261 |            bn=False,
262 |            bn_decay=None,
263 |            is_training=None):
264 |   """ 3D convolution with non-linear operation.
265 | 
266 |   Args:
267 |     inputs: 5-D tensor variable BxDxHxWxC
268 |     num_output_channels: int
269 |     kernel_size: a list of 3 ints
270 |     scope: string
271 |     stride: a list of 3 ints
272 |     padding: 'SAME' or 'VALID'
273 |     use_xavier: bool, use xavier_initializer if true
274 |     stddev: float, stddev for truncated_normal init
275 |     weight_decay: float
276 |     activation_fn: function
277 |     bn: bool, whether to use batch norm
278 |     bn_decay: float or float tensor variable in [0,1]
279 |     is_training: bool Tensor variable
280 | 
281 |   Returns:
282 |     Variable tensor
283 |   """
284 |   with tf.variable_scope(scope) as sc:
285 |     kernel_d, kernel_h, kernel_w = kernel_size
286 |     num_in_channels = inputs.get_shape()[-1].value
287 |     kernel_shape = [kernel_d, kernel_h, kernel_w,
288 |                     num_in_channels, num_output_channels]
289 |     kernel = _variable_with_weight_decay('weights',
290 |                                          shape=kernel_shape,
291 |                                          use_xavier=use_xavier,
292 |                                          stddev=stddev,
293 |                                          wd=weight_decay)
294 |     stride_d, stride_h, stride_w = stride
295 |     outputs = tf.nn.conv3d(inputs, kernel,
296 |                            [1, stride_d, stride_h, stride_w, 1],
297 |                            padding=padding)
298 |     biases = _variable_on_cpu('biases', [num_output_channels],
299 |                               tf.constant_initializer(0.0))
300 |     outputs = tf.nn.bias_add(outputs, biases)
301 |     
302 |     if bn:
303 |       outputs = batch_norm_for_conv3d(outputs, is_training,
304 |                                       bn_decay=bn_decay, scope='bn')
305 | 
306 |     if activation_fn is not None:
307 |       outputs = activation_fn(outputs)
308 |     return outputs
309 | 
310 | def fully_connected(inputs,
311 |                     num_outputs,
312 |                     scope,
313 |                     use_xavier=True,
314 |                     stddev=1e-3,
315 |                     weight_decay=0.0,
316 |                     activation_fn=tf.nn.relu,
317 |                     bn=False,
318 |                     bn_decay=None,
319 |                     is_training=None):
320 |   """ Fully connected layer with non-linear operation.
321 |   
322 |   Args:
323 |     inputs: 2-D tensor BxN
324 |     num_outputs: int
325 |   
326 |   Returns:
327 |     Variable tensor of size B x num_outputs.
328 |   """
329 |   with tf.variable_scope(scope) as sc:
330 |     num_input_units = inputs.get_shape()[-1].value
331 |     weights = _variable_with_weight_decay('weights',
332 |                                           shape=[num_input_units, num_outputs],
333 |                                           use_xavier=use_xavier,
334 |                                           stddev=stddev,
335 |                                           wd=weight_decay)
336 |     outputs = tf.matmul(inputs, weights)
337 |     biases = _variable_on_cpu('biases', [num_outputs],
338 |                              tf.constant_initializer(0.0))
339 |     outputs = tf.nn.bias_add(outputs, biases)
340 |      
341 |     if bn:
342 |       outputs = batch_norm_for_fc(outputs, is_training, bn_decay, 'bn')
343 | 
344 |     if activation_fn is not None:
345 |       outputs = activation_fn(outputs)
346 |     return outputs
347 | 
348 | 
349 | def max_pool2d(inputs,
350 |                kernel_size,
351 |                scope,
352 |                stride=[2, 2],
353 |                padding='VALID'):
354 |   """ 2D max pooling.
355 | 
356 |   Args:
357 |     inputs: 4-D tensor BxHxWxC
358 |     kernel_size: a list of 2 ints
359 |     stride: a list of 2 ints
360 |   
361 |   Returns:
362 |     Variable tensor
363 |   """
364 |   with tf.variable_scope(scope) as sc:
365 |     kernel_h, kernel_w = kernel_size
366 |     stride_h, stride_w = stride
367 |     outputs = tf.nn.max_pool(inputs,
368 |                              ksize=[1, kernel_h, kernel_w, 1],
369 |                              strides=[1, stride_h, stride_w, 1],
370 |                              padding=padding,
371 |                              name=sc.name)
372 |     return outputs
373 | 
374 | def avg_pool2d(inputs,
375 |                kernel_size,
376 |                scope,
377 |                stride=[2, 2],
378 |                padding='VALID'):
379 |   """ 2D avg pooling.
380 | 
381 |   Args:
382 |     inputs: 4-D tensor BxHxWxC
383 |     kernel_size: a list of 2 ints
384 |     stride: a list of 2 ints
385 |   
386 |   Returns:
387 |     Variable tensor
388 |   """
389 |   with tf.variable_scope(scope) as sc:
390 |     kernel_h, kernel_w = kernel_size
391 |     stride_h, stride_w = stride
392 |     outputs = tf.nn.avg_pool(inputs,
393 |                              ksize=[1, kernel_h, kernel_w, 1],
394 |                              strides=[1, stride_h, stride_w, 1],
395 |                              padding=padding,
396 |                              name=sc.name)
397 |     return outputs
398 | 
399 | 
400 | def max_pool3d(inputs,
401 |                kernel_size,
402 |                scope,
403 |                stride=[2, 2, 2],
404 |                padding='VALID'):
405 |   """ 3D max pooling.
406 | 
407 |   Args:
408 |     inputs: 5-D tensor BxDxHxWxC
409 |     kernel_size: a list of 3 ints
410 |     stride: a list of 3 ints
411 |   
412 |   Returns:
413 |     Variable tensor
414 |   """
415 |   with tf.variable_scope(scope) as sc:
416 |     kernel_d, kernel_h, kernel_w = kernel_size
417 |     stride_d, stride_h, stride_w = stride
418 |     outputs = tf.nn.max_pool3d(inputs,
419 |                                ksize=[1, kernel_d, kernel_h, kernel_w, 1],
420 |                                strides=[1, stride_d, stride_h, stride_w, 1],
421 |                                padding=padding,
422 |                                name=sc.name)
423 |     return outputs
424 | 
425 | def avg_pool3d(inputs,
426 |                kernel_size,
427 |                scope,
428 |                stride=[2, 2, 2],
429 |                padding='VALID'):
430 |   """ 3D avg pooling.
431 | 
432 |   Args:
433 |     inputs: 5-D tensor BxDxHxWxC
434 |     kernel_size: a list of 3 ints
435 |     stride: a list of 3 ints
436 |   
437 |   Returns:
438 |     Variable tensor
439 |   """
440 |   with tf.variable_scope(scope) as sc:
441 |     kernel_d, kernel_h, kernel_w = kernel_size
442 |     stride_d, stride_h, stride_w = stride
443 |     outputs = tf.nn.avg_pool3d(inputs,
444 |                                ksize=[1, kernel_d, kernel_h, kernel_w, 1],
445 |                                strides=[1, stride_d, stride_h, stride_w, 1],
446 |                                padding=padding,
447 |                                name=sc.name)
448 |     return outputs
449 | 
450 | 
451 | 
452 | 
453 | 
454 | def batch_norm_template(inputs, is_training, scope, moments_dims, bn_decay):
455 |   """ Batch normalization on convolutional maps and beyond...
456 |   Ref.: http://stackoverflow.com/questions/33949786/how-could-i-use-batch-normalization-in-tensorflow
457 |   
458 |   Args:
459 |       inputs:        Tensor, k-D input ... x C could be BC or BHWC or BDHWC
460 |       is_training:   boolean tf.Varialbe, true indicates training phase
461 |       scope:         string, variable scope
462 |       moments_dims:  a list of ints, indicating dimensions for moments calculation
463 |       bn_decay:      float or float tensor variable, controling moving average weight
464 |   Return:
465 |       normed:        batch-normalized maps
466 |   """
467 |   with tf.variable_scope(scope) as sc:
468 |     num_channels = inputs.get_shape()[-1].value
469 |     beta = tf.Variable(tf.constant(0.0, shape=[num_channels]),
470 |                        name='beta', trainable=True)
471 |     gamma = tf.Variable(tf.constant(1.0, shape=[num_channels]),
472 |                         name='gamma', trainable=True)
473 |     batch_mean, batch_var = tf.nn.moments(inputs, moments_dims, name='moments')
474 |     decay = bn_decay if bn_decay is not None else 0.9
475 |     ema = tf.train.ExponentialMovingAverage(decay=decay)
476 |     # Operator that maintains moving averages of variables.
477 |     ema_apply_op = tf.cond(is_training,
478 |                            lambda: ema.apply([batch_mean, batch_var]),
479 |                            lambda: tf.no_op())
480 |     
481 |     # Update moving average and return current batch's avg and var.
482 |     def mean_var_with_update():
483 |       with tf.control_dependencies([ema_apply_op]):
484 |         return tf.identity(batch_mean), tf.identity(batch_var)
485 |     
486 |     # ema.average returns the Variable holding the average of var.
487 |     mean, var = tf.cond(is_training,
488 |                         mean_var_with_update,
489 |                         lambda: (ema.average(batch_mean), ema.average(batch_var)))
490 |     normed = tf.nn.batch_normalization(inputs, mean, var, beta, gamma, 1e-3)
491 |   return normed
492 | 
493 | 
494 | def batch_norm_for_fc(inputs, is_training, bn_decay, scope):
495 |   """ Batch normalization on FC data.
496 |   
497 |   Args:
498 |       inputs:      Tensor, 2D BxC input
499 |       is_training: boolean tf.Varialbe, true indicates training phase
500 |       bn_decay:    float or float tensor variable, controling moving average weight
501 |       scope:       string, variable scope
502 |   Return:
503 |       normed:      batch-normalized maps
504 |   """
505 |   return batch_norm_template(inputs, is_training, scope, [0,], bn_decay)
506 | 
507 | 
508 | def batch_norm_for_conv1d(inputs, is_training, bn_decay, scope):
509 |   """ Batch normalization on 1D convolutional maps.
510 |   
511 |   Args:
512 |       inputs:      Tensor, 3D BLC input maps
513 |       is_training: boolean tf.Varialbe, true indicates training phase
514 |       bn_decay:    float or float tensor variable, controling moving average weight
515 |       scope:       string, variable scope
516 |   Return:
517 |       normed:      batch-normalized maps
518 |   """
519 |   return batch_norm_template(inputs, is_training, scope, [0,1], bn_decay)
520 | 
521 | 
522 | 
523 |   
524 | def batch_norm_for_conv2d(inputs, is_training, bn_decay, scope):
525 |   """ Batch normalization on 2D convolutional maps.
526 |   
527 |   Args:
528 |       inputs:      Tensor, 4D BHWC input maps
529 |       is_training: boolean tf.Varialbe, true indicates training phase
530 |       bn_decay:    float or float tensor variable, controling moving average weight
531 |       scope:       string, variable scope
532 |   Return:
533 |       normed:      batch-normalized maps
534 |   """
535 |   return batch_norm_template(inputs, is_training, scope, [0,1,2], bn_decay)
536 | 
537 | 
538 | 
539 | def batch_norm_for_conv3d(inputs, is_training, bn_decay, scope):
540 |   """ Batch normalization on 3D convolutional maps.
541 |   
542 |   Args:
543 |       inputs:      Tensor, 5D BDHWC input maps
544 |       is_training: boolean tf.Varialbe, true indicates training phase
545 |       bn_decay:    float or float tensor variable, controling moving average weight
546 |       scope:       string, variable scope
547 |   Return:
548 |       normed:      batch-normalized maps
549 |   """
550 |   return batch_norm_template(inputs, is_training, scope, [0,1,2,3], bn_decay)
551 | 
552 | 
553 | def dropout(inputs,
554 |             is_training,
555 |             scope,
556 |             keep_prob=0.5,
557 |             noise_shape=None):
558 |   """ Dropout layer.
559 | 
560 |   Args:
561 |     inputs: tensor
562 |     is_training: boolean tf.Variable
563 |     scope: string
564 |     keep_prob: float in [0,1]
565 |     noise_shape: list of ints
566 | 
567 |   Returns:
568 |     tensor variable
569 |   """
570 |   with tf.variable_scope(scope) as sc:
571 |     outputs = tf.cond(is_training,
572 |                       lambda: tf.nn.dropout(inputs, keep_prob, noise_shape),
573 |                       lambda: inputs)
574 |     return outputs
575 | 
576 | 
577 | def pairwise_distance(point_cloud):
578 |     """Compute pairwise distance of a point cloud.
579 | 
580 | 	Args:
581 | 	  point_cloud: tensor (batch_size, num_points, num_dims)
582 | 
583 | 	Returns:
584 | 	  pairwise distance: (batch_size, num_points, num_points)
585 | 	"""
586 |     og_batch_size = point_cloud.get_shape().as_list()[0]
587 |     point_cloud = tf.squeeze(point_cloud)
588 |     if og_batch_size == 1:
589 |         point_cloud = tf.expand_dims(point_cloud, 0)
590 | 
591 |     point_cloud_transpose = tf.transpose(point_cloud, perm=[0, 2, 1])
592 |     point_cloud_inner = tf.matmul(point_cloud, point_cloud_transpose)
593 |     point_cloud_inner = -2 * point_cloud_inner
594 |     point_cloud_square = tf.reduce_sum(tf.square(point_cloud), axis=-1, keep_dims=True)
595 |     point_cloud_square_tranpose = tf.transpose(point_cloud_square, perm=[0, 2, 1])
596 |     return point_cloud_square + point_cloud_inner + point_cloud_square_tranpose
597 | 
598 | 
599 | def knn(adj_matrix, k=20):
600 |     """Get KNN based on the pairwise distance.
601 | 	Args:
602 | 	  pairwise distance: (batch_size, num_points, num_points)
603 | 	  k: int
604 | 
605 | 	Returns:
606 | 	  nearest neighbors: (batch_size, num_points, k)
607 | 	"""
608 |     neg_adj = -adj_matrix
609 |     _, nn_idx = tf.nn.top_k(neg_adj, k=k)
610 |     return nn_idx
611 | 
612 | 
613 | def get_edge_feature(point_cloud, nn_idx, k=20):
614 |     """Construct edge feature for each point
615 | 	Args:
616 | 	  point_cloud: (batch_size, num_points, 1, num_dims)
617 | 	  nn_idx: (batch_size, num_points, k)
618 | 	  k: int
619 | 
620 | 	Returns:
621 | 	  edge features: (batch_size, num_points, k, num_dims)
622 | 	"""
623 |     og_batch_size = point_cloud.get_shape().as_list()[0]
624 |     point_cloud = tf.squeeze(point_cloud)
625 |     if og_batch_size == 1:
626 |         point_cloud = tf.expand_dims(point_cloud, 0)
627 | 
628 |     point_cloud_central = point_cloud
629 | 
630 |     point_cloud_shape = point_cloud.get_shape()
631 |     batch_size = point_cloud_shape[0].value
632 |     num_points = point_cloud_shape[1].value
633 |     num_dims = point_cloud_shape[2].value
634 | 
635 |     idx_ = tf.range(batch_size) * num_points
636 |     idx_ = tf.reshape(idx_, [batch_size, 1, 1])
637 | 
638 |     point_cloud_flat = tf.reshape(point_cloud, [-1, num_dims])
639 |     point_cloud_neighbors = tf.gather(point_cloud_flat, nn_idx + idx_)
640 |     point_cloud_central = tf.expand_dims(point_cloud_central, axis=-2)
641 | 
642 |     point_cloud_central = tf.tile(point_cloud_central, [1, 1, k, 1])
643 | 
644 |     edge_feature = tf.concat([point_cloud_central, point_cloud_neighbors - point_cloud_central], axis=-1)
645 |     return edge_feature
646 | 


--------------------------------------------------------------------------------
/utils/transform_nets.py:
--------------------------------------------------------------------------------
  1 | ##Taken from Charles Qi's PointNet code base##
  2 | import tensorflow as tf
  3 | import numpy as np
  4 | import sys
  5 | import os
  6 | BASE_DIR = os.path.dirname(os.path.abspath(__file__))
  7 | sys.path.append(BASE_DIR)
  8 | sys.path.append(os.path.join(BASE_DIR, '../utils'))
  9 | import tf_util
 10 | 
 11 | 
 12 | def input_transform_net(point_cloud, is_training, bn_decay=None, K=3):
 13 |     """ Input (XYZ) Transform Net, input is BxNx3 gray image
 14 |         Return:
 15 |             Transformation matrix of size 3xK """
 16 |     batch_size = point_cloud.get_shape()[0].value
 17 |     num_point = point_cloud.get_shape()[1].value
 18 | 
 19 |     input_image = tf.expand_dims(point_cloud, -1)
 20 |     net = tf_util.conv2d(input_image, 64, [1, 3],
 21 |                          padding='VALID', stride=[1,1],
 22 |                          bn=True, is_training=is_training,
 23 |                          scope='tconv1', bn_decay=bn_decay)
 24 |     net = tf_util.conv2d(net, 128, [1, 1],
 25 |                          padding='VALID', stride=[1,1],
 26 |                          bn=True, is_training=is_training,
 27 |                          scope='tconv2', bn_decay=bn_decay)
 28 |     net = tf_util.conv2d(net, 1024, [1, 1],
 29 |                          padding='VALID', stride=[1,1],
 30 |                          bn=True, is_training=is_training,
 31 |                          scope='tconv3', bn_decay=bn_decay)
 32 |     net = tf_util.max_pool2d(net, [num_point, 1],
 33 |                              padding='VALID', scope='tmaxpool')
 34 | 
 35 |     net = tf.reshape(net, [batch_size, -1])
 36 |     net = tf_util.fully_connected(net, 512, bn=True, is_training=is_training,
 37 |                                   scope='tfc1', bn_decay=bn_decay)
 38 |     net = tf_util.fully_connected(net, 256, bn=True, is_training=is_training,
 39 |                                   scope='tfc2', bn_decay=bn_decay)
 40 | 
 41 |     with tf.variable_scope('transform_XYZ') as sc:
 42 |         assert(K==3)
 43 |         weights = tf.get_variable('weights', [256, 3*K],
 44 |                                   initializer=tf.constant_initializer(0.0),
 45 |                                   dtype=tf.float32)
 46 |         biases = tf.get_variable('biases', [3*K],
 47 |                                  initializer=tf.constant_initializer(0.0),
 48 |                                  dtype=tf.float32)
 49 |         biases += tf.constant([1,0,0,0,1,0,0,0,1], dtype=tf.float32)
 50 |         transform = tf.matmul(net, weights)
 51 |         transform = tf.nn.bias_add(transform, biases)
 52 | 
 53 |     transform = tf.reshape(transform, [batch_size, 3, K])
 54 |     return transform
 55 | 
 56 | 
 57 | def feature_transform_net(inputs, is_training, bn_decay=None, K=64):
 58 |     """ Feature Transform Net, input is BxNx1xK
 59 |         Return:
 60 |             Transformation matrix of size KxK """
 61 |     batch_size = inputs.get_shape()[0].value
 62 |     num_point = inputs.get_shape()[1].value
 63 | 
 64 |     net = tf_util.conv2d(inputs, 64, [1, 1],
 65 |                          padding='VALID', stride=[1,1],
 66 |                          bn=True, is_training=is_training,
 67 |                          scope='tconv1', bn_decay=bn_decay)
 68 |     net = tf_util.conv2d(net, 128, [1, 1],
 69 |                          padding='VALID', stride=[1,1],
 70 |                          bn=True, is_training=is_training,
 71 |                          scope='tconv2', bn_decay=bn_decay)
 72 |     net = tf_util.conv2d(net, 1024, [1, 1],
 73 |                          padding='VALID', stride=[1,1],
 74 |                          bn=True, is_training=is_training,
 75 |                          scope='tconv3', bn_decay=bn_decay)
 76 |     net = tf_util.max_pool2d(net, [num_point, 1],
 77 |                              padding='VALID', scope='tmaxpool')
 78 | 
 79 |     net = tf.reshape(net, [batch_size, -1])
 80 |     net = tf_util.fully_connected(net, 512, bn=True, is_training=is_training,
 81 |                                   scope='tfc1', bn_decay=bn_decay)
 82 |     net = tf_util.fully_connected(net, 256, bn=True, is_training=is_training,
 83 |                                   scope='tfc2', bn_decay=bn_decay)
 84 | 
 85 |     with tf.variable_scope('transform_feat') as sc:
 86 |         weights = tf.get_variable('weights', [256, K*K],
 87 |                                   initializer=tf.constant_initializer(0.0),
 88 |                                   dtype=tf.float32)
 89 |         biases = tf.get_variable('biases', [K*K],
 90 |                                  initializer=tf.constant_initializer(0.0),
 91 |                                  dtype=tf.float32)
 92 |         biases += tf.constant(np.eye(K).flatten(), dtype=tf.float32)
 93 |         transform = tf.matmul(net, weights)
 94 |         transform = tf.nn.bias_add(transform, biases)
 95 | 
 96 |     transform = tf.reshape(transform, [batch_size, K, K])
 97 |     return transform
 98 | 
 99 | def neural_feature_net(point_cloud, is_training, bn_decay=None, knn_k=20, F=10):
100 |     """ Input (XYZ) Neural Feature Net, input is BxNx3 point cloud
101 | 			Return:
102 | 				Input Pointcloud's Neural Features of size BxNxF """
103 |     point_cloud_shape = point_cloud.get_shape()
104 |     batch_size = point_cloud_shape[0].value
105 |     num_point = point_cloud_shape[1].value
106 |     num_dim = point_cloud_shape[2].value
107 | 
108 |     adj_matrix = tf_util.pairwise_distance(point_cloud)
109 |     nn_idx = tf_util.knn(adj_matrix, k=knn_k) # shape: BxNxK
110 | 
111 |     idx_ = tf.range(batch_size) * num_point
112 |     idx_ = tf.reshape(idx_, [batch_size, 1, 1])
113 | 
114 |     point_cloud_flat = tf.reshape(point_cloud, [-1, num_dim])
115 |     point_cloud_neighbors = tf.gather(point_cloud_flat, nn_idx + idx_) # shape: BxNxKx3
116 | 
117 |     point_cloud_neighbors = tf.reshape(point_cloud_neighbors, [batch_size*num_point, knn_k, num_dim])
118 | 
119 |     input_image = tf.expand_dims(point_cloud_neighbors, -1)
120 |     net = tf_util.conv2d(input_image, 64, [1, 3],
121 |                          padding='VALID', stride=[1, 1],
122 |                          bn=True, is_training=is_training,
123 |                          scope='tconv1', bn_decay=bn_decay)
124 |     net = tf_util.conv2d(net, 64, [1, 1],
125 |                          padding='VALID', stride=[1, 1],
126 |                          bn=True, is_training=is_training,
127 |                          scope='tconv2', bn_decay=bn_decay)
128 |     net = tf_util.conv2d(net, 64, [1, 1],
129 |                          padding='VALID', stride=[1, 1],
130 |                          bn=True, is_training=is_training,
131 |                          scope='tconv3', bn_decay=bn_decay)
132 |     net = tf_util.max_pool2d(net, [knn_k, 1],
133 |                              padding='VALID', scope='tmaxpool')
134 | 
135 |     net = tf.reshape(net, [batch_size*num_point, -1])
136 |     net = tf_util.fully_connected(net, 32, bn=True, is_training=is_training,
137 |                                   scope='tfc1', bn_decay=bn_decay)
138 |     net = tf_util.fully_connected(net, F, bn=True, is_training=is_training,
139 |                                   scope='tfc2', bn_decay=bn_decay)
140 | 
141 |     feature = tf.reshape(net, [batch_size, num_point, F])
142 |     return feature
143 | 
144 | 


--------------------------------------------------------------------------------