├── .gitignore
├── Brisbane Event VPR.ipynb
├── README.md
├── code_helpers_public.py
├── code_helpers_public_pr_curve.py
├── convert_rosbags.py
├── correspondence_event_camera_frame_camera.py
├── dataset.png
├── export_frames_from_rosbag.py
├── read_gps.py
└── reconstruct_videos.py


/.gitignore:
--------------------------------------------------------------------------------
  1 | # vs-code
  2 | .vscode
  3 | 
  4 | # Byte-compiled / optimized / DLL files
  5 | __pycache__/
  6 | *.py[cod]
  7 | *$py.class
  8 | 
  9 | # C extensions
 10 | *.so
 11 | 
 12 | # Distribution / packaging
 13 | .Python
 14 | build/
 15 | develop-eggs/
 16 | dist/
 17 | downloads/
 18 | eggs/
 19 | .eggs/
 20 | lib/
 21 | lib64/
 22 | parts/
 23 | sdist/
 24 | var/
 25 | wheels/
 26 | pip-wheel-metadata/
 27 | share/python-wheels/
 28 | *.egg-info/
 29 | .installed.cfg
 30 | *.egg
 31 | MANIFEST
 32 | 
 33 | # PyInstaller
 34 | #  Usually these files are written by a python script from a template
 35 | #  before PyInstaller builds the exe, so as to inject date/other infos into it.
 36 | *.manifest
 37 | *.spec
 38 | 
 39 | # Installer logs
 40 | pip-log.txt
 41 | pip-delete-this-directory.txt
 42 | 
 43 | # Unit test / coverage reports
 44 | htmlcov/
 45 | .tox/
 46 | .nox/
 47 | .coverage
 48 | .coverage.*
 49 | .cache
 50 | nosetests.xml
 51 | coverage.xml
 52 | *.cover
 53 | *.py,cover
 54 | .hypothesis/
 55 | .pytest_cache/
 56 | 
 57 | # Translations
 58 | *.mo
 59 | *.pot
 60 | 
 61 | # Django stuff:
 62 | *.log
 63 | local_settings.py
 64 | db.sqlite3
 65 | db.sqlite3-journal
 66 | 
 67 | # Flask stuff:
 68 | instance/
 69 | .webassets-cache
 70 | 
 71 | # Scrapy stuff:
 72 | .scrapy
 73 | 
 74 | # Sphinx documentation
 75 | docs/_build/
 76 | 
 77 | # PyBuilder
 78 | target/
 79 | 
 80 | # Jupyter Notebook
 81 | .ipynb_checkpoints
 82 | 
 83 | # IPython
 84 | profile_default/
 85 | ipython_config.py
 86 | 
 87 | # pyenv
 88 | .python-version
 89 | 
 90 | # pipenv
 91 | #   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
 92 | #   However, in case of collaboration, if having platform-specific dependencies or dependencies
 93 | #   having no cross-platform support, pipenv may install dependencies that don't work, or not
 94 | #   install all needed dependencies.
 95 | #Pipfile.lock
 96 | 
 97 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow
 98 | __pypackages__/
 99 | 
100 | # Celery stuff
101 | celerybeat-schedule
102 | celerybeat.pid
103 | 
104 | # SageMath parsed files
105 | *.sage.py
106 | 
107 | # Environments
108 | .env
109 | .venv
110 | env/
111 | venv/
112 | ENV/
113 | env.bak/
114 | venv.bak/
115 | 
116 | # Spyder project settings
117 | .spyderproject
118 | .spyproject
119 | 
120 | # Rope project settings
121 | .ropeproject
122 | 
123 | # mkdocs documentation
124 | /site
125 | 
126 | # mypy
127 | .mypy_cache/
128 | .dmypy.json
129 | dmypy.json
130 | 
131 | # Pyre type checker
132 | .pyre/
133 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Event-Based Visual Place Recognition With Ensembles of Temporal Windows
 2 | 
 3 | [![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg?style=flat-square)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
 4 | [![stars](https://img.shields.io/github/stars/Tobias-Fischer/ensemble-event-vpr.svg?style=flat-square)](https://github.com/Tobias-Fischer/ensemble-event-vpr/stargazers)
 5 | [![GitHub issues](https://img.shields.io/github/issues/Tobias-Fischer/ensemble-event-vpr?style=flat-square)](https://github.com/Tobias-Fischer/ensemble-event-vpr/issues)
 6 | [![GitHub repo size](https://img.shields.io/github/repo-size/Tobias-Fischer/ensemble-event-vpr.svg?style=flat-square)](./README.md)
 7 | <a href="https://qcr.github.io" alt="QUT Centre for Robotics Open Source"><img src="https://img.shields.io/badge/collection-QUT%20Robotics-%23043d71?style=flat-square" /></a>
 8 | 
 9 | ### License + Attribution
10 | This code is licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). Commercial usage is not permitted. If you use this dataset or the code in a scientific publication, please cite the following [paper](http://doi.org/10.1109/LRA.2020.3025505) ([preprint and additional material](https://arxiv.org/abs/2006.02826)):
11 | 
12 | ```
13 | @article{fischer2020event,
14 |   title={Event-Based Visual Place Recognition With Ensembles of Temporal Windows},
15 |   author={Fischer, Tobias and Milford, Michael},
16 |   journal={IEEE Robotics and Automation Letters},
17 |   volume={5},
18 |   number={4},
19 |   pages={6924--6931},
20 |   year={2020}
21 | }
22 | ```
23 | 
24 | The [Brisbane-Event-VPR dataset](https://zenodo.org/record/4302805) accompanies this code repository: https://zenodo.org/record/4302805
25 | 
26 | ![Dataset preview](./dataset.png)
27 | 
28 | 
29 | ## Code overview
30 | The following code is available:
31 | - The [correspondence_event_camera_frame_camera.py](./correspondence_event_camera_frame_camera.py) file contains the mapping between the rosbag names and the consumer camera video names. The variable `video_beginning` indicates the ROS timestamp within the bag file that corresponds to the first frame of the consumer camera video file.
32 | - The [read_gps.py](./read_gps.py) file contains some helper functions to read in GPS data from the provided nmea files, and find matches between two traverses.
33 | - The [code_helpers_public.py](./code_helpers_public.py) file contains some helper codes.
34 | - The [code_helpers_public_pr_curve.py](./code_helpers_public_pr_curve.py) file contains some helper codes to get precision, recall and precision-recall curves given a distance matrix.
35 | - The main code is contained in the [Brisbane Event VPR.ipynb](./Brisbane%20Event%20VPR.ipynb) notebook.
36 | 
37 | Please note that in our paper we used manually annotated and then interpolated correspondences; instead here we provide matches based on the GPS data. Therefore, the results between what is reported in the paper and what is obtained using the methods here will be slightly different.
38 | 
39 | ## Reconstruct videos from events
40 | 1. Clone this repository: `git clone https://github.com/Tobias-Fischer/ensemble-event-vpr.git`
41 | 
42 | 1. Clone https://github.com/cedric-scheerlinck/rpg_e2vid and follow the instructions to create a conda environment and download the pretrained models.
43 | 
44 | 1. Download the [Brisbane-Event-VPR dataset](https://zenodo.org/record/4302805).
45 | 
46 | 1. Now convert the bag files to txt/zip files that can be used by the event2video code: `python convert_rosbags.py`. Make sure to adjust the path to the `extract_events_from_rosbag.py` file from the rpg_e2vid repository.
47 | 
48 | 1. Now do the event to video conversion: `python reconstruct_videos.py`. Make sure to adjust the path to the `run_reconstruction.py` file from the rpg_e2vid repository.
49 | 
50 | ## Create suitable conda environment
51 | 1. Create a new conda environment with the dependencies: `conda create --name brisbaneeventvpr tensorflow-gpu pynmea2 scipy matplotlib numpy tqdm jupyterlab opencv pip ros-noetic-rosbag ros-noetic-cv-bridge python=3.8 -c conda-forge -c robostack`
52 | 
53 | ## Export RGB frames from rosbags
54 | 1. `conda activate brisbaneeventvpr`
55 | 
56 | 1. `python export_frames_from_rosbag.py`
57 | 
58 | ## Event-based VPR with ensembles
59 | 1. Create a new conda environment with the dependencies: `conda create --name brisbaneeventvpr tensorflow-gpu pynmea2 scipy matplotlib numpy tqdm jupyterlab opencv pip`
60 | 
61 | 1. `conda activate brisbaneeventvpr`
62 | 
63 | 1. `git clone https://github.com/QVPR/netvlad_tf_open.git`
64 | 
65 | 1. `cd netvlad_tf_open && pip install -e .`
66 | 
67 | 1. Download the NetVLAD checkpoint [here](http://rpg.ifi.uzh.ch/datasets/netvlad/vd16_pitts30k_conv5_3_vlad_preL2_intra_white.zip) (1.1 GB). Extract the zip and move its contents to the checkpoints folder of the `netvlad_tf_open` repository.
68 | 
69 | 1. Open the [Brisbane Event VPR.ipynb](./Brisbane%20Event%20VPR.ipynb) and adjust the path to the `dataset_folder`.
70 | 
71 | 1. You can now run the code in [Brisbane Event VPR.ipynb](./Brisbane%20Event%20VPR.ipynb).
72 | 
73 | # Related works
74 | Please check out [this collection](https://qcr.github.io/collection/vpr_overview/) of related works on place recognition.
75 | 


--------------------------------------------------------------------------------
/code_helpers_public.py:
--------------------------------------------------------------------------------
  1 | import cv2
  2 | import numpy as np
  3 | from os.path import join, isfile, basename, abspath
  4 | import matplotlib.pyplot as plt
  5 | 
  6 | 
  7 | def compare_images(image_paths1, image_paths2, image_num1, image_num2):
  8 |     if isinstance(image_paths1[image_num1], np.ndarray):
  9 |         img_A = image_paths1[image_num1]
 10 |     else:
 11 |         img_A = cv2.imread(image_paths1[image_num1])
 12 | 
 13 |     if isinstance(image_paths2[image_num2], np.ndarray):
 14 |         img_B = image_paths2[image_num2]
 15 |     else:
 16 |         img_B = cv2.imread(image_paths2[image_num2])
 17 | 
 18 |     fig, ax = plt.subplots(1, 2, figsize=(2 * 7.2, 5.4))
 19 | 
 20 |     if isinstance(image_paths1[image_num1], str):
 21 |         ax[0].set_title(basename(image_paths1[image_num1]) + ' (Place ' + str(image_num1) + ')')
 22 |     else:
 23 |         ax[0].set_title('Place ' + str(image_num1))
 24 |     if isinstance(image_paths2[image_num2], str):
 25 |         ax[1].set_title(basename(image_paths2[image_num2]) + ' (Place ' + str(image_num2) + ')')
 26 |     else:
 27 |         ax[1].set_title('Place ' + str(image_num2))
 28 |     if len(img_A.shape) == 3 and img_A.shape[2] == 3:
 29 |         ax[0].imshow(img_A[..., ::-1])
 30 |     else:
 31 |         ax[0].imshow(img_A, 'gray', vmin=0, vmax=255)
 32 |     if len(img_B.shape) == 3 and img_B.shape[2] == 3:
 33 |         ax[1].imshow(img_B[..., ::-1])
 34 |     else:
 35 |         ax[1].imshow(img_B, 'gray', vmin=0, vmax=255)
 36 | 
 37 | 
 38 | def get_timestamps(comparison_folder1, comparison_folder2):
 39 |     def read_timestamps(comparison_folder):
 40 |         with open(join(comparison_folder, 'timestamps.txt'), "r") as file_in:
 41 |             timestamps_in = [float(line.rstrip()) for line in file_in]
 42 |         return timestamps_in
 43 |     return np.array(read_timestamps(comparison_folder1)), np.array(read_timestamps(comparison_folder2))
 44 | 
 45 | 
 46 | def get_timestamp_matches(timestamps, timestamps_to_match):
 47 |     timestamps_matched = np.array([np.abs(timestamps - ts).argmin() for ts in timestamps_to_match])
 48 |     return timestamps_matched
 49 | 
 50 | 
 51 | class ListOnDemand(object):
 52 |     def __init__(self):
 53 |         self.path_list = []
 54 |         self.image_list = []
 55 | 
 56 |     def append(self, item):
 57 |         self.path_list.append(item)
 58 |         self.image_list.append(None)
 59 | 
 60 |     def get_path_list(self):
 61 |         return self.path_list
 62 | 
 63 |     def __len__(self):
 64 |         return len(self.path_list)
 65 | 
 66 |     def __getitem__(self, idx):
 67 |         if isinstance(idx, slice):
 68 |             #Get the start, stop, and step from the slice
 69 |             return [self[ii] for ii in range(*idx.indices(len(self)))]
 70 |         elif isinstance(idx, list):
 71 |             return [self[ii] for ii in idx]
 72 |         elif isinstance(idx, np.ndarray):
 73 |             return self[idx.tolist()]
 74 |         elif isinstance(idx, str):
 75 |             return self[int(idx)]
 76 |         elif isinstance(idx, int) or isinstance(idx, np.generic):
 77 |             if idx < 0 : #Handle negative indices
 78 |                 idx += len( self )
 79 |             if idx < 0 or idx >= len( self ) :
 80 |                 raise IndexError("Index %d is out of range."%idx)
 81 | 
 82 |             if self.image_list[idx] is None:
 83 |                 self.image_list[idx] = cv2.imread(self.path_list[idx])
 84 | 
 85 |             return self.image_list[idx]
 86 |         else:
 87 |             raise TypeError("Invalid argument type:", type(idx))
 88 | 
 89 | def get_image_sets_on_demand(image_paths1, image_paths2, matches1, matches2):
 90 |     images_set1 = ListOnDemand()
 91 |     images_set2 = ListOnDemand()
 92 |     for idx1 in range(len(matches1)):
 93 |         image_num1 = matches1[idx1]
 94 |         images_set1.append(image_paths1[image_num1])
 95 |     for idx2 in range(len(matches2)):
 96 |         image_num2 = matches2[idx2]
 97 |         images_set2.append(image_paths2[image_num2])
 98 |     return images_set1, images_set2
 99 | 
100 | 
101 | def get_vlad_features(sess, net_out, image_batch, images_set, save_name=None, batch_size=4, tqdm_position=1):
102 |     if save_name and isfile(save_name):
103 |         pre_extracted_features = np.load(save_name)
104 |         if pre_extracted_features.shape[0] == len(images_set):
105 |             return pre_extracted_features
106 |         else:
107 |             print('Warning: shapes of pre_extracted_features and images_set do not match, re-compute!')
108 | 
109 |     descs = []
110 |     for batch_offset in tqdm(range(0, len(images_set), batch_size), position=tqdm_position, leave=False):
111 |     # for batch_offset in range(0, len(images_set), batch_size):
112 |         images = []
113 |         for i in range(batch_offset, batch_offset + batch_size):
114 |             if i == len(images_set):
115 |                 break
116 | 
117 |             image = images_set[i]
118 |             if not isinstance(image, (np.ndarray, np.generic) ):
119 |                 image = cv2.imread(image)
120 | 
121 |             if image_batch.shape[3] == 1:  # grayscale
122 |                 images.append(np.expand_dims(np.expand_dims(image, axis=0), axis=-1))
123 |             else:  # color
124 |                 image_inference = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
125 |                 images.append(np.expand_dims(image_inference, axis=0))
126 | 
127 |         batch = np.concatenate(images, 0)
128 |         descs = descs + list(sess.run(net_out, feed_dict={image_batch: batch}))
129 | 
130 |     netvlad_feature_list = np.array(descs)
131 | 
132 |     if save_name:
133 |         np.save(save_name, netvlad_feature_list)
134 | 
135 |     return netvlad_feature_list
136 | 


--------------------------------------------------------------------------------
/code_helpers_public_pr_curve.py:
--------------------------------------------------------------------------------
 1 | # PR code stuff
 2 | # code taken from https://github.com/oravus/DeltaDescriptors/blob/master/src/outFuncs.py
 3 | # MIT License; copyright Sourav Garg
 4 | 
 5 | import numpy as np
 6 | 
 7 | 
 8 | def getPR(mInds,gt,locRad):
 9 |     positives = np.argwhere(mInds!=-1)[:,0]
10 |     tp = np.sum(gt[positives] <= locRad)
11 |     fp = len(positives) - tp
12 | 
13 |     negatives = np.argwhere(mInds==-1)[:,0]
14 |     tn = np.sum(gt[negatives]>locRad)
15 |     fn = len(negatives) - tn
16 | 
17 |     assert(tp+tn+fp+fn==len(gt))
18 | 
19 |     if tp == 0:
20 |         return 0,0,0 # what else?
21 | 
22 |     prec = tp/float(tp+fp)
23 |     recall = tp/float(tp+fn)
24 |     fscore = 2*prec*recall/(prec+recall)
25 | 
26 |     return prec, recall, fscore
27 | 
28 | 
29 | def getPRCurve(mInds,mDists,gt,locRad):
30 |     prfData = []
31 |     lb, ub = mDists.min(),mDists.max()
32 |     step = (ub-lb)/100.0
33 |     for thresh in np.arange(lb,ub+step,step):
34 |         matchFlags = mDists<=thresh
35 |         outVals = mInds.copy()
36 |         outVals[~matchFlags] = -1
37 | 
38 |         p,r,f = getPR(outVals,gt,locRad)
39 |         prfData.append([p,r,f])
40 |     return np.array(prfData)
41 | 
42 | 
43 | def getPAt100R(dists,maxLocRad):
44 |     pAt100R = []
45 |     for i1 in range(maxLocRad):
46 |         pAt100R.append([np.sum(dists<=i1)])
47 |     pAt100R = np.array(pAt100R) / float(len(dists))
48 |     return pAt100R
49 | 
50 | 
51 | def getRAt99P(prfData):
52 |     # select all rows where precision == 1 and then get max of recall
53 |     if len(prfData[prfData[:, 0]>0.99]) == 0:
54 |         return 0
55 |     else:
56 |         return np.max(prfData[prfData[:, 0]>0.99][:, 1])
57 | 
58 | 
59 | def getPRCurveWrapper(dist_matrix, plot_threshold_fps):
60 |     dist_matrix_plot_pr = dist_matrix.copy()
61 | 
62 |     gt = np.abs(np.arange(len(dist_matrix_plot_pr))-np.argmin(dist_matrix_plot_pr, axis=0))
63 |     prvals = getPRCurve(np.argmin(dist_matrix_plot_pr, axis=0),np.min(dist_matrix_plot_pr, axis=0), gt, plot_threshold_fps)
64 |     return prvals
65 | 
66 | 
67 | def get_recall_helper(dist_matrix, max_dist=None, transpose=True, progress_bar_position=None):
68 |     if max_dist is None:
69 |         max_dist = len(dist_matrix)
70 |     
71 |     tps = []
72 |     best_matches = []
73 |     
74 |     if progress_bar_position is None:
75 |         list_to_iterate = range(dist_matrix.shape[0])
76 |     else:
77 |         list_to_iterate = tqdm(range(dist_matrix.shape[0]), position=progress_bar_position, leave=False)
78 | 
79 |     for idx in list_to_iterate:
80 |         if len(dist_matrix.shape) == 2:
81 |             if transpose:
82 |                 best_match = np.argmin(dist_matrix.T[idx])
83 |             else:
84 |                 best_match = np.argmin(dist_matrix[idx])
85 |         elif len(dist_matrix.shape) == 1:  # for SeqSLAM
86 |             best_match = dist_matrix[idx]
87 |         else:
88 |             raise Exception('Not supported')
89 | 
90 |         best_matches.append(best_match)
91 |         # tp = np.array([np.count_nonzero(abs(sorted_matches[:i+1]-idx)==0) for i in range(0, max_dist+1)])  # Consider top-k matches
92 |         tp = np.array([np.count_nonzero(abs(best_match-idx)<i) for i in range(1, max_dist + 1)])  # Only consider top match
93 |         tps.append(tp)
94 |     tps_summed = np.sum(np.array(tps), axis=0)
95 |     recall = tps_summed / len(tps)
96 |     return recall, np.array(tps), np.array(best_matches)


--------------------------------------------------------------------------------
/convert_rosbags.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | 
 3 | import os
 4 | 
 5 | filenames = ["dvs_vpr_2020-04-21-17-03-03.bag",
 6 |              "dvs_vpr_2020-04-22-17-24-21.bag",
 7 |              "dvs_vpr_2020-04-24-15-12-03.bag",
 8 |              "dvs_vpr_2020-04-27-18-13-29.bag",
 9 |              "dvs_vpr_2020-04-28-09-14-11.bag",
10 |              "dvs_vpr_2020-04-29-06-20-23.bag"]
11 | 
12 | for filename in filenames:
13 |     os.system("python scripts/extract_events_from_rosbag.py " + str(filename) + " --output_folder=./ --event_topic=/dvs/events")
14 | 


--------------------------------------------------------------------------------
/correspondence_event_camera_frame_camera.py:
--------------------------------------------------------------------------------
 1 | traverse_to_name = {
 2 |     'run_tobi3_dvs_vpr_2020-04-21-17-03-03': 'sunset1',
 3 |     'run_tobi3_dvs_vpr_2020-04-22-17-24-21': 'sunset2',
 4 |     'run_tobi3_dvs_vpr_2020-04-24-15-12-03': 'daytime',
 5 |     'run_tobi3_dvs_vpr_2020-04-28-09-14-11': 'morning',
 6 |     'run_tobi3_dvs_vpr_2020-04-29-06-20-23': 'sunrise',
 7 |     'run_tobi3_dvs_vpr_2020-04-27-18-13-29': 'night'
 8 | }
 9 | 
10 | name_to_consumervideo = {
11 |     'sunset1': '20200421_170039-sunset1',
12 |     'sunset2': '20200422_172431-sunset2',
13 |     'daytime': '20200424_151015-daytime',
14 |     'night': '20200427_181204-night',
15 |     'morning': '20200428_091154-morning',
16 |     'sunrise': '20200429_061912-sunrise'
17 | }
18 | 
19 | video_beginning = {
20 |     'sunset1': 1587452582.35,
21 |     'sunset2': 1587540271.65,
22 |     'daytime': 1587705130.80,
23 |     'morning': 1588029265.73,
24 |     'sunrise': 1588105232.91,
25 |     'night': 1587975221.10
26 | }
27 | 


--------------------------------------------------------------------------------
/dataset.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Tobias-Fischer/ensemble-event-vpr/fa427842c2d7bf2e3dbcc4faf729519725b95dea/dataset.png


--------------------------------------------------------------------------------
/export_frames_from_rosbag.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | 
 3 | import rosbag
 4 | import os
 5 | import cv2
 6 | from tqdm.auto import tqdm as tqdm
 7 | from cv_bridge import CvBridge
 8 | 
 9 | rosbag_location = './'
10 | cv_bridge = CvBridge()
11 | 
12 | 
13 | def timestamp_str(ts):
14 |     t = ts.secs + ts.nsecs / float(1e9)
15 |     return '{:.12f}'.format(t)
16 | 
17 | filenames = ["dvs_vpr_2020-04-21-17-03-03.bag",
18 |              "dvs_vpr_2020-04-22-17-24-21.bag",
19 |              "dvs_vpr_2020-04-24-15-12-03.bag",
20 |              "dvs_vpr_2020-04-27-18-13-29.bag",
21 |              "dvs_vpr_2020-04-28-09-14-11.bag",
22 |              "dvs_vpr_2020-04-29-06-20-23.bag"]
23 | 
24 | for bagname in filenames:
25 |     bagname = bagname.replace('.bag', '')
26 |     with rosbag.Bag(rosbag_location+'/'+bagname+'.bag', 'r') as bag:
27 |         topics = bag.get_type_and_topic_info().topics
28 |         for topic_name, topic_info in topics.iteritems():
29 |             if topic_name == '/dvs/image_raw':
30 |                 total_num_frames = topic_info.message_count
31 |                 print('Found {} frames in rosbag'.format(total_num_frames))
32 | 
33 |         if not os.path.isdir(rosbag_location+bagname+'/frames'):
34 |             os.makedirs(rosbag_location+bagname+'/frames')
35 | 
36 |         with tqdm(total=total_num_frames) as pbar:
37 |             for topic, msg, t in bag.read_messages(topics=['/dvs/image_raw']):
38 |                 if msg.header.stamp.secs < 100:
39 |                     pbar.update(1)
40 |                     continue
41 | 
42 |                 cv_image = cv_bridge.imgmsg_to_cv2(msg, "bgr8")
43 | 
44 |                 if cv_image.shape[0] == 260 and cv_image.shape[1] == 346:
45 |                     cv2.imwrite(rosbag_location + bagname + '/frames/' + timestamp_str(msg.header.stamp) + '.png', cv_image)
46 | 
47 |                 pbar.update(1)
48 | 


--------------------------------------------------------------------------------
/read_gps.py:
--------------------------------------------------------------------------------
 1 | import pynmea2
 2 | import numpy as np
 3 | 
 4 | 
 5 | def get_gps(nmea_file_path):
 6 |     nmea_file = open(nmea_file_path, encoding='utf-8')
 7 | 
 8 |     latitudes, longitudes, timestamps = [], [], []
 9 | 
10 |     first_timestamp = None
11 |     previous_lat, previous_lon = 0, 0
12 | 
13 |     for line in nmea_file.readlines():
14 |         try:
15 |             msg = pynmea2.parse(line)
16 |             if first_timestamp is None:
17 |                 first_timestamp = msg.timestamp
18 |             if msg.sentence_type not in ['GSV', 'VTG', 'GSA']:
19 |                 # print(msg.timestamp, msg.latitude, msg.longitude)
20 |                 # print(repr(msg.latitude))
21 |                 dist_to_prev = np.linalg.norm(np.array([msg.latitude, msg.longitude]) - np.array([previous_lat, previous_lon]))
22 |                 if msg.latitude != 0 and msg.longitude != 0 and msg.latitude != previous_lat and msg.longitude != previous_lon and dist_to_prev > 0.0001:
23 |                     timestamp_diff = (msg.timestamp.hour - first_timestamp.hour) * 3600 + (msg.timestamp.minute - first_timestamp.minute) * 60 + (msg.timestamp.second - first_timestamp.second)
24 |                     latitudes.append(msg.latitude); longitudes.append(msg.longitude); timestamps.append(timestamp_diff)
25 |                     previous_lat, previous_lon = msg.latitude, msg.longitude
26 | 
27 |         except pynmea2.ParseError as e:
28 |             # print('Parse error: {} {}'.format(msg.sentence_type, e))
29 |             continue
30 | 
31 |     return np.array(np.vstack((latitudes, longitudes, timestamps))).T
32 | 


--------------------------------------------------------------------------------
/reconstruct_videos.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | 
 3 | import os
 4 | 
 5 | filenames = ["dvs_vpr_2020-04-21-17-03-03.zip",
 6 |              "dvs_vpr_2020-04-22-17-24-21.zip",
 7 |              "dvs_vpr_2020-04-24-15-12-03.zip",
 8 |              "dvs_vpr_2020-04-27-18-13-29.zip",
 9 |              "dvs_vpr_2020-04-28-09-14-11.zip",
10 |              "dvs_vpr_2020-04-29-06-20-23.zip"]
11 | 
12 | model = 'firenet_1000.pth.tar'
13 | # model = 'E2VID_lightweight.pth.tar'
14 | 
15 | for filename in filenames:
16 |     for num_events_per_pixel in [0.1, 0.3, 0.6, 0.8]:
17 |         os.system("python run_reconstruction.py -c firenet_1000.pth.tar -i " + str(filename) + " --auto_hdr --color --num_events_per_pixel " + str(num_events_per_pixel) + " --hot_pixels_file " + str(filename.replace('.zip', '_hot_pixels.txt') + " --output_folder N_" + str(num_events_per_pixel) + "/" + str(filename.replace('.zip', '')) + " --dataset_name " + str(filename.replace('.zip', ''))))
18 |     for window_duration in [44, 66, 88, 120, 140]:
19 |         os.system("python run_reconstruction.py -c firenet_1000.pth.tar -i " + str(filename) + " --auto_hdr --color --window_duration " + str(window_duration) + " --hot_pixels_file " + str(filename.replace('.zip', '_hot_pixels.txt') + " --output_folder t_" + str(window_duration) + "/" + str(filename.replace('.zip', '')) + " --dataset_name " + str(filename.replace('.zip', ''))))
20 | 


--------------------------------------------------------------------------------