.
675 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | # An Asynchronous Kalman Filter for Hybrid Event Cameras
4 |
5 |
6 | 
7 |
8 |
9 |
10 |
11 | Ziwei Wang, Yonhon Ng, Cedric Scheerlinck and Robert Mahony
12 |
13 | The paper was accepted by the 2021 IEEE Int. Conf. Computer Vision (ICCV), 2021
14 |
15 | [[Paper](https://openaccess.thecvf.com/content/ICCV2021/papers/Wang_An_Asynchronous_Kalman_Filter_for_Hybrid_Event_Cameras_ICCV_2021_paper.pdf)] [[ArXiv](https://arxiv.org/abs/2012.05590)] [[Supplementary Materials](https://openaccess.thecvf.com/content/ICCV2021/supplemental/Wang_An_Asynchronous_Kalman_ICCV_2021_supplemental.pdf)] [[GitHub](https://github.com/ziweiWWANG/AKF)]
16 |
17 | ## Citation
18 | If you use or discuss our AKF, please cite our paper as follows:
19 |
20 | @inproceedings{wang2021asynchronous,
21 | title={An asynchronous kalman filter for hybrid event cameras},
22 | author={Wang, Ziwei and Ng, Yonhon and Scheerlinck, Cedric and Mahony, Robert},
23 | booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
24 | pages={448--457},
25 | year={2021}
26 | }
27 |
28 |
29 |
30 | @article{wang2023asynchronous,
31 | title={An Asynchronous Linear Filter Architecture for Hybrid Event-Frame Cameras},
32 | author={Wang, Ziwei and Ng, Yonhon and Scheerlinck, Cedric and Mahony, Robert},
33 | journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
34 | year={2023},
35 | publisher={IEEE}
36 | }
37 |
38 |
39 | Please check out our more recent [Event-Asynchronous-Filter](https://github.com/ziweiWWANG/Event-Asynchronous-Filter) (TPAMI 2023) in the same line of work.
40 |
41 | The [Event-based Complementary Filter](https://github.com/cedric-scheerlinck/dvs_image_reconstruction) is integrated into a unified pipeline and tested on more datasets.
42 |
43 |
44 |
45 | 
46 |
47 |
48 |
49 |
50 | ## Code - How to use
51 |
52 |
53 | There are a few parameters that users can specify:
54 |
55 | ### In file [run_akf.m](https://github.com/ziweiWWANG/AKF/blob/main/run_akf.m):
56 |
57 | | Variables | Description | Default Value |
58 | |----------------------|----------------------|-----------------------------|
59 | | `deblur_option` | 1: deblur, 0: no deblur. Use the deblur option if the input images are blurry | 1 |
60 | | `framerate` | the frame rate of the output image sequence in Hz | 300 |
61 | | `use_median_filter` | a flag of applying a 3-by-3 median filter to the output images | 0 |
62 | | `output_high_frame_rate_flag` | 1: output images of the pre-defined framerate, 0: output images of the frame intensity framerat. | 1 |
63 | | `sigma_p` | the process noise parameter | 0.0005 |
64 | | `sigma_i` | the isolated noise parameter | 0.03 |
65 | | `sigma_r` | the refractory noise parameter | 0.05 |
66 | | `refractory_period` | the refractory period in microsecond. It models the circuit limitations in each pixel of an event camera limit the response time of events| 10000|
67 | |`min_ct_scale`| the minimal value for the contrast threshold scaling factor | 0.6|
68 | |`max_ct_scale`| the maximal value for the contrast threshold scaling factor | 100|
69 | |`p_ini` | initial value for state covariance P | 0.09|
70 |
71 | ### In file [akf_reconstruction.m](https://github.com/ziweiWWANG/AKF/blob/main/akf_reconstruction.m):
72 | 1. `post_process`: 0 for no normalization; 1 for (image-min/(max-min)); 2 for user-defined maximum and minimum value for extremely bright view; 3 for user-defined maximum and minimum value for extremely dark view. Post-processing methods are important in displaying the reconstructed HDR images since the intensity values can go beyond 0 and 1; 4 for contrast-limited adaptive histogram equalization (using matlab inbuilt function `adapthisteq()`). Without a proper post-processing method, the details in the HDR part of the image (higher than 1 or lower than 0) can not be displayed. Users can adjust the pre-defined maximum and minimum value in file [output_img.m](https://github.com/ziweiWWANG/AKF/blob/main/output_img.m) to have the best visualization.
73 | 2. The `f_Q` is the most important parameter for image noise. It represents the inverse of the `R_bar` function in `equation (6)` in the paper. You can simply treat it as the image confidence function of intensity. For example, for an image in the range [0 255], the extreme values around 0 and 255 would have lower confidence. The `f_Q` is included in the provided dataset. If you are using your own dataset, you need to tune it carefully.
74 | 3. The preset exposure time for each intensity image is included in the provided datasets (some datasets are recorded with auto-exposure, e.g., [interlaken_01a_events_1_150.mat](https://drive.google.com/drive/folders/1vhgi5h4lwIjhTi-7szzpuzsJq2ewpzUm?usp=sharing)). If you want to use your own dataset, please set or estimate the exposure time as well.
75 | 4. If the exposure time for the intensity images are very short and there is almost no blurry, you can disable the deblur function by setting `deblur_option = 0`. But you still need to define an `exposure` time.
76 |
77 | ## Datasets
78 | Download the datasets and save them in folder `data/`.
79 | If you want to use your datasets, define `post_process` method, `f_Q`, `exposure`, contrast threshold `ct` at the beginning of [akf_reconstruction.m](https://github.com/ziweiWWANG/AKF/blob/main/akf_reconstruction.m). See notes in the next section.
80 |
81 | ### [Click Here To Download Example Datasets](https://drive.google.com/drive/folders/1vhgi5h4lwIjhTi-7szzpuzsJq2ewpzUm?usp=sharing)
82 | Dataset name convention: DatasetName_StartFrame_EndFrame of the original dataset (we only keep the fast motion part or highly HDR part in the sample datasets. You can download the whole dataset sequence from the website of the following papers, and test if you like). The example datasets are publicly available datasets from:
83 | [[Mueggler et al., IJRR 2017]](https://rpg.ifi.uzh.ch/davis_data.html),
84 | [[Scherlinck et al., ACCV 2018]](https://drive.google.com/drive/folders/1Jv73p1-Hi56HXyal4SHQbzs2zywISOvc),
85 | [[Gehrig et al., ICRA 2021]](https://dsec.ifi.uzh.ch/).
86 |
87 | ### Our HDR Hybrid Event-Frame Dataset
88 | Selected Images from Our HDR Hybrid Event-Frame Dataset:
89 |
90 |
91 | 
92 |
93 |
94 | First row shows the low dynamic range frames and the second row shows the high dynamic range ground truth (with tone-mapping for display only).
95 |
96 | #### [Click Here To Download Raw Data](https://drive.google.com/drive/folders/1JYdvY2GqgD3RC-rczgf8t1JoIuBqoJvp?usp=sharing)
97 | Raw data includes raw events, HDR ground truth images, LDR images, exposure time, etc. This can be directly used for AKF.
98 |
99 | Example of loading our HDR dataset in Python:
100 | ```
101 | import h5py
102 | file_path = '../Downloads/tree3.mat'
103 | with h5py.File(file_path, 'r') as f:
104 | events = f['events'][:]
105 | HDR_hdr_rgb = f['HDR_mat'][:]
106 | image_ldr_bw = f['image'][:]
107 | time_image = f['time_image'][:]
108 | ```
109 |
110 | #### [Click Here To Download Event and Frame Pairs](https://drive.google.com/drive/folders/1fH7TurmZ68WQunNP4RfTUeeMOiD7RGfX?usp=sharing)
111 | For some methods which require event reconstruction and frame pairs, we provide reconstructed event data using the E2VID event reconstruction algorithm [[Rebecq et al., TPAMI 2019]](https://github.com/uzh-rpg/rpg_e2vid).
112 |
113 | ### Our AHDR Hybrid Event-Frame Dataset
114 |
115 |
116 | 
117 |
118 |
119 | For our AHDR dataset, we apply an artificial camera
120 | response function to RGB camera output frames to simulate
121 | a low dynamic range camera. The resulting image noise covariance should be high for the ‘cropped’ intensity values. You can adjust `f_Q` if needed.
122 |
123 | Download our mountain dataset, lake dataset, and artificial camera response functions `s_curve` here:
124 | [link](https://drive.google.com/drive/folders/1RXHQGjyit5vqO3l1LEl76XKh9MfkWwbn?usp=sharing). The datasets include raw events and SDR RGB images. Please feel free to use your own data here.
125 |
126 | Process SDR to LDR images:
127 | ```
128 | cut_num = 80; # Choose from 30, 50, 80, 100, 115
129 | for i = 1:size(img_color,4)
130 | load(['Path_to_s_curve/s_curve_' num2str(cut_num) '.mat']);
131 | image(:,:,i) = s_curve(rgb2gray(img_color(:,:,:,i)) - 14);
132 | end
133 | ```
134 |
135 | ### Notes
136 | 1. Make sure your event and image timestamps are well aligned.
137 | 2. As a nature of the filtering methods, the quality of the reconstruction results is relevant to the quality of event camera datasets. Datasets with obvious noise recorded by hybrid event-frame cameras or lower resolution/sensitivity cameras such as [DAVIS 240](https://inivation.com/wp-content/uploads/2019/08/DAVIS240.pdf) might lead to unsatisfied results in high temporal resolution video reconstruction. The method requires a short time to adapt and converge to the optimal Kalman filter parameters for each dataset.
138 | 3. For academic use only. Should you have any questions or suggestions regarding this code and the corresponding results, please don't hesitate to get in touch with ziwei.wang1@anu.edu.au
139 |
--------------------------------------------------------------------------------
/akf_reconstruction.m:
--------------------------------------------------------------------------------
1 | function akf_reconstruction(DataName, deblur_option, ...
2 | framerate, use_median_filter, output_high_frame_rate_flag,...
3 | sigma_p, sigma_i, sigma_r,refractory_period, min_ct_scale, max_ct_scale,...
4 | p_ini)
5 |
6 | %% the post_process method, exposure and 'f_Q' is included in the provided datasets.
7 | % if you are using your datasets, carefully define these parameters here.
8 |
9 | %% post_process method
10 | % 0: no normalization,
11 | % 1: (image-min/(max-min)),
12 | % 2: user-defined max and min value for extremely bright view
13 | % 3: user-defined max and min value for extremely dark view
14 |
15 | % for example:
16 | % post_process = 1;
17 |
18 | %% f_Q, exposure time and contrast threshold
19 | % The 'f_Q' is the most important parameter for image noise. It
20 | % represents the inverse of the R_bar function in equation (6) in the
21 | % paper. You can simply treat it as the image confidence function of
22 | % intensity. For example, for an image in range [0 255], the extreme
23 | % values around 0 and 255 would have lower confidence. You can find the
24 | % sample weight function from the provided dataset link.
25 |
26 | % The preset exposure time for each intensity image is included in the
27 | % provided datasets (some datasets are recorded with auto-exposure,
28 | % e.g., interlaken_01a_events_1_150.mat). If you want to use your own
29 | % dataset, please set or estimate the exposure time as well.
30 |
31 | % for example:
32 | % load('../src/mat/weight_function.mat'); f_Q = max(weight_func,0.09) * 7e5;
33 | % exposure = zeros(10)+500; % define exposure time for 10 input images
34 | % ct = 0.1; % set contrast threshold
35 |
36 |
37 | %% set output folders
38 | folder = sprintf(['./reconstruction/' DataName]);
39 | if ~exist(folder, 'dir')
40 | mkdir(folder)
41 | end
42 | fileID_bw = fopen(sprintf(['./' folder '/timestamps.txt']),'w');
43 | if output_high_frame_rate_flag
44 | high_frame_rate_folder = sprintf([folder '_hfr']);
45 | if ~exist(high_frame_rate_folder, 'dir')
46 | mkdir(high_frame_rate_folder)
47 | end
48 | end
49 |
50 | %% load dataset
51 | image = []; events = [];
52 | load_data_add = sprintf(['./data/' DataName '.mat']);
53 | load(load_data_add);
54 | %% initialization
55 | [ts_array_,ts_array_2,frame_idx,R_inv_array,C,weight]...
56 | = initialization(im_height, im_width);
57 | frame_time = floor(1/framerate * 1e6);
58 | frame_start = 2;
59 | frame_end = length(time_image);
60 | safety_offset = 1;
61 | P = zeros(im_height,im_width) + p_ini;
62 | new_frame_f = 1;
63 | write_output_f = 0;
64 | img_idx_now = frame_start;
65 | img_idx_next = frame_start+1; % intensity range [1 256]
66 | img_now_ts_1 = time_image(img_idx_now)-exposure(img_idx_now)/2;
67 | img_now_ts_mid = time_image(img_idx_now);
68 | img_now_ts_2 = time_image(img_idx_now)+exposure(img_idx_now)/2;
69 | img_next_ts_1 = time_image(img_idx_now+1)-exposure(img_idx_now)/2;
70 | img1_round_in_0255 = double(image(:,:,img_idx_now))+1;
71 | img2_round_in_0255 = double(image(:,:,img_idx_next))+1;
72 | img1_round_in_01 = img1_round_in_0255/255;
73 | img2_round_in_01 = img2_round_in_0255/255;
74 | log_intensity_now_ = log(double(image(:,:,frame_start))/255 + safety_offset);
75 |
76 | t1 = time_image(frame_start);
77 | e_start_idx = find(events(:,1) >= t1,1,'first');
78 | t2 = time_image(frame_end);
79 | if t2 >= events(end,1)
80 | e_end_idx = size(events,1); % last event
81 | else
82 | e_end_idx = find(events(:,1) >= t2,1,'first');
83 | end
84 | t_next_publish_ = time_image(frame_start)+exposure(img_idx_now)/2;
85 | output_deblur_t1 = double(image(:,:,frame_start))/255;
86 | output_deblur_t2 = double(image(:,:,frame_start+1))/255;
87 | log_deblur_image_now(:,:,1) = log(double(output_deblur_t1) + safety_offset);
88 | log_deblur_image_now(:,:,2) = log(double(output_deblur_t2) + safety_offset);
89 | log_intensity_state_ = log_deblur_image_now(:,:,1);
90 | ct_scale = compute_ct_scale(ct,exposure,log_deblur_image_now,time_image,events,img_idx_now);
91 | % assume forwards and backbards interpolation are the same at the beginning
92 | log_output_interp_t1 = log_deblur_image_now(:,:,1);
93 | log_output_interp_t2 = log_output_interp_t1;
94 |
95 | %% event Update
96 | for i=e_start_idx:e_end_idx
97 | ts = events(i,1); % microsecond
98 | x = events(i,2);
99 | y = events(i,3);
100 | polarity = events(i,4); % [-1, 1]
101 |
102 | %% update raw image & deblur image & ct_scale
103 | if events(i,1) > time_image(frame_end) || img_idx_next >= (frame_end-1)
104 | break;
105 | %% need deblur => update reference image inside exposure time
106 | elseif deblur_option && (ts > img_now_ts_1) && (ts <= img_now_ts_2)
107 | if new_frame_f
108 | output_deblur_1_log = deblur_edge_left(min_ct_scale,max_ct_scale,img_idx_now,time_image,exposure(img_idx_now),events,image,ct,safety_offset);
109 | log_intensity_now_ = output_deblur_1_log;
110 | new_frame_f = 0;
111 | else
112 | if (polarity == 1)
113 | contrast_threshold = ct;
114 | else
115 | contrast_threshold = -ct;
116 | end
117 | log_intensity_now_(y,x) = log_intensity_now_(y,x) + contrast_threshold;
118 | end
119 |
120 | %% time to output image
121 | if (ts > img_now_ts_mid) && write_output_f
122 | intensity_now_ = exp(log_intensity_now_) - safety_offset;
123 | intensity_now_(intensity_now_>1) = 1;
124 | intensity_now_(intensity_now_<0) = 0;
125 | log_intensity_now_ = log(intensity_now_ + safety_offset);
126 |
127 | %% global update
128 | delta_t = (ts - ts_array_)/1e6;
129 | C = (log_intensity_state_ - log_intensity_now_) ./ P;
130 | log_intensity_state_ = (1 ./ (1 ./ P + R_inv_array .* delta_t)) .* C + log_intensity_now_;
131 | P = 1 ./ ((1 ./ P) + R_inv_array .* delta_t);
132 | P = P + sigma_p .* delta_t;
133 | ts_array_(:) = ts;
134 |
135 | % output images and timestamps
136 | output_img(log_intensity_now_,img_idx_now,use_median_filter,safety_offset,post_process,folder,P,log_intensity_state_)
137 | output_img_ts = time_image(img_idx_now) / 1e6;
138 | output_img_ts = sprintf('%0.9f',output_img_ts);
139 | outputIdxTs = sprintf(['image_' num2str(img_idx_now) '.png ' num2str(output_img_ts) '\n']);
140 | fprintf(fileID_bw,outputIdxTs);
141 | write_output_f = 0;
142 | end
143 |
144 | %% no deblur + time to output
145 | elseif (~deblur_option) && (ts > img_now_ts_1) && (ts <= img_now_ts_2) && write_output_f
146 | log_intensity_now_ = log(double(image(:,:,img_idx_now))/255+safety_offset);
147 |
148 | %% global update
149 | delta_t = (ts - ts_array_)/1e6;
150 | if (min(delta_t)<0)
151 | fprintf('ERROR! Check code! \n')
152 | end
153 | C = (log_intensity_state_ - log_intensity_now_) ./ P;
154 | log_intensity_state_ = (1 ./ (1 ./ P + R_inv_array .* delta_t)) .* C + log_intensity_now_;
155 | P = 1 ./ ((1 ./ P) + R_inv_array .* delta_t);
156 | P = P + sigma_p .* delta_t;
157 | ts_array_(:) = ts;
158 |
159 | % output images and timestamps
160 | output_img(log_intensity_now_,img_idx_now,use_median_filter,safety_offset,post_process,folder,P,log_intensity_state_)
161 | output_img_ts = time_image(img_idx_now) / 1e6;
162 | output_img_ts = sprintf('%0.9f',output_img_ts);
163 | outputIdxTs = sprintf(['image_' num2str(img_idx_now) '.png ' num2str(output_img_ts) '\n']);
164 | fprintf(fileID_bw,outputIdxTs);
165 | write_output_f = 0;
166 |
167 | %% time to update to next frame
168 | elseif (ts > img_next_ts_1) && (img_idx_next < length(time_image))
169 | new_frame_f = 1;
170 | write_output_f = 1;
171 | img_idx_next = img_idx_next + 1;
172 | img_idx_now = img_idx_now + 1;
173 | img_now_ts_1 = time_image(img_idx_now) - exposure(img_idx_now)/2;
174 | img_now_ts_mid = time_image(img_idx_now);
175 | img_now_ts_2 = time_image(img_idx_now) + exposure(img_idx_now)/2;
176 | img_next_ts_1 = time_image(img_idx_now + 1) - exposure(img_idx_now)/2;
177 | img_next_ts_2 = time_image(img_idx_now + 1) + exposure(img_idx_now)/2;
178 |
179 | % deblur the first and the second image
180 | if deblur_option
181 | [output_deblur_t1,output_deblur_t2] = deblur_edge_outer(min_ct_scale,max_ct_scale,img_idx_now,time_image,exposure(img_idx_now),events,image,ct,safety_offset);
182 | else
183 | output_deblur_t1 = double(image(:,:,img_idx_now))/255;
184 | output_deblur_t2 = double(image(:,:,img_idx_now+1))/255;
185 | end
186 | log_deblur_image_now(:,:,1) = log(double(output_deblur_t1) + safety_offset);
187 | log_deblur_image_now(:,:,2) = log(double(output_deblur_t2) + safety_offset);
188 |
189 | % update ct_scale between the first and the second image
190 | ct_scale = compute_ct_scale(ct,exposure,log_deblur_image_now,time_image,events,img_idx_now);
191 |
192 | % update parameters for interpolation forward -- current time: time_image(img_idx_now)
193 | log_output_interp_t1 = log_deblur_image_now(:,:,1);
194 |
195 | % update parameters for interpolation backward -- current time: time_image(img_idx_now)
196 | log_output_interp_t2 = log_deblur_image_now(:,:,2);
197 | t1 = img_now_ts_1;
198 | t2 = img_next_ts_2;
199 | e_start_idx_interp = find(events(:,1) >= t1,1,'first');
200 | e_end_idx_interp = find(events(:,1) >= t2,1,'first');
201 | for id = e_start_idx_interp:e_end_idx_interp
202 | if events(id,4) < 1
203 | if ct_scale(events(id,3),events(id,2)) >= min_ct_scale && ct_scale(events(id,3),events(id,2)) <= max_ct_scale
204 | log_output_interp_t2(events(id,3),events(id,2)) = log_output_interp_t2(events(id,3),events(id,2)) + ct/ct_scale(events(id,3),events(id,2));
205 | else
206 | log_output_interp_t2(events(id,3),events(id,2)) = log_output_interp_t2(events(id,3),events(id,2)) + ct;
207 | end
208 | else
209 | if ct_scale(events(id,3),events(id,2)) >= min_ct_scale && ct_scale(events(id,3),events(id,2)) <= max_ct_scale
210 | log_output_interp_t2(events(id,3),events(id,2)) = log_output_interp_t2(events(id,3),events(id,2)) - ct/ct_scale(events(id,3),events(id,2));
211 | else
212 | log_output_interp_t2(events(id,3),events(id,2)) = log_output_interp_t2(events(id,3),events(id,2)) - ct;
213 | end
214 | end
215 | end
216 |
217 | % image noise update
218 | img1_round_in_0255 = round(double(image(:,:,img_idx_now))) + 1; % intensity range [1 256]
219 | img1_round_in_01 = img1_round_in_0255 / 255; % intensity range [0 1]
220 | R_img_idx = img_idx_next;
221 | img2_round_in_0255 = round(double(image(:,:,R_img_idx))) + 1; % intensity range [1 256]
222 | R_bar_inv = f_Q(img1_round_in_0255);
223 | R_inv_array = R_bar_inv .* (img1_round_in_01 + safety_offset).^2;
224 | end
225 |
226 | %% local update
227 | log_intensity_now = log_intensity_now_(y,x);
228 | delta_t = (ts - ts_array_(y,x))/1e6;
229 | if (polarity == 1)
230 | contrast_threshold = ct;
231 | else
232 | contrast_threshold = -ct;
233 | end
234 | C(y,x) = (log_intensity_state_(y,x) - log_intensity_now) ./ P(y,x);
235 | log_intensity_state_(y,x) = (1 ./ (1 ./ P(y,x) + R_inv_array(y,x) .* delta_t)) .* C(y,x) + log_intensity_now;
236 | log_intensity_state_(y,x) = log_intensity_state_(y,x) + contrast_threshold;
237 | if x > 1 && x < im_width && y > 1 && y < im_height && ((ts - ts_array_2(y,x)) > refractory_period)
238 | ts_neig = ts_array_2(y-1:y+1,x-1:x+1);
239 | Q_i = sigma_i * min(ts - ts_neig(:))/1e6; % in second, depends on neighbour pixels
240 | Q_r = 0;
241 | elseif x > 1 && x < im_width && y > 1 && y < im_height && ((ts - ts_array_2(y,x)) <= refractory_period)
242 | ts_neig = ts_array_2(y-1:y+1,x-1:x+1);
243 | Q_i = sigma_i * min(ts - ts_neig(:))/1e6; % in second, depends on neighbour pixels
244 | Q_r = sigma_r * min(ts - ts_array_2(y,x))/1e6;
245 | else
246 | Q_i = 10; % border trust interpolation
247 | Q_r = 10;
248 | end
249 | P(y,x) = 1 ./ ((1 ./ P(y,x)) + R_inv_array(y,x) .* delta_t);
250 | P(y,x) = P(y,x) + sigma_p .* delta_t;
251 | P(y,x) = P(y,x) + Q_i + Q_r;
252 | ts_array_(y,x) = ts;
253 | ts_array_2(y,x) = ts;
254 |
255 | %% continuous-time interpolation
256 | if polarity < 1
257 | if ct_scale(y,x) >= min_ct_scale && ct_scale(y,x) <= max_ct_scale
258 | log_output_interp_t1(y,x) = log_output_interp_t1(y,x) + (-ct/ct_scale(y,x));
259 | log_output_interp_t2(y,x) = log_output_interp_t2(y,x) + (-ct/ct_scale(y,x));
260 | else
261 | log_output_interp_t1(y,x) = log_output_interp_t1(y,x) + (-ct);
262 | log_output_interp_t2(y,x) = log_output_interp_t2(y,x) + (-ct);
263 | end
264 | else
265 | if ct_scale(y,x) >= min_ct_scale && ct_scale(y,x) <= max_ct_scale
266 | log_output_interp_t1(y,x) = log_output_interp_t1(y,x) + (ct/ct_scale(y,x));
267 | log_output_interp_t2(y,x) = log_output_interp_t2(y,x) + (ct/ct_scale(y,x));
268 | else
269 | log_output_interp_t1(y,x) = log_output_interp_t1(y,x) + ct;
270 | log_output_interp_t2(y,x) = log_output_interp_t2(y,x) + ct;
271 | end
272 | end
273 | output_interp_t1 = exp(log_output_interp_t1(y,x)) - safety_offset;
274 | output_interp_t2 = exp(log_output_interp_t2(y,x)) - safety_offset;
275 |
276 | %% interpolation update
277 | if (ts >= img_now_ts_2) && (ts <= img_next_ts_1)
278 | weight = (ts - img_now_ts_2) / (img_next_ts_1 - img_now_ts_2);
279 | output_interp_mid = output_interp_t1 * (1 - weight) + output_interp_t2 * (weight);
280 | elseif (ts >= img_now_ts_1) && (ts < img_now_ts_2)
281 | % inside of the exposure time of the first image
282 | % no interpolation, use forward integration
283 | % ts cannot go beyond img_next_ts_1
284 | weight = 0;
285 | output_interp_mid = output_interp_t1;
286 | end
287 | log_intensity_now_(y,x) = log(output_interp_mid+safety_offset);
288 |
289 | %% high frame rate reconstruction - update globally with a certain framerate
290 | if (ts > t_next_publish_)
291 | %% compute R
292 | R_bar_inv = 1./(1./f_Q(img1_round_in_0255) .* (1 - weight) + 1./f_Q(img2_round_in_0255) .* weight);
293 | R_inv_array = R_bar_inv .* (img1_round_in_01 .* (1 - weight) + img2_round_in_01 .* weight + safety_offset).^2;
294 |
295 | %% gobal update
296 | delta_t = (ts - ts_array_)/1e6;
297 | C = (log_intensity_state_ - log_intensity_now_) ./ P;
298 | log_intensity_state_ = (1 ./ (1 ./ P + R_inv_array .* delta_t)) .* C + log_intensity_now_;
299 | P = 1 ./ ((1 ./ P) + R_inv_array .* delta_t);
300 | P = P + sigma_p .* delta_t;
301 | ts_array_(:) = ts;
302 |
303 | %% high frequency image output
304 | if output_high_frame_rate_flag
305 | output_img(log_intensity_now_,frame_idx,use_median_filter,safety_offset,post_process,high_frame_rate_folder,P,log_intensity_state_)
306 | frame_idx = frame_idx + 1;
307 | end
308 | ts_array_(:) = ts;
309 | t_next_publish_ = t_next_publish_ + frame_time;
310 | end
311 | end
312 | end
--------------------------------------------------------------------------------
/compute_ct_scale.m:
--------------------------------------------------------------------------------
1 | function ct_scale = compute_ct_scale(ct,exposure,log_deblur_image,time_image,events,frame)
2 | i_idx = frame;
3 | t1 = time_image(i_idx) - exposure(i_idx)/2;
4 | t2 = time_image(i_idx+1) + exposure(i_idx+1)/2;
5 | e_start_idx = find(events(:,1) >= t1,1,'first');
6 | e_end_idx = find(events(:,1) >= t2,1,'first');
7 |
8 | % only generate raw image
9 | x = events(:,2);
10 | y = events(:,3);
11 | p = events(:,4);
12 | sum_p = zeros(size(log_deblur_image,1),size(log_deblur_image,2));
13 |
14 | for i = e_start_idx:e_end_idx
15 | if p(i) < 1
16 | sum_p(y(i),x(i)) = sum_p(y(i),x(i)) - ct;
17 | else
18 | sum_p(y(i),x(i)) = sum_p(y(i),x(i)) + ct;
19 | end
20 | end
21 | ct_scale = sum_p ./ (log_deblur_image(:,:,2) - log_deblur_image(:,:,1));
22 | end
23 |
24 |
25 |
--------------------------------------------------------------------------------
/deblur_edge_left.m:
--------------------------------------------------------------------------------
1 | function output_deblur_1_log = deblur_edge_left(min_ct_scale,max_ct_scale,img_idx_now,time_image,exposure,events,image,c_deblur,safety_offset)
2 | output_deblur_1 = deblur_mid(min_ct_scale,max_ct_scale,img_idx_now,time_image(img_idx_now),exposure,events,image,c_deblur,safety_offset);
3 | output_deblur_1_log = log(output_deblur_1 + safety_offset);
4 | t1 = time_image(img_idx_now,1) - exposure/2;
5 | t2 = time_image(img_idx_now,1);
6 | e_start_idx = find(events(:,1) >= t1,1,'first');
7 | e_end_idx = find(events(:,1) >= t2,1,'first');
8 |
9 | for i_e = e_start_idx:e_end_idx
10 | xe = events(i_e,2); % [1 -> width]
11 | ye = events(i_e,3); % [1 -> height]
12 | polarity_e = events(i_e,4); % [-1, 1]
13 | if (polarity_e == 1)
14 | contrast_threshold_e = c_deblur;
15 | else
16 | contrast_threshold_e = -c_deblur;
17 | end
18 | output_deblur_1_log(ye,xe) = output_deblur_1_log(ye,xe) - contrast_threshold_e;
19 | end
20 | end
21 |
--------------------------------------------------------------------------------
/deblur_edge_outer.m:
--------------------------------------------------------------------------------
1 | function [output_deblur_t1,output_deblur_t2] = deblur_edge_outer(min_ct_scale,max_ct_scale,img_idx_now,time_image,exposure,events,image,c_deblur,safety_offset)
2 | %% outer edge deblur frame 1
3 | output_deblur_on_1 = deblur_mid(min_ct_scale,max_ct_scale,img_idx_now,time_image(img_idx_now),exposure,events,image,c_deblur,safety_offset);
4 | output_deblur_1_log = log(output_deblur_on_1+1); % add safety_offset 1
5 | t1 = time_image(img_idx_now) - exposure/2;
6 | t2 = time_image(img_idx_now);
7 | e_start_idx = find(events(:,1) >= t1,1,'first');
8 | e_end_idx = find(events(:,1) >= t2,1,'first');
9 |
10 | for i_e = e_start_idx:e_end_idx
11 | xe = events(i_e,2); % [1 -> width]
12 | ye = events(i_e,3); % [1 -> height]
13 | polarity_e = events(i_e,4); % [-1, 1]
14 | if (polarity_e == 1)
15 | contrast_threshold_e = c_deblur;
16 | else
17 | contrast_threshold_e = -c_deblur;
18 | end
19 | output_deblur_1_log(ye,xe) = output_deblur_1_log(ye,xe) - contrast_threshold_e;
20 | end
21 | output_deblur_t1 = exp(output_deblur_1_log)-safety_offset;
22 |
23 | %% outer edge deblur frame 2
24 | output_deblur_on_2 = deblur_mid(min_ct_scale,max_ct_scale,img_idx_now+1,time_image(img_idx_now+1),exposure,events,image,c_deblur,safety_offset);
25 | output_deblur_2_log = log(output_deblur_on_2+1);
26 | t1 = time_image(img_idx_now+1);
27 | t2 = time_image(img_idx_now+1) + exposure/2;
28 | e_start_idx = find(events(:,1) >= t1,1,'first');
29 | e_end_idx = find(events(:,1) >= t2,1,'first');
30 |
31 | for i_e = e_start_idx:e_end_idx
32 | xe = events(i_e,2); % [1 -> width]
33 | ye = events(i_e,3); % [1 -> height]
34 | polarity_e = events(i_e,4); % [-1, 1]
35 | if (polarity_e == 1)
36 | contrast_threshold_e = c_deblur;
37 | else
38 | contrast_threshold_e = -c_deblur;
39 | end
40 | output_deblur_2_log(ye,xe) = output_deblur_2_log(ye,xe) + contrast_threshold_e;
41 | end
42 | output_deblur_t2 = exp(output_deblur_2_log)-safety_offset;
43 | end
--------------------------------------------------------------------------------
/deblur_mid.m:
--------------------------------------------------------------------------------
1 | function [output_deblur_now] = deblur_mid(min_ct_scale,max_ct_scale,frame,time_image,exposure,events,image,c_deblur,safety_offset)
2 | height = size(image,1);
3 | width = size(image,2);
4 | ct_scale = ones(height,width); % don't consider ct scale in deblur
5 | delta_t = zeros(height,width);
6 | if numel(exposure) == 1
7 | t1 = time_image - exposure/2;
8 | t2 = time_image;
9 | t3 = time_image + exposure/2;
10 | else
11 | t1 = time_image - exposure(frame)/2;
12 | t2 = time_image;
13 | t3 = time_image + exposure(frame)/2;
14 | end
15 | e_start_idx = find(events(:,1) >= t1,1,'first');
16 | e_mid_idx = find(events(:,1) >= t2,1,'first');
17 | e_end_idx = find(events(:,1) >= t3,1,'first');
18 | Et = zeros(height,width);
19 | sum_t_Et = zeros(height,width);
20 | et_array = zeros(height,width) + t2;
21 | T_deblur = events(e_end_idx,1) - events(e_start_idx,1);
22 |
23 | % define events for deblur
24 | t = events(e_start_idx:e_end_idx,1);
25 | x = events(e_start_idx:e_end_idx,2);
26 | y = events(e_start_idx:e_end_idx,3);
27 | p = events(e_start_idx:e_end_idx,4);
28 | for i = e_mid_idx-e_start_idx+1:-1:e_start_idx-e_start_idx+1
29 | if p(i) ~= 1
30 | if ct_scale(y(i),x(i)) > min_ct_scale && ct_scale(y(i),x(i)) < max_ct_scale
31 | Et(y(i),x(i)) = Et(y(i),x(i)) + c_deblur / ct_scale(y(i),x(i));
32 | else
33 | Et(y(i),x(i)) = Et(y(i),x(i)) + c_deblur;
34 | end
35 | else
36 | if ct_scale(y(i),x(i)) > min_ct_scale && ct_scale(y(i),x(i)) < max_ct_scale
37 | Et(y(i),x(i)) = Et(y(i),x(i)) - c_deblur / ct_scale(y(i),x(i));
38 | else
39 | Et(y(i),x(i)) = Et(y(i),x(i)) - c_deblur;
40 | end
41 | end
42 | delta_t(y(i),x(i)) = abs(t(i) - et_array(y(i),x(i)));
43 | sum_t_Et(y(i),x(i)) = sum_t_Et(y(i),x(i)) + delta_t(y(i),x(i)) .* exp(Et(y(i),x(i)));
44 | et_array(y(i),x(i)) = t(i);
45 | end
46 |
47 | sum_t_Et = sum_t_Et + abs(et_array - t1) .* exp(Et);
48 | Et = zeros(height,width);
49 | et_array = zeros(height,width) + t2;
50 |
51 | for i = e_mid_idx-e_start_idx+1:e_end_idx-e_start_idx+1
52 | if p(i) ~= 1
53 | if ct_scale(y(i),x(i)) > min_ct_scale && ct_scale(y(i),x(i)) < max_ct_scale
54 | Et(y(i),x(i)) = Et(y(i),x(i)) - c_deblur / ct_scale(y(i),x(i));
55 | else
56 | Et(y(i),x(i)) = Et(y(i),x(i)) - c_deblur;
57 | end
58 | else
59 | if ct_scale(y(i),x(i)) > min_ct_scale && ct_scale(y(i),x(i)) < max_ct_scale
60 | Et(y(i),x(i)) = Et(y(i),x(i)) + c_deblur / ct_scale(y(i),x(i));
61 | else
62 | Et(y(i),x(i)) = Et(y(i),x(i)) + c_deblur;
63 | end
64 | end
65 | delta_t(y(i),x(i)) = abs(et_array(y(i),x(i)) - t(i));
66 | sum_t_Et(y(i),x(i)) = sum_t_Et(y(i),x(i)) + delta_t(y(i),x(i)) .* exp(Et(y(i),x(i)));
67 | et_array(y(i),x(i)) = t(i);
68 | end
69 | sum_t_Et = sum_t_Et + abs(t3 - et_array) .* exp(Et);
70 | motion = sum_t_Et / T_deblur;
71 | deblur_image = double(image(:,:,frame))/255 + safety_offset;
72 | output_deblur(:,:,frame) = (deblur_image ./ motion - safety_offset);
73 | output_deblur_now = output_deblur(:,:,frame);
74 | end
75 |
76 |
--------------------------------------------------------------------------------
/figures/ahdr-dataset_.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ziweiWWANG/AKF/41ed371e2a1397733670ed4b2e9e2279a78b4149/figures/ahdr-dataset_.png
--------------------------------------------------------------------------------
/figures/hdr-dataset.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ziweiWWANG/AKF/41ed371e2a1397733670ed4b2e9e2279a78b4149/figures/hdr-dataset.png
--------------------------------------------------------------------------------
/figures/tpami_video_thumbnail.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ziweiWWANG/AKF/41ed371e2a1397733670ed4b2e9e2279a78b4149/figures/tpami_video_thumbnail.png
--------------------------------------------------------------------------------
/figures/video_thumbnail.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ziweiWWANG/AKF/41ed371e2a1397733670ed4b2e9e2279a78b4149/figures/video_thumbnail.png
--------------------------------------------------------------------------------
/initialization.m:
--------------------------------------------------------------------------------
1 | function [ts_array_,ts_array_2,frame_idx,R,C,weight]...
2 | = initialization(height, width)
3 | ts_array_ = zeros(height, width);
4 | ts_array_2 = zeros(height, width);
5 | R = ones(height,width);
6 | C = zeros(height,width);
7 | frame_idx = 0;
8 | weight = 0;
9 | end
--------------------------------------------------------------------------------
/output_img.m:
--------------------------------------------------------------------------------
1 | function output_img(log_intensity_now_,img_idx_now,use_median_filter,safety_offset,post_process,folder,P,log_intensity_state_)
2 | td_img = exp(log_intensity_state_) - safety_offset;
3 | if use_median_filter
4 | td_img = medfilt2(td_img,[3,3]);
5 | end
6 |
7 | for ii = 1:size(td_img,1)
8 | for jj = 1:size(td_img,2)
9 | if (P(ii,jj)) > 100 || isnan(P(ii,jj))
10 | P(ii,jj) = 0.25;
11 | td_img(ii,jj) = exp(log_intensity_now_(ii,jj)) - safety_offset;
12 | elseif (P(ii,jj)) < 0
13 | P(ii,jj) = 0;
14 | end
15 | end
16 | end
17 | td_img((td_img>3) | (td_img<-3)) = NaN;
18 | td_img(isnan(td_img)) = exp(log_intensity_now_(isnan(td_img))) - safety_offset;
19 |
20 | %% post process
21 | if post_process == 1
22 | tmp = sort(td_img(:));
23 | intensity_lower_bound = tmp(floor(numel(tmp)*0.03));
24 | intensity_upper_bound = tmp(ceil(numel(tmp)*0.97));
25 | td_img = (td_img - intensity_lower_bound) / (intensity_upper_bound - intensity_lower_bound);
26 | elseif post_process == 2 % extremely bright view
27 | intensity_lower_bound = -0.001;
28 | intensity_upper_bound = 1.2;
29 | td_img = (td_img - intensity_lower_bound) / (intensity_upper_bound - intensity_lower_bound);
30 | elseif post_process == 3 % extremely dark view
31 | intensity_lower_bound = -0.001;
32 | intensity_upper_bound = 0.4;
33 | td_img = (td_img - intensity_lower_bound) / (intensity_upper_bound - intensity_lower_bound);
34 | elseif post_process == 4 % use histogram equalization for display
35 | % save raw output without any post-processing for evalution
36 | save(sprintf([folder '/image_' num2str(img_idx_now) '.mat']), 'td_img');
37 | % adaptive histogram equalization for display only
38 | td_img = adapthisteq(td_img);
39 | end
40 |
41 | %% write images
42 | imwrite(td_img, sprintf([folder '/image_' num2str(img_idx_now) '.png']));
43 | end
--------------------------------------------------------------------------------
/run_akf.m:
--------------------------------------------------------------------------------
1 | clear
2 | close all
3 |
4 | %% Dataset
5 | % you can download example datasets:
6 | % public dataset: city_09d_150_200, boxes_6dof_500_800, night_drive, interlaken_01a_1_150.
7 | % our HDR dataset: carpark1, carpark2, tree1, tree2, tree3, city.
8 | % or you can process and use your own dataset
9 | DataName = 'city_09d_150_200';
10 |
11 | %% There are a few parameters that can be specified by users:
12 | % deblur_option: true for deblur and false for no deblur
13 | deblur_option = true;
14 | % framerate: the frame rate of the output image sequence in Hz
15 | framerate = 300;
16 | % use_median_filter: true for apply a 3-by-3 median filter to the output
17 | % images, false for not apply
18 | use_median_filter = false;
19 | % output_high_frame_rate_flag:
20 | % true: output images of the pre-defined framerate,
21 | % false: output images of the frame intensity framerate
22 | output_high_frame_rate_flag = false;
23 | % sigma_p: the process noise parameter
24 | sigma_p = 0.005;
25 | % sigma_i: the isolated noise parameter
26 | sigma_i = 0.03;
27 | % sigma_r: the refractory noise parameter
28 | sigma_r = 0.05;
29 | % refractory_period: the refractory period in microsecond. It models the
30 | % circuit limitations in each pixel of an event camera limit the response
31 | % time of events
32 | refractory_period = 1*10^4;
33 | % min_ct_scale: the minimal value for the contrast threshold scaling factor
34 | min_ct_scale = 0.6;
35 | % max_ct_scale: the maximum value for the contrast threshold scaling factor
36 | max_ct_scale = 100;
37 | % p_ini: the initial value for state covariance P
38 | p_ini = 0.09;
39 |
40 | %% run akf reconstruction
41 | akf_reconstruction(DataName, deblur_option, ...
42 | framerate, use_median_filter, output_high_frame_rate_flag,...
43 | sigma_p, sigma_i, sigma_r,refractory_period, min_ct_scale, max_ct_scale,...
44 | p_ini)
45 |
46 |
47 |
--------------------------------------------------------------------------------