├── LICENSE.md ├── README.md ├── data_samples └── interp │ └── office_zigzag │ ├── 000.png │ ├── 001.png │ ├── 002.png │ ├── 003.png │ ├── 004.png │ ├── 005.png │ ├── 006.png │ ├── 007.png │ ├── 008.png │ ├── 009.png │ ├── 010.png │ └── info.txt ├── main.py └── src ├── config.py ├── simulator.py ├── simulator_utils.py └── visualize.py /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Songnan Lin 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DVS-Voltmeter 2 | 3 | Code repo for the paper 'DVS-Voltmeter: Stochastic Process-based Event Simulator for Dynamic Vision Sensors'. 4 | 5 | ## Prerequisites 6 | 7 | ``` 8 | easydict == 1.9 9 | pytorch >= 1.8 10 | numpy >= 1.20.1 11 | opencv-python == 4.5.1.48 12 | tqdm == 4.49.0 13 | ``` 14 | 15 | The code may be compatible with lower versions, while the aforementioned ones have been tested. 16 | 17 | ## Dataset 18 | 19 | The sample input video frames to try DVS-Voltmeter with are in [samples](https://drive.google.com/drive/folders/1puqNjrdDrk69RsSsNL3CyERtJSJFmX5p?usp=sharing) on google drive. Download samples and put them in ***data_samples/interp*** folder. 20 | 21 | To simulate events from other videos, put high frame-rate videos (can be obtained by video interpolation) in ***data_samples/interp*** folder and modify the data index tree as following: 22 | 23 | ``` 24 | ├── [data_samples] 25 | │ ├── interp 26 | │ │ ├── videoname1 27 | │ │ │ ├── info.txt 28 | │ │ │ ├── framename11.png 29 | │ │ │ ├── framename12.png 30 | │ │ ├── videoname2 31 | │ │ │ ├── info.txt 32 | │ │ │ ├── framename21.png 33 | │ │ │ ├── framename22.png 34 | ``` 35 | 36 | The video info file ***info.txt*** records the path and timestamp ($\mu s$) of each frame. 37 | 38 | ## Usage 39 | 40 | 1. Configure the src/config.py file. For detailed configuration, please refer to the config file. 41 | 2. Run ```python main.py``` 42 | 43 | ## Biblography 44 | 45 | If you find our work useful, please use the following citation. 46 | 47 | ``` 48 | @inproceedings{lin2022dvsvoltmeter, 49 | title={DVS-Voltmeter: Stochastic Process-based Event Simulator for Dynamic Vision Sensors}, 50 | author={Lin, Songnan and Ma, Ye and Guo, Zhenhua and Wen, Bihan}, 51 | booktitle={ECCV}, 52 | year={2022} 53 | } 54 | ``` 55 | 56 | ## License 57 | 58 | MIT License 59 | -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/000.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/001.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/001.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/002.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/002.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/003.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/003.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/004.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/004.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/005.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/005.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/006.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/007.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/008.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/008.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/009.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/009.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/010.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Lynn0306/DVS-Voltmeter/9d86353ecfaaa1af82b9eb66b162f0bf80cd4861/data_samples/interp/office_zigzag/010.png -------------------------------------------------------------------------------- /data_samples/interp/office_zigzag/info.txt: -------------------------------------------------------------------------------- 1 | ./data_samples/interp/office_zigzag/000.png 000000013767 2 | ./data_samples/interp/office_zigzag/001.png 000000018173 3 | ./data_samples/interp/office_zigzag/002.png 000000022580 4 | ./data_samples/interp/office_zigzag/003.png 000000026986 5 | ./data_samples/interp/office_zigzag/004.png 000000031393 6 | ./data_samples/interp/office_zigzag/005.png 000000035799 7 | ./data_samples/interp/office_zigzag/006.png 000000040206 8 | ./data_samples/interp/office_zigzag/007.png 000000044612 9 | ./data_samples/interp/office_zigzag/008.png 000000049019 10 | ./data_samples/interp/office_zigzag/009.png 000000053425 11 | ./data_samples/interp/office_zigzag/010.png 000000057832 -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | #! python3 2 | # -*- encoding: utf-8 -*- 3 | ''' 4 | @File : main.py 5 | @Time : 2022/7/12 17:30 6 | @Author : Songnan Lin, Ye Ma 7 | @Contact : songnan.lin@ntu.edu.sg, my17@tsinghua.org.cn 8 | @Note : 9 | @inproceedings{lin2022dvsvoltmeter, 10 | title={DVS-Voltmeter: Stochastic Process-based Event Simulator for Dynamic Vision Sensors}, 11 | author={Lin, Songnan and Ma, Ye and Guo, Zhenhua and Wen, Bihan}, 12 | booktitle={ECCV}, 13 | year={2022} 14 | } 15 | ''' 16 | 17 | import argparse 18 | import os 19 | import numpy as np 20 | import cv2 21 | import tqdm 22 | from src.config import cfg 23 | from src.simulator import EventSim 24 | from src.visualize import events_to_voxel_grid, visual_voxel_grid 25 | 26 | def get_args_from_command_line(): 27 | parser = argparse.ArgumentParser(description='Parser of Runner of Network') 28 | parser.add_argument('--camera_type', type=str, help='Camera type, such as DVS346', default='DVS346') 29 | parser.add_argument('--model_para', type=float, nargs='+', help='Set parameters for a specific camera type', default=None) 30 | parser.add_argument('--input_dir', type=str, help='Set dataset root_path', default=None) 31 | parser.add_argument('--output_dir', type=str, help='Set output path', default=None) 32 | args = parser.parse_args() 33 | return args 34 | 35 | def integrate_cfg(cfg, command_line_args): 36 | args = command_line_args 37 | cfg.SENSOR.CAMERA_TYPE = args.camera_type if args.camera_type is not None else cfg.SENSOR.CAMERA_TYPE 38 | cfg.SENSOR.K = args.model_para if args.model_para is not None else cfg.SENSOR.K 39 | cfg.DIR.IN_PATH = args.input_dir if args.input_dir is not None else cfg.DIR.IN_PATH 40 | cfg.DIR.OUT_PATH = args.output_dir if args.output_dir is not None else cfg.DIR.OUT_PATH 41 | if cfg.SENSOR.K is None or len(cfg.SENSOR.K) != 6: 42 | raise Exception('No model parameters given for sensor type %s' % cfg.SENSOR.CAMERA_TYPE) 43 | print(cfg) 44 | return cfg 45 | 46 | def is_valid_dir(dirs): 47 | return os.path.exists(os.path.join(dirs, 'info.txt')) 48 | 49 | def process_dir(cfg, file_info, video_name): 50 | indir = os.path.join(cfg.DIR.IN_PATH, video_name) 51 | outdir = os.path.join(cfg.DIR.OUT_PATH, video_name) 52 | print(f"Processing folder {indir}... Generating events in file {outdir}") 53 | 54 | # file info 55 | file_timestamps_us = [int(info_i.split()[1]) for info_i in file_info] 56 | file_paths = [info_i.split()[0] for info_i in file_info] 57 | 58 | # set simulator 59 | sim = EventSim(cfg=cfg, output_folder=cfg.DIR.OUT_PATH, video_name=video_name) 60 | 61 | # process 62 | pbar = tqdm.tqdm(total=len(file_paths)) 63 | num_events, num_on_events, num_off_events = 0, 0, 0 64 | events = [] 65 | for i in range(0, len(file_paths)): 66 | timestamp_us = file_timestamps_us[i] 67 | image = cv2.imread(file_paths[i], cv2.IMREAD_GRAYSCALE) 68 | 69 | # event generation!!! 70 | event = sim.generate_events(image, timestamp_us) 71 | 72 | if event is not None: 73 | events.append(event) 74 | num_events += event.shape[0] 75 | num_on_events += np.sum(event[:, -1] == 1) 76 | num_off_events += np.sum(event[:, -1] == 0) 77 | 78 | pbar.set_description(f"Events generated: {num_events}") 79 | pbar.update(1) 80 | 81 | events = np.concatenate(events, axis=0) 82 | np.savetxt(os.path.join(cfg.DIR.OUT_PATH, video_name + '.txt'), events, fmt='%1.0f') 83 | sim.reset() 84 | 85 | 86 | if __name__ == "__main__": 87 | args = get_args_from_command_line() 88 | cfg = integrate_cfg(cfg, args) 89 | 90 | video_list = sorted(os.listdir(cfg.DIR.IN_PATH)) 91 | for video_i in video_list: 92 | video_i_path = os.path.join(cfg.DIR.IN_PATH, video_i) 93 | os.makedirs(os.path.join(cfg.DIR.OUT_PATH, video_i), exist_ok=True) 94 | 95 | if is_valid_dir(video_i_path): 96 | # video info 97 | with open(os.path.join(cfg.DIR.IN_PATH, video_i, 'info.txt'), 'r') as f: 98 | video_info = f.readlines() 99 | # simulation 100 | process_dir(cfg=cfg, file_info=video_info, video_name=video_i) 101 | -------------------------------------------------------------------------------- /src/config.py: -------------------------------------------------------------------------------- 1 | #! python3 2 | # -*- encoding: utf-8 -*- 3 | ''' 4 | @File : config.py 5 | @Time : 2022/7/12 18:17 6 | @Author : Songnan Lin, Ye Ma 7 | @Contact : songnan.lin@ntu.edu.sg, my17@tsinghua.org.cn 8 | @Note : 9 | @inproceedings{lin2022dvsvoltmeter, 10 | title={DVS-Voltmeter: Stochastic Process-based Event Simulator for Dynamic Vision Sensors}, 11 | author={Lin, Songnan and Ma, Ye and Guo, Zhenhua and Wen, Bihan}, 12 | booktitle={ECCV}, 13 | year={2022} 14 | } 15 | ''' 16 | 17 | from easydict import EasyDict as edict 18 | 19 | __C = edict() 20 | cfg = __C 21 | 22 | # SENSOR 23 | __C.SENSOR = edict() 24 | __C.SENSOR.CAMERA_TYPE = 'DVS346' 25 | 26 | __C.SENSOR.K = None 27 | if cfg.SENSOR.CAMERA_TYPE == 'DVS346': 28 | __C.SENSOR.K = [0.00018 * 29250, 20, 0.0001, 1e-7, 5e-9, 0.00001] 29 | elif cfg.SENSOR.CAMERA_TYPE == 'DVS240': 30 | __C.SENSOR.K = [0.000094 * 47065, 23, 0.0002, 1e-7, 5e-8, 0.00001] 31 | 32 | 33 | # Directories 34 | __C.DIR = edict() 35 | __C.DIR.IN_PATH = 'data_samples/interp/' 36 | __C.DIR.OUT_PATH = 'data_samples/output/' 37 | 38 | 39 | # Visualize 40 | __C.Visual = edict() 41 | __C.Visual.FRAME_STEP = 5 42 | -------------------------------------------------------------------------------- /src/simulator.py: -------------------------------------------------------------------------------- 1 | #! python3 2 | # -*- encoding: utf-8 -*- 3 | ''' 4 | @File : simulator.py 5 | @Time : 2022/7/12 22:42 6 | @Author : Songnan Lin, Ye Ma 7 | @Contact : songnan.lin@ntu.edu.sg, my17@tsinghua.org.cn 8 | @Note : 9 | @inproceedings{lin2022dvsvoltmeter, 10 | title={DVS-Voltmeter: Stochastic Process-based Event Simulator for Dynamic Vision Sensors}, 11 | author={Lin, Songnan and Ma, Ye and Guo, Zhenhua and Wen, Bihan}, 12 | booktitle={ECCV}, 13 | year={2022} 14 | } 15 | ''' 16 | 17 | import os 18 | import numpy as np 19 | import torch 20 | from .simulator_utils import event_generation 21 | 22 | class EventSim(object): 23 | def __init__( 24 | self, 25 | cfg, 26 | output_folder: str = None, video_name: str = None 27 | ): 28 | """ 29 | Parameters 30 | ---------- 31 | cfg: config 32 | output_folder: str 33 | folder of output data file 34 | video_name: str 35 | name of input video / output data file 36 | """ 37 | 38 | # set parameters in model 39 | self.k1, self.k2, self.k3, self.k4, self.k5, self.k6 \ 40 | = cfg.SENSOR.K[0], cfg.SENSOR.K[1], cfg.SENSOR.K[2], cfg.SENSOR.K[3], cfg.SENSOR.K[4], cfg.SENSOR.K[5] 41 | 42 | # output file 43 | path = os.path.join(output_folder, video_name + '.npy') 44 | 45 | # init 46 | self.reset() 47 | 48 | def reset(self): 49 | ''' 50 | resets so that next use will reinitialize the base frame 51 | ''' 52 | self.baseFrame = None 53 | self.t_previous = None # time of previous frame 54 | 55 | def generate_events( 56 | self, new_frame: np.ndarray, 57 | t_frame: int) -> np.ndarray: 58 | """ 59 | Notes: 60 | Compute events in new frame. 61 | 62 | Parameters 63 | new_frame: np.ndarray 64 | [height, width] 65 | t_frame: int 66 | timestamp of new frame in us (1e6) 67 | 68 | Returns 69 | events: np.ndarray if any events, else None 70 | [N, 4], each row contains [timestamp (us), x cordinate, y cordinate, sign of event]. 71 | """ 72 | 73 | new_frame = torch.from_numpy(new_frame).to(torch.float64) 74 | t_frame = float(t_frame) 75 | 76 | # ------------------ 77 | # Initialization 78 | if self.baseFrame is None: 79 | self.baseFrame = new_frame 80 | self.t_previous = t_frame 81 | self.delta_vd_res = torch.zeros_like(new_frame) # initialize residual voltage change $\Delta V_d^{res}$ 82 | self.t_now = torch.ones_like(new_frame, dtype=torch.float) * self.t_previous 83 | self.thres_off = torch.ones_like(new_frame) 84 | self.thres_on = torch.ones_like(new_frame) 85 | return None 86 | 87 | if t_frame <= self.t_previous: 88 | raise ValueError("this frame time={} must be later than " 89 | "previous frame time={}".format(t_frame, self.t_previous)) 90 | 91 | # ------------------ 92 | # Calculte distribution parameters of Brownian Motion with Drift in Eq. (10)(11) 93 | delta_light = (new_frame - self.baseFrame) # delta L 94 | avg_light = (new_frame + self.baseFrame) / 2.0 # average L 95 | denominator = 1 / (avg_light + self.k2) 96 | mu_clean = (self.k1 * delta_light / (t_frame - self.t_previous)) * denominator 97 | mu = mu_clean + self.k4 + self.k5 * avg_light 98 | var_clean = (self.k3 * torch.sqrt(avg_light)) * denominator 99 | var = var_clean + self.k6 100 | ori_shape = mu.shape 101 | 102 | # ------------------ 103 | # Event Generation! 104 | e_t, e_x, e_y, e_p, e_dvd = event_generation(self.thres_on, self.thres_off, 105 | mu, var, 106 | self.delta_vd_res, self.t_now, t_frame) 107 | if e_t.shape[0] > 0: 108 | e_t = torch.round(e_t).int() 109 | event_tensor = torch.stack([e_t, e_x, e_y, e_p], dim=1) 110 | _, sorted_idx = torch.sort(e_t) 111 | event_tensor = event_tensor[sorted_idx, :] 112 | event_tensor = event_tensor.contiguous().numpy().astype(np.int32) 113 | else: 114 | event_tensor = None 115 | 116 | # Update 117 | self.delta_vd_res = e_dvd.reshape(ori_shape) 118 | self.t_now = torch.ones_like(self.t_now, device=self.t_now.device) * t_frame 119 | self.t_previous = t_frame 120 | self.baseFrame = new_frame 121 | 122 | return event_tensor 123 | -------------------------------------------------------------------------------- /src/simulator_utils.py: -------------------------------------------------------------------------------- 1 | #! python3 2 | # -*- encoding: utf-8 -*- 3 | ''' 4 | @File : simulator_utils.py 5 | @Time : 2022/7/13 01:27 6 | @Author : Songnan Lin, Ye Ma 7 | @Contact : songnan.lin@ntu.edu.sg, my17@tsinghua.org.cn 8 | @Note : Sample events by alternating between polarity selection and timestamp sampling, in Algorithm 1, Section 4 9 | @inproceedings{lin2022dvsvoltmeter, 10 | title={DVS-Voltmeter: Stochastic Process-based Event Simulator for Dynamic Vision Sensors}, 11 | author={Lin, Songnan and Ma, Ye and Guo, Zhenhua and Wen, Bihan}, 12 | booktitle={ECCV}, 13 | year={2022} 14 | } 15 | ''' 16 | 17 | import torch 18 | import numpy as np 19 | import random 20 | import math 21 | 22 | 23 | def sample_non_c_zero(ep, c_in, sigma_in): 24 | # 取inverse gaussian的分布 25 | # https://en.wikipedia.org/wiki/Inverse_Gaussian_distribution 26 | pos_c_position = c_in > 0 27 | neg_c_position = c_in < 0 28 | 29 | X = torch.empty_like(ep, dtype=ep.dtype, 30 | device=ep.device) 31 | 32 | # same direction 33 | X_Pos = torch.randn(size=(torch.sum(pos_c_position),), dtype=ep.dtype, device=ep.device) 34 | X[pos_c_position] = X_Pos 35 | 36 | # opposite direction # why? 37 | ep_neg = ep[neg_c_position] 38 | c_neg = c_in[neg_c_position] 39 | sigma_in_neg = sigma_in[neg_c_position] 40 | x_max_thres = -1 * torch.sqrt(-4 * ep_neg * 41 | c_neg / torch.pow(sigma_in_neg, 2.0)) 42 | sample_mean = torch.zeros_like(x_max_thres, device=x_max_thres.device) 43 | sample_sigma = torch.ones_like(x_max_thres, device=x_max_thres.device) 44 | sample_inf = sample_sigma * (-1) * np.inf 45 | x_truncated = sample_truncated_normal( 46 | sample_mean, sample_sigma, sample_inf, x_max_thres) 47 | X[neg_c_position] = x_truncated 48 | 49 | mean = ep / c_in 50 | lambda_ig = torch.pow(ep / sigma_in, 2.0) 51 | scale = lambda_ig 52 | mu = mean / 2 / scale 53 | 54 | Y = mean * X * X 55 | Z = (4*scale*Y + Y * Y) 56 | X = mean + mu * (Y - torch.sqrt(Z)) 57 | 58 | U = torch.empty_like(ep, device=ep.device) 59 | U.uniform_() 60 | out = torch.where(U > mean/(mean + X), mean * mean / X, X) 61 | return out 62 | 63 | 64 | def sample_levy(c, mu=0): 65 | # standard brown movement without drift 66 | # 分布服从c=(alpha/sigma)^2 的levy分布,详情请看 67 | # https://en.wikipedia.org/wiki/Inverse_Gaussian_distribution 68 | # https://en.wikipedia.org/wiki/L%C3%A9vy_distribution#Random_sample_generation 69 | u = torch.empty_like(c, device=c.device) 70 | u.uniform_() 71 | ev = torch.erfinv(1-u) 72 | out = c / torch.pow(ev, 2) + mu 73 | return out 74 | 75 | 76 | def sample_IG_torch(episilon: torch.Tensor, c: torch.Tensor, sigma: torch.Tensor): 77 | assert isinstance(episilon, torch.Tensor) 78 | assert isinstance(c, torch.Tensor) 79 | delta_t_tensor = torch.zeros_like(episilon, device=episilon.device) 80 | 81 | # 当有没有漂移的时候有不同分布,有漂移是inverse gaussian,没有是Levy 82 | # 以下是levy代码 83 | zeros_mask = c == 0.0 84 | ep_zeros_c = episilon[zeros_mask] 85 | sigma_zeros_c = sigma[zeros_mask] 86 | scale_levy = torch.pow(ep_zeros_c / sigma_zeros_c, 2.0) 87 | delta_t_c_zero = sample_levy(scale_levy).reshape(-1) 88 | 89 | # where_non_zeros = torch.nonzero(torch.logical_not(zeros_mask)) 90 | where_non_zeros = ~zeros_mask 91 | ep_non_zeros = episilon[where_non_zeros] 92 | c_non_zeros = c[where_non_zeros] 93 | sigma_non_zeros = sigma[where_non_zeros] 94 | delta_t_c_non_zero = sample_non_c_zero( 95 | ep_non_zeros, c_non_zeros, sigma_non_zeros).reshape(-1) 96 | 97 | delta_t_tensor[zeros_mask] = delta_t_c_zero 98 | delta_t_tensor[where_non_zeros] = delta_t_c_non_zero 99 | 100 | return delta_t_tensor 101 | 102 | 103 | def test_scipy(n: int, epsilon: np.ndarray, c: np.ndarray): 104 | # pdf = invgauss.rvs(c, size=n) 105 | # pdf = pdf 106 | # geninvgauss() 107 | mean = epsilon / c 108 | scale = epsilon ** 2 109 | mu = mean / 2 / scale 110 | x = np.random.normal(size=(n,)) 111 | u = np.random.uniform(size=(n,)) 112 | 113 | y = mean * x * x 114 | x = mean + mu * (y - np.sqrt(4*scale*y + y*y)) 115 | print(np.sum(np.isnan(x))) 116 | b = np.where(u > mean/(mean + x), mean*mean/x, x) 117 | b_w = np.where(np.logical_not(np.isnan(b))) 118 | b = b[b_w] 119 | print(b.shape) 120 | return b 121 | 122 | 123 | def sample_truncated_normal(mean: torch.Tensor, scale: torch.Tensor, a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: 124 | """[summary] 125 | [1]https://discuss.pytorch.org/t/implementing-truncated-normal-initializer/4778/21 126 | [2]The Truncated Normal Distribution 127 | Args: 128 | mean (torch.Tensor): [description] 129 | scale (torch.Tensor): [description] 130 | a (torch.Tensor): [description] 131 | b (torch.Tensor): [description] 132 | 133 | Returns: 134 | torch.Tensor: [description] 135 | """ 136 | assert mean.shape == scale.shape 137 | assert scale.shape == a.shape 138 | assert a.shape == b.shape 139 | 140 | def normal_cdf(v, mu, sigma): 141 | cdf = 0.5 * (1 + torch.erf((v - mu) * 142 | sigma.reciprocal() / math.sqrt(2))) 143 | return cdf 144 | 145 | uni = torch.empty_like(mean).uniform_() 146 | alpha_cdf = normal_cdf(a, mean, scale) 147 | beta_cdf = normal_cdf(b, mean, scale) 148 | p = alpha_cdf + (beta_cdf - alpha_cdf) * uni 149 | v = (2 * p - 1).clamp(-1, 1) 150 | out = mean + scale * math.sqrt(2) * torch.erfinv(v) 151 | out = torch.where(out < a, a, out) 152 | out = torch.where(out > b, b, out) 153 | return out 154 | 155 | 156 | def gaussian_cdf(x, mu=0.0, sigma=1.0): 157 | fai = (torch.erf((x - mu) / sigma / math.sqrt(2)) + 1) * 0.5 158 | return fai 159 | 160 | 161 | def inverse_gaussian_cdf_diff(x1, x2, lamb, mu): 162 | # assume x1 < x2 163 | k = 2 * lamb / mu 164 | 165 | lx1 = torch.sqrt(lamb / x1) 166 | x1_1 = lx1 * (x1 / mu - 1) 167 | x1_2 = -1 * lx1 * (x1 / mu + 1) 168 | 169 | lx2 = torch.sqrt(lamb / x2) 170 | x2_1 = lx2 * (x2 / mu - 1) 171 | x2_2 = -1 * lx2 * (x2 / mu + 1) 172 | 173 | # log1 = torch.log(gaussian_cdf(x1)) 174 | # log2 = torch.log(gaussian_cdf(x2)) + k 175 | 176 | # cdf = log1 + torch.log1p(torch.exp(log2-log1)) 177 | # cdf = torch.exp(cdf) 178 | part1 = gaussian_cdf(x2_1) - gaussian_cdf(x1_1) 179 | part2 = gaussian_cdf(x2_2) - gaussian_cdf(x1_2) 180 | part2_log = torch.log(part2) + k 181 | part2 = torch.exp(part2_log) 182 | cdf = part1 + part2 183 | cdf = torch.where(x <= 0, torch.zeros_like(x), cdf) 184 | return cdf 185 | 186 | 187 | def inverse_gaussian_pdf(x, lamb, mu): 188 | expin = -1 * lamb * torch.pow(x - mu, 2) / 2 / torch.pow(mu, 2) / x 189 | k = torch.sqrt(lamb / 2 / math.pi / torch.pow(x, 3)) 190 | return k * torch.exp(expin) 191 | 192 | 193 | def ig_prob_a_b_NC(a, b, lamb, mu, h: int = 4): 194 | # using Newton-Cotes equations 195 | if h == 1: 196 | para = [0.5, 0.5] 197 | elif h == 2: 198 | para = [1.0/6, 2.0/3, 1.0/6] 199 | elif h == 3: 200 | para = [0.125, 0.375, 0.375, 0.125] 201 | elif h == 4: 202 | para = [7/90.0, 16/45, 2/15, 16/45, 7/90] 203 | else: 204 | # if h != 4: 205 | raise NotImplementedError('h> 4 methods not implemented.') 206 | a = torch.clamp_min(a, 0) 207 | sub = b - a 208 | inter_tensors = [a] 209 | inter_tensors.extend([a + sub * i / h for i in range(h-1)]) 210 | inter_tensors.append(b) 211 | 212 | assert len(inter_tensors) == len(para) == h+1 213 | 214 | out = torch.zeros_like(a) 215 | for para_single, inter_tensor_single in zip(para, inter_tensors): 216 | out = out + para_single * \ 217 | inverse_gaussian_pdf(inter_tensor_single, lamb, mu) 218 | out = out * sub 219 | return out 220 | 221 | 222 | def event_generation(ep_on, ep_off, c, sigma, delta_vd_legacy, start_t, end_t, x=None, y=None): 223 | assert ep_on.shape == ep_off.shape 224 | assert ep_on.shape == c.shape 225 | assert ep_on.shape == delta_vd_legacy.shape 226 | delta_vd_legacy = delta_vd_legacy.double() 227 | 228 | if x is None: 229 | assert y is None # stands for 2 dim tensors 230 | assert len(ep_on.shape) == 2 231 | h, w = ep_on.shape 232 | h = torch.arange(h, device=ep_on.device) 233 | w = torch.arange(w, device=ep_on.device) 234 | yy, xx = torch.meshgrid(h, w) 235 | x = xx.reshape(-1) 236 | y = yy.reshape(-1) 237 | 238 | if len(ep_on.shape) == 2: 239 | ep_on = ep_on.reshape(-1) 240 | ep_off = ep_off.reshape(-1) 241 | c = c.reshape(-1) 242 | delta_vd_legacy = delta_vd_legacy.reshape(-1) 243 | start_t = start_t.reshape(-1).to(torch.float64) 244 | sigma = sigma.reshape(-1) 245 | 246 | # first on probs 247 | # 判断是先触发ON/OFF事件 248 | # ref: 随机过程 249 | ep_on_real = ep_on - delta_vd_legacy # ep to trigger positive OFF 250 | ep_off_real = ep_off + delta_vd_legacy # absolute ep to trigger negative ON 251 | sigma_squared = torch.pow(sigma, 2.0) 252 | exp_2uB = torch.exp(2 * c * ep_off_real / sigma_squared) 253 | exp_2uA = torch.exp(-2 * c * ep_on_real / sigma_squared) 254 | p_first_on = (exp_2uB - 1) / (exp_2uB - exp_2uA) # trigger OFF(A) before ON(-B) 255 | p_first_on = torch.where(torch.isnan(p_first_on), torch.ones_like( 256 | p_first_on, device=p_first_on.device), p_first_on) 257 | p_first_on = torch.where(c == 0, torch.ones_like( 258 | p_first_on, device=p_first_on.device) * 0.5, p_first_on) 259 | 260 | u = torch.empty_like(p_first_on, device=p_first_on.device) 261 | u.uniform_() 262 | on_mask = u <= p_first_on # 先触发Positive (OFF)事件的mask 263 | 264 | ep_input = torch.where(on_mask, ep_on_real, ep_off_real).to(torch.float64) 265 | 266 | c_input = torch.where(on_mask, c, -1 * c) 267 | delta_t_step = sample_IG_torch(ep_input, c_input, sigma) # 采样 $\Delta t$ 268 | t_end_ideal = delta_t_step.to(torch.float64) + start_t.to(torch.float64) 269 | 270 | delta_t_2_ends = t_end_ideal - end_t 271 | delta_vd_legacy_new = delta_vd_legacy 272 | 273 | still_possible_events = delta_t_2_ends < 0 # 没到end time, 仍有新event可能 274 | 275 | # last_event_positions = torch.logical_not(still_possible_events) 276 | # 超过end time,计算 $\Delta V_d_legacy$ 277 | last_event_positions = ~still_possible_events 278 | # ep_ori = torch.where(on_mask, ep_on, -1 * ep_off) 279 | sign = on_mask.to(torch.float64) * 2 - 1 280 | ep_signed = sign * ep_input 281 | delta_vd_last = ep_signed * (end_t - start_t) / delta_t_step 282 | delta_vd_legacy_new[last_event_positions] = delta_vd_legacy_new[last_event_positions] + \ 283 | delta_vd_last[last_event_positions] 284 | 285 | # collect events 286 | # this_event_positions = torch.nonzero(still_possible_events) 287 | this_event_positions = still_possible_events 288 | events_t = t_end_ideal[this_event_positions] 289 | events_x = x[this_event_positions] 290 | events_y = y[this_event_positions] 291 | events_p = on_mask[this_event_positions] 292 | 293 | # if len(this_event_positions[0]) > 0: 294 | if torch.sum(this_event_positions) > 0: 295 | ep_on_next = ep_on[this_event_positions] 296 | ep_off_next = ep_off[this_event_positions] 297 | c_next = c[this_event_positions] 298 | delta_vd_next = torch.zeros_like(ep_on_next, device=ep_on_next.device) 299 | # delta_vd_next_ideal = torch.zeros_like( 300 | # ep_on_next, device=ep_on_next.device) 301 | sigma_next = sigma[this_event_positions] 302 | x_next = events_x 303 | y_next = events_y 304 | t_next = events_t 305 | e_t, e_x, e_y, e_p, e_delta_vd = event_generation( 306 | ep_on_next, ep_off_next, c_next, sigma_next, delta_vd_next, t_next, end_t, x_next, y_next) 307 | 308 | events_t = torch.cat([events_t, e_t]) 309 | events_x = torch.cat([events_x, e_x]) 310 | events_y = torch.cat([events_y, e_y]) 311 | events_p = torch.cat([events_p, e_p]) 312 | 313 | delta_vd_legacy_new[still_possible_events] = e_delta_vd 314 | 315 | return events_t, events_x, events_y, events_p, delta_vd_legacy_new 316 | -------------------------------------------------------------------------------- /src/visualize.py: -------------------------------------------------------------------------------- 1 | #! python3 2 | # -*- encoding: utf-8 -*- 3 | ''' 4 | @File : visualize.py 5 | @Time : 2022/7/14 02:28 6 | @Author : Songnan Lin, Ye Ma 7 | @Contact : songnan.lin@ntu.edu.sg, my17@tsinghua.org.cn 8 | @Note : https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/inference_utils.py 9 | @inproceedings{lin2022dvsvoltmeter, 10 | title={DVS-Voltmeter: Stochastic Process-based Event Simulator for Dynamic Vision Sensors}, 11 | author={Lin, Songnan and Ma, Ye and Guo, Zhenhua and Wen, Bihan}, 12 | booktitle={ECCV}, 13 | year={2022} 14 | } 15 | ''' 16 | 17 | 18 | import os.path 19 | import cv2 20 | import numpy as np 21 | 22 | def events_to_voxel_grid(events, num_bins, width, height): 23 | """ 24 | https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/inference_utils.py 25 | Build a voxel grid with bilinear interpolation in the time domain from a set of events. 26 | 27 | :param events: a [N x 4] NumPy array (np.int32) containing one event per row in the form: 28 | [timestamp(us), x, y, polarity(0 or 1)] 29 | :param num_bins: number of bins in the temporal axis of the voxel grid 30 | :param width, height: dimensions of the voxel grid 31 | """ 32 | 33 | assert(events.shape[1] == 4) 34 | assert(num_bins > 0) 35 | assert(width > 0) 36 | assert(height > 0) 37 | 38 | voxel_grid = np.zeros((num_bins, height, width), np.float32).ravel() 39 | 40 | # normalize the event timestamps so that they lie between 0 and num_bins 41 | last_stamp = events[-1, 0] 42 | first_stamp = events[0, 0] 43 | deltaT = last_stamp - first_stamp 44 | 45 | if deltaT == 0: 46 | deltaT = 1.0 47 | 48 | ts = events[:, 0].astype(np.float32) 49 | ts = (num_bins - 1) * (ts - first_stamp) / deltaT 50 | xs = events[:, 1].astype(np.int32) 51 | ys = events[:, 2].astype(np.int32) 52 | pols = events[:, 3].astype(np.float32) 53 | pols[pols == 0] = -1 # polarity should be +1 / -1 54 | 55 | tis = ts.astype(np.int32) 56 | dts = ts - tis 57 | vals_left = pols * (1.0 - dts) 58 | vals_right = pols * dts 59 | 60 | valid_indices = tis < num_bins 61 | np.add.at(voxel_grid, xs[valid_indices] + ys[valid_indices] * width 62 | + tis[valid_indices] * width * height, vals_left[valid_indices]) 63 | 64 | valid_indices = (tis + 1) < num_bins 65 | np.add.at(voxel_grid, xs[valid_indices] + ys[valid_indices] * width 66 | + (tis[valid_indices] + 1) * width * height, vals_right[valid_indices]) 67 | 68 | voxel_grid = np.reshape(voxel_grid, (num_bins, height, width)) 69 | 70 | return voxel_grid 71 | 72 | def visual_voxel_grid(voxel_grid, output_folder, filename_key): 73 | for i in range(voxel_grid.shape[0]): 74 | path = os.path.join(output_folder, '%s_%02d.png' % (filename_key, i)) 75 | normalize_im = (voxel_grid[i] - np.min(voxel_grid[i])) / (np.max(voxel_grid[i]) - np.min(voxel_grid[i])) 76 | cv2.imwrite(path, (normalize_im * 255).astype(np.uint8)) --------------------------------------------------------------------------------