├── LICENSE ├── README.md ├── cnlos_reconstruction.m ├── demo.m ├── hio_reconstruction.m ├── nonconfocal_reconstruction.m ├── nonplanar_reconstruction.m ├── scenes ├── bike_1.png ├── bike_2.png ├── discoball_1.png ├── discoball_2.png ├── dragon_1.png ├── dragon_2.png ├── interactive_1.png ├── interactive_2.png ├── nonplanar_1.png ├── nonplanar_2.png ├── nonplanar_3.png ├── outdoor_1.jpg ├── outdoor_2.jpg ├── outdoor_3.jpg ├── outdoor_4.png ├── resolution_1.png ├── resolution_2.png ├── statue_1.png ├── statue_2.png ├── teaser_1.png ├── teaser_2.png └── teaser_3.png └── util ├── fkback3.m ├── fkforward3.m ├── imresize3d.m ├── progress.m └── waveconv.m /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018, Stanford University 2 | 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms for academic and other non-commercial purposes with or without modification, are permitted provided that the following conditions are met: 6 | 7 | Redistributions of source code, including modified source code, must retain the above copyright notice, this list of conditions and the following disclaimer. 8 | 9 | Redistributions in binary form or a modified form of the source code must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 10 | 11 | Neither the name of The Leland Stanford Junior University, any of its trademarks, the names of its employees, nor contributors to the source code may be used to endorse or promote products derived from this software without specific prior written permission. 12 | 13 | Where a modified version of the source code is redistributed publicly in source or binary forms, the modified source code must be published in a freely accessible manner, or otherwise redistributed at no charge to anyone requesting a copy of the modified source code, subject to the same terms as this agreement. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE LELAND STANFORD JUNIOR UNIVERSITY OR ITS TRUSTEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Non-Line-of-Sight Imaging Code & Datasets 2 | 3 | This repository contains code for the paper _Wave-Based Non-Line-of-Sight Imaging using Fast f-k Migration_ by David B. Lindell, Gordon Wetzstein, and Matthew O'Toole. The captured datasets can be downloaded separately from our [project webpage](http://www.computationalimaging.org/publications/nlos-fk/). 4 | 5 | ## Captured Scenes 6 | ### Bike 7 | ||| 8 | |---|---| 9 | 10 | - Description: A white stone statue captured at approximately 1 m distance from the wall. 11 | - Resolution: 512 x 512 12 | - Scanned Area: 2 m x 2 m planar wall 13 | - Integration times: 10 min., 30 min., 60 min., 180 min. 14 | 15 | ### Discoball 16 | ||| 17 | |---|---| 18 | 19 | - Description: A specular disoball captured at approximately 1 m distance from the wall. 20 | - Resolution: 512 x 512 21 | - Scanned Area: 2 m x 2 m planar wall 22 | - Integration times: 10 min., 30 min., 60 min., 180 min. 23 | 24 | 25 | ### Dragon 26 | ||| 27 | |---|---| 28 | 29 | - Description: A glossy dragon captured at approximately 1 m distance from the wall. 30 | - Resolution: 512 x 512 31 | - Scanned Area: 2 m x 2 m planar wall 32 | - Integration times: 15 sec., 1 min., 2 min., 10 min., 30 min., 60 min., 180 min. 33 | 34 | ### Interactive 35 | ||| 36 | |---|---| 37 | 38 | - Description: Two video captures of a person moving around in a retroreflective outfit. 39 | - Resolution: 32 x 32 (4 fps), 64 x 64 (2 fps) 40 | - Scanned Area: 2 m x 2 m planar wall 41 | 42 | ### Nonplanar 43 | |||| 44 | |---|---|---| 45 | 46 | - Description: Two retroreflective letters captured at approximately 1 m distance from the wall by scanning a non-planar surface. 47 | - Resolution: 128 x 128 48 | - Scanned Area: 1 m x 1 m non-planar surface. 49 | - Integration times: 150 sec. 50 | 51 | ### Outdoor 52 | ||||| 53 | |---|---|---|---| 54 | 55 | - Description: An outdoor scene captured by scanning the exterior of a building. 56 | - Resolution: 128 x 128 57 | - Scanned Area: 2 m x 2 m stone wall 58 | - Integration times: 10 min., 30 min., 50 min. 59 | 60 | ### Resolution 61 | ||| 62 | |---|---| 63 | 64 | - Description: A resolution chart captured at approximately 1 m distance from the wall. 65 | - Resolution: 512 x 512 66 | - Scanned Area: 2 m x 2 m planar wall 67 | - Integration times: 10 min., 30 min., 60 min., 180 min. 68 | 69 | ### Statue 70 | ||| 71 | |---|---| 72 | 73 | - Description: A white stone statue captured at approximately 1 m distance from the wall. 74 | - Resolution: 512 x 512 75 | - Scanned Area: 2 m x 2 m planar wall 76 | - Integration times: 10 min., 30 min., 60 min., 180 min. 77 | 78 | ### Teaser 79 | |||| 80 | |---|---|---| 81 | 82 | - Description: The teaser scene used in the paper which includes a number of objects, including a bookshelf, statue, dragon, and discoball. 83 | - Resolution: 512 x 512 84 | - Scanned Area: 2 m x 2 m planar wall 85 | - Integration times: 10 min., 30 min., 60 min., 180 min. 86 | 87 | ## Description of Files 88 | 89 | The code/dataset should be organized as in the following directory tree 90 | 91 | ./dataset 92 | bike/ 93 | discoball/ 94 | dragon/ 95 | interactive/ 96 | nonplanar/ 97 | outdoor/ 98 | phase_retrieval/ 99 | resolution/ 100 | scenes/ 101 | statue/ 102 | teaser/ 103 | util/ 104 | cnlos_reconstruction.m 105 | demo.m 106 | hio_reconstruction.m 107 | LICENSE 108 | nonconfocal_reconstruction.m 109 | nonplanar_reconstruction.m 110 | README.md 111 | 112 | #### Bike, Discoball, Dragon, Outdoor, Resolution, Statue, and Teaser Scenes 113 | Each of the folders contains a calibration file, measurement files, and example reconstructions. 114 | 115 | The calibration files are named `tof.mat` and contain an image variable named `tofgrid`. This variable gives the time of flight in picoseconds to each scan point. Our example processing algorithms use this information to 'rectify' the measurements such that time zero coincides with the moment the light reflects off of the wall for all scan points. For the non-planar scan scenes we also include a variable, `pos`, which contains the 3D coordinate of each scan point relative to the confocal scanning hardware, and a `tof_raw.mat` file, which contains the raw time-of-flight measurement volume. 116 | 117 | The measurement files are named `meas_[10,30,60,180]min.mat`, where the file name indicates the length of the exposure time in minutes. During the acquisition, we produce a complete scan every fifteen seconds, and so variable-length exposures can be compiled by accumulating the measurements over multiple scans. The files contain a 3D variable named `meas` of dimension 512x512x2048, with time being the last dimension. The time resolution of the measurements is 32 ps. 118 | 119 | Note that our acquisition procedure involves using a gated sensor to prevent recording the overwhelmingly bright direct reflection from the wall. The time at which the gate turns on varies with distance to the scan surface and is visible in the measurements as a transition to non-zero photon counts. Latency in changing the gate delay results in a small subset of the measurements being gated out (set to zero) at the beginning of the scan period. 120 | 121 | Finally, we include example reconstructions (at 512x512 spatial resolution) for the planar scenes using filtered backprojection (FBP) the Light-Cone Transform (LCT) and f-k migration methods. The corresponding files contain the reconstructed albedo volumes for each exposure time; the reconstruction method and exposure time are indicated by the filename. 122 | 123 | #### Interactive Scenes 124 | These scenes were captured by rapidly scanning the wall at 2 or 4 fps and capturing 32 x 32 or 64 x64 spatial samples. The measurements capture a person moving around in a retroreflective outfit. The directory contains a set of 4D datafiles with 3D measurement volumes and f-k migration reconstructions for each captured frame. Videos of the scene captured with a conventional camera are also included. 125 | 126 | #### Non-Confocal Scene 127 | The non-confocal processing is shown for an example scene from the [Zaragosa NLOS dataset](https://graphics.unizar.es/nlos_dataset.html). Before running the function, download the non-confocal dataset from the Zaragosa webpage or using this [direct link](https://drive.google.com/uc?export=download&id=1HZrlcSnme0qDLWanOlHgK2OgwLBeU6JX). We also provide example code for the NLOS dataset of [Klein et al.](https://nlos.cs.uni-bonn.de/). For convenience, we provide a direct [download link](https://drive.google.com/uc?export=download&id=1DISUkCfiA_NSBRrF7w8cBXduyH_m_EQa) to the example dataset we work with in a .mat format. 128 | 129 | #### Non-Planar Scene 130 | A non-planar scene consisting of two retroreflective letters was captured by scanning a deformed projector screen. We include the measurements, calibration files, and reconstructed 3D volumes using f-k migration or filtered backprojection. `The nonplanar_reconstruction.m` file can be run to load the measurements and generate the reconstructions. See the file for additional details. 131 | 132 | #### Phase Retrieval 133 | A hybrid-input output phase retrieval algorithm can be used to process the teaser scene by running the `hio_reconstruction.m` file. We include the measurement volume used for processing, as well as full-resolution reconstructions of the initial volume and after 50 iterations of the algorithm. 134 | 135 | #### Reconstruction Demo Program 136 | 137 | Run the `demo.m` script to reconstruct a sample scene with FBP, LCT, f-k migration, and the [phasor fields method](https://www.nature.com/articles/s41586-019-1461-3) by Liu et al. We also demonstrate how to reconstruct the interactive results, the non-planar scene, and how to run the phase retrieval algorithm. 138 | 139 | **License** 140 | The code and dataset are licensed under the following license: 141 | > Copyright (c) 2018, Stanford University 142 | > 143 | > All rights reserved. 144 | > 145 | > Redistribution and use in source and binary forms for academic and other non-commercial purposes with or without modification, are permitted provided that the following conditions are met: 146 | > 147 | > Redistributions of source code, including modified source code, must retain the above copyright notice, this list of conditions and the following disclaimer. 148 | > 149 | > Redistributions in binary form or a modified form of the source code must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 150 | > 151 | > Neither the name of The Leland Stanford Junior University, any of its trademarks, the names of its employees, nor contributors to the source code may be used to endorse or promote products derived from this software without specific prior written permission. 152 | > 153 | > Where a modified version of the source code is redistributed publicly in source or binary forms, the modified source code must be published in a freely accessible manner, or otherwise redistributed at no charge to anyone requesting a copy of the modified source code, subject to the same terms as this agreement. 154 | > 155 | > THIS SOFTWARE IS PROVIDED BY THE TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE LELAND STANFORD JUNIOR UNIVERSITY OR ITS TRUSTEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 156 | 157 | **Contact** 158 | Questions can be addressed to [David Lindell](mailto:lindell@stanford.edu) 159 | 160 | ## tl;dr 161 | Clone the git repo, download the datasets from our project webpage, and run `demo.m` in Matlab. 162 | -------------------------------------------------------------------------------- /cnlos_reconstruction.m: -------------------------------------------------------------------------------- 1 | function result = cnlos_reconstruction(meas, tofgrid, wall_size, alg, crop, bin_resolution) 2 | % Reconstruction procedures for "Confocal Non-Line-of-Sight Imaging Based on the Light Cone Transform" 3 | % by Matthew O'Toole, David B. Lindell, and Gordon Wetzstein. 4 | % and for "Wave-Based Non-Line-of-Sight Imaging using Fast f-k Migration" 5 | % by David B. Lindell, Matthew O'Toole, and Gordon Wetzstein. 6 | % 7 | % Usage/Parameters: 8 | % cnlos_reconstruction(meas, tofgrid, wall_size, alg) 9 | % meas: measurements provided from the data files for each scene 10 | % tofgrid: time-of-flight delays in picoseconds provided by the associated 11 | % datafiles for each scene. Pass in an empty array if data is 12 | % already corrected. 13 | % wall_size: size of scanned wall along one dimension (assumes 14 | % identical horizontal, vertical scanned dimensions) 15 | % alg == 0 : FBP 16 | % alg == 1 : LCT 17 | % alg == 2 : f-k migration 18 | % alg == 3 : phasor fields 19 | % crop: optional argument, bin index to crop measurements after correcting 20 | % for time-of-flight delays 21 | 22 | % Constants 23 | c = 3e8; % Speed of light (meters per second) 24 | width = wall_size / 2; 25 | if ~exist('crop', 'var') % bin index to crop measurements after aligning 26 | crop = 512; % so that direct component is at t=0 27 | end 28 | if ~exist('bin_resolution', 'var') 29 | bin_resolution = 32e-12; % temporal bin resolution of measurements 30 | end 31 | 32 | % Parameters 33 | isdiffuse = 1; % Toggle diffuse reflection (LCT only) 34 | isbackproj = 0; % Toggle backprojection vs LCT (LCT/phasor fields only) 35 | if alg == 0 || alg == 3 36 | isbackproj = 1; 37 | end 38 | snr = 1e-1; % SNR value (LCT only) 39 | 40 | % adjust so that t=0 is when light reaches the scan surface 41 | if ~isempty(tofgrid) 42 | for ii = 1:size(meas, 1) 43 | for jj = 1:size(meas,2 ) 44 | meas(ii, jj, :) = circshift(meas(ii, jj, :), [0, 0, -floor(tofgrid(ii, jj) / (bin_resolution*1e12))]); 45 | end 46 | end 47 | end 48 | meas = meas(:, :, 1:crop); 49 | 50 | N = size(meas,1); % Spatial resolution of data 51 | M = size(meas,3); % Temporal resolution of data 52 | range = M.*c.*bin_resolution; % Maximum range for histogram 53 | 54 | % Permute data dimensions 55 | data = permute(meas,[3 2 1]); 56 | 57 | % Define volume representing voxel distance from wall 58 | grid_z = repmat(linspace(0,1,M)',[1 N N]); 59 | 60 | display('Inverting...'); 61 | tic; 62 | 63 | if (alg == 2) % f-k migration 64 | [z,y,x] = ndgrid(-M:M-1,-N:N-1,-N:N-1); 65 | z = z./M; y = y./N; x = x./N; 66 | 67 | % Step 0: Pad data 68 | data = data .* grid_z.^2; 69 | data = sqrt(data); 70 | tdata = zeros(2.*M,2.*N,2.*N); 71 | tdata(1:end./2,1:end./2,1:end./2) = data; 72 | 73 | % Step 1: FFT 74 | tdata = fftshift(fftn(tdata)); 75 | 76 | % Step 2: Stolt trick 77 | tvol = interpn(z,y,x,tdata,sqrt(abs((((N.*range)./(M.*width.*4)).^2).*(x.^2+y.^2)+z.^2)),y,x,'linear',0); 78 | tvol = tvol.*(z > 0); 79 | tvol = tvol.*abs(z)./max(sqrt(abs((((N.*range)./(M.*width.*4)).^2).*(x.^2+y.^2)+z.^2)),1e-6); 80 | 81 | % Step 3: IFFT 82 | tvol = ifftn(ifftshift(tvol)); 83 | tvol = abs(tvol).^2; 84 | vol = abs(tvol(1:end./2,1:end./2,1:end./2)); 85 | 86 | elseif (alg == 0 || alg == 1) % backprojection or LCT 87 | % Define NLOS blur kernel 88 | psf = definePsf(N,M,width./range); 89 | 90 | % Compute inverse filter of NLOS blur kernel 91 | fpsf = fftn(psf); 92 | if isbackproj 93 | invpsf = conj(fpsf); 94 | else 95 | invpsf = conj(fpsf) ./ (abs(fpsf).^2 + 1./snr); 96 | end 97 | 98 | % Define transform operators 99 | [mtx,mtxi] = resamplingOperator(M); 100 | mtx = full(mtx); 101 | mtxi = full(mtxi); 102 | 103 | % Step 1: Scale radiometric component 104 | if (isdiffuse) 105 | data = data.*(grid_z.^4); 106 | else 107 | data = data.*(grid_z.^2); 108 | end 109 | 110 | % Step 2: Resample time axis and pad result 111 | tdata = zeros(2.*M,2.*N,2.*N); 112 | tdata(1:end./2,1:end./2,1:end./2) = reshape(mtx*data(:,:),[M N N]); 113 | 114 | % Step 3: Convolve with inverse filter and unpad result 115 | tvol = ifftn(fftn(tdata).*invpsf); 116 | tvol = tvol(1:end./2,1:end./2,1:end./2); 117 | 118 | % Step 4: Resample depth axis and clamp results 119 | vol = reshape(mtxi*tvol(:,:),[M N N]); 120 | vol = max(real(vol),0); 121 | 122 | if isbackproj 123 | % apply laplacian of gaussian filter 124 | vol = filterLaplacian(vol); 125 | 126 | % normalize, apply optional threshold 127 | vol = max(vol./max(vol(:)),0); 128 | end 129 | 130 | elseif (alg == 3) % phasor fields 131 | % Define backprojection blur kernel 132 | psf = definePsf(N,M,width./range); 133 | 134 | % Compute psf for backprojection 135 | bp_psf = conj(fftn(psf)); 136 | 137 | % Define transform operators 138 | [mtx,mtxi] = resamplingOperator(M); 139 | mtx = full(mtx); 140 | mtxi = full(mtxi); 141 | 142 | % Step 0: define virtual wavelet properties 143 | s_lamda_limit = wall_size/(N - 1); % sample spacing on the wall 144 | sampling_coeff = 2; % scale the size of the virtual wavelength (usually 2, optionally 3 for noisy scenes) 145 | virtual_wavelength = sampling_coeff * (s_lamda_limit * 2); % virtual wavelength in units of cm 146 | cycles = 5; % number of wave cycles in the wavelet, typically 4-6 147 | 148 | % Step 1: convolve measurement volume with virtual wave 149 | [phasor_data_cos, phasor_data_sin] = waveconv(bin_resolution, virtual_wavelength, cycles, data); 150 | phasor_data_cos = single(phasor_data_cos); 151 | phasor_data_sin = single(phasor_data_sin); 152 | 153 | % Step 2: transform virtual wavefield into LCT domain 154 | phasor_tdata_cos = single(zeros(2.*M,2.*N,2.*N)); 155 | phasor_tdata_sin = single(zeros(2.*M,2.*N,2.*N)); 156 | phasor_tdata_cos(1:end./2,1:end./2,1:end./2) = reshape(mtx*phasor_data_cos(:,:),[M N N]); 157 | phasor_tdata_sin(1:end./2,1:end./2,1:end./2) = reshape(mtx*phasor_data_sin(:,:),[M N N]); 158 | 159 | % Step 3: convolve with backprojection kernel 160 | tvol_phasorbp_sin = ifftn(fftn(phasor_tdata_sin).*bp_psf); 161 | tvol_phasorbp_sin = tvol_phasorbp_sin(1:end./2,1:end./2,1:end./2); 162 | phasor_tdata_cos = ifftn(fftn(phasor_tdata_cos).*bp_psf); 163 | phasor_tdata_cos = phasor_tdata_cos(1:end./2,1:end./2,1:end./2); 164 | 165 | % Step 4: compute phasor field magnitude and inverse LCT 166 | tvol = sqrt(tvol_phasorbp_sin.^2 + phasor_tdata_cos.^2); 167 | vol = reshape(mtxi*tvol(:,:),[M N N]); 168 | vol = max(real(vol),0); 169 | end 170 | 171 | display('... done.'); 172 | time_elapsed = toc; 173 | 174 | display(sprintf(['Reconstructed volume of size %d x %d x %d '... 175 | 'in %f seconds'], size(vol,3),size(vol,2),size(vol,1),time_elapsed)); 176 | 177 | tic_z = linspace(0,range./2,size(vol,1)); 178 | tic_y = linspace(width,-width,size(vol,2)); 179 | tic_x = linspace(width,-width,size(vol,3)); 180 | 181 | % clip artifacts at boundary, rearrange for visualization 182 | vol(end-10:end, :, :) = 0; 183 | vol = permute(vol, [1, 3, 2]); 184 | result = permute(vol, [2, 3, 1]); 185 | vol = flip(vol, 2); 186 | vol = flip(vol, 3); 187 | 188 | % View result 189 | figure 190 | 191 | subplot(1,3,1); 192 | imagesc(tic_x,tic_y,squeeze(max(vol,[],1))); 193 | title('Front view'); 194 | set(gca,'XTick',linspace(min(tic_x),max(tic_x),3)); 195 | set(gca,'YTick',linspace(min(tic_y),max(tic_y),3)); 196 | xlabel('x (m)'); 197 | ylabel('y (m)'); 198 | colormap('gray'); 199 | axis square; 200 | 201 | subplot(1,3,2); 202 | imagesc(tic_x,tic_z,squeeze(max(vol,[],2))); 203 | title('Top view'); 204 | set(gca,'XTick',linspace(min(tic_x),max(tic_x),3)); 205 | set(gca,'YTick',linspace(min(tic_z),max(tic_z),3)); 206 | xlabel('x (m)'); 207 | ylabel('z (m)'); 208 | colormap('gray'); 209 | axis square; 210 | 211 | subplot(1,3,3); 212 | imagesc(tic_z,tic_y,squeeze(max(vol,[],3))') 213 | title('Side view'); 214 | set(gca,'XTick',linspace(min(tic_z),max(tic_z),3)); 215 | set(gca,'YTick',linspace(min(tic_y),max(tic_y),3)); 216 | xlabel('z (m)'); 217 | ylabel('y (m)'); 218 | colormap('gray'); 219 | axis square; 220 | 221 | drawnow; 222 | 223 | end 224 | 225 | function psf = definePsf(U,V,slope) 226 | % Local function to compute NLOS blur kernel 227 | x = linspace(-1,1,2.*U); 228 | y = linspace(-1,1,2.*U); 229 | z = linspace(0,2,2.*V); 230 | [grid_z,grid_y,grid_x] = ndgrid(z,y,x); 231 | 232 | % Define PSF 233 | psf = abs(((4.*slope).^2).*(grid_x.^2 + grid_y.^2) - grid_z); 234 | psf = double(psf == repmat(min(psf,[],1),[2.*V 1 1])); 235 | psf = psf./sum(psf(:,U,U)); 236 | psf = psf./norm(psf(:)); 237 | psf = circshift(psf,[0 U U]); 238 | end 239 | 240 | function [mtx,mtxi] = resamplingOperator(M) 241 | % Local function that defines resampling operators 242 | mtx = sparse([],[],[],M.^2,M,M.^2); 243 | 244 | x = 1:M.^2; 245 | mtx(sub2ind(size(mtx),x,ceil(sqrt(x)))) = 1; 246 | mtx = spdiags(1./sqrt(x)',0,M.^2,M.^2)*mtx; 247 | mtxi = mtx'; 248 | 249 | K = log(M)./log(2); 250 | for k = 1:round(K) 251 | mtx = 0.5.*(mtx(1:2:end,:) + mtx(2:2:end,:)); 252 | mtxi = 0.5.*(mtxi(:,1:2:end) + mtxi(:,2:2:end)); 253 | end 254 | end 255 | 256 | function out = filterLaplacian(vol) 257 | % set filter parameters 258 | hsize = 5; 259 | std1 = 1; 260 | 261 | % calculate filter weights 262 | lim = (hsize - 1) / 2; 263 | std2 = std1^2; 264 | 265 | [x,y,z] = ndgrid(-lim:lim,-lim:lim, -lim:lim); 266 | w = exp(-(x.*x + y.*y + z.*z)/(2*std2)); 267 | w(w < eps * max(w(:))) = 0; 268 | sumw = sum(w(:)); 269 | if sumw ~= 0 270 | w = w/sumw; 271 | end; 272 | 273 | w1 = w.*(x.*x + y.*y + z.*z - 3*std2)/(std2^2); 274 | w = w1 - sum(w1(:))/hsize^3; % make the filter sum to zero 275 | 276 | out = convn(vol, w, 'same'); 277 | end -------------------------------------------------------------------------------- /demo.m: -------------------------------------------------------------------------------- 1 | %% Run this demo script to step through the included reconstruction procedures 2 | % Assumes MATLAB R2016b or higher (lower versions may work but have not 3 | % been tested). 4 | addpath('./util'); 5 | 6 | % First, run FBP, LCT, and f-k migration reconstructions for one of the 7 | % captured datasets 8 | 9 | % Optionally replace the below filenames with files from other scenes: 10 | % bike, discoball, dragon, outdoor, resolution, statue, teaser 11 | load('statue/tof.mat'); 12 | load('statue/meas_10min.mat'); 13 | 14 | % resize to low resolution to reduce memory requirements 15 | measlr = imresize3d(meas, 128, 128, 2048); % y, x, t 16 | tofgridlr = imresize(tofgrid, [128, 128]); 17 | wall_size = 2; % scanned area is 2 m x 2 m 18 | 19 | % run FBP 20 | fprintf('\nRunning FBP\n'); 21 | algorithm = 0; 22 | fbp = cnlos_reconstruction(measlr, tofgridlr, wall_size, algorithm); 23 | 24 | % run LCT 25 | fprintf('\nRunning LCT\n'); 26 | algorithm = 1; 27 | lct = cnlos_reconstruction(measlr, tofgridlr, wall_size, algorithm); 28 | 29 | % run f-k migration 30 | fprintf('\nRunning f-k migration\n'); 31 | algorithm = 2; 32 | fk = cnlos_reconstruction(measlr, tofgridlr, wall_size, algorithm); 33 | 34 | % run phasor fields reconstruction by Liu et al. 35 | % (https://www.nature.com/articles/s41586-019-1461-3) 36 | fprintf('\nRunning phasor fields\n'); 37 | algorithm = 3; 38 | fk = cnlos_reconstruction(measlr, tofgridlr, wall_size, algorithm); 39 | 40 | %% Reconstruct a frame from the interactive results 41 | % perform reconstruction and visualization for 32 x 32 resolution scene 42 | % captured at 4 Hz 43 | fprintf('\nReconstructing interactive results\n'); 44 | 45 | % this processes the frame, but it could also be directly loaded from the 46 | % preprocessed version in 'interactive/fk_32.mat' 47 | load('interactive/meas_32.mat'); 48 | load('interactive/tof_32.mat'); 49 | v = VideoReader('interactive/video_32.mov'); 50 | 51 | video_idx = 1400; 52 | frame_idx = 73; 53 | frame = squeeze(meas(frame_idx, :, :, :)); 54 | crop = 1024; % crop measurements to 1024 bins 55 | cnlos_reconstruction(frame, tofgrid, wall_size, algorithm, crop); 56 | figure; imshow(read(v, video_idx),'InitialMagnification', 20); 57 | title('Photo of interactive capture'); 58 | clear v; 59 | 60 | %% Run non-planar reconstruction 61 | % reconstruction of a scene from a scanned non-planar surface 62 | fprintf('\nRunning non-planar reconstructions\n'); 63 | nonplanar_reconstruction; 64 | 65 | %% Run phase retrieval 66 | % run the phase retrieval algorithm on one of the captured datasets 67 | fprintf('\nRunning phase-retrieval reconstruction\n'); 68 | hio_reconstruction; -------------------------------------------------------------------------------- /hio_reconstruction.m: -------------------------------------------------------------------------------- 1 | function hio_reconstruction() 2 | % Phase retrieval algorithm described in "Wave-Based Non-Line-of-Sight 3 | % Imaging using Fast f-k Migration" by David B. Lindell, Matthew O'Toole, 4 | % and Gordon Wetzstein. 5 | % 6 | % This function runs the hybrid input-output (HIO) phase retrieval procedure 7 | % to reproduce the result shown in the supplementary material 8 | % 9 | % Reconstruction results are also provided for f-k migration without phase 10 | % and after 50 iterations of HIO in the 'hio_init.mat', and 11 | % 'hio_50.mat' files. 12 | % 13 | % The reconstruction runs at lower spatial resolution by default to 14 | % decrease the amount of computation time. Change this by adjusting the 15 | % 'use_low_resolution' flag below. 16 | 17 | addpath('util/'); 18 | use_low_resolution = 1; % run at lower resolution to decrease computation time 19 | 20 | load('phase_retrieval/meas_180min.mat'); 21 | meas = sqrt(meas512); 22 | if use_low_resolution 23 | meas = imresize3d(meas, 32, 32, 512); 24 | end 25 | 26 | N = size(meas,1); % Spatial resolution of data 27 | M = size(meas,3); % Temporal resolution of data 28 | meas = permute(meas,[3 2 1]); 29 | 30 | % constants 31 | width = 1; 32 | bin_resolution = 32e-12; % Native bin resolution for SPAD is 4 ps 33 | c = 3e8; % Speed of light (meters per second) 34 | range = M.*c.*bin_resolution; % Maximum range for histogram 35 | 36 | % falloff correction factor 37 | [z,y,x] = ndgrid(0:M-1,0:N-1,0:N-1); 38 | meas = meas .* (z+1); 39 | 40 | % compute initial reconstruction 41 | geom = fkback3(meas, width, range); 42 | geom_init = geom; 43 | mag = meas; 44 | 45 | N_iters = 50; 46 | figure; 47 | for ii = 1:N_iters 48 | progress(ii, N_iters); 49 | % recover geometry and enforce zero phase 50 | grad = geom - fkback3(meas, width, range); 51 | geom = geom - 0.005 * grad; 52 | 53 | % keep mag and add phase from measurements 54 | phase = angle(fkforward3(geom, width, range)); 55 | meas = mag .* exp(1j .* phase); 56 | 57 | if mod(ii, 10) == 0 58 | subplot(231); 59 | imagesc(squeeze(max(abs(geom_init(1:end-5, :, :)).^2, [], 1)).'); 60 | axis square; 61 | title('Initial Reconstruction (xy)'); 62 | 63 | subplot(232); 64 | imagesc(squeeze(max(abs(geom_init(1:end-5, :, :)).^2, [], 2)).'); 65 | axis square; 66 | title('(yz)'); 67 | 68 | subplot(233); 69 | imagesc(squeeze(max(abs(geom_init(1:end-5, :, :)).^2, [],3)).'); 70 | axis square; 71 | title('(xz)'); 72 | 73 | subplot(234); 74 | imagesc(squeeze(max(abs(geom(1:end-5, :, :)).^2, [], 1)).'); 75 | axis square; 76 | title(sprintf('HIO Reconstruction (xy) (%d Iters)', ii)); 77 | 78 | subplot(235); 79 | imagesc(squeeze(max(abs(geom(1:end-5, :, :)).^2, [], 2)).'); 80 | axis square; 81 | title('(yz)'); 82 | 83 | subplot(236); 84 | imagesc(squeeze(max(abs(geom(1:end-5, :, :)).^2, [], 3)).'); 85 | axis square; 86 | title('(xz)'); 87 | drawnow; 88 | end 89 | end 90 | -------------------------------------------------------------------------------- /nonconfocal_reconstruction.m: -------------------------------------------------------------------------------- 1 | function nonconfocal_reconstruction(varargin) 2 | % Non-confocal reconstruction procedure described in "Wave-Based 3 | % Non-Line-of-Sight Imaging using Fast f-k Migration" by David B. Lindell, 4 | % Matthew O'Toole, and Gordon Wetzstein. 5 | % 6 | % The simulated data for this procedure can be downloaded from the Zaragoza 7 | % NLOS dataset at https://graphics.unizar.es/nlos_dataset.html 8 | % The direct download link to the dataset is 9 | % https://drive.google.com/uc?export=download&id=1HZrlcSnme0qDLWanOlHgK2OgwLBeU6JX 10 | % 11 | % We also provide example code for the nonconfocal dataset of Klein et al. 12 | % This can be downloaded at https://nlos.cs.uni-bonn.de/challenges/Geometry or 13 | % we provide a direct link to an example dataset at 14 | % https://drive.google.com/uc?export=download&id=1DISUkCfiA_NSBRrF7w8cBXduyH_m_EQa 15 | % 16 | % This function loads simulated nonconfocal measurements, calculates the 17 | % normal moveout correction, which results in emulated confocal 18 | % measurements, and then processes the data using f-k migration. 19 | 20 | fprintf('Running non-confocal processing...\n'); 21 | 22 | if length(varargin) == 0 23 | method = 'zaragoza'; 24 | else 25 | method = varargin{1}; 26 | end 27 | 28 | % Zaragoza dataset (exhaustively scanned 2D laser + 2D camera measurements) 29 | if strcmp(method, 'zaragoza') 30 | 31 | fname = './Z_l[0.00,-0.50,0.00]_r[1.57,0.00,3.14]_v[0.81,0.01,0.81]_s[16]_l[16]_gs[1.00].hdf5'; 32 | if ~exist(fname, 'file') 33 | error(['Download the simualted non-confocal dataset from the Zaragoza NLOS webpage' ... 34 | ' at https://graphics.unizar.es/nlos_dataset.html, or use the direct download' ... 35 | ' link https://drive.google.com/uc?export=download&id=1HZrlcSnme0qDLWanOlHgK2OgwLBeU6JX']); 36 | end 37 | 38 | fprintf('\tProcessing Zaragoza dataset\n') 39 | fprintf('\tLoading data\n'); 40 | laserGridPositions = h5read(fname, '/laserGridPositions'); 41 | laserGridPositions = reshape(laserGridPositions, [], 3); 42 | 43 | laserPosition = h5read(fname, '/laserPosition'); 44 | 45 | cameraPosition = h5read(fname, '/cameraPosition'); 46 | 47 | % simulated data is laser_x, laser_y, cam_x, cam_y, bounces, time 48 | % collapse to laser_pos, cam_pos, time 49 | data = h5read(fname, '/data'); 50 | dims = size(data); 51 | data = reshape(data, dims(1)*dims(2), dims(3)*dims(4), dims(5), dims(6)); 52 | data = squeeze(sum(data, 3)); 53 | 54 | deltaT = h5read(fname, '/deltaT'); 55 | 56 | fprintf('\tReparamaterize + normal moveout correction\n'); 57 | meas = reparameterize(laserGridPositions, laserPosition, cameraPosition, data, deltaT); 58 | 59 | fprintf('\tf-k migration\n'); 60 | alg = 2; % f-k migration 61 | crop = 1024; % crop measurements to 1024 bins 62 | wall_size = 1; % simulated scanned area is 1 x 1 m 63 | tofgrid = []; 64 | bin_resolution = double(deltaT / 3e8); 65 | cnlos_reconstruction(meas, tofgrid, wall_size, alg, crop, bin_resolution); 66 | 67 | % Klein dataset (single laser position, grid of 2D camera positions) 68 | elseif strcmp(method, 'klein') 69 | fname = 'letterk.mat'; 70 | if ~exist(fname, 'file') 71 | error(['Download the simualted non-confocal dataset from the NLOS Benchmark webpage' ... 72 | ' at https://nlos.cs.uni-bonn.de/challenges/Geometry, or use the direct download' ... 73 | ' link https://drive.google.com/uc?export=download&id=1DISUkCfiA_NSBRrF7w8cBXduyH_m_EQa']); 74 | end 75 | 76 | fprintf('\tProcessing Klein dataset\n') 77 | fprintf('\tLoading data\n'); 78 | load(fname) 79 | 80 | zero_append = tMin / tDelta; 81 | data = cat(3, zeros(size(data, 1), size(data, 2), zero_append), data); 82 | 83 | fprintf('\tReparameterize + normal moveout correction\n'); 84 | R = 256; 85 | data = data(:, :, 1:1024); 86 | data = imresize3(data, [R, R, 1024]); 87 | [Y, X] = meshgrid(linspace(-0.256, 0.256, size(data, 1)), linspace(-0.256, 0.256, size(data, 2))); 88 | 89 | x = X(:); 90 | y = Y(:); 91 | 92 | mx = X(:) / 2; 93 | my = Y(:) / 2; 94 | 95 | mx = mx(:); 96 | my = my(:); 97 | 98 | midpoints = [mx my]; 99 | midpoints = round(midpoints, 4); 100 | [midpoints] = unique(midpoints, 'rows'); 101 | 102 | hx = abs(X(:)) / 2; 103 | hy = abs(Y(:)) / 2; 104 | 105 | hx = hx(:); 106 | hy = hy(:); 107 | 108 | offsets = [hx hy]; 109 | offsets = round(offsets, 4); 110 | offsets = unique(offsets, 'rows'); 111 | 112 | t = tDelta * (0:(size(data,3)-1)); 113 | v = 1; 114 | 115 | meas = single(zeros(size(midpoints, 1), size(data,3))); 116 | num_added = zeros(size(offsets, 1), 1); 117 | m_added = zeros(size(midpoints, 1), 1); 118 | data_flat = reshape(data, size(data,1)*size(data,2), []); 119 | for ii = 1:length(y) 120 | % find closest midpoint and offset 121 | m = [x(ii) / 2, y(ii) / 2]; 122 | h = abs([x(ii) / 2, y(ii) / 2]); 123 | [~, m_idx] = min(sqrt((m(1) - midpoints(:, 1)).^2 + (m(2) - midpoints(:, 2)).^2)); 124 | [~, h_idx] = min(sqrt((h(1) - offsets(:, 1)).^2 + (h(2) - offsets(:, 2)).^2)); 125 | 126 | % assign nmo-corrected measurement to new array 127 | offset_x = offsets(h_idx, 1); 128 | offset_y = offsets(h_idx, 2); 129 | val = squeeze(data_flat(ii, :)); 130 | meas(m_idx, :) = meas(m_idx, :) + NMO(t, v, offset_x, offset_y, val); 131 | m_added(m_idx) = m_added(m_idx) + 1; 132 | end 133 | 134 | for ii = 1:size(m_added, 1) 135 | meas(ii, :) = meas(ii, :) / m_added(ii); 136 | end 137 | 138 | meas(isnan(meas)) = 0; 139 | m = int32(sqrt(size(midpoints, 1))); 140 | meas = reshape(meas, m, m, []); 141 | 142 | fprintf('\tf-k migration\n'); 143 | alg = 2; % f-k migration 144 | crop = 1024; % crop measurements to 1024 bins 145 | wall_size = 0.256; 146 | tofgrid = []; 147 | bin_resolution = double(tDelta / 3e8); 148 | cnlos_reconstruction(meas, tofgrid, wall_size, alg, crop, bin_resolution); 149 | end 150 | end 151 | 152 | function meas = reparameterize(laserGridPositions, laserPosition, cameraPosition, data, deltaT) 153 | 154 | % correct for laser/camera positions 155 | x = laserGridPositions(:, 1); 156 | y = laserGridPositions(:, 3); 157 | z = laserGridPositions(:, 2); 158 | 159 | xc = cameraPosition(1); 160 | yc = cameraPosition(3); 161 | zc = cameraPosition(2); 162 | 163 | xl = laserPosition(1); 164 | yl = laserPosition(3); 165 | zl = laserPosition(2); 166 | 167 | dist = sqrt((x-xl).^2 + (y-yl).^2 + (z-zl).^2) + sqrt((x'-xc).^2 + (y'-yc).^2 + (z'-zc).^2); 168 | for ii = 1:length(y) 169 | for jj = 1:length(x) 170 | data(ii, jj, :) = circshift(data(ii, jj, :), -floor(dist(ii,jj) / (deltaT))); 171 | end 172 | end 173 | 174 | min_x = min(x); 175 | max_x = max(x); 176 | min_y = min(y); 177 | max_y = max(y); 178 | [X, Y] = meshgrid(linspace(min_x, max_x, sqrt(size(x, 1))), linspace(min_y, max_y, sqrt(size(y, 1)))); 179 | 180 | mx = (X(:) + X(:)') / 2; 181 | my = (Y(:) + Y(:)') / 2; 182 | 183 | mx = mx(:); 184 | my = my(:); 185 | 186 | midpoints = [mx my]; 187 | midpoints = round(midpoints, 3); 188 | [midpoints] = unique(midpoints, 'rows'); 189 | 190 | hx = abs(X(:) - X(:)') / 2; 191 | hy = abs(Y(:) - Y(:)') / 2; 192 | 193 | hx = hx(:); 194 | hy = hy(:); 195 | 196 | offsets = [hx hy]; 197 | offsets = round(offsets, 3); 198 | offsets = unique(offsets, 'rows'); 199 | 200 | t = deltaT * (0:(size(data,3)-1)); 201 | v = 1; 202 | 203 | meas = single(zeros(size(midpoints, 1), size(offsets, 1), size(data,3))); 204 | m_added = zeros(size(midpoints, 1), 1); 205 | for ii = 1:length(y) 206 | % disp(ii); 207 | for jj = 1:length(x) 208 | % find closest midpoint and offset 209 | m = [(x(ii) + x(jj)) / 2, (y(ii) + y(jj)) / 2]; 210 | h = abs([(x(ii) - x(jj)) / 2, (y(ii) - y(jj)) / 2]); 211 | [~, m_idx] = min(sqrt((m(1) - midpoints(:, 1)).^2 + (m(2) - midpoints(:, 2)).^2)); 212 | [~, h_idx] = min(sqrt((h(1) - offsets(:, 1)).^2 + (h(2) - offsets(:, 2)).^2)); 213 | 214 | % assign nmo-corrected measurement to new array 215 | offset_x = offsets(h_idx, 1); 216 | offset_y = offsets(h_idx, 2); 217 | val = squeeze(data(ii, jj, :)); 218 | meas(m_idx, h_idx, :) = squeeze(meas(m_idx, h_idx, :))' + NMO(t, v, offset_x, offset_y, val); 219 | m_added(m_idx) = m_added(m_idx) + 1; 220 | end 221 | end 222 | 223 | meas = squeeze(sum(meas, 2)); 224 | meas = reshape(meas, 31, 31, []); 225 | m_added = reshape(m_added, 31, 31); 226 | for ii = 1:size(m_added, 1) 227 | for jj = 1:size(m_added, 2) 228 | meas(ii, jj, :) = meas(ii, jj, :) / m_added(ii, jj); 229 | end 230 | end 231 | 232 | meas(isnan(meas)) = 0; 233 | m = int32(sqrt(size(midpoints, 1))); 234 | meas = reshape(meas, m, m, []); 235 | 236 | end 237 | 238 | % perform normal moveout correction 239 | function nmo = NMO(t, v, offset_x, offset_y, meas) 240 | tq = sqrt(t.^2 + 4*offset_x.^2 / v.^2 + 4*offset_y.^2 / v.^2); 241 | tq(isnan(tq)) = 0; 242 | nmo = interp1(t, meas, tq, 'linear', 0); 243 | end 244 | -------------------------------------------------------------------------------- /nonplanar_reconstruction.m: -------------------------------------------------------------------------------- 1 | function nonplanar_reconstruction() 2 | % Non-planar reconstruction procedure described in "Wave-Based 3 | % Non-Line-of-Sight Imaging using Fast f-k Migration" by David B. Lindell, 4 | % Matthew O'Toole, and Gordon Wetzstein. 5 | % 6 | % This function loads the measurements and calibration files and displays 7 | % the reconstructed scene with and without the non-planar correction. A 3D 8 | % visualization of the scanned surface is also displayed. The full 9 | % wave-based non-planar correction can be applied (default), or a faster, 10 | % approximate histogram-shifting-based approach can be used (adjust 11 | % parameter settings accordingly). To speed up the reconstruction, the 12 | % volumes are processed at low resolution by default. 13 | % 14 | % Reconstructed volumes at full resolution are also provided in the 15 | % 'nonplanar' directory for the filtered backprojection approach as well 16 | % for f-k migration applied to the uncorrected data, with the 17 | % naive histogram-shifting correction, or with the full wave-based 18 | % non-planar correction. 19 | 20 | % load data files 21 | addpath('util/'); 22 | load('nonplanar/tof.mat'); 23 | load('nonplanar/meas_150sec.mat'); 24 | load('nonplanar/calibration.mat'); 25 | 26 | % Parameters 27 | use_naive = 0; % toggle using wave correction or naive histogram shifting 28 | use_low_resolution = 1; % process at lower resolution to reduce computation time 29 | 30 | % Constants 31 | bin_resolution = 32e-12; 32 | c = 3e8; 33 | 34 | % realign measurements so that time zero corresponds to arrival of direct 35 | % component to the surface 36 | rect_data = meas; 37 | for ii = 1:size(rect_data, 1) 38 | for jj = 1:size(rect_data,2 ) 39 | rect_data(ii, jj, :) = circshift(rect_data(ii, jj, :), -floor(tofgrid(ii, jj) / (bin_resolution*1e12))); 40 | end 41 | end 42 | rect_data = sqrt(rect_data(:, :, 1:512)); 43 | 44 | if use_low_resolution 45 | rect_data = imresize3d(rect_data, 32, 32, 512); 46 | z_offset = imresize(z_offset, [32, 32]); 47 | end 48 | 49 | % set dimension sizes and space/fourier domain indices 50 | Nx = size(rect_data, 1); 51 | Ny = size(rect_data, 2); 52 | Nz = size(rect_data, 3); 53 | Nx_iter = Nx; 54 | Ny_iter = Ny; 55 | Nz_iter = Nz; 56 | 57 | spectra = zeros(size(rect_data)); 58 | meas_rect = zeros(size(rect_data)); 59 | [N, ~, M] = size(rect_data); 60 | range = M.*c.*bin_resolution; 61 | 62 | x = linspace(-width/2, width/2, Nx); 63 | y = linspace(-width/2, width/2, Ny); 64 | z = linspace(0, range/2, Nz); 65 | [X, Y] = ndgrid(x, y); 66 | 67 | kx = fftshift(fftfreq(Nx, x(2) - x(1))); 68 | ky = fftshift(fftfreq(Ny, y(2) - y(1))); 69 | kz = fftshift(fftfreq(Nz, z(2) - z(1))); 70 | [KX, KY, KZ] = ndgrid(kx, ky, kz); 71 | vec = @(x) x(:); 72 | 73 | % run the non-planar correction 74 | for ii = 1:Nx_iter 75 | progress(ii, Nx_iter); 76 | for jj = 1:Ny_iter 77 | if use_naive 78 | meas_rect(ii, jj, :) = circshift(rect_data(ii, jj, :), 2*floor(z_offset(ii, jj) / (bin_resolution*c))); 79 | else 80 | tmp_vol = zeros(size(rect_data)); 81 | tmp_vol(ii, jj, :) = rect_data(ii, jj, :); 82 | tmp_spectra = fftshift(fftn(tmp_vol)); 83 | phase_mask = exp(2*pi*1j * (sqrt(KZ.^2 - KX.^2 - KY.^2)) .* z_offset(ii, jj)); 84 | spectra = spectra + tmp_spectra.*phase_mask; 85 | end 86 | end 87 | 88 | if ~use_naive 89 | meas_rect = real( rifft3(ifftshift(spectra))); 90 | end 91 | end 92 | 93 | % run reconstruction 94 | algorithm = 2; 95 | wall_size = 1; 96 | 97 | fprintf('\nReconstructing corrected volume\n'); 98 | cnlos_reconstruction(meas_rect.^2, [], wall_size, algorithm); 99 | fprintf('\nReconstructing volume without correction\n'); 100 | cnlos_reconstruction(rect_data.^2, [], wall_size, algorithm); 101 | 102 | fprintf('\nVisualizing non-planar scan surface\n'); 103 | load('nonplanar/scan_points.mat'); 104 | 105 | figure; 106 | scan_grid(3, :) = -(scan_grid(3, :) - max(scan_grid(3, :))); 107 | scan_grid_pc = pointCloud(scan_grid'); 108 | 109 | pcshow(scan_grid_pc, 'VerticalAxis', 'Y', 'MarkerSize', 14); 110 | set(gca, 'zlim', [0.0, 0.11]); 111 | set(gca, 'ztick', [0.0 0.1], 'zticklabel', {'0', '0.1'}); 112 | xlabel('x'); ylabel('y'); zlabel('z'); 113 | camup([0 1 0]); 114 | campos([-1, 1, 1]) 115 | set(gca, 'clim', [0, 0.1]); 116 | title('Scan Surface'); 117 | end 118 | 119 | function result = fftfreq(n, d) 120 | val = 1.0 / (n * d); 121 | N = floor((n-1) / 2) + 1; 122 | p1 = 0:N-1; 123 | p2 = -floor(n/2):-1; 124 | result = [p1 p2] .* val; 125 | end 126 | 127 | function spectra = rifft3(spectra) 128 | % do ifft in x 129 | spectra = ifft(spectra, [], 1); 130 | % ifft in y 131 | spectra = ifft(spectra, [], 2); 132 | 133 | % real ifft in z (enforce conj symmetry, then do fft) 134 | N = size(spectra, 3); 135 | spectra(:, :, 2:N/2) = conj(flip(spectra(:, :, N/2+2:end), 3)); 136 | spectra = ifft(spectra, [], 3); 137 | end -------------------------------------------------------------------------------- /scenes/bike_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/bike_1.png -------------------------------------------------------------------------------- /scenes/bike_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/bike_2.png -------------------------------------------------------------------------------- /scenes/discoball_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/discoball_1.png -------------------------------------------------------------------------------- /scenes/discoball_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/discoball_2.png -------------------------------------------------------------------------------- /scenes/dragon_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/dragon_1.png -------------------------------------------------------------------------------- /scenes/dragon_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/dragon_2.png -------------------------------------------------------------------------------- /scenes/interactive_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/interactive_1.png -------------------------------------------------------------------------------- /scenes/interactive_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/interactive_2.png -------------------------------------------------------------------------------- /scenes/nonplanar_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/nonplanar_1.png -------------------------------------------------------------------------------- /scenes/nonplanar_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/nonplanar_2.png -------------------------------------------------------------------------------- /scenes/nonplanar_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/nonplanar_3.png -------------------------------------------------------------------------------- /scenes/outdoor_1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/outdoor_1.jpg -------------------------------------------------------------------------------- /scenes/outdoor_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/outdoor_2.jpg -------------------------------------------------------------------------------- /scenes/outdoor_3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/outdoor_3.jpg -------------------------------------------------------------------------------- /scenes/outdoor_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/outdoor_4.png -------------------------------------------------------------------------------- /scenes/resolution_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/resolution_1.png -------------------------------------------------------------------------------- /scenes/resolution_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/resolution_2.png -------------------------------------------------------------------------------- /scenes/statue_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/statue_1.png -------------------------------------------------------------------------------- /scenes/statue_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/statue_2.png -------------------------------------------------------------------------------- /scenes/teaser_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/teaser_1.png -------------------------------------------------------------------------------- /scenes/teaser_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/teaser_2.png -------------------------------------------------------------------------------- /scenes/teaser_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/computational-imaging/nlos-fk/d34d49fed5a86f4ebddf85d461cff6923de5367d/scenes/teaser_3.png -------------------------------------------------------------------------------- /util/fkback3.m: -------------------------------------------------------------------------------- 1 | function geom = fkback3(meas, width, range) 2 | N = size(meas,2); % Spatial resolution of data 3 | M = size(meas,1); % Temporal resolution of data 4 | data = meas; 5 | 6 | [z,y,x] = ndgrid(-M:M-1,-N:N-1,-N:N-1); 7 | z = z./M; x = x./N; y = y./N; 8 | 9 | % Step 0: Pad data 10 | tdata = zeros(2.*M,2.*N,2.*N); 11 | tdata(1:end/2,1:end/2,1:end/2) = data; 12 | 13 | % Step 1: FFT 14 | tdata = fftshift(fftn(tdata)); 15 | 16 | % Step 2: Stolt trick 17 | tvol = interpn(z,y,x,tdata,sqrt(abs((((N.*range)./(M.*width.*4)).^2).*(x.^2+y.^2)+z.^2)),y,x,'linear',0); 18 | tvol = tvol.*abs(z)./max(sqrt(abs((((N.*range)./(M.*width.*4)).^2).*(x.^2+y.^2)+z.^2)),1e-8); 19 | tvol = tvol.*(z > 0); 20 | geom = ifftn(ifftshift(tvol)); 21 | geom = geom(1:end/2,1:end/2,1:end/2); 22 | end -------------------------------------------------------------------------------- /util/fkforward3.m: -------------------------------------------------------------------------------- 1 | function meas = fkforward3(geom, width, range) 2 | N = size(geom,2); % Spatial resolution of data 3 | M = size(geom,1); % Temporal resolution of data 4 | data = geom; 5 | 6 | [z,y,x] = ndgrid(-M:M-1,-N:N-1,-N:N-1); 7 | z = z./M; x = x./N; y = y./N; 8 | 9 | % Step 0: Pad data 10 | tdata = zeros(2.*M,2.*N,2.*N); 11 | tdata(1:end./2,1:end./2,1:end./2) = data; 12 | 13 | % Forward 14 | tvol = fftshift(fftn(tdata)); 15 | tdata = interpn(z,y,x,tvol,sqrt(abs((((N.*range)./(M.*width.*4)).^2).*(x.^2+y.^2)-z.^2)),y,x,'linear',0); 16 | tdata = tdata.*(abs((((N.*range)./(M.*4*width)).^2).*(x.^2+y.^2)) 0); 18 | tdata = ifftn(ifftshift(tdata)); 19 | meas = tdata(1:end/2,1:end/2,1:end/2); 20 | end 21 | -------------------------------------------------------------------------------- /util/imresize3d.m: -------------------------------------------------------------------------------- 1 | function out = imresize3d(vol, Nx, Ny, Nz) 2 | x = linspace(0, 1, size(vol, 1)); 3 | y = linspace(0, 1, size(vol, 2)); 4 | z = linspace(0, 1, size(vol, 3)); 5 | [X, Y, Z] = ndgrid(x, y, z); 6 | F = griddedInterpolant(X, Y, Z, vol, 'nearest', 'none'); 7 | 8 | xq = linspace(0, 1, Nx); 9 | yq = linspace(0, 1, Ny); 10 | zq = linspace(0, 1, Nz); 11 | [Xq, Yq, Zq] = ndgrid(xq, yq, zq); 12 | out = F(Xq, Yq, Zq); 13 | end 14 | -------------------------------------------------------------------------------- /util/progress.m: -------------------------------------------------------------------------------- 1 | function progress(cur_iter, num_iters) 2 | persistent old_tmp; 3 | 4 | if cur_iter == 1 5 | msg = sprintf('%.3f%% done.',cur_iter/num_iters*100); 6 | old_tmp = msg; 7 | elseif cur_iter == num_iters 8 | tmp = sprintf('%.3f%% done.',cur_iter/num_iters*100); 9 | msg = [char(8)*ones(1,length(old_tmp)),tmp]; 10 | old_tmp = tmp; 11 | fprintf(['%' num2str(length(msg)) 's\n'],msg); 12 | return; 13 | else 14 | tmp = sprintf('%.3f%% done.',cur_iter/num_iters*100); 15 | msg = [char(8)*ones(1,length(old_tmp)),tmp]; 16 | old_tmp = tmp; 17 | end 18 | fprintf(['%' num2str(length(msg)) 's'],msg); 19 | 20 | end -------------------------------------------------------------------------------- /util/waveconv.m: -------------------------------------------------------------------------------- 1 | function [wave_cos, wave_sin] = waveconv(bin_resolution, virtual_wavelength, cycles, data) 2 | c = 3e8; 3 | s_z = bin_resolution * c; 4 | samples = round(cycles * virtual_wavelength / (bin_resolution * c)); 5 | num_cycles = samples * s_z / virtual_wavelength; 6 | sigma = 0.3; 7 | 8 | % generate sin/cos signals 9 | sin_wave = sin(2*pi*(num_cycles * linspace(1,samples,samples)')/samples); 10 | cos_wave = cos(2*pi*(num_cycles * linspace(1,samples,samples)')/samples); 11 | 12 | % apply window function 13 | window = single(gausswin(samples, 1/sigma)); 14 | virtual_wave_sin = sin_wave .* window; 15 | virtual_wave_cos = cos_wave .* window; 16 | 17 | % convolve with measurements 18 | wave_cos = single(zeros(size(data))); 19 | wave_sin = single(zeros(size(data))); 20 | for laser_index = 1 : size(data,2) 21 | for camera_index = 1 : size(data,3) 22 | time_response = squeeze(data(:,laser_index, camera_index)); 23 | tmp_real = conv(time_response,virtual_wave_sin, 'same'); 24 | tmp_img = conv(time_response,virtual_wave_cos, 'same'); 25 | wave_cos(:,laser_index, camera_index) = tmp_real; 26 | wave_sin(:,laser_index, camera_index) = tmp_img; 27 | end 28 | end 29 | end 30 | 31 | 32 | 33 | 34 | 35 | 36 | function [real, img] = waveconvolution(time_resolution, virtual_guassian_size, virtual_lamda, data_t) 37 | % Input parameter: 38 | % time_resolution: unit second, time resolution per time bin 39 | % virtual_guassian_size: integer, represent as virtual phasor wave guassian envelop 40 | % virtual_lamda: unit cm, virtual phasor wavelength 41 | % data_t: 3D temporal measurement 42 | 43 | % Output parameter: 44 | % real, img: real and imaginary part of the virtual phasor wave 45 | 46 | c = 3e8; % speed of light 47 | s_z = time_resolution * c * 100; % unit: cm 48 | 49 | Sinusoid_pattern = virtual_guassian_size * s_z / virtual_lamda; 50 | 51 | Gauss_sigma = 0.3; 52 | 53 | Sinusoid_pattern = single(Sinusoid_pattern); 54 | virtual_guassian_size = single(virtual_guassian_size); 55 | 56 | sin_wave = sin(2*pi*(Sinusoid_pattern * linspace(1,virtual_guassian_size,virtual_guassian_size)')/virtual_guassian_size); 57 | cos_wave = cos(2*pi*(Sinusoid_pattern * linspace(1,virtual_guassian_size,virtual_guassian_size)')/virtual_guassian_size); 58 | gauss_wave = single(gausswin(virtual_guassian_size, 1/Gauss_sigma)); 59 | 60 | Virtual_Wave_sin = sin_wave .* gauss_wave; 61 | Virtual_Wave_cos = cos_wave .* gauss_wave; 62 | 63 | real = single(zeros(size(data_t))); 64 | img = single(zeros(size(data_t))); 65 | for laser_index = 1 : size(data_t,2) 66 | for camera_index = 1 : size(data_t,3) 67 | time_response = squeeze(data_t(:,laser_index, camera_index)); 68 | 69 | % Wave convolution 70 | tmp_real = conv(time_response,Virtual_Wave_sin, 'same'); 71 | tmp_img = conv(time_response,Virtual_Wave_cos, 'same'); 72 | real(:,laser_index, camera_index) = tmp_real; 73 | img(:,laser_index, camera_index) = tmp_img; 74 | 75 | end 76 | end 77 | 78 | end 79 | --------------------------------------------------------------------------------