├── .DS_Store ├── .gitignore ├── .idea └── vcs.xml ├── LICENSE ├── README.md ├── epipolar_geometry.py ├── example ├── .DS_Store ├── features_poster.txt ├── poster1.jpg ├── poster2.JPG └── poster3.JPG ├── image_sequence.py ├── img ├── .DS_Store └── demo.png ├── model.py ├── rec_model_cloud.txt ├── reconstructed_model.stl └── uncalibrated_rec.py /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nbarba/py3DRec/369f26d807631e21b405d19697fe158158784ffb/.DS_Store -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | -------------------------------------------------------------------------------- /.idea/vcs.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Nikolaos Barmpalios 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # py3DRec 2 | 3 | This repository contains a revised implementation of the structure from motion algorithm used in [Barbalios et al., "3D Human Face Modelling From Uncalibrated Images Using Spline Based Deformation", VISSAP 2008](http://poseidon.csd.auth.gr/papers/PUBLISHED/CONFERENCE/pdf/VISAPP_final.pdf). Given at least three views of an object, and identified feature points across these views, the algorithm generates a 3D reconstruction of the object. 4 | 5 | 6 | ## Overview 7 | 8 | The algorithm takes a series of features (i.e. 2D coordinates), tracked through a sequence of images (i.e. taken with a common handheld camera), and returns the 3D coordinates of these features in the metric space. To make this happen, the algorithm consists of the following steps: 9 | * Selecting two views (images), e.g. i-th and j-th view, the fundamental matrix and the epipoles are computed from the corresponding 2D features of the i-th and j-th view. For best results, the fartherst views in the sequence are selected. 10 | * Estimating the projection matrices for the i-th and j-th view. To do so, the i-th view is assumed to be aligned with the world frame, and the projection matrix for the j-th view can be deduced using the fundamental matrix, the epipole and the reference frame of the reconstruction. 11 | * Triangulation of the 2D features of the i-th and j-th views to get an initial estimate of the 3D point coordinates of the 2D features. 12 | * Estimating the projection matrices for all the remaining views, using the 3D points we got from triangulation. 13 | * Bundle adjustment, i.e. an overall optimization to refine the 3D points and the projection matrices by minimizing the reprojection error of the 3D points back to each view. 14 | * Self-calibration to estimate the camera intrisic parameters for each view, and transform the 3D point coordinates from projective to metric space. 15 | * Delaunay triangulation of the 3D points, to get a 3D structure. 16 | 17 | Note: The algorithm used in the above referenced paper included an additional step, where the 3D points were used to deform a generic face model. 18 | 19 | ![Overview](./img/demo.png) 20 | 21 | ## Prerequisites 22 | The following python packages are required to run the projet: 23 | * [NumPy](http://www.numpy.org) 24 | * [pandas](http://pandas.pydata.org) 25 | * [SciPy](https://www.scipy.org) 26 | * [numpy-stl](https://pypi.python.org/pypi/numpy-stl) 27 | * [pillow](https://pillow.readthedocs.io/en/4.3.x/) 28 | 29 | 30 | ## Usage 31 | ### Code Structure 32 | 33 | The code consists of four classes, each one designed for a specific task: 34 | * __ImageSequence:__ holds all the image sequence related stuff, like sequence length, width and height of each image, 2D feature coordinates across all images. It also contains a method (i.e. show()), to visualize the image sequence with the 2D features highlighted. 35 | * __EpipolarGeometry:__ implements epipolar geometry related operations, such as computing the fundamental matrix, finding the epipoles, triangulation, computing homographies, and the reprojection error for one or multiple views. 36 | * __UncalibratedReconstruction:__ the main class that implements the 3D modeling algorithm. Constructor arguments include: 37 | * _sequence length:_ the length of the image sequence 38 | * _width:_ the width of images in the sequence 39 | * _length:_ the length of the images in the sequence 40 | * _triang_method:_ triangulation method (0: standard triangulation, 1: polynomial triangulation) 41 | * _opt_triang:_ optimize triangulation result (i.e. 3D points) 42 | * _opt_f:_ optimize fundamental matrix estimation 43 | * _self_foc:_ for self-calibration, defines the type of focal length expected across views (0: fixed, 1: varying) 44 | * __RecModel:__ holds the result of the reconstruction, i.e. projection matrices, rotation matrices, translation vectors, camera intrisic parameters vectors and the 3D structure point coordinates. It also contains a method, named export_stl_file, that performs Delaunay triangulation and saves the result in stl format. 45 | 46 | ## Example 47 | A toy-example is provided in the examples folder. It is a sequence of pictures of a poster on a wall (3 images). The 2D features were automatically selected using the SIFT algorithm, and are given in a txt file that has the following format: 48 | | x coordinates | y coordinates | feature id | image id |. 49 | 50 | To run the example, use the following command: 51 | ``` 52 | python uncalibrated_rec.py --input_file=./example/features_poster.txt --show 53 | ``` 54 | The first argument (i.e. --input_file) defines the txt file with the features, the second flag (--show) displays the image sequence together with the 2D features (remove this argument to not show the sequence). 55 | 56 | After the algorithm is executed, two files should be generated: 57 | * rec_model_cloud.txt : contains the 3D homogeneous coordinates of the reconstructed 3D model. 58 | * reconstructed_model.stl: an stl file of the reconstructed 3D model 59 | 60 | 61 | ## License 62 | 63 | This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details 64 | 65 | -------------------------------------------------------------------------------- /epipolar_geometry.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from numpy.linalg import inv 3 | from numpy.linalg import svd 4 | from numpy.linalg import eig 5 | from scipy.optimize import leastsq,least_squares,fmin 6 | 7 | 8 | 9 | class EpipolarGeometry: 10 | ''' 11 | Class that implements basic epipolar geometry operations 12 | ''' 13 | 14 | 15 | def fundamental_matrix(self,view1_feat2D,view2_feat2D,optimize): 16 | ''' 17 | Method to computes the fundamental matrix betwen two images. 18 | Args: 19 | view1_feat2D: 2D feature coordinates in view 1 20 | view2_feat2D: 2D feature coordinates in view 2 21 | optimize: optimize the F estimation using non-linear least squares 22 | Returns: 23 | the fundamendal matrix F 24 | ''' 25 | number_of_features=view1_feat2D.shape[1] 26 | 27 | #convert to homogeneous coordinates 28 | view1_feat2D=np.vstack((view1_feat2D,np.ones(number_of_features))) 29 | view2_feat2D=np.vstack((view2_feat2D,np.ones(number_of_features))) 30 | 31 | # create a homogeneous system Af=0 (f : elements of fundamental matrix), using mFm'=0 32 | A=np.zeros((number_of_features,9)) 33 | A[:,0]=np.transpose(view2_feat2D[0,:])*(view1_feat2D[0,:]) 34 | A[:,1]=np.transpose(view2_feat2D[0,:])*(view1_feat2D[1,:]) 35 | A[:,2]=np.transpose(view2_feat2D[0,:]) 36 | A[:,3]=np.transpose(view1_feat2D[0,:])*(view1_feat2D[1,:]) 37 | A[:,4]=np.transpose(view2_feat2D[1,:])*(view1_feat2D[1,:]) 38 | A[:,5]=np.transpose(view2_feat2D[1,:]) 39 | A[:,6]=np.transpose(view1_feat2D[0,:]) 40 | A[:,7]=np.transpose(view1_feat2D[1,:]) 41 | A[:,8]=np.ones(number_of_features) 42 | 43 | 44 | # use the eigenvector that corresponds to the smallest eigenvalue as initial estimate for F. 45 | U, s, Vh = svd(A) 46 | V=np.transpose(Vh) 47 | 48 | # fundamental matrix is now the eigenvector corresponding to the smallest eigen value. 49 | F=np.reshape(V[:,8],(3,3)) 50 | 51 | # make sure that fundamental matrix is of rank 2 52 | U,s,V = svd(F); 53 | s[2]=0 54 | S = np.diag(s) 55 | F=np.dot(U, np.dot(S, V)) 56 | 57 | if optimize==1: 58 | #Optimize initial estimate using the algebraic error 59 | f=np.append(np.concatenate((F[0,:],F[1,:]),axis=0),F[2,0])/F[2,2] 60 | 61 | result=least_squares(self.fundamental_matrix_error,f, args=(view1_feat2D[0:2,:],view2_feat2D[0:2,:])) 62 | f=result.x 63 | 64 | F=np.asarray([np.transpose(f[0:3]), np.transpose(f[3:6]), [f[6],-(-f[0]*f[4]+f[6]*f[2]*f[4]+f[3]*f[1]-f[6]*f[1]*f[5])/(-f[3]*f[2]+f[0]*f[5]),1]]); 65 | F=F/sum(sum(F))*9; 66 | 67 | return F 68 | 69 | 70 | def fundamental_matrix_error(self,f,view1_feat2D,view2_feat2D): 71 | ''' 72 | Method to compute the fundamental matrix error based on the epipolar constraint (mFm'=0) 73 | Args: 74 | f : a vector with the elements of the fundamental matrix 75 | view1_feat2D: 2D feature coordinates in view 1 76 | view2_feat2D: 2D feature coordinates in view 2 77 | Returns: 78 | the error based on mFm'=0 79 | 80 | ''' 81 | F=np.asarray([np.transpose(f[0:3]), np.transpose(f[3:6]), [f[6],-(-f[0]*f[4]+f[6]*f[2]*f[4]+f[3]*f[1]-f[6]*f[1]*f[5])/(-f[3]*f[2]+f[0]*f[5]),1]]); 82 | 83 | number_of_features=view1_feat2D.shape[1] 84 | #convert to homogeneous coordinates 85 | view1_feat2D=np.vstack((view1_feat2D,np.ones(number_of_features))) 86 | view2_feat2D=np.vstack((view2_feat2D,np.ones(number_of_features))) 87 | #compute error 88 | error=np.zeros((number_of_features,1)) 89 | for i in range(0,number_of_features): 90 | error[i]=np.dot(view2_feat2D[:,i],np.dot(F,np.transpose(view1_feat2D[:,i]))); 91 | 92 | return error.flatten() 93 | 94 | def get_epipole(self,F): 95 | ''' 96 | Method to return the epipole from the fundamental matrix 97 | Args: 98 | F: the fundamental matrix 99 | Retruns: 100 | the epipole 101 | ''' 102 | u,v=eig(F) 103 | dd=np.square(u); 104 | min_index=np.argmin(np.abs(dd)) # SOS: These are complex numbers, so compare their modulus 105 | epipole=v[:,min_index] 106 | epipole=epipole/epipole[2] 107 | return epipole.real 108 | 109 | def compute_homography(self,epipole,F): 110 | ''' 111 | Method that computes the homograhpy [epipole]x[F] 112 | Args: 113 | epipole: the epipole 114 | F : the fundamental matrix 115 | Returns: 116 | the homography H 117 | ''' 118 | e_12=np.asarray([[0,-epipole[2], epipole[1]],[epipole[2],0,-epipole[0]],[-epipole[1],epipole[0],0]]); 119 | H=np.dot(e_12,F) 120 | H=H*np.sign(np.trace(H)) 121 | return H 122 | 123 | 124 | def triangulate_points(self,view1_feat2D,view2_feat2D,P1,P2): 125 | ''' 126 | Method that triangulates using two projection matrices, and the 127 | normalized 2D features coordinates. 128 | Args: 129 | view1_feat2D: 2D feature coordinates in view 1 130 | view2_feat2D: 2D feature coordinates in view 2 131 | P1: projection matrix for view 1 132 | P2: projection matrix for view 2 133 | Returns: 134 | the 3D points 135 | ''' 136 | A=np.zeros((4,4)); 137 | A[0,:]=P1[2,:]*view1_feat2D[0]-P1[0,:] 138 | A[1,:]=P1[2,:]*view1_feat2D[1]-P1[1,:] 139 | A[2,:]=P2[2,:]*view2_feat2D[0]-P2[0,:] 140 | A[3,:]=P2[2,:]*view2_feat2D[1]-P2[1,:] 141 | 142 | U, s, Vh = svd(A) 143 | V=np.transpose(Vh) 144 | feat3D=V[:,V.shape[0]-1] 145 | feat3D=feat3D/feat3D[3] 146 | return feat3D 147 | 148 | 149 | def compute_L_R(self,view_feat2D,epipole): 150 | ''' 151 | Utility method used in polynomial triangulation 152 | ''' 153 | L=np.eye(3); 154 | L[0:2,2]=-view_feat2D; 155 | 156 | th = np.arctan(-(epipole[1]-epipole[2]*view_feat2D[1])/(epipole[0]-epipole[2]*view_feat2D[0])); 157 | R=np.eye(3) 158 | R[0,0]=np.cos(th) 159 | R[1,1]=np.cos(th) 160 | R[0,1]=-np.sin(th) 161 | R[1,0]=np.sin(th) 162 | return L,R 163 | 164 | def polynomial_triangulation(self,feat1,feat2,epipole_1,epipole_2,F,P1,P2): 165 | ''' 166 | Method to perform 'polynomial' triangulation, as suggested in 167 | R. Hartley and P. Sturm, "Triangulation", Computer Vision and Image Understanding, 68(2):146-157, 1997 168 | The method searches for the epipolar line which fits best to both feature points, and then project both points on tis line. Triangulation 169 | is then performed on these new points. (This now happens through SVD since the error will be zero) 170 | Args: 171 | feat1: 2D feature coordinates in view 1 172 | feat2: 2D feature coordinates in view 2 173 | epipole_1: the epipole of view 1 174 | epipole_2: the epipole of view 2 175 | F: the fundamental matrix 176 | P1: projection matrix for view 1 177 | P2: projection matrix for view 2 178 | Returns: 179 | the 3D points 180 | ''' 181 | L1,R1=self.compute_L_R(feat1,epipole_1) 182 | L2,R2=self.compute_L_R(feat2,epipole_2) 183 | 184 | F1=R2.dot(np.transpose(inv(L2))).dot(F).dot(inv(L1)).dot(np.transpose(R1)) 185 | a=F1[1,1]; 186 | b=F1[1,2]; 187 | c=F1[2,1]; 188 | d=F1[2,2]; 189 | f1=-F1[1,0]/b; 190 | f2=-F1[0,1]/c; 191 | 192 | p=np.zeros(7); 193 | p[0]=-2*np.power(f1,4)*np.power(a,2)*c*d+2*np.power(f1,4)*a*np.power(c,2)*b; 194 | p[1]=2*np.power(a,4)+2*np.power(f2,4)*np.power(c,4)-2*np.power(f1,4)*np.power(a,2)*np.power(d,2)+2*np.power(f1,4)*np.power(b,2)*np.power(c,2)+4*np.power(a,2)*np.power(f2,2)*np.power(c,2); 195 | p[2]=8*np.power(f2,4)*d*np.power(c,3)+8*b*np.power(a,3)+8*b*a*np.power(f2,2)*np.power(c,2)+8*np.power(f2,2)*d*c*np.power(a,2)-4*np.power(f1,2)*np.power(a,2)*c*d+4*np.power(f1,2)*a*np.power(c,2)*b-2*np.power(f1,4)*b*np.power(d,2)*a+2*np.power(f1,4)*np.power(b,2)*d*c; 196 | p[3]=-4*np.power(f1,2)*np.power(a,2)*np.power(d,2)+4*np.power(f1,2)*np.power(b,2)*np.power(c,2)+4*np.power(b,2)*np.power(f2,2)*np.power(c,2)+16*b*a*np.power(f2,2)*d*c+12*np.power(f2,4)*np.power(d,2)*np.power(c,2)+12*np.power(b,2)*np.power(a,2)+4*np.power(f2,2)*np.power(d,2)*np.power(a,2); 197 | p[4]=8*np.power(f2,4)*np.power(d,3)*c+8*np.power(b,2)*np.power(f2,2)*d*c+8*np.power(f2,2)*np.power(d,2)*b*a-4*np.power(f1,2)*b*np.power(d,2)*a+4*np.power(f1,2)*np.power(b,2)*d*c+2*a*np.power(c,2)*b+8*np.power(b,3)*a-2*np.power(a,2)*c*d; 198 | p[5]=4*np.power(b,2)*np.power(f2,2)*np.power(d,2)+2*np.power(f2,4)*np.power(d,4)+2*np.power(b,2)*np.power(c,2)-2*np.power(a,2)*np.power(d,2)+2*np.power(b,4); 199 | p[6]=-2*b*d*(a*d-b*c); 200 | 201 | r=np.roots(p) 202 | y=np.polyval(p,r) 203 | 204 | aa=max(y) 205 | i=0; 206 | ii=0; 207 | 208 | for i in range(0,6): 209 | if ((np.imag(r[i])==0) and (abs(y[i]) Estimating fundamental matrix....") 437 | F, epipole_1, epipole_2 = rec_engine.two_view_geometry_computation(norm_feat_2d[0], norm_feat_2d[1]) 438 | 439 | print("> Computing reference plane....") 440 | # Step 2: compute the reconstruction reference plane using the epipole in the second image 441 | p, H = rec_engine.compute_reference_frame(epipole_2, F) 442 | 443 | print("> Estimating projection matrices for first two views....") 444 | # Step 3: Estimate projection matrices for the first two views 445 | P = rec_engine.estimate_initial_projection_matrices(H, epipole_2, p) 446 | 447 | print("> 3D point estimate triangulation....") 448 | # Step 4: triangulate points to get an initial estimate of the 3D point cloud 449 | feat3D = rec_engine.get_initial_structure(norm_feat_2d, P, epipole_1, epipole_2, F) 450 | 451 | print("> Estimating projection matrices for additional views....") 452 | # Step 5: Use the 3D point estimates to estimate the projection matrices of the remaining views 453 | P = rec_engine.projective_pose_estimation(norm_feat_2d, P, feat3D) 454 | 455 | print("> Bundle Adjustment....") 456 | # Step 5: Optimize 3D points and projection matrices using the reprojection error 457 | P, feat3D, error = rec_engine.bundle_adjustment(norm_feat_2d, P, feat3D) 458 | print(" - Bundle adjustment error: ", error) 459 | 460 | print("> Self-calibration") 461 | # Step 6: Self-calibration 462 | Tm, K, error = rec_engine.self_calibration(P) 463 | print(" - Self-calibration error: ", error) 464 | print(" - Tranformation Matrix (Projective -> Metric): ") 465 | 466 | print("> Converting to metric space") 467 | metric_feat3D, metric_P = rec_engine.convert_to_metric_space(Tm, feat3D, P, K) 468 | 469 | print("> Saving model...") 470 | 471 | recModel = RecModel() 472 | recModel.P = P 473 | recModel.points3D = metric_feat3D 474 | recModel.Tm = Tm 475 | 476 | np.savetxt('rec_model_cloud.txt', metric_feat3D, delimiter=',') 477 | print(" - 3D point cloud saved in rec_model_cloud.txt ") 478 | 479 | recModel.export_stl_file('reconstructed_model.stl') 480 | print(" - STL model saved in reconstructed_model.stl") 481 | 482 | if (show == True): 483 | sequence.show() 484 | 485 | print("> 3D reconstruction completed in " + str(round(time.time() - start_time, 1)) + " sec!") 486 | 487 | 488 | if __name__ == "__main__": 489 | parser = argparse.ArgumentParser(description='3D Reconstruction from uncalibrated images') 490 | parser.add_argument('--input_file', metavar='path', required=True, 491 | help='Input file containing image point correspondences') 492 | parser.add_argument('--show', required=False, action="store_true", 493 | help="Display the image sequence with the 2D features") 494 | args = parser.parse_args() 495 | main(input_file=args.input_file, show=args.show) 496 | --------------------------------------------------------------------------------