├── .DS_Store
├── .gitignore
├── .idea
└── vcs.xml
├── LICENSE
├── README.md
├── epipolar_geometry.py
├── example
├── .DS_Store
├── features_poster.txt
├── poster1.jpg
├── poster2.JPG
└── poster3.JPG
├── image_sequence.py
├── img
├── .DS_Store
└── demo.png
├── model.py
├── rec_model_cloud.txt
├── reconstructed_model.stl
└── uncalibrated_rec.py
/.DS_Store:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nbarba/py3DRec/369f26d807631e21b405d19697fe158158784ffb/.DS_Store
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | # Byte-compiled / optimized / DLL files
2 | __pycache__/
3 | *.py[cod]
4 | *$py.class
5 |
6 | # C extensions
7 | *.so
8 |
9 | # Distribution / packaging
10 | .Python
11 | env/
12 | build/
13 | develop-eggs/
14 | dist/
15 | downloads/
16 | eggs/
17 | .eggs/
18 | lib/
19 | lib64/
20 | parts/
21 | sdist/
22 | var/
23 | wheels/
24 | *.egg-info/
25 | .installed.cfg
26 | *.egg
27 |
28 | # PyInstaller
29 | # Usually these files are written by a python script from a template
30 | # before PyInstaller builds the exe, so as to inject date/other infos into it.
31 | *.manifest
32 | *.spec
33 |
34 | # Installer logs
35 | pip-log.txt
36 | pip-delete-this-directory.txt
37 |
38 | # Unit test / coverage reports
39 | htmlcov/
40 | .tox/
41 | .coverage
42 | .coverage.*
43 | .cache
44 | nosetests.xml
45 | coverage.xml
46 | *.cover
47 | .hypothesis/
48 |
49 | # Translations
50 | *.mo
51 | *.pot
52 |
53 | # Django stuff:
54 | *.log
55 | local_settings.py
56 |
57 | # Flask stuff:
58 | instance/
59 | .webassets-cache
60 |
61 | # Scrapy stuff:
62 | .scrapy
63 |
64 | # Sphinx documentation
65 | docs/_build/
66 |
67 | # PyBuilder
68 | target/
69 |
70 | # Jupyter Notebook
71 | .ipynb_checkpoints
72 |
73 | # pyenv
74 | .python-version
75 |
76 | # celery beat schedule file
77 | celerybeat-schedule
78 |
79 | # SageMath parsed files
80 | *.sage.py
81 |
82 | # dotenv
83 | .env
84 |
85 | # virtualenv
86 | .venv
87 | venv/
88 | ENV/
89 |
90 | # Spyder project settings
91 | .spyderproject
92 | .spyproject
93 |
94 | # Rope project settings
95 | .ropeproject
96 |
97 | # mkdocs documentation
98 | /site
99 |
100 | # mypy
101 | .mypy_cache/
102 |
--------------------------------------------------------------------------------
/.idea/vcs.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2017 Nikolaos Barmpalios
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # py3DRec
2 |
3 | This repository contains a revised implementation of the structure from motion algorithm used in [Barbalios et al., "3D Human Face Modelling From Uncalibrated Images Using Spline Based Deformation", VISSAP 2008](http://poseidon.csd.auth.gr/papers/PUBLISHED/CONFERENCE/pdf/VISAPP_final.pdf). Given at least three views of an object, and identified feature points across these views, the algorithm generates a 3D reconstruction of the object.
4 |
5 |
6 | ## Overview
7 |
8 | The algorithm takes a series of features (i.e. 2D coordinates), tracked through a sequence of images (i.e. taken with a common handheld camera), and returns the 3D coordinates of these features in the metric space. To make this happen, the algorithm consists of the following steps:
9 | * Selecting two views (images), e.g. i-th and j-th view, the fundamental matrix and the epipoles are computed from the corresponding 2D features of the i-th and j-th view. For best results, the fartherst views in the sequence are selected.
10 | * Estimating the projection matrices for the i-th and j-th view. To do so, the i-th view is assumed to be aligned with the world frame, and the projection matrix for the j-th view can be deduced using the fundamental matrix, the epipole and the reference frame of the reconstruction.
11 | * Triangulation of the 2D features of the i-th and j-th views to get an initial estimate of the 3D point coordinates of the 2D features.
12 | * Estimating the projection matrices for all the remaining views, using the 3D points we got from triangulation.
13 | * Bundle adjustment, i.e. an overall optimization to refine the 3D points and the projection matrices by minimizing the reprojection error of the 3D points back to each view.
14 | * Self-calibration to estimate the camera intrisic parameters for each view, and transform the 3D point coordinates from projective to metric space.
15 | * Delaunay triangulation of the 3D points, to get a 3D structure.
16 |
17 | Note: The algorithm used in the above referenced paper included an additional step, where the 3D points were used to deform a generic face model.
18 |
19 | 
20 |
21 | ## Prerequisites
22 | The following python packages are required to run the projet:
23 | * [NumPy](http://www.numpy.org)
24 | * [pandas](http://pandas.pydata.org)
25 | * [SciPy](https://www.scipy.org)
26 | * [numpy-stl](https://pypi.python.org/pypi/numpy-stl)
27 | * [pillow](https://pillow.readthedocs.io/en/4.3.x/)
28 |
29 |
30 | ## Usage
31 | ### Code Structure
32 |
33 | The code consists of four classes, each one designed for a specific task:
34 | * __ImageSequence:__ holds all the image sequence related stuff, like sequence length, width and height of each image, 2D feature coordinates across all images. It also contains a method (i.e. show()), to visualize the image sequence with the 2D features highlighted.
35 | * __EpipolarGeometry:__ implements epipolar geometry related operations, such as computing the fundamental matrix, finding the epipoles, triangulation, computing homographies, and the reprojection error for one or multiple views.
36 | * __UncalibratedReconstruction:__ the main class that implements the 3D modeling algorithm. Constructor arguments include:
37 | * _sequence length:_ the length of the image sequence
38 | * _width:_ the width of images in the sequence
39 | * _length:_ the length of the images in the sequence
40 | * _triang_method:_ triangulation method (0: standard triangulation, 1: polynomial triangulation)
41 | * _opt_triang:_ optimize triangulation result (i.e. 3D points)
42 | * _opt_f:_ optimize fundamental matrix estimation
43 | * _self_foc:_ for self-calibration, defines the type of focal length expected across views (0: fixed, 1: varying)
44 | * __RecModel:__ holds the result of the reconstruction, i.e. projection matrices, rotation matrices, translation vectors, camera intrisic parameters vectors and the 3D structure point coordinates. It also contains a method, named export_stl_file, that performs Delaunay triangulation and saves the result in stl format.
45 |
46 | ## Example
47 | A toy-example is provided in the examples folder. It is a sequence of pictures of a poster on a wall (3 images). The 2D features were automatically selected using the SIFT algorithm, and are given in a txt file that has the following format:
48 | | x coordinates | y coordinates | feature id | image id |.
49 |
50 | To run the example, use the following command:
51 | ```
52 | python uncalibrated_rec.py --input_file=./example/features_poster.txt --show
53 | ```
54 | The first argument (i.e. --input_file) defines the txt file with the features, the second flag (--show) displays the image sequence together with the 2D features (remove this argument to not show the sequence).
55 |
56 | After the algorithm is executed, two files should be generated:
57 | * rec_model_cloud.txt : contains the 3D homogeneous coordinates of the reconstructed 3D model.
58 | * reconstructed_model.stl: an stl file of the reconstructed 3D model
59 |
60 |
61 | ## License
62 |
63 | This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details
64 |
65 |
--------------------------------------------------------------------------------
/epipolar_geometry.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | from numpy.linalg import inv
3 | from numpy.linalg import svd
4 | from numpy.linalg import eig
5 | from scipy.optimize import leastsq,least_squares,fmin
6 |
7 |
8 |
9 | class EpipolarGeometry:
10 | '''
11 | Class that implements basic epipolar geometry operations
12 | '''
13 |
14 |
15 | def fundamental_matrix(self,view1_feat2D,view2_feat2D,optimize):
16 | '''
17 | Method to computes the fundamental matrix betwen two images.
18 | Args:
19 | view1_feat2D: 2D feature coordinates in view 1
20 | view2_feat2D: 2D feature coordinates in view 2
21 | optimize: optimize the F estimation using non-linear least squares
22 | Returns:
23 | the fundamendal matrix F
24 | '''
25 | number_of_features=view1_feat2D.shape[1]
26 |
27 | #convert to homogeneous coordinates
28 | view1_feat2D=np.vstack((view1_feat2D,np.ones(number_of_features)))
29 | view2_feat2D=np.vstack((view2_feat2D,np.ones(number_of_features)))
30 |
31 | # create a homogeneous system Af=0 (f : elements of fundamental matrix), using mFm'=0
32 | A=np.zeros((number_of_features,9))
33 | A[:,0]=np.transpose(view2_feat2D[0,:])*(view1_feat2D[0,:])
34 | A[:,1]=np.transpose(view2_feat2D[0,:])*(view1_feat2D[1,:])
35 | A[:,2]=np.transpose(view2_feat2D[0,:])
36 | A[:,3]=np.transpose(view1_feat2D[0,:])*(view1_feat2D[1,:])
37 | A[:,4]=np.transpose(view2_feat2D[1,:])*(view1_feat2D[1,:])
38 | A[:,5]=np.transpose(view2_feat2D[1,:])
39 | A[:,6]=np.transpose(view1_feat2D[0,:])
40 | A[:,7]=np.transpose(view1_feat2D[1,:])
41 | A[:,8]=np.ones(number_of_features)
42 |
43 |
44 | # use the eigenvector that corresponds to the smallest eigenvalue as initial estimate for F.
45 | U, s, Vh = svd(A)
46 | V=np.transpose(Vh)
47 |
48 | # fundamental matrix is now the eigenvector corresponding to the smallest eigen value.
49 | F=np.reshape(V[:,8],(3,3))
50 |
51 | # make sure that fundamental matrix is of rank 2
52 | U,s,V = svd(F);
53 | s[2]=0
54 | S = np.diag(s)
55 | F=np.dot(U, np.dot(S, V))
56 |
57 | if optimize==1:
58 | #Optimize initial estimate using the algebraic error
59 | f=np.append(np.concatenate((F[0,:],F[1,:]),axis=0),F[2,0])/F[2,2]
60 |
61 | result=least_squares(self.fundamental_matrix_error,f, args=(view1_feat2D[0:2,:],view2_feat2D[0:2,:]))
62 | f=result.x
63 |
64 | F=np.asarray([np.transpose(f[0:3]), np.transpose(f[3:6]), [f[6],-(-f[0]*f[4]+f[6]*f[2]*f[4]+f[3]*f[1]-f[6]*f[1]*f[5])/(-f[3]*f[2]+f[0]*f[5]),1]]);
65 | F=F/sum(sum(F))*9;
66 |
67 | return F
68 |
69 |
70 | def fundamental_matrix_error(self,f,view1_feat2D,view2_feat2D):
71 | '''
72 | Method to compute the fundamental matrix error based on the epipolar constraint (mFm'=0)
73 | Args:
74 | f : a vector with the elements of the fundamental matrix
75 | view1_feat2D: 2D feature coordinates in view 1
76 | view2_feat2D: 2D feature coordinates in view 2
77 | Returns:
78 | the error based on mFm'=0
79 |
80 | '''
81 | F=np.asarray([np.transpose(f[0:3]), np.transpose(f[3:6]), [f[6],-(-f[0]*f[4]+f[6]*f[2]*f[4]+f[3]*f[1]-f[6]*f[1]*f[5])/(-f[3]*f[2]+f[0]*f[5]),1]]);
82 |
83 | number_of_features=view1_feat2D.shape[1]
84 | #convert to homogeneous coordinates
85 | view1_feat2D=np.vstack((view1_feat2D,np.ones(number_of_features)))
86 | view2_feat2D=np.vstack((view2_feat2D,np.ones(number_of_features)))
87 | #compute error
88 | error=np.zeros((number_of_features,1))
89 | for i in range(0,number_of_features):
90 | error[i]=np.dot(view2_feat2D[:,i],np.dot(F,np.transpose(view1_feat2D[:,i])));
91 |
92 | return error.flatten()
93 |
94 | def get_epipole(self,F):
95 | '''
96 | Method to return the epipole from the fundamental matrix
97 | Args:
98 | F: the fundamental matrix
99 | Retruns:
100 | the epipole
101 | '''
102 | u,v=eig(F)
103 | dd=np.square(u);
104 | min_index=np.argmin(np.abs(dd)) # SOS: These are complex numbers, so compare their modulus
105 | epipole=v[:,min_index]
106 | epipole=epipole/epipole[2]
107 | return epipole.real
108 |
109 | def compute_homography(self,epipole,F):
110 | '''
111 | Method that computes the homograhpy [epipole]x[F]
112 | Args:
113 | epipole: the epipole
114 | F : the fundamental matrix
115 | Returns:
116 | the homography H
117 | '''
118 | e_12=np.asarray([[0,-epipole[2], epipole[1]],[epipole[2],0,-epipole[0]],[-epipole[1],epipole[0],0]]);
119 | H=np.dot(e_12,F)
120 | H=H*np.sign(np.trace(H))
121 | return H
122 |
123 |
124 | def triangulate_points(self,view1_feat2D,view2_feat2D,P1,P2):
125 | '''
126 | Method that triangulates using two projection matrices, and the
127 | normalized 2D features coordinates.
128 | Args:
129 | view1_feat2D: 2D feature coordinates in view 1
130 | view2_feat2D: 2D feature coordinates in view 2
131 | P1: projection matrix for view 1
132 | P2: projection matrix for view 2
133 | Returns:
134 | the 3D points
135 | '''
136 | A=np.zeros((4,4));
137 | A[0,:]=P1[2,:]*view1_feat2D[0]-P1[0,:]
138 | A[1,:]=P1[2,:]*view1_feat2D[1]-P1[1,:]
139 | A[2,:]=P2[2,:]*view2_feat2D[0]-P2[0,:]
140 | A[3,:]=P2[2,:]*view2_feat2D[1]-P2[1,:]
141 |
142 | U, s, Vh = svd(A)
143 | V=np.transpose(Vh)
144 | feat3D=V[:,V.shape[0]-1]
145 | feat3D=feat3D/feat3D[3]
146 | return feat3D
147 |
148 |
149 | def compute_L_R(self,view_feat2D,epipole):
150 | '''
151 | Utility method used in polynomial triangulation
152 | '''
153 | L=np.eye(3);
154 | L[0:2,2]=-view_feat2D;
155 |
156 | th = np.arctan(-(epipole[1]-epipole[2]*view_feat2D[1])/(epipole[0]-epipole[2]*view_feat2D[0]));
157 | R=np.eye(3)
158 | R[0,0]=np.cos(th)
159 | R[1,1]=np.cos(th)
160 | R[0,1]=-np.sin(th)
161 | R[1,0]=np.sin(th)
162 | return L,R
163 |
164 | def polynomial_triangulation(self,feat1,feat2,epipole_1,epipole_2,F,P1,P2):
165 | '''
166 | Method to perform 'polynomial' triangulation, as suggested in
167 | R. Hartley and P. Sturm, "Triangulation", Computer Vision and Image Understanding, 68(2):146-157, 1997
168 | The method searches for the epipolar line which fits best to both feature points, and then project both points on tis line. Triangulation
169 | is then performed on these new points. (This now happens through SVD since the error will be zero)
170 | Args:
171 | feat1: 2D feature coordinates in view 1
172 | feat2: 2D feature coordinates in view 2
173 | epipole_1: the epipole of view 1
174 | epipole_2: the epipole of view 2
175 | F: the fundamental matrix
176 | P1: projection matrix for view 1
177 | P2: projection matrix for view 2
178 | Returns:
179 | the 3D points
180 | '''
181 | L1,R1=self.compute_L_R(feat1,epipole_1)
182 | L2,R2=self.compute_L_R(feat2,epipole_2)
183 |
184 | F1=R2.dot(np.transpose(inv(L2))).dot(F).dot(inv(L1)).dot(np.transpose(R1))
185 | a=F1[1,1];
186 | b=F1[1,2];
187 | c=F1[2,1];
188 | d=F1[2,2];
189 | f1=-F1[1,0]/b;
190 | f2=-F1[0,1]/c;
191 |
192 | p=np.zeros(7);
193 | p[0]=-2*np.power(f1,4)*np.power(a,2)*c*d+2*np.power(f1,4)*a*np.power(c,2)*b;
194 | p[1]=2*np.power(a,4)+2*np.power(f2,4)*np.power(c,4)-2*np.power(f1,4)*np.power(a,2)*np.power(d,2)+2*np.power(f1,4)*np.power(b,2)*np.power(c,2)+4*np.power(a,2)*np.power(f2,2)*np.power(c,2);
195 | p[2]=8*np.power(f2,4)*d*np.power(c,3)+8*b*np.power(a,3)+8*b*a*np.power(f2,2)*np.power(c,2)+8*np.power(f2,2)*d*c*np.power(a,2)-4*np.power(f1,2)*np.power(a,2)*c*d+4*np.power(f1,2)*a*np.power(c,2)*b-2*np.power(f1,4)*b*np.power(d,2)*a+2*np.power(f1,4)*np.power(b,2)*d*c;
196 | p[3]=-4*np.power(f1,2)*np.power(a,2)*np.power(d,2)+4*np.power(f1,2)*np.power(b,2)*np.power(c,2)+4*np.power(b,2)*np.power(f2,2)*np.power(c,2)+16*b*a*np.power(f2,2)*d*c+12*np.power(f2,4)*np.power(d,2)*np.power(c,2)+12*np.power(b,2)*np.power(a,2)+4*np.power(f2,2)*np.power(d,2)*np.power(a,2);
197 | p[4]=8*np.power(f2,4)*np.power(d,3)*c+8*np.power(b,2)*np.power(f2,2)*d*c+8*np.power(f2,2)*np.power(d,2)*b*a-4*np.power(f1,2)*b*np.power(d,2)*a+4*np.power(f1,2)*np.power(b,2)*d*c+2*a*np.power(c,2)*b+8*np.power(b,3)*a-2*np.power(a,2)*c*d;
198 | p[5]=4*np.power(b,2)*np.power(f2,2)*np.power(d,2)+2*np.power(f2,4)*np.power(d,4)+2*np.power(b,2)*np.power(c,2)-2*np.power(a,2)*np.power(d,2)+2*np.power(b,4);
199 | p[6]=-2*b*d*(a*d-b*c);
200 |
201 | r=np.roots(p)
202 | y=np.polyval(p,r)
203 |
204 | aa=max(y)
205 | i=0;
206 | ii=0;
207 |
208 | for i in range(0,6):
209 | if ((np.imag(r[i])==0) and (abs(y[i]) Estimating fundamental matrix....")
437 | F, epipole_1, epipole_2 = rec_engine.two_view_geometry_computation(norm_feat_2d[0], norm_feat_2d[1])
438 |
439 | print("> Computing reference plane....")
440 | # Step 2: compute the reconstruction reference plane using the epipole in the second image
441 | p, H = rec_engine.compute_reference_frame(epipole_2, F)
442 |
443 | print("> Estimating projection matrices for first two views....")
444 | # Step 3: Estimate projection matrices for the first two views
445 | P = rec_engine.estimate_initial_projection_matrices(H, epipole_2, p)
446 |
447 | print("> 3D point estimate triangulation....")
448 | # Step 4: triangulate points to get an initial estimate of the 3D point cloud
449 | feat3D = rec_engine.get_initial_structure(norm_feat_2d, P, epipole_1, epipole_2, F)
450 |
451 | print("> Estimating projection matrices for additional views....")
452 | # Step 5: Use the 3D point estimates to estimate the projection matrices of the remaining views
453 | P = rec_engine.projective_pose_estimation(norm_feat_2d, P, feat3D)
454 |
455 | print("> Bundle Adjustment....")
456 | # Step 5: Optimize 3D points and projection matrices using the reprojection error
457 | P, feat3D, error = rec_engine.bundle_adjustment(norm_feat_2d, P, feat3D)
458 | print(" - Bundle adjustment error: ", error)
459 |
460 | print("> Self-calibration")
461 | # Step 6: Self-calibration
462 | Tm, K, error = rec_engine.self_calibration(P)
463 | print(" - Self-calibration error: ", error)
464 | print(" - Tranformation Matrix (Projective -> Metric): ")
465 |
466 | print("> Converting to metric space")
467 | metric_feat3D, metric_P = rec_engine.convert_to_metric_space(Tm, feat3D, P, K)
468 |
469 | print("> Saving model...")
470 |
471 | recModel = RecModel()
472 | recModel.P = P
473 | recModel.points3D = metric_feat3D
474 | recModel.Tm = Tm
475 |
476 | np.savetxt('rec_model_cloud.txt', metric_feat3D, delimiter=',')
477 | print(" - 3D point cloud saved in rec_model_cloud.txt ")
478 |
479 | recModel.export_stl_file('reconstructed_model.stl')
480 | print(" - STL model saved in reconstructed_model.stl")
481 |
482 | if (show == True):
483 | sequence.show()
484 |
485 | print("> 3D reconstruction completed in " + str(round(time.time() - start_time, 1)) + " sec!")
486 |
487 |
488 | if __name__ == "__main__":
489 | parser = argparse.ArgumentParser(description='3D Reconstruction from uncalibrated images')
490 | parser.add_argument('--input_file', metavar='path', required=True,
491 | help='Input file containing image point correspondences')
492 | parser.add_argument('--show', required=False, action="store_true",
493 | help="Display the image sequence with the 2D features")
494 | args = parser.parse_args()
495 | main(input_file=args.input_file, show=args.show)
496 |
--------------------------------------------------------------------------------