├── README.md ├── images ├── 1.png ├── 2.png ├── fin1.jpg ├── fin2.jpg ├── s ├── 室内1.jpg ├── 室内2.jpg ├── 景深大1.jpg ├── 景深大2.jpg ├── 景深小A1.jpg ├── 景深小A2.jpg ├── 景深小B1.jpg └── 景深小B2.jpg └── t3.py /README.md: -------------------------------------------------------------------------------- 1 | # PythonComputerVision-4-ImageMosaic 2 | 全景图像拼接技术———在同一位置（即图像的照相机位置相同）拍摄的两幅或者多福图像是单应性相关的（p.s上一篇文章中详细介绍了“单应性相关概念”），我们经常使用该约束将很多图像缝补起来，拼成一个大的图像来创建全景图像。在本文中，将要探讨如何创建全景图像。 3 | ## 一.原理介绍 4 | 在进行图像拼接时，首先要解决的是找到图像之间的匹配的对应点。本文采用SIFT算法来实现特征点的匹配，SIFT算法的具体内容参照之前的文章：https://github.com/Nocami/PythonComputerVision-2-SIFT SIFT是很强大的描述子，它能产生很少的错误的匹配，但仍然还是存在错误的对应点。所以需要用一种算法对SIFT算法产生的特征描述符进行剔除误匹配点。 5 | ### 1)RANSAC 6 | RANSAC是“RANdom SAmple Consensus”(随机一致性采样)的缩写。RANSAC算法是一种经典的消除误匹配的方法,具有匹配精度高、可靠度强等优点，该方法是用来找到正确模型来拟合带有噪声数据的迭代方法。RANSAC的标准例子：用一条直线拟合带有噪声数据的点集。简单的最小二乘在该例子中可能会失效,但RANSAC可以挑选出正确的点，然后获取能够正确拟合的直线。 7 | #### 示例 8 | 从一组观测数据中找出合适的2维直线。所给出的观测数据中包含正确点和错误点，正确点可以相似的被直线所通过，而错误点远离于直线，分布在其两侧。普通的最小二乘法找不到那条贯穿全部点的直线，因为它会努力的去适应包括错误点在内的所有点。而RANSAC算法能得出一个仅仅用正确点的计算模型，且命中率很高。但尽管如此，它也不能保证100%正确。 9 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/1.png)![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/2.png) 10 | 图-包含很多点的数据集               图-RANSAC拟合的直线 11 | ### 2)单应性矩阵估计 12 | 在任何模型中都可以使用RANSAC模块，这里使用可能的对应点集来自动找到用于全景图像的单应性矩阵--使用SIFT特征自动找到匹配对应，可也使用如下代码： 13 | ~~~python 14 | import sift 15 | 16 | featname = ['./images5/'+str(i+1)+'.sift' for i in range(2)] 17 | imname = ['./images5/'+str(i+1)+'.jpg' for i in range(2)] 18 | l = {} 19 | d = {} 20 | for i in range(2): 21 | sift.process_image(imname[i],featname[i]) 22 | l[i],d[i] = sift.read_features_from_file(featname[i]) 23 | 24 | matches = {} 25 | for i in range(1): 26 | matches[i] = sift.match(d[i+1],d[i]) 27 | ~~~ 28 | **其中，第一个range（i）中i的值为要拼接的图像个数，第二个为第一个i-1。** 29 | 运行此例代码，我们可以看到，图像中的对应点并不是完全正确，还存在很多错误配对： 30 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%B0%8FA1.jpg) 31 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E5%AE%A4%E5%86%851.jpg) 32 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%A4%A71.jpg) 33 | ## 二.图像拼接 34 | 估计出图像见的单应性矩阵（使用RANSAC算法）后，将所有图像扭曲到一个公共平面上，就完成了一副简单的全景图像。一般的，这个公共平面选择为中心图像的平面，不然会发生大量的形变。因为我们的图像是由照相机水平旋转拍摄成的，所以我们可以使用一个简单的步骤：将中心图像左边或右边的区域填充0，为扭曲图像腾出空间。 35 | **p.s需要注意的是：**若拼接图为两张，则影响不会很大，中心图像做为平面中心的操作在多福图像拼接时会有明显的效果。 36 | ### 1)代码： 37 | ~~~python 38 | from pylab import * 39 | from numpy import * 40 | from PIL import Image 41 | 42 | # If you have PCV installed, these imports should work 43 | from PCV.geometry import homography, warp 44 | from PCV.localdescriptors import sift 45 | 46 | """ 47 | This is the panorama example from section 3.3. 48 | """ 49 | 50 | # set paths to data folder 51 | featname = ['./images5/'+str(i+1)+'.sift' for i in range(2)] 52 | imname = ['./images5/'+str(i+1)+'.jpg' for i in range(2)] 53 | 54 | # extract features and match 55 | l = {} 56 | d = {} 57 | for i in range(2): 58 | sift.process_image(imname[i],featname[i]) 59 | l[i],d[i] = sift.read_features_from_file(featname[i]) 60 | 61 | matches = {} 62 | for i in range(1): 63 | matches[i] = sift.match(d[i+1],d[i]) 64 | 65 | # visualize the matches (Figure 3-11 in the book) 66 | for i in range(1): 67 | im1 = array(Image.open(imname[i])) 68 | im2 = array(Image.open(imname[i+1])) 69 | figure() 70 | sift.plot_matches(im2,im1,l[i+1],l[i],matches[i],show_below=True) 71 | 72 | 73 | # function to convert the matches to hom. points 74 | def convert_points(j): 75 | ndx = matches[j].nonzero()[0] 76 | fp = homography.make_homog(l[j+1][ndx,:2].T) 77 | ndx2 = [int(matches[j][i]) for i in ndx] 78 | tp = homography.make_homog(l[j][ndx2,:2].T) 79 | 80 | # switch x and y - TODO this should move elsewhere 81 | fp = vstack([fp[1],fp[0],fp[2]]) 82 | tp = vstack([tp[1],tp[0],tp[2]]) 83 | return fp,tp 84 | 85 | 86 | # estimate the homographies 87 | model = homography.RansacModel() 88 | 89 | fp,tp = convert_points(0) 90 | H_01 = homography.H_from_ransac(fp,tp,model)[0] #im 0 to 1 91 | 92 | #fp,tp = convert_points(1) 93 | #H_12 = homography.H_from_ransac(fp,tp,model)[0] #im 1 to 2 94 | 95 | #tp,fp = convert_points(2) #NB: reverse order 96 | #H_32 = homography.H_from_ransac(fp,tp,model)[0] #im 3 to 2 97 | 98 | #tp,fp = convert_points(3) #NB: reverse order 99 | #H_43 = homography.H_from_ransac(fp,tp,model)[0] #im 4 to 3 100 | 101 | 102 | # warp the images 103 | delta = 2000 # for padding and translation 104 | 105 | im1 = array(Image.open(imname[0]), "uint8") 106 | im2 = array(Image.open(imname[1]), "uint8") 107 | im_12 = warp.panorama(H_01,im1,im2,delta,delta) 108 | #im1 = array(Image.open(imname[0]), "f") 109 | #im_02 = warp.panorama(dot(H_12,H_01),im1,im_12,delta,delta) 110 | 111 | #im1 = array(Image.open(imname[3]), "f") 112 | #im_32 = warp.panorama(H_32,im1,im_02,delta,delta) 113 | 114 | #im1 = array(Image.open(imname[4]), "f") 115 | #im_42 = warp.panorama(dot(H_32,H_43),im1,im_32,delta,2*delta) 116 | 117 | 118 | figure() 119 | imshow(array(im_12, "uint8")) 120 | axis('off') 121 | savefig("example1.png",dpi=300) 122 | show() 123 | 124 | 125 | ~~~ 126 | 此代码段为2图图像拼接，若需要多幅图，只需将其中的注释部分取消即可，图像顺序为自右向左。 127 | ### 2)实例效果 128 | 下面我们看看实际效果： 129 | #### 双图-室外情况下、景深较小： 130 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%B0%8FA1.jpg) 131 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%B0%8FA2.jpg) 132 | nice！看起来毫无PS痕迹！ 133 | 再看一组同样条件下的照片： 134 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%B0%8FB1.jpg) 135 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%B0%8FB2.jpg) 136 | 这一组可以看到明显的拼接缝隙，这是由照片的色差造成的，我们可以看到整体效果还不错。 137 | #### 双图-室外情况下、景深较大： 138 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%A4%A71.jpg) 139 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E6%99%AF%E6%B7%B1%E5%A4%A72.jpg) 140 | 当物体景深过大时，会产生尺度问题，影响拼接图像的质量，我们可以看到右下角树干有明显瑕疵。 141 | #### 双图-室内情况、焦距近、图像杂乱： 142 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E5%AE%A4%E5%86%851.jpg) 143 | ![image](https://github.com/Nocami/PythonComputerVision-4-ImageMosaic/blob/master/images/%E5%AE%A4%E5%86%852.jpg) 144 | 在这种情况下，算法效果就不是很好了，出现成功拼接的概率大大降低，很容易出现拼接错误等情况。 145 | #### 多图拼接： 146 | ![image](https://github.com/Nocami/PythonComputerVision-4-Panorama/blob/master/images/fin1.jpg) 147 | ![image](https://github.com/Nocami/PythonComputerVision-4-Panorama/blob/master/images/fin2.jpg) 148 | 同一点拍摄的四幅图像拼接成一张全景图，这种情况下，全景拼接的效果表现得特别明显。 149 | ## 后话： 150 | 本文代码运行环境为 python2.7,环境配置以相关文件请访问之前的PythonComputerVision系列文章，链接：https://github.com/Nocami?tab=repositories 151 | 本文所用实例图片为JiMei University。 152 | -------------------------------------------------------------------------------- /images/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/1.png -------------------------------------------------------------------------------- /images/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/2.png -------------------------------------------------------------------------------- /images/fin1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/fin1.jpg -------------------------------------------------------------------------------- /images/fin2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/fin2.jpg -------------------------------------------------------------------------------- /images/s: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /images/室内1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/室内1.jpg -------------------------------------------------------------------------------- /images/室内2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/室内2.jpg -------------------------------------------------------------------------------- /images/景深大1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/景深大1.jpg -------------------------------------------------------------------------------- /images/景深大2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/景深大2.jpg -------------------------------------------------------------------------------- /images/景深小A1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/景深小A1.jpg -------------------------------------------------------------------------------- /images/景深小A2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/景深小A2.jpg -------------------------------------------------------------------------------- /images/景深小B1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/景深小B1.jpg -------------------------------------------------------------------------------- /images/景深小B2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nocami/PythonComputerVision-4-Panorama/9830f6757d8653659bb60003409a39e08fd2de82/images/景深小B2.jpg -------------------------------------------------------------------------------- /t3.py: -------------------------------------------------------------------------------- 1 | from pylab import * 2 | from numpy import * 3 | from PIL import Image 4 | 5 | # If you have PCV installed, these imports should work 6 | from PCV.geometry import homography, warp 7 | from PCV.localdescriptors import sift 8 | 9 | """ 10 | This is the panorama example from section 3.3. 11 | """ 12 | 13 | # set paths to data folder 14 | featname = ['./images5/'+str(i+1)+'.sift' for i in range(2)] 15 | imname = ['./images5/'+str(i+1)+'.jpg' for i in range(2)] 16 | 17 | # extract features and match 18 | l = {} 19 | d = {} 20 | for i in range(2): 21 | sift.process_image(imname[i],featname[i]) 22 | l[i],d[i] = sift.read_features_from_file(featname[i]) 23 | 24 | matches = {} 25 | for i in range(1): 26 | matches[i] = sift.match(d[i+1],d[i]) 27 | 28 | # visualize the matches (Figure 3-11 in the book) 29 | for i in range(1): 30 | im1 = array(Image.open(imname[i])) 31 | im2 = array(Image.open(imname[i+1])) 32 | figure() 33 | sift.plot_matches(im2,im1,l[i+1],l[i],matches[i],show_below=True) 34 | 35 | 36 | # function to convert the matches to hom. points 37 | def convert_points(j): 38 | ndx = matches[j].nonzero()[0] 39 | fp = homography.make_homog(l[j+1][ndx,:2].T) 40 | ndx2 = [int(matches[j][i]) for i in ndx] 41 | tp = homography.make_homog(l[j][ndx2,:2].T) 42 | 43 | # switch x and y - TODO this should move elsewhere 44 | fp = vstack([fp[1],fp[0],fp[2]]) 45 | tp = vstack([tp[1],tp[0],tp[2]]) 46 | return fp,tp 47 | 48 | 49 | # estimate the homographies 50 | model = homography.RansacModel() 51 | 52 | fp,tp = convert_points(0) 53 | H_01 = homography.H_from_ransac(fp,tp,model)[0] #im 0 to 1 54 | 55 | #fp,tp = convert_points(1) 56 | #H_12 = homography.H_from_ransac(fp,tp,model)[0] #im 1 to 2 57 | 58 | #tp,fp = convert_points(2) #NB: reverse order 59 | #H_32 = homography.H_from_ransac(fp,tp,model)[0] #im 3 to 2 60 | 61 | #tp,fp = convert_points(3) #NB: reverse order 62 | #H_43 = homography.H_from_ransac(fp,tp,model)[0] #im 4 to 3 63 | 64 | 65 | # warp the images 66 | delta = 2000 # for padding and translation 67 | 68 | im1 = array(Image.open(imname[0]), "uint8") 69 | im2 = array(Image.open(imname[1]), "uint8") 70 | im_12 = warp.panorama(H_01,im1,im2,delta,delta) 71 | #im1 = array(Image.open(imname[0]), "f") 72 | #im_02 = warp.panorama(dot(H_12,H_01),im1,im_12,delta,delta) 73 | 74 | #im1 = array(Image.open(imname[3]), "f") 75 | #im_32 = warp.panorama(H_32,im1,im_02,delta,delta) 76 | 77 | #im1 = array(Image.open(imname[4]), "f") 78 | #im_42 = warp.panorama(dot(H_32,H_43),im1,im_32,delta,2*delta) 79 | 80 | 81 | figure() 82 | imshow(array(im_12, "uint8")) 83 | axis('off') 84 | savefig("example1.png",dpi=300) 85 | show() 86 | 87 | --------------------------------------------------------------------------------