├── .gitignore ├── LICENSE ├── README.md ├── doc ├── doc.md ├── en.md └── img │ ├── back_proj.png │ ├── bad.png │ ├── car.mp4 │ ├── choose_back.png │ ├── choose_front.png │ ├── example1.png │ ├── example2.png │ ├── example3.png │ ├── front_proj.png │ ├── images │ ├── back.png │ ├── car.png │ ├── front.png │ ├── left.png │ └── right.png │ ├── mask.png │ ├── mask_dilate.png │ ├── masks.png │ ├── original.png │ ├── overlap.png │ ├── overlap_gray.png │ ├── paramsettings.png │ ├── polyA.png │ ├── polyB.png │ ├── result.mp4 │ ├── result.png │ ├── smallcar.mp4 │ ├── weight_for_FL.png │ └── weights.png ├── images ├── back.png ├── car.png ├── front.png ├── left.png └── right.png ├── masks.png ├── run_calibrate_camera.py ├── run_get_projection_maps.py ├── run_get_weight_matrices.py ├── run_live_demo.py ├── surround_view ├── __init__.py ├── base_thread.py ├── birdview.py ├── capture_thread.py ├── fisheye_camera.py ├── imagebuffer.py ├── param_settings.py ├── process_thread.py ├── simple_gui.py ├── structures.py └── utils.py ├── test_cameras.py ├── weights.png └── yaml ├── back.yaml ├── front.yaml ├── left.yaml └── right.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | */__pycache__/ 4 | *.py[cod] 5 | *$py.class 6 | 7 | # C extensions 8 | *.so 9 | 10 | # Distribution / packaging 11 | .Python 12 | env/ 13 | build/ 14 | develop-eggs/ 15 | dist/ 16 | downloads/ 17 | eggs/ 18 | .eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *,cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # IPython Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # dotenv 80 | .env 81 | 82 | # virtualenv 83 | venv/ 84 | ENV/ 85 | 86 | # Spyder project settings 87 | .spyderproject 88 | 89 | # Rope project settings 90 | .ropeproject 91 | 92 | *-data.inc 93 | src/**/*.py.bak 94 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Zhao Liang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 一份简单的环视系统制作实现，包含完整的标定、投影、拼接和实时运行流程，详细文档见 `doc` 目录。环视系统的开源代码很少见，希望大家积极提 issue 和 pull request，让这个项目能更好地有益于入门者学习。 2 | 3 | 4 | This is a simple implementation of a surround view system, including calibration, projection, stitching, and real-time running processes. The [English documentation](https://github.com/hynpu/surround-view-system-introduction/blob/master/doc/en.md) can be found in the `doc` folder. 5 | 6 | 7 | -------------------------------------------------------------------------------- /doc/doc.md: -------------------------------------------------------------------------------- 1 | 关于车辆的全景环视系统网上已经有很多的资料，然而几乎没有可供参考的代码，这一点对入门的新人来说非常不友好。这个项目的目的就是介绍全景系统的原理，并给出一份基本要素齐全的、可以实际运行的 Python 实现供大家参考。环视全景系统涉及的知识并不复杂，只需要读者了解相机的标定、透视变换，并懂得如何使用 OpenCV。 2 | 3 | 这个程序最初是在一辆搭载了一台 AGX Xavier 的无人小车上开发的，运行效果如下：(见 `img/smallcar.mp4`) 4 | 5 | 6 | 7 | 小车上搭载了四个 USB 环视鱼眼摄像头，相机传回的画面分辨率为 640x480，图像首先经过畸变校正，然后在射影变换下转换为对地面的鸟瞰图，最后拼接起来经过平滑处理后得到了上面的效果。全部过程在 CPU 中进行处理，整体运行流畅。 8 | 9 | 后来我把代码重构以后移植到一辆乘用车上 (处理器是同型号的 AGX)，得到了差不多的效果：(见 `img/car.mp4`) 10 | 11 | 12 | 13 | 这个版本使用的是四个 960x640 的 csi 摄像头，输出的全景图分辨率为 1200x1600，在不进行亮度均衡处理时全景图处理线程运行速度大约为 17 fps，加入亮度均衡处理后骤降到只有 7 fps。我认为适当缩小分辨率的话 (比如采用 480x640 的输出可以将像素个数降低到原来的 1/6) 应该也可以获得流畅的视觉效果。 14 | 15 | 16 | > **注**：画面中黑色的部分是相机投影后出现的盲区，这是因为前面的相机为了避开车标的部位安装在了车头左侧且角度倾斜，所以视野受到了限制。想象一个人歪着脖子还斜着眼走路的样子 ... 17 | 18 | 这个项目的实现比较粗糙，仅作为演示项目展示生成环视全景图的基本要素，大家领会精神即可。我开发这个项目的目的是为了在自动泊车时实时显示车辆的轨迹，同时也用来作为我指导的实习生的实习项目。由于之前没有经验和参照，大多数算法和流程都是琢磨着写的，不见得高明，请大家多指教。代码是 Python 写成的，效率上肯定不如 C++，所以仅适合作学习和验证想法使用。 19 | 20 | 下面就来一一介绍我的实现步骤。 21 | 22 | 23 | # 硬件和软件配置 24 | 25 | 26 | 我想首先强调的是，硬件配置是这个项目中最不需要费心的事情，在第一个小车项目中使用的硬件如下： 27 | 28 | 1. 四个 USB 鱼眼相机，支持的分辨率为 640x480|800x600|1920x1080 三种。我这里因为是需要在 Python 下面实时运行，为了效率考虑设置的分辨率是 640x480。 29 | 2. 一台 AGX Xavier。实际上在普通笔记本上跑一样溜得飞起。 30 | 31 | 第二个乘用车项目使用的硬件如下： 32 | 33 | 1. 四个 csi 摄像头，设置的分辨率是 960x640。型号是 Sekonix 的 [SF3326-100-RCCB](http://sekolab.com/products/camera/)。 34 | 2. 一台 AGX Xavier，型号同上面的小车项目一样，不过外加了一个工控机接收 csi 摄像头画面。 35 | 36 | 我认为你只要有四个视野足够覆盖车周围的摄像头，再加一个普通笔记本电脑就足够进行全部的离线开发了。需要注意的是，由于传输速率的限制，笔记本的一个 USB 口可能无法同时带动四个摄像头，所以你可能需要用 hub 将摄像头分配在两个 USB 端口上。 37 | 38 | 软件配置如下： 39 | 40 | 1. 操作系统 Ubuntu 16.04/18.04. 41 | 2. Python>=3. 42 | 3. OpenCV>=3. 43 | 4. PyQt5. 44 | 45 | 其中 PyQt5 主要用来实现多线程，方便将来移植到 Qt 环境。 46 | 47 | 48 | # 项目采用的若干约定 49 | 50 | 51 | 为了方便起见，在本项目中四个环视相机分别用 `front`、`back`、`left`、`right` 来指代，并假定其对应的设备号是整数，例如 0, 1, 2, 3。实际开发中请针对具体情况进行修改。 52 | 53 | 相机的内参矩阵记作 `camera_matrix`，这是一个 3x3 的矩阵。畸变系数记作 `dist_coeffs`，这是一个 1x4 的向量。相机的投影矩阵记作 `project_matrix`，这是一个 3x3 的射影矩阵。 54 | 55 | 56 | # 准备工作：获得原始图像与相机内参 57 | 58 | 59 | 首先我们需要获取每个相机的内参矩阵与畸变系数。我在项目中附上了一个脚本 [run_calibrate_camera.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_calibrate_camera.py)，你只需要运行这个脚本，通过命令行参数告诉它相机设备号，是否是鱼眼相机，以及标定板的网格大小，然后手举标定板在相机面前摆几个姿势即可。 60 | 61 | 以下是视频中四个相机分别拍摄的原始画面，顺序依次为前、后、左、右，并命名为 `front.png`、`back.png`、`left.png`、`right.png` 保存在项目的 `images/` 目录下。 62 | 63 | | | | | | 64 | |:-:|:-:|:-:|:-:| 65 | |front|back|left|right| 66 | ||||| 67 | 68 | 四个相机的内参文件分别为 `front.yaml`、`back.yaml`、`left.yaml`、`right.yaml`，这些内参文件都存放在项目的 [yaml](https://github.com/neozhaoliang/surround-view-system-introduction/tree/master/yaml) 子目录下。 69 | 70 | 你可以看到图中地面上铺了一张标定布，这个布的尺寸是 `6mx10m`，每个黑白方格的尺寸为 `40cmx40cm`，每个圆形图案所在的方格是 `80cmx80cm`。我们将利用这个标定物来手动选择对应点获得投影矩阵。 71 | 72 | 73 | # 设置投影范围和参数 74 | 75 | 76 | 接下来我们需要获取每个相机到地面的投影矩阵，这个投影矩阵会把相机校正后的画面转换为对地面上某个矩形区域的鸟瞰图。这四个相机的投影矩阵不是独立的，它们必须保证投影后的区域能够正好拼起来。 77 | 78 | 这一步是通过联合标定实现的，即在车的四周地面上摆放标定物，拍摄图像，手动选取对应点，然后获取投影矩阵。 79 | 80 | 请看下图： 81 | 82 | 83 | 84 | 首先在车身的四角摆放四个标定板，标定板的图案大小并无特殊要求，只要尺寸一致，能在图像中清晰看到即可。每个标定板应当恰好位于相邻的两个相机视野的重合区域中。 85 | 86 | 在上面拍摄的相机画面中车的四周铺了一张标定布，这个具体是标定板还是标定布不重要，只要能清楚的看到特征点就可以了。 87 | 88 | 然后我们需要设置几个参数：(以下所有参数均以厘米为单位) 89 | 90 | + `innerShiftWidth`, `innerShiftHeight`：标定板内侧边缘与车辆左右两侧的距离，标定板内侧边缘与车辆前后方的距离。 91 | + `shiftWidth`, `shiftHeight`：这两个参数决定了在鸟瞰图中向标定板的外侧看多远。这两个值越大，鸟瞰图看的范围就越大，相应地远处的物体被投影后的形变也越严重，所以应酌情选择。 92 | + `totalWidth`, `totalHeight`：这两个参数代表鸟瞰图的总宽高，在我们这个项目中标定布宽 6m 高 10m，于是鸟瞰图中地面的范围为 `(600 + 2 * shiftWidth, 1000 + 2 * shiftHeight)`。为方便计我们让每个像素对应 1 厘米，于是鸟瞰图的总宽高为 93 | 94 | ``` 95 | totalWidth = 600 + 2 * shiftWidth 96 | totalHeight = 1000 + 2 * shiftHeight 97 | ``` 98 | 99 | + 车辆所在矩形区域的四角 (图中标注的红色圆点)，这四个角点的坐标分别为 `(xl, yt)`, `(xr, yt)`, `(xl, yb)`, `(xr, yb)` (`l` 表示 left, `r` 表示 right，`t` 表示 top，`b` 表示 bottom)。这个矩形区域相机是看不到的，我们会用一张车辆的图标来覆盖此处。 100 | 101 | 注意这个车辆区域四边的延长线将整个鸟瞰图分为前左 (FL)、前中 (F)、前右 (FR)、左 (L)、右 (R)、后左 (BL)、后中 (B)、后右 (BR) 八个部分，其中 FL (区域 I)、FR (区域 II)、BL (区域 III)、BR (区域 IV) 是相邻相机视野的重合区域，也是我们重点需要进行融合处理的部分。F、R、L、R 四个区域属于每个相机单独的视野，不需要进行融合处理。 102 | 103 | 以上参数存放在 [param_settings.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/param_settings.py) 中。 104 | 105 | 设置好参数以后，每个相机的投影区域也就确定了，比如前方相机对应的投影区域如下： 106 | 107 | 108 | 109 | 接下来我们需要通过手动选取标志点来获取到地面的投影矩阵。 110 | 111 | 112 | # 手动标定获取投影矩阵 113 | 114 | 115 | 首先运行项目中 [run_get_projection_maps.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_get_projection_maps.py) 这个脚本，这个脚本需要你输入如下的参数： 116 | 117 | + `-camera`: 指定是哪个相机。 118 | + `-scale`: 校正后画面的横向和纵向放缩比。 119 | + `-shift`: 校正后画面中心的横向和纵向平移距离。 120 | 121 | 为什么需要 `scale` 和 `shift` 这两个参数呢？这是因为默认的 OpenCV 的校正方式是在鱼眼相机校正后的图像中裁剪出一个 OpenCV "认为" 合适的区域并将其返回，这必然会丢失一部分像素，特别地可能会把我们希望选择的特征点给裁掉。幸运的是 [cv2.fisheye.initUndistortRectifyMap](https://docs.opencv.org/master/db/d58/group__calib3d__fisheye.html#ga0d37b45f780b32f63ed19c21aa9fd333) 这个函数允许我们再传入一个新的内参矩阵，对校正后但是裁剪前的画面作一次放缩和平移。你可以尝试调整并选择合适的横向、纵向压缩比和图像中心的位置使得地面上的标志点出现在画面中舒服的位置上，以方便进行标定。 122 | 123 | 运行 124 | 125 | ```bash 126 | python run_get_projection_maps.py -camera front -scale 0.7 0.8 -shift -150 -100 127 | ``` 128 | 129 | 后显示前方相机校正后的画面如下： 130 | 131 | 132 | 133 | 然后依次点击事先确定好的四个标志点 (顺序不能错！)，得到的效果如下： 134 | 135 | 136 | 137 | 注意标志点的设置代码在[这里](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/param_settings.py#L40)。 138 | 139 | 这四个点是可以自由设置的，但是你需要在程序中手动修改它们在鸟瞰图中的像素坐标。当你在校正图中点击这四个点时，OpenCV 会根据它们在校正图中的像素坐标和在鸟瞰图中的像素坐标的对应关系计算一个射影矩阵。这里用到的原理就是四点对应确定一个射影变换 (四点对应可以给出八个方程，从而求解出射影矩阵的八个未知量。注意射影矩阵的最后一个分量总是固定为 1)。 140 | 141 | 如果你不小心点歪了的话可以按 `d` 键删除上一个错误的点。选择好以后点回车，就会显示投影后的效果图: 142 | 143 | 144 | 145 | 觉得效果可以的话敲回车，就会将投影矩阵写入 `front.yaml` 中，这个矩阵的名字为 `project_matrix`。失败的话就按 `q` 退出再来一次。 146 | 147 | 再比如后面相机的标定如下图所示： 148 | 149 | 150 | 151 | 对应的投影图为 152 | 153 | 154 | 155 | 对四个相机分别采用此操作，我们就得到了四个相机的鸟瞰图，以及对应的四个投影矩阵。下一步我们的任务是把这四个鸟瞰图拼起来。 156 | 157 | > **重要注意事项**：有些用户反映在按照上面的方法进行标定时，得到的拼接效果不够好。经过询问，发现原因是他选择的四个标定点都集中在图像的中心部分。你应该尽可能让四个标定点围成的矩形尽可能覆盖图像更大的范围。原因是鱼眼图像在矫正后也是有误差的，边缘的误差更大。所以要尽可能让 OpenCV 在更大的范围内计算一个全局最优的射影矩阵。 158 | 159 | 160 | # 鸟瞰图的拼接与平滑 161 | 162 | 163 | 如果你前面操作一切正常的话，运行 [run_get_weight_matrices.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_get_weight_matrices.py) 后应该会显示如下的拼接图： 164 | 165 | 166 | 167 | 我来逐步介绍它是怎么做到的： 168 | 169 | 1. 由于相邻相机之间有重叠的区域，所以这部分的融合是关键。如果直接采取两幅图像加权平均 (权重各自为 1/2) 的方式融合的话你会得到类似下面的结果： 170 | 171 | 172 | 173 | 你可以看到由于校正和投影的误差，相邻相机在重合区域的投影结果并不能完全吻合，导致拼接的结果出现乱码和重影。这里的关键在于权重系数应该是随像素变化而变化的，并且是随着像素连续变化。 174 | 175 | 2. 以左上角区域为例，这个区域是 `front`, `left` 两个相机视野的重叠区域。我们首先将投影图中的重叠部分取出来： 176 | 177 | 178 | 179 | 灰度化并二值化： 180 | 181 | 182 | 183 | 注意这里面有噪点，可以用形态学操作去掉 (不必特别精细，大致去掉即可)： 184 | 185 | 186 | 187 | 至此我们就得到了重叠区域的一个完整 mask。 188 | 189 | 3. 将 `front`, `left` 图像各自位于重叠区域外部的边界检测出来，这一步是通过先调用 `cv2.findContours` 求出最外围的边界，再用 `cv2.approxPolyDP` 获得逼近的多边形轮廓： 190 | 191 | ||| 192 | |:-:|:-:| 193 | ||| 194 | 195 | 我们把 `front` 相机减去重叠区域后的轮廓记作 `polyA` (左上图中蓝色边界)，`left` 相机减去重叠区域后的轮廓记作 `polyB` (右上图中绿色边界)。 196 | 197 | 4. 对重叠区域中的每个像素，利用 `cv2.pointPolygonTest` 计算其到这两个多边形 `polyA` 和 `polyB` 的距离 $d_A,d_B$，则该像素对应的权值为 $w=d_B^2/(d_A^2+d_B^2)$，即如果这个像素落在 `front` 画面内，则它与 `polyB` 的距离就更远，从而权值更大。 198 | 199 | 5. 对不在重叠区域内的像素，若其属于 `front` 相机的范围则其权值为 1，否则权值为 0。于是我们得到了一个连续变化的，取值范围在 0~1 之间的矩阵 $G$，其灰度图如下： 200 | 201 | 202 | 203 | 将 $G$ 作为权值可得融合后的图像为 `front * G + (1- G) * left`。 204 | 205 | 6. 注意由于重叠区域中的像素值是来自两幅图像的加权平均，所以出现在这个区域内的物体会不可避免出现虚影的现象，所以我们需要尽量压缩重叠区域的范围，尽可能只对拼接缝周围的像素计算权值，拼接缝上方的像素尽量使用来自 `front` 的原像素，拼接缝下方的像素尽量使用来自 `back` 的原像素。这一步可以通过控制 $d_B$ 的值得到。 206 | 207 | 7. 我们还漏掉了重要的一步：由于不同相机的曝光度不同，导致不同的区域会出现明暗的亮度差，影响美观。我们需要调整每个区域的亮度，使得整个拼接图像的亮度趋于一致。这一步做法不唯一，自由发挥的空间很大。我查阅了一下网上提到的方法，发现它们要么过于复杂，几乎不可能是实时的；要么过于简单，无法达到理想的效果。特别在上面第二个视频的例子中，由于前方相机的视野被车标遮挡导致感光范围不足，导致其与其它三个相机的画面亮度差异很大，调整起来很费劲。 208 | 209 | 一个基本的想法是这样的：每个相机传回的画面有 `BGR` 三个通道，四个相机传回的画面总共有 12 个通道。我们要计算 12 个系数，将这 12 个系数分别乘到这 12 个通道上，然后再合并起来形成调整后的画面。过亮的通道要调暗一些所以乘的系数小于 1，过暗的通道要调亮一些所以乘的系数大于 1。这些系数可以通过四个画面在四个重合区域内的亮度比值得出，你可以自由设计计算系数的方式，只要满足这个基本原理即可。 210 | 211 | 我的实现见[这里](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/birdview.py#L210)。感觉就像一段 shader 代码。 212 | 213 | 还有一种偷懒的办法是事先计算一个 tone mapping 函数 (比如逐段线性的，或者 AES tone mapping function)，然后强制把所有像素进行转换，这个方法最省力，但是得到的画面色调会与真实场景有较大差距。似乎有的市面产品就是采用的这种方法。 214 | 215 | 8. 最后由于有些情况下摄像头不同通道的强度不同，还需要进行一次色彩平衡，见下图: 216 | 217 | | | | | 218 | |:-:|:-:|:-:| 219 | |拼接后原始画面| 亮度平衡画面 | 亮度平衡+色彩平衡画面| 220 | |||| 221 | 222 | 在第二个视频的例子中，画面的颜色偏红，加入色彩平衡后画面恢复了正常。 223 | 224 | 225 | # 具体实现的注意事项 226 | 227 | 228 | 1. 多线程与线程同步。本文的两个例子中四个摄像头都不是硬件触发保证同步的，而且即便是硬件同步的，四个画面的处理线程也未必同步，所以需要有一个线程同步机制。这个项目的实现采用的是比较原始的一种，其核心代码如下： 229 | ```python 230 | class MultiBufferManager(object): 231 | 232 | ... 233 | 234 | def sync(self, device_id): 235 | # only perform sync if enabled for specified device/stream 236 | self.mutex.lock() 237 | if device_id in self.sync_devices: 238 | # increment arrived count 239 | self.arrived += 1 240 | # we are the last to arrive: wake all waiting threads 241 | if self.do_sync and self.arrived == len(self.sync_devices): 242 | self.wc.wakeAll() 243 | # still waiting for other streams to arrive: wait 244 | else: 245 | self.wc.wait(self.mutex) 246 | # decrement arrived count 247 | self.arrived -= 1 248 | self.mutex.unlock() 249 | ``` 250 | 这里使用了一个 `MultiBufferManager` 对象来管理所有的线程，每个摄像头的线程在每次循环时会调用它的 `sync` 方法，并通过将计数器加 1 的方法来通知这个对象 "报告，我已做完上次的任务，请将我加入休眠池等待下次任务"。一旦计数器达到 4 就会触发唤醒所有线程进入下一轮的任务循环。 251 | 252 | 2. 建立查找表 (lookup table) 以加快运算速度。鱼眼镜头的画面需要经过校正、投影、翻转以后才能用于拼接，这三步涉及频繁的图像内存分配和销毁，非常费时间。在我的测试中抓取线程始终稳定在 30fps 多一点左右，但是每个画面的处理线程只有 20 fps 左右。这一步最好是通过预计算一个查找表来加速。你还记得 `cv2.fisheye.initUndistortRectifyMap` 这个函数吗？它返回的 `mapx, mapy` 就是两个查找表。比如当你指定它返回的矩阵类型为 `cv2.CV_16SC2` 的时候，它返回的 `mapx` 就是一个逐像素的查找表，`mapy` 是一个用于插值平滑的一维数组 (可以扔掉不要)。同理对于 `project_matrix` 也不难获得一个查找表，两个合起来就可以得到一个直接从原始画面到投影画面的查找表 (当然损失了用于插值的信息)。 253 | 在这个项目中由于采用的是 Python 实现，而 Python 的 `for` 循环效率不高，所以没有采用这种查找表的方式。 254 | 255 | 3. 四个权重矩阵可以作为 `RGBA` 四个通道压缩到一张图片中，这样存储和读取都很方便。四个重叠区域对应的 mask 矩阵也是如此： 256 | 257 | 258 | 259 | 260 | 261 | 262 | # 实车运行 263 | 264 | 265 | 你可以在实车上运行 [run_live_demo.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_live_demo.py) 来验证最终的效果。 266 | 267 | 你需要注意修改相机设备号，以及 OpenCV 打开摄像头的方式。usb 相机可以直接用 `cv2.VideoCapture(i)` (`i` 是 usb 设备号) 的方式打开，csi 相机则需要调用 `gstreamer` 打开，对应的代码在[这里](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/utils.py#L5)和[这里](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/capture_thread.py#L75)。 268 | 269 | 270 | # 附录：项目各脚本一览 271 | 272 | 273 | 项目中目前的脚本根据执行顺序排列如下: 274 | 275 | 1. `run_calibrate_camera.py`：用于相机内参标定。 276 | 2. `param_settings.py`：用于设置投影区域的各参数。 277 | 3. `run_get_projection_maps.py`：用于手动标定获取到地面的投影矩阵。 278 | 4. `run_get_weight_matrices.py`：用于计算四个重叠区域对应的权重矩阵以及 mask 矩阵，并显示拼接效果。 279 | 6. `run_live_demo.py`：用于在实车上运行的最终版本。 280 | -------------------------------------------------------------------------------- /doc/en.md: -------------------------------------------------------------------------------- 1 | This project is a simple, runnable, and reproducible demo to show how to develop a surround-view system in Python. 2 | 3 | It contains all the key steps: camera calibration, image stitching, and real-time processing. 4 | 5 | This project was originally developed on a small car with an AGX Xavier and four USB fisheye cameras：(see `img/smallcar.mp4`) 6 | 7 | 8 | 9 | The camera resolution was set to 640x480, everything was done in Python. 10 | 11 | Later I improved the project and migrated it to a [EU5 car](https://en.wikipedia.org/wiki/Beijing_U5), still processing in a Xavier AGX, and got a better result: (see `img/car.mp4`) 12 | 13 | 14 | 15 | This EU5 car version used the four CSI cameras of resolution 960x640. The full review image has a resolution 1200x1600, the fps is about 17/7 without/with post-precessing, respectively. 16 | 17 | 18 | > **Remark**：The black area in front of the car is the blind area after projection, it's because the front camera wasn't installed correctly. 19 | 20 | The project is not very complex, but it does involve some careful computations. Now we explain the whole process step by step. 21 | 22 | 23 | # Hardware and software 24 | 25 | The hardware I used in the small car project is: 26 | 27 | 1. Four USB fisheye cameras, supporting three different modes of resolution: 640x480|800x600|1920x1080. I used 640x480 because it suffices for a car of this size. 28 | 2. AGX Xavier. 29 | 30 | Indeed you can do all the development on your laptop, an AGX is not a strict prerequisite to reproduce this project. 31 | 32 | The hardware I used in the EU5 car project is: 33 | 34 | 1. Four CSI cameras of resolution 960x640。I used Sekonix's [SF3326-100-RCCB camera](http://sekolab.com/products/camera/). 35 | 2. Also, AGX Xavier is the same as in the small car. 36 | 37 | The software: 38 | 39 | 1. Ubuntu 16.04/18.04. 40 | 2. Python>=3. 41 | 3. OpenCV>=3. 42 | 4. PyQt5. 43 | 44 | `PyQt5` is used mainly for multi-threading. 45 | 46 | 47 | # Conventions 48 | 49 | The four cameras will be named `front`、`back`、`left`、`right`，and with device numbers 0, 1, 2, and 3, respectively. Please modify this according to your actual device numbers. 50 | 51 | The camera intrinsic matrix is denoted as `camera_matrix`， this is a 3x3 matrix. 52 | The distorted coefficients are stored in `dist_coeffs`, this is a 1x4 vector. 53 | The projection matrix is denoted as `project_matrix`， this is a 3x3 projective matrix. 54 | 55 | 56 | # Prepare work: camera calibration 57 | 58 | 59 | There is a script [run_calibrate_camera.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_calibrate_camera.py) in this project to help 60 | you calibrate the camera. I'm not going to discuss how to calibrate a camera here, as there are lots of resources on the web. 61 | 62 | Below are the images taken by the four cameras, in the order `front.png`、`back.png`、`left.png`、`right.png`, they are in the `images/` directory. 63 | 64 | | | | | | 65 | |:-:|:-:|:-:|:-:| 66 | |front|back|left|right| 67 | ||||| 68 | 69 | The parameters of these cameras are stored in the yaml files `front.yaml`、`back.yaml`、`left.yaml`、`right.yaml`, these files can be found in the [yaml](https://github.com/neozhaoliang/surround-view-system-introduction/tree/master/yaml) directory. 70 | 71 | You can see there is a black-white calibration pattern on the ground, the size of the pattern is `6mx10m`, the size of each black/white square is `40cmx40cm`, the size of each square with a circle in it is `80cmx80cm`. 72 | 73 | 74 | # Setting projection parameters 75 | 76 | 77 | Now we compute the projection matrix for each camera. This matrix will transform the undistorted image into a bird's view of the ground. All four projection matrices must fit together to make sure the four projected images can be stitched together. 78 | 79 | This is done by putting calibration patterns on the ground, taking the camera images, manually choosing the feature points, and then computing the matrix. 80 | 81 | See the illustration below: 82 | 83 | 84 | 85 | Firstly you put four calibration boards at the four corners around the car (the blue squares). There are no particular restrictions on how large the board must be, only make sure you can see it clearly in the image. 86 | 87 | OF course, each board must be seen by the two adjacent cameras. 88 | 89 | Now we need to set a few parameters: (in `cm` units) 90 | 91 | + `innerShiftWidth`, `innerShiftHeight`：distance between the inner edges of the left/right calibration boards and the car， the distance between the inner edges of the front/back calibration boards and the car。 92 | + `shiftWidth`, `shiftHeight`：How far you will want to look at out of the boards. The bigger these values, the larger the area the birdview image will cover. 93 | + `totalWidth`, `totalHeight`：Size of the area that the birdview image covers. In this project, the calibration pattern is of width `600cm` and height `1000cm`, hence the bird view image will cover an area of size `(600 + 2 * shiftWidth, 1000 + 2 * shiftHeight)`. For simplicity, 94 | we let each pixel correspond to 1cm, so the final bird-view image also has a resolution 95 | 96 | ``` 97 | totalWidth = 600 + 2 * shiftWidth 98 | totalHeight = 1000 + 2 * shiftHeight 99 | ``` 100 | 101 | + The four corners of the rectangular area where the vehicle is located (marked with red dots in the image) are denoted by the coordinates (xl, yt), (xr, yt), (xl, yb), and (xr, yb), where "l" stands for left, "r" stands for right, "t" stands for top, and "b" stands for bottom. The camera cannot see this rectangular area, and we will use an icon of the vehicle to cover it. 102 | 103 | Note that the extension lines of the four sides of the vehicle area divide the entire bird's-eye view into eight parts: front-left (FL), front-center (F), front-right (FR), left (L), right (R), back-left (BL), back-center (B), and back-right (BR). Among them, FL (area I), FR (area II), BL (area III), and BR (area IV) are the overlapping areas of adjacent camera views, and they are the parts that we need to focus on for fusion processing. The areas F, R, L, and R belong to the individual views of each camera and do not require fusion processing. 104 | 105 | The above parameters are saved in [param_settings.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/param_settings.py) 106 | 107 | 108 | Once the parameters are set, the projection area for each camera is determined. For example, the projection area for the front camera is as follows: 109 | 110 | 111 | 112 | Next, we need to manually select the feature points to obtain the projection matrix for the ground plane. 113 | 114 | 115 | # Manually select feature points for the projection matrix 116 | 117 | 118 | Firstly you need to run this script, [run_get_projection_maps.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_get_projection_maps.py), with the following parameters： 119 | 120 | + `-camera`: specify the camera (left, right, front, rear) 121 | + `-scale`: The horizontal and vertical scaling ratios of the corrected image after undistortion 122 | + `-shift`: The horizontal and vertical distances of the corrected image center after undistortion。 123 | 124 | 125 | The scale and shift parameters are needed because the default OpenCV calibration method for fisheye cameras involves cropping the corrected image to a region that OpenCV "thinks" is appropriate. This inevitably results in the loss of some pixels, especially the feature points that we may want to select. 126 | 127 | Fortunately, the function [`cv2.fisheye.initUndistortRectifyMap`](https://docs.opencv.org/master/db/d58/group__calib3d__fisheye.html#ga0d37b45f780b32f63ed19c21aa9fd333) allows us to provide a new intrinsic matrix, which can be used to perform a scaling and translation of the un-cropped corrected image. By adjusting the horizontal and vertical scaling ratios and the position of the image center, we can ensure that the feature points on the ground plane appear in comfortable places in the image, making it easier to perform calibration. 128 | 129 | 130 | ```bash 131 | python run_get_projection_maps.py -camera front -scale 0.7 0.8 -shift -150 -100 132 | ``` 133 | 134 | The undistorted image of the front camera： 135 | 136 | 137 | 138 | Then, click on the four predetermined feature points in order (the order cannot be wrong!), and the result will look like this: 139 | 140 | 141 | 142 | The script for setting up the points is [here](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/param_settings.py#L40)。 143 | 144 | These four points can be freely set, but you need to manually modify their pixel coordinates in the bird's-eye view in the program. When you click on these four points in the corrected image, OpenCV will calculate a perspective transformation matrix based on the correspondence between their pixel coordinates in the corrected image and their corresponding coordinates in the bird view. The principle used here is that a perspective transformation can be uniquely determined by four corresponding points (four points can give eight equations, from which the eight unknowns in the perspective matrix can be solved. Note that the last component of the perspective matrix is always fixed to 1). 145 | 146 | If you accidentally click the wrong point, you can press the d key to delete the last selected point. After selecting the four points, press Enter, and the program will display the resulting bird's-eye view image: 147 | 148 | 149 | 150 | If you are satisfied with the result, press the Enter key to write the projection matrix to the front.yaml file. The name of the matrix is project_matrix. If you are not satisfied, press 'q' to exit and start over. 151 | 152 | The four points of the rear camera image： 153 | 154 | 155 | 156 | The corresponding undistorted image: 157 | 158 | 159 | 160 | We will stitch the four bird's-eye view images together using the same procedure and get their projection matrix respectively. 161 | 162 | > **Important**: It is crucial to select four points that cover as largest possible area in the image to ensure seamless stitching. Failure to do so may result in poor stitching. Despite being called as undistorted, the image may still contain distortions due to various errors in the undistortion process, particularly noticeable towards the image periphery. Therefore, we should ask OpenCV to find a globally optimized projective matrix by leveraging information from a broader image area, rather than relying solely on local and limited regions. 163 | 164 | 165 | # Stitching and smoothing of the birdseye view image 166 | 167 | 168 | If everything goes well from the previous section, and after executing this script: [run_get_weight_matrices.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_get_weight_matrices.py), you will notice the stitched birdseye view image： 169 | 170 | 171 | 172 | The logic behind this script is as follows： 173 | 174 | 1. Due to the overlapping areas between adjacent cameras, the fusion of these overlapping parts is crucial for this task. If we directly use a simple weighted averaging approach with weights of 0.5 for each image, we would observe the output like the following image: 175 | 176 | 177 | 178 | You can see that due to the errors in calibration and projection, the projected results of adjacent cameras in the overlapping area do not match perfectly, resulting in garbled and ghosting effects in the stitched image. The key point is the weighting coefficients which should vary with the pixels and change continuously with them. 179 | 180 | 2. Let's take the upper-left corner as an example, which is the overlapping area of the front and left cameras. We first extract the overlapping area from the projected images: 181 | 182 | 183 | 184 | grayscale and thresholding： 185 | 186 | 187 | 188 | We can use morphological operations to remove the noise (it doesn't need to be very precise, rough removal is enough): 189 | 190 | 191 | 192 | Then we have the mask for this overlapping area。 193 | 194 | 3. To obtain the outer boundaries of the front and left images that lie outside the overlapping region, we first use cv2.findContours to find the outermost contour and then use cv2.approxPolyDP to obtain the approximated polygonal contour： 195 | 196 | ||| 197 | |:-:|:-:| 198 | ||| 199 | 200 | We denote the contour obtained by subtracting the overlapping region from the front camera as polyA (blue boundary in the top left image), and the contour obtained by subtracting the overlapping region from the left camera as polyB (green boundary in the top right image). 201 | 202 | 4. For each pixel in the overlapping area, we can calculate its distance to the two polygons polyA and polyB using cv2.pointPolygonTest, denoting the distances as $d_A$ and $d_B$, respectively. The weight of the pixel is then given by $w=d_B^2/(d_A^2+d_B^2)$. If the pixel falls inside the front image, then its distance to polyB will be greater, giving it a larger weight. 203 | 204 | 5. For each pixel in the overlapping region, we can use cv2.pointPolygonTest to calculate its distance to the two polygons polyA and polyB. Let $d_A$ and $d_B$ be the distances from the pixel to polyA and polyB, respectively. Then the weight of the pixel is calculated as $w=d_B^2/(d_A^2+d_B^2)$. This means that if the pixel is in the front camera view, it will have a larger weight if it is farther away from polyB. For pixels outside the overlapping region, their weight is 1 if they belong to the front camera's view, and 0 otherwise. Thus, we obtain a continuous matrix $G$ with values ranging between 0 and 1. Here is the grayscale image of $G$: 205 | 206 | 207 | 208 | By using $G$ as the weight matrix, we can get the fused image: `front * G + (1- G) * left`。 209 | 210 | 6. Please note that since the pixel values in the overlapping region are the weighted average of two images, there will inevitably be ghosting artifacts for objects in this region. Therefore, we need to minimize the size of the overlapping region as much as possible and only calculate the weight values for pixels around the stitching seam. We should use the original pixels from the front image as much as possible for the pixels above the seam and the original pixels from the back image for the pixels below the seam. This step can be achieved by controlling the value of $d_B$. 211 | 212 | 7. Due to the different exposure levels of different cameras, there will be brightness differences in different areas, which will affect the performance of the final stitched image. We need to adjust the brightness of each area to make the overall brightness of the stitched image tend to be consistent. And there is no unique method. After doing several searches online and then realized that the methods mentioned are either too complicated and computationally expensive or too simple and unable to achieve the ideal performance. In particular, in the example of the second video above, the field of view of the front camera is insufficient due to the obstruction of the car logo, resulting in a large difference in brightness between its image and the other three cameras, which is very difficult to adjust. 213 | 214 | One basic idea is as follows: Each camera returns an image with three channels in BGR format, and the four cameras together provide a total of 12 channels. We need to calculate 12 coefficients, which are then multiplied with each of the 12 channels, and then combined to form the adjusted image. Channels that are too bright need to be darkened, so the coefficients are less than 1, and channels that are too dark need to be brightened, so the coefficients are greater than 1. These coefficients can be obtained from the brightness ratio of the four images in their overlapping regions. You can design the method for calculating these coefficients as you wish as long as it satisfies this basic principle. 215 | 216 | Here is my [implementation](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/birdview.py#L210). 217 | 218 | There is also another simple way to adjust the brightness, which is to pre-calculate a tone mapping function (such as piecewise linear or AES tone mapping function) and then force all pixels to be converted using this function. This method is the most simple one, but the color tone in the output frame may differ significantly from the actual environment. 219 | 220 | 8. The final step is color balance, which is needed in some cases where the intensity of different channels may vary between cameras. Please refer to the image below: 221 | 222 | | | | | 223 | |:-:|:-:|:-:| 224 | |Raw frame after stitching| After white balance | After brightness and color balance| 225 | |||| 226 | 227 | In the example of the second video, the image turns to be more red than normal. After applying color balancing, the image is back to normal. 228 | 229 | 230 | # Attentions 231 | 232 | 233 | 1. Multi-threading and thread synchronization. In the two examples in this article, none of the four cameras are hardware-triggered for synchronization, and even if hardware synchronization is used, the processing threads of the four images may not be synchronized, so a thread synchronization mechanism is needed. The implementation of this project uses a relatively primitive method, and its core code is as follows: 234 | 235 | ```python 236 | class MultiBufferManager(object): 237 | 238 | ... 239 | 240 | def sync(self, device_id): 241 | # only perform sync if enabled for specified device/stream 242 | self.mutex.lock() 243 | if device_id in self.sync_devices: 244 | # increment arrived count 245 | self.arrived += 1 246 | # we are the last to arrive: wake all waiting threads 247 | if self.do_sync and self.arrived == len(self.sync_devices): 248 | self.wc.wakeAll() 249 | # still waiting for other streams to arrive: wait 250 | else: 251 | self.wc.wait(self.mutex) 252 | # decrement arrived count 253 | self.arrived -= 1 254 | self.mutex.unlock() 255 | ``` 256 | Here, a MultiBufferManager object is used to manage all the threads. Each camera thread calls its sync method at each iteration and notifies the object by incrementing a counter, saying "report, I have completed the previous task, please put me in the sleep pool and wait for the next task." Once the counter reaches 4, all threads are awakened to enter the next round of task iteration. 257 | 258 | 2. Creating a lookup table can speed up the processing. To stitch images from fisheye lenses, the captured images need to go through calibration, projection, and flipping before they can be used for stitching. These three steps involve frequent memory allocation and deallocation, which is time-consuming. In our experiment, the capturing threads were stable at around 30 fps, but each processing thread is only about 20 fps. To speed up this processing, it's best to precompute a lookup table. For example, the cv2.fisheye.initUndistortRectifyMap returns two lookup tables, mapx and mapy, and when you specify the matrix type cv2.CV_16SC2, mapx returned is a per-pixel lookup table, and mapy is a one-dimensional array for interpolation smoothing (which can be discarded). Similarly, a lookup table can also be obtained for the project_matrix, and combining the two will give you a lookup table that directly maps the original image to the projected image (although losing information for interpolation). 259 | 260 | In this project, Python was used for implementation; however, the for loops in Python are not very efficient, this lookup table method was not used. 261 | 262 | 3. The four weight matrices can be compressed into a single image with four channels (RGBA), which is convenient for storage and retrieval. The same applied for the four overlapping mask matrices: 263 | 264 | 265 | 266 | 267 | 268 | 269 | # On vehicle demo 270 | 271 | 272 | You can run this algo on vehicle to test performance: [run_live_demo.py](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/run_live_demo.py) 273 | 274 | But you need to specify the correct porf of the cameras. For USB camera, you can directly call `cv2.VideoCapture(i)` (`i` indicates the USB port for this camera)，for CSI camera, you need to use `gstreamer` , and here is the script [CSI camera scripts](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/utils.py#L5) and [Capture thred](https://github.com/neozhaoliang/surround-view-system-introduction/blob/master/surround_view/capture_thread.py#L75)。 275 | 276 | 277 | # Appendix: scripts in this repo and their description 278 | 279 | 280 | The script squence of running this repo: 281 | 282 | 1. `run_calibrate_camera.py`：intrinsic matrix calibration 283 | 2. `param_settings.py`：set up projection matrix and related parameters 284 | 3. `run_get_projection_maps.py`：manually select projection points and area 285 | 4. `run_get_weight_matrices.py`：calculate the weight matrix and mask matrix for the four overlapping regions, and to display the stitching results。 286 | 6. `run_live_demo.py`：on vehicle demo。 287 | -------------------------------------------------------------------------------- /doc/img/back_proj.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/back_proj.png -------------------------------------------------------------------------------- /doc/img/bad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/bad.png -------------------------------------------------------------------------------- /doc/img/car.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/car.mp4 -------------------------------------------------------------------------------- /doc/img/choose_back.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/choose_back.png -------------------------------------------------------------------------------- /doc/img/choose_front.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/choose_front.png -------------------------------------------------------------------------------- /doc/img/example1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/example1.png -------------------------------------------------------------------------------- /doc/img/example2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/example2.png -------------------------------------------------------------------------------- /doc/img/example3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/example3.png -------------------------------------------------------------------------------- /doc/img/front_proj.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/front_proj.png -------------------------------------------------------------------------------- /doc/img/images/back.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/images/back.png -------------------------------------------------------------------------------- /doc/img/images/car.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/images/car.png -------------------------------------------------------------------------------- /doc/img/images/front.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/images/front.png -------------------------------------------------------------------------------- /doc/img/images/left.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/images/left.png -------------------------------------------------------------------------------- /doc/img/images/right.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/images/right.png -------------------------------------------------------------------------------- /doc/img/mask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/mask.png -------------------------------------------------------------------------------- /doc/img/mask_dilate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/mask_dilate.png -------------------------------------------------------------------------------- /doc/img/masks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/masks.png -------------------------------------------------------------------------------- /doc/img/original.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/original.png -------------------------------------------------------------------------------- /doc/img/overlap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/overlap.png -------------------------------------------------------------------------------- /doc/img/overlap_gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/overlap_gray.png -------------------------------------------------------------------------------- /doc/img/paramsettings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/paramsettings.png -------------------------------------------------------------------------------- /doc/img/polyA.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/polyA.png -------------------------------------------------------------------------------- /doc/img/polyB.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/polyB.png -------------------------------------------------------------------------------- /doc/img/result.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/result.mp4 -------------------------------------------------------------------------------- /doc/img/result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/result.png -------------------------------------------------------------------------------- /doc/img/smallcar.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/smallcar.mp4 -------------------------------------------------------------------------------- /doc/img/weight_for_FL.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/weight_for_FL.png -------------------------------------------------------------------------------- /doc/img/weights.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/doc/img/weights.png -------------------------------------------------------------------------------- /images/back.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/images/back.png -------------------------------------------------------------------------------- /images/car.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/images/car.png -------------------------------------------------------------------------------- /images/front.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/images/front.png -------------------------------------------------------------------------------- /images/left.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/images/left.png -------------------------------------------------------------------------------- /images/right.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/images/right.png -------------------------------------------------------------------------------- /masks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/masks.png -------------------------------------------------------------------------------- /run_calibrate_camera.py: -------------------------------------------------------------------------------- 1 | """ 2 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 3 | Fisheye Camera calibration 4 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 5 | 6 | Usage: 7 | python calibrate_camera.py \ 8 | -i 0 \ 9 | -grid 9x6 \ 10 | -out fisheye.yaml \ 11 | -framestep 20 \ 12 | --resolution 640x480 13 | --fisheye 14 | """ 15 | import argparse 16 | import os 17 | import numpy as np 18 | import cv2 19 | from surround_view import CaptureThread, MultiBufferManager 20 | import surround_view.utils as utils 21 | 22 | 23 | # we will save the camera param file to this directory 24 | TARGET_DIR = os.path.join(os.getcwd(), "yaml") 25 | 26 | # default param file 27 | DEFAULT_PARAM_FILE = os.path.join(TARGET_DIR, "camera_params.yaml") 28 | 29 | 30 | def main(): 31 | parser = argparse.ArgumentParser() 32 | 33 | # input video stream 34 | parser.add_argument("-i", "--input", type=int, default=0, 35 | help="input camera device") 36 | 37 | # chessboard pattern size 38 | parser.add_argument("-grid", "--grid", default="9x6", 39 | help="size of the calibrate grid pattern") 40 | 41 | parser.add_argument("-r", "--resolution", default="640x480", 42 | help="resolution of the camera image") 43 | 44 | parser.add_argument("-framestep", type=int, default=20, 45 | help="use every nth frame in the video") 46 | 47 | parser.add_argument("-o", "--output", default=DEFAULT_PARAM_FILE, 48 | help="path to output yaml file") 49 | 50 | parser.add_argument("-fisheye", "--fisheye", action="store_true", 51 | help="set true if this is a fisheye camera") 52 | 53 | parser.add_argument("-flip", "--flip", default=0, type=int, 54 | help="flip method of the camera") 55 | 56 | parser.add_argument("--no_gst", action="store_true", 57 | help="set true if not use gstreamer for the camera capture") 58 | 59 | args = parser.parse_args() 60 | 61 | if not os.path.exists(TARGET_DIR): 62 | os.mkdir(TARGET_DIR) 63 | 64 | text1 = "press c to calibrate" 65 | text2 = "press q to quit" 66 | text3 = "device: {}".format(args.input) 67 | font = cv2.FONT_HERSHEY_SIMPLEX 68 | fontscale = 0.6 69 | 70 | resolution_str = args.resolution.split("x") 71 | W = int(resolution_str[0]) 72 | H = int(resolution_str[1]) 73 | grid_size = tuple(int(x) for x in args.grid.split("x")) 74 | grid_points = np.zeros((1, np.prod(grid_size), 3), np.float32) 75 | grid_points[0, :, :2] = np.indices(grid_size).T.reshape(-1, 2) 76 | 77 | objpoints = [] # 3d point in real world space 78 | imgpoints = [] # 2d points in image plane 79 | 80 | device = args.input 81 | cap_thread = CaptureThread(device_id=device, 82 | flip_method=args.flip, 83 | resolution=(W, H), 84 | use_gst=not args.no_gst, 85 | ) 86 | buffer_manager = MultiBufferManager() 87 | buffer_manager.bind_thread(cap_thread, buffer_size=8) 88 | if cap_thread.connect_camera(): 89 | cap_thread.start() 90 | else: 91 | print("cannot open device") 92 | return 93 | 94 | quit = False 95 | do_calib = False 96 | i = -1 97 | while True: 98 | i += 1 99 | img = buffer_manager.get_device(device).get().image 100 | if i % args.framestep != 0: 101 | continue 102 | 103 | print("searching for chessboard corners in frame " + str(i) + "...") 104 | gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) 105 | found, corners = cv2.findChessboardCorners( 106 | gray, 107 | grid_size, 108 | cv2.CALIB_CB_ADAPTIVE_THRESH + 109 | cv2.CALIB_CB_NORMALIZE_IMAGE + 110 | cv2.CALIB_CB_FILTER_QUADS 111 | ) 112 | if found: 113 | term = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_COUNT, 30, 0.01) 114 | cv2.cornerSubPix(gray, corners, (5, 5), (-1, -1), term) 115 | print("OK") 116 | imgpoints.append(corners) 117 | objpoints.append(grid_points) 118 | cv2.drawChessboardCorners(img, grid_size, corners, found) 119 | 120 | cv2.putText(img, text1, (20, 70), font, fontscale, (255, 200, 0), 2) 121 | cv2.putText(img, text2, (20, 110), font, fontscale, (255, 200, 0), 2) 122 | cv2.putText(img, text3, (20, 30), font, fontscale, (255, 200, 0), 2) 123 | cv2.imshow("corners", img) 124 | key = cv2.waitKey(1) & 0xFF 125 | if key == ord("c"): 126 | print("\nPerforming calibration...\n") 127 | N_OK = len(objpoints) 128 | if N_OK < 12: 129 | print("Less than 12 corners (%d) detected, calibration failed" %(N_OK)) 130 | continue 131 | else: 132 | do_calib = True 133 | break 134 | 135 | elif key == ord("q"): 136 | quit = True 137 | break 138 | 139 | if quit: 140 | cap_thread.stop() 141 | cap_thread.disconnect_camera() 142 | cv2.destroyAllWindows() 143 | 144 | if do_calib: 145 | N_OK = len(objpoints) 146 | K = np.zeros((3, 3)) 147 | D = np.zeros((4, 1)) 148 | rvecs = [np.zeros((1, 1, 3), dtype=np.float64) for _ in range(N_OK)] 149 | tvecs = [np.zeros((1, 1, 3), dtype=np.float64) for _ in range(N_OK)] 150 | calibration_flags = (cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC + 151 | cv2.fisheye.CALIB_CHECK_COND + 152 | cv2.fisheye.CALIB_FIX_SKEW) 153 | 154 | if args.fisheye: 155 | ret, mtx, dist, rvecs, tvecs = cv2.fisheye.calibrate( 156 | objpoints, 157 | imgpoints, 158 | (W, H), 159 | K, 160 | D, 161 | rvecs, 162 | tvecs, 163 | calibration_flags, 164 | (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-6) 165 | ) 166 | else: 167 | ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera( 168 | objpoints, 169 | imgpoints, 170 | (W, H), 171 | None, 172 | None) 173 | 174 | if ret: 175 | fs = cv2.FileStorage(args.output, cv2.FILE_STORAGE_WRITE) 176 | fs.write("resolution", np.int32([W, H])) 177 | fs.write("camera_matrix", K) 178 | fs.write("dist_coeffs", D) 179 | fs.release() 180 | print("successfully saved camera data") 181 | cv2.putText(img, "Success!", (220, 240), font, 2, (0, 0, 255), 2) 182 | 183 | else: 184 | cv2.putText(img, "Failed!", (220, 240), font, 2, (0, 0, 255), 2) 185 | 186 | cv2.imshow("corners", img) 187 | cv2.waitKey(0) 188 | 189 | 190 | if __name__ == "__main__": 191 | main() 192 | -------------------------------------------------------------------------------- /run_get_projection_maps.py: -------------------------------------------------------------------------------- 1 | """ 2 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3 | Manually select points to get the projection map 4 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5 | """ 6 | import argparse 7 | import os 8 | import numpy as np 9 | import cv2 10 | from surround_view import FisheyeCameraModel, PointSelector, display_image 11 | import surround_view.param_settings as settings 12 | 13 | 14 | def get_projection_map(camera_model, image): 15 | und_image = camera_model.undistort(image) 16 | name = camera_model.camera_name 17 | gui = PointSelector(und_image, title=name) 18 | dst_points = settings.project_keypoints[name] 19 | choice = gui.loop() 20 | if choice > 0: 21 | src = np.float32(gui.keypoints) 22 | dst = np.float32(dst_points) 23 | camera_model.project_matrix = cv2.getPerspectiveTransform(src, dst) 24 | proj_image = camera_model.project(und_image) 25 | 26 | ret = display_image("Bird's View", proj_image) 27 | if ret > 0: 28 | return True 29 | if ret < 0: 30 | cv2.destroyAllWindows() 31 | 32 | return False 33 | 34 | 35 | def main(): 36 | parser = argparse.ArgumentParser() 37 | parser.add_argument("-camera", required=True, 38 | choices=["front", "back", "left", "right"], 39 | help="The camera view to be projected") 40 | parser.add_argument("-scale", nargs="+", default=None, 41 | help="scale the undistorted image") 42 | parser.add_argument("-shift", nargs="+", default=None, 43 | help="shift the undistorted image") 44 | args = parser.parse_args() 45 | 46 | if args.scale is not None: 47 | scale = [float(x) for x in args.scale] 48 | else: 49 | scale = (1.0, 1.0) 50 | 51 | if args.shift is not None: 52 | shift = [float(x) for x in args.shift] 53 | else: 54 | shift = (0, 0) 55 | 56 | camera_name = args.camera 57 | camera_file = os.path.join(os.getcwd(), "yaml", camera_name + ".yaml") 58 | image_file = os.path.join(os.getcwd(), "images", camera_name + ".png") 59 | image = cv2.imread(image_file) 60 | camera = FisheyeCameraModel(camera_file, camera_name) 61 | camera.set_scale_and_shift(scale, shift) 62 | success = get_projection_map(camera, image) 63 | if success: 64 | print("saving projection matrix to yaml") 65 | camera.save_data() 66 | else: 67 | print("failed to compute the projection map") 68 | 69 | 70 | if __name__ == "__main__": 71 | main() 72 | -------------------------------------------------------------------------------- /run_get_weight_matrices.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import cv2 4 | from PIL import Image 5 | from surround_view import FisheyeCameraModel, display_image, BirdView 6 | import surround_view.param_settings as settings 7 | 8 | 9 | def main(): 10 | names = settings.camera_names 11 | images = [os.path.join(os.getcwd(), "images", name + ".png") for name in names] 12 | yamls = [os.path.join(os.getcwd(), "yaml", name + ".yaml") for name in names] 13 | camera_models = [FisheyeCameraModel(camera_file, camera_name) for camera_file, camera_name in zip (yamls, names)] 14 | 15 | projected = [] 16 | for image_file, camera in zip(images, camera_models): 17 | img = cv2.imread(image_file) 18 | img = camera.undistort(img) 19 | img = camera.project(img) 20 | img = camera.flip(img) 21 | projected.append(img) 22 | 23 | birdview = BirdView() 24 | Gmat, Mmat = birdview.get_weights_and_masks(projected) 25 | birdview.update_frames(projected) 26 | birdview.make_luminance_balance().stitch_all_parts() 27 | birdview.make_white_balance() 28 | birdview.copy_car_image() 29 | ret = display_image("BirdView Result", birdview.image) 30 | if ret > 0: 31 | Image.fromarray((Gmat * 255).astype(np.uint8)).save("weights.png") 32 | Image.fromarray(Mmat.astype(np.uint8)).save("masks.png") 33 | 34 | 35 | if __name__ == "__main__": 36 | main() 37 | -------------------------------------------------------------------------------- /run_live_demo.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | from surround_view import CaptureThread, CameraProcessingThread 4 | from surround_view import FisheyeCameraModel, BirdView 5 | from surround_view import MultiBufferManager, ProjectedImageBuffer 6 | import surround_view.param_settings as settings 7 | 8 | 9 | yamls_dir = os.path.join(os.getcwd(), "yaml") 10 | camera_ids = [4, 3, 5, 6] 11 | flip_methods = [0, 2, 0, 2] 12 | names = settings.camera_names 13 | cameras_files = [os.path.join(yamls_dir, name + ".yaml") for name in names] 14 | camera_models = [FisheyeCameraModel(camera_file, name) for camera_file, name in zip(cameras_files, names)] 15 | 16 | 17 | def main(): 18 | capture_tds = [CaptureThread(camera_id, flip_method) 19 | for camera_id, flip_method in zip(camera_ids, flip_methods)] 20 | capture_buffer_manager = MultiBufferManager() 21 | for td in capture_tds: 22 | capture_buffer_manager.bind_thread(td, buffer_size=8) 23 | if (td.connect_camera()): 24 | td.start() 25 | 26 | proc_buffer_manager = ProjectedImageBuffer() 27 | process_tds = [CameraProcessingThread(capture_buffer_manager, 28 | camera_id, 29 | camera_model) 30 | for camera_id, camera_model in zip(camera_ids, camera_models)] 31 | for td in process_tds: 32 | proc_buffer_manager.bind_thread(td) 33 | td.start() 34 | 35 | birdview = BirdView(proc_buffer_manager) 36 | birdview.load_weights_and_masks("./weights.png", "./masks.png") 37 | birdview.start() 38 | while True: 39 | img = cv2.resize(birdview.get(), (300, 400)) 40 | cv2.imshow("birdview", img) 41 | key = cv2.waitKey(1) & 0xFF 42 | if key == ord("q"): 43 | break 44 | 45 | for td in capture_tds: 46 | print("camera {} fps: {}\n".format(td.device_id, td.stat_data.average_fps), end="\r") 47 | 48 | for td in process_tds: 49 | print("process {} fps: {}\n".format(td.device_id, td.stat_data.average_fps), end="\r") 50 | 51 | print("birdview fps: {}".format(birdview.stat_data.average_fps)) 52 | 53 | 54 | for td in process_tds: 55 | td.stop() 56 | 57 | for td in capture_tds: 58 | td.stop() 59 | td.disconnect_camera() 60 | 61 | 62 | if __name__ == "__main__": 63 | main() 64 | -------------------------------------------------------------------------------- /surround_view/__init__.py: -------------------------------------------------------------------------------- 1 | from .fisheye_camera import FisheyeCameraModel 2 | from .imagebuffer import MultiBufferManager 3 | from .capture_thread import CaptureThread 4 | from .process_thread import CameraProcessingThread 5 | from .simple_gui import display_image, PointSelector 6 | from .birdview import BirdView, ProjectedImageBuffer 7 | -------------------------------------------------------------------------------- /surround_view/base_thread.py: -------------------------------------------------------------------------------- 1 | from queue import Queue 2 | import cv2 3 | from PyQt5.QtCore import (QThread, QTime, QMutex, pyqtSignal, QMutexLocker) 4 | 5 | from .structures import ThreadStatisticsData 6 | 7 | 8 | class BaseThread(QThread): 9 | 10 | """ 11 | Base class for all types of threads (capture, processing, stitching, ..., 12 | etc). Mainly for collecting statistics of the threads. 13 | """ 14 | 15 | FPS_STAT_QUEUE_LENGTH = 32 16 | 17 | update_statistics_gui = pyqtSignal(ThreadStatisticsData) 18 | 19 | def __init__(self, parent=None): 20 | super(BaseThread, self).__init__(parent) 21 | self.init_commons() 22 | 23 | def init_commons(self): 24 | self.stopped = False 25 | self.stop_mutex = QMutex() 26 | self.clock = QTime() 27 | self.fps = Queue() 28 | self.processing_time = 0 29 | self.processing_mutex = QMutex() 30 | self.fps_sum = 0 31 | self.stat_data = ThreadStatisticsData() 32 | 33 | def stop(self): 34 | with QMutexLocker(self.stop_mutex): 35 | self.stopped = True 36 | 37 | def update_fps(self, dt): 38 | # add instantaneous fps value to queue 39 | if dt > 0: 40 | self.fps.put(1000 / dt) 41 | 42 | # discard redundant items in the fps queue 43 | if self.fps.qsize() > self.FPS_STAT_QUEUE_LENGTH: 44 | self.fps.get() 45 | 46 | # update statistics 47 | if self.fps.qsize() == self.FPS_STAT_QUEUE_LENGTH: 48 | while not self.fps.empty(): 49 | self.fps_sum += self.fps.get() 50 | 51 | self.stat_data.average_fps = round(self.fps_sum / self.FPS_STAT_QUEUE_LENGTH, 2) 52 | self.fps_sum = 0 53 | -------------------------------------------------------------------------------- /surround_view/birdview.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import cv2 4 | from PIL import Image 5 | from PyQt5.QtCore import QMutex, QWaitCondition, QMutexLocker 6 | from .base_thread import BaseThread 7 | from .imagebuffer import Buffer 8 | from . import param_settings as settings 9 | from .param_settings import xl, xr, yt, yb 10 | from . import utils 11 | 12 | 13 | class ProjectedImageBuffer(object): 14 | 15 | """ 16 | Class for synchronizing processing threads from different cameras. 17 | """ 18 | 19 | def __init__(self, drop_if_full=True, buffer_size=8): 20 | self.drop_if_full = drop_if_full 21 | self.buffer = Buffer(buffer_size) 22 | self.sync_devices = set() 23 | self.wc = QWaitCondition() 24 | self.mutex = QMutex() 25 | self.arrived = 0 26 | self.current_frames = dict() 27 | 28 | def bind_thread(self, thread): 29 | with QMutexLocker(self.mutex): 30 | self.sync_devices.add(thread.device_id) 31 | 32 | name = thread.camera_model.camera_name 33 | shape = settings.project_shapes[name] 34 | self.current_frames[thread.device_id] = np.zeros(shape[::-1] + (3,), np.uint8) 35 | thread.proc_buffer_manager = self 36 | 37 | def get(self): 38 | return self.buffer.get() 39 | 40 | def set_frame_for_device(self, device_id, frame): 41 | if device_id not in self.sync_devices: 42 | raise ValueError("Device not held by the buffer: {}".format(device_id)) 43 | self.current_frames[device_id] = frame 44 | 45 | def sync(self, device_id): 46 | # only perform sync if enabled for specified device/stream 47 | self.mutex.lock() 48 | if device_id in self.sync_devices: 49 | # increment arrived count 50 | self.arrived += 1 51 | # we are the last to arrive: wake all waiting threads 52 | if self.arrived == len(self.sync_devices): 53 | self.buffer.add(self.current_frames, self.drop_if_full) 54 | self.wc.wakeAll() 55 | # still waiting for other streams to arrive: wait 56 | else: 57 | self.wc.wait(self.mutex) 58 | # decrement arrived count 59 | self.arrived -= 1 60 | self.mutex.unlock() 61 | 62 | def wake_all(self): 63 | with QMutexLocker(self.mutex): 64 | self.wc.wakeAll() 65 | 66 | def __contains__(self, device_id): 67 | return device_id in self.sync_devices 68 | 69 | def __str__(self): 70 | return (self.__class__.__name__ + ":\n" + \ 71 | "devices: {}\n".format(self.sync_devices)) 72 | 73 | 74 | def FI(front_image): 75 | return front_image[:, :xl] 76 | 77 | 78 | def FII(front_image): 79 | return front_image[:, xr:] 80 | 81 | 82 | def FM(front_image): 83 | return front_image[:, xl:xr] 84 | 85 | 86 | def BIII(back_image): 87 | return back_image[:, :xl] 88 | 89 | 90 | def BIV(back_image): 91 | return back_image[:, xr:] 92 | 93 | 94 | def BM(back_image): 95 | return back_image[:, xl:xr] 96 | 97 | 98 | def LI(left_image): 99 | return left_image[:yt, :] 100 | 101 | 102 | def LIII(left_image): 103 | return left_image[yb:, :] 104 | 105 | 106 | def LM(left_image): 107 | return left_image[yt:yb, :] 108 | 109 | 110 | def RII(right_image): 111 | return right_image[:yt, :] 112 | 113 | 114 | def RIV(right_image): 115 | return right_image[yb:, :] 116 | 117 | 118 | def RM(right_image): 119 | return right_image[yt:yb, :] 120 | 121 | 122 | class BirdView(BaseThread): 123 | 124 | def __init__(self, 125 | proc_buffer_manager=None, 126 | drop_if_full=True, 127 | buffer_size=8, 128 | parent=None): 129 | super(BirdView, self).__init__(parent) 130 | self.proc_buffer_manager = proc_buffer_manager 131 | self.drop_if_full = drop_if_full 132 | self.buffer = Buffer(buffer_size) 133 | self.image = np.zeros((settings.total_h, settings.total_w, 3), np.uint8) 134 | self.weights = None 135 | self.masks = None 136 | self.car_image = settings.car_image 137 | self.frames = None 138 | 139 | def get(self): 140 | return self.buffer.get() 141 | 142 | def update_frames(self, images): 143 | self.frames = images 144 | 145 | def load_weights_and_masks(self, weights_image, masks_image): 146 | GMat = np.asarray(Image.open(weights_image).convert("RGBA"), dtype=np.float) / 255.0 147 | self.weights = [np.stack((GMat[:, :, k], 148 | GMat[:, :, k], 149 | GMat[:, :, k]), axis=2) 150 | for k in range(4)] 151 | 152 | Mmat = np.asarray(Image.open(masks_image).convert("RGBA"), dtype=np.float) 153 | Mmat = utils.convert_binary_to_bool(Mmat) 154 | self.masks = [Mmat[:, :, k] for k in range(4)] 155 | 156 | def merge(self, imA, imB, k): 157 | G = self.weights[k] 158 | return (imA * G + imB * (1 - G)).astype(np.uint8) 159 | 160 | @property 161 | def FL(self): 162 | return self.image[:yt, :xl] 163 | 164 | @property 165 | def F(self): 166 | return self.image[:yt, xl:xr] 167 | 168 | @property 169 | def FR(self): 170 | return self.image[:yt, xr:] 171 | 172 | @property 173 | def BL(self): 174 | return self.image[yb:, :xl] 175 | 176 | @property 177 | def B(self): 178 | return self.image[yb:, xl:xr] 179 | 180 | @property 181 | def BR(self): 182 | return self.image[yb:, xr:] 183 | 184 | @property 185 | def L(self): 186 | return self.image[yt:yb, :xl] 187 | 188 | @property 189 | def R(self): 190 | return self.image[yt:yb, xr:] 191 | 192 | @property 193 | def C(self): 194 | return self.image[yt:yb, xl:xr] 195 | 196 | def stitch_all_parts(self): 197 | front, back, left, right = self.frames 198 | np.copyto(self.F, FM(front)) 199 | np.copyto(self.B, BM(back)) 200 | np.copyto(self.L, LM(left)) 201 | np.copyto(self.R, RM(right)) 202 | np.copyto(self.FL, self.merge(FI(front), LI(left), 0)) 203 | np.copyto(self.FR, self.merge(FII(front), RII(right), 1)) 204 | np.copyto(self.BL, self.merge(BIII(back), LIII(left), 2)) 205 | np.copyto(self.BR, self.merge(BIV(back), RIV(right), 3)) 206 | 207 | def copy_car_image(self): 208 | np.copyto(self.C, self.car_image) 209 | 210 | def make_luminance_balance(self): 211 | 212 | def tune(x): 213 | if x >= 1: 214 | return x * np.exp((1 - x) * 0.5) 215 | else: 216 | return x * np.exp((1 - x) * 0.8) 217 | 218 | front, back, left, right = self.frames 219 | m1, m2, m3, m4 = self.masks 220 | Fb, Fg, Fr = cv2.split(front) 221 | Bb, Bg, Br = cv2.split(back) 222 | Lb, Lg, Lr = cv2.split(left) 223 | Rb, Rg, Rr = cv2.split(right) 224 | 225 | a1 = utils.mean_luminance_ratio(RII(Rb), FII(Fb), m2) 226 | a2 = utils.mean_luminance_ratio(RII(Rg), FII(Fg), m2) 227 | a3 = utils.mean_luminance_ratio(RII(Rr), FII(Fr), m2) 228 | 229 | b1 = utils.mean_luminance_ratio(BIV(Bb), RIV(Rb), m4) 230 | b2 = utils.mean_luminance_ratio(BIV(Bg), RIV(Rg), m4) 231 | b3 = utils.mean_luminance_ratio(BIV(Br), RIV(Rr), m4) 232 | 233 | c1 = utils.mean_luminance_ratio(LIII(Lb), BIII(Bb), m3) 234 | c2 = utils.mean_luminance_ratio(LIII(Lg), BIII(Bg), m3) 235 | c3 = utils.mean_luminance_ratio(LIII(Lr), BIII(Br), m3) 236 | 237 | d1 = utils.mean_luminance_ratio(FI(Fb), LI(Lb), m1) 238 | d2 = utils.mean_luminance_ratio(FI(Fg), LI(Lg), m1) 239 | d3 = utils.mean_luminance_ratio(FI(Fr), LI(Lr), m1) 240 | 241 | t1 = (a1 * b1 * c1 * d1)**0.25 242 | t2 = (a2 * b2 * c2 * d2)**0.25 243 | t3 = (a3 * b3 * c3 * d3)**0.25 244 | 245 | x1 = t1 / (d1 / a1)**0.5 246 | x2 = t2 / (d2 / a2)**0.5 247 | x3 = t3 / (d3 / a3)**0.5 248 | 249 | x1 = tune(x1) 250 | x2 = tune(x2) 251 | x3 = tune(x3) 252 | 253 | Fb = utils.adjust_luminance(Fb, x1) 254 | Fg = utils.adjust_luminance(Fg, x2) 255 | Fr = utils.adjust_luminance(Fr, x3) 256 | 257 | y1 = t1 / (b1 / c1)**0.5 258 | y2 = t2 / (b2 / c2)**0.5 259 | y3 = t3 / (b3 / c3)**0.5 260 | 261 | y1 = tune(y1) 262 | y2 = tune(y2) 263 | y3 = tune(y3) 264 | 265 | Bb = utils.adjust_luminance(Bb, y1) 266 | Bg = utils.adjust_luminance(Bg, y2) 267 | Br = utils.adjust_luminance(Br, y3) 268 | 269 | z1 = t1 / (c1 / d1)**0.5 270 | z2 = t2 / (c2 / d2)**0.5 271 | z3 = t3 / (c3 / d3)**0.5 272 | 273 | z1 = tune(z1) 274 | z2 = tune(z2) 275 | z3 = tune(z3) 276 | 277 | Lb = utils.adjust_luminance(Lb, z1) 278 | Lg = utils.adjust_luminance(Lg, z2) 279 | Lr = utils.adjust_luminance(Lr, z3) 280 | 281 | w1 = t1 / (a1 / b1)**0.5 282 | w2 = t2 / (a2 / b2)**0.5 283 | w3 = t3 / (a3 / b3)**0.5 284 | 285 | w1 = tune(w1) 286 | w2 = tune(w2) 287 | w3 = tune(w3) 288 | 289 | Rb = utils.adjust_luminance(Rb, w1) 290 | Rg = utils.adjust_luminance(Rg, w2) 291 | Rr = utils.adjust_luminance(Rr, w3) 292 | 293 | self.frames = [cv2.merge((Fb, Fg, Fr)), 294 | cv2.merge((Bb, Bg, Br)), 295 | cv2.merge((Lb, Lg, Lr)), 296 | cv2.merge((Rb, Rg, Rr))] 297 | return self 298 | 299 | def get_weights_and_masks(self, images): 300 | front, back, left, right = images 301 | G0, M0 = utils.get_weight_mask_matrix(FI(front), LI(left)) 302 | G1, M1 = utils.get_weight_mask_matrix(FII(front), RII(right)) 303 | G2, M2 = utils.get_weight_mask_matrix(BIII(back), LIII(left)) 304 | G3, M3 = utils.get_weight_mask_matrix(BIV(back), RIV(right)) 305 | self.weights = [np.stack((G, G, G), axis=2) for G in (G0, G1, G2, G3)] 306 | self.masks = [(M / 255.0).astype(int) for M in (M0, M1, M2, M3)] 307 | return np.stack((G0, G1, G2, G3), axis=2), np.stack((M0, M1, M2, M3), axis=2) 308 | 309 | def make_white_balance(self): 310 | self.image = utils.make_white_balance(self.image) 311 | 312 | def run(self): 313 | if self.proc_buffer_manager is None: 314 | raise ValueError("This thread requires a buffer of projected images to run") 315 | 316 | while True: 317 | self.stop_mutex.lock() 318 | if self.stopped: 319 | self.stopped = False 320 | self.stop_mutex.unlock() 321 | break 322 | self.stop_mutex.unlock() 323 | self.processing_time = self.clock.elapsed() 324 | self.clock.start() 325 | 326 | self.processing_mutex.lock() 327 | 328 | self.update_frames(self.proc_buffer_manager.get().values()) 329 | self.make_luminance_balance().stitch_all_parts() 330 | self.make_white_balance() 331 | self.copy_car_image() 332 | self.buffer.add(self.image.copy(), self.drop_if_full) 333 | self.processing_mutex.unlock() 334 | 335 | # update statistics 336 | self.update_fps(self.processing_time) 337 | self.stat_data.frames_processed_count += 1 338 | # inform GUI of updated statistics 339 | self.update_statistics_gui.emit(self.stat_data) 340 | -------------------------------------------------------------------------------- /surround_view/capture_thread.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | from PyQt5.QtCore import qDebug 3 | 4 | from .base_thread import BaseThread 5 | from .structures import ImageFrame 6 | from .utils import gstreamer_pipeline 7 | 8 | 9 | class CaptureThread(BaseThread): 10 | 11 | def __init__(self, 12 | device_id, 13 | flip_method=2, 14 | drop_if_full=True, 15 | api_preference=cv2.CAP_GSTREAMER, 16 | resolution=None, 17 | use_gst=True, 18 | parent=None): 19 | """ 20 | device_id: device number of the camera. 21 | flip_method: 0 for identity, 2 for 180 degree rotation (if the camera is installed 22 | up-side-down). 23 | drop_if_full: drop the frame if buffer is full. 24 | api_preference: cv2.CAP_GSTREAMER for csi cameras, usually cv2.CAP_ANY would suffice. 25 | resolution: camera resolution (width, height). 26 | """ 27 | super(CaptureThread, self).__init__(parent) 28 | self.device_id = device_id 29 | self.flip_method = flip_method 30 | self.use_gst = use_gst 31 | self.drop_if_full = drop_if_full 32 | self.api_preference = api_preference 33 | self.resolution = resolution 34 | self.cap = cv2.VideoCapture() 35 | # an instance of the MultiBufferManager object, 36 | # for synchronizing this thread with other cameras. 37 | self.buffer_manager = None 38 | 39 | def run(self): 40 | if self.buffer_manager is None: 41 | raise ValueError("This thread has not been binded to any buffer manager yet") 42 | 43 | while True: 44 | self.stop_mutex.lock() 45 | if self.stopped: 46 | self.stopped = False 47 | self.stop_mutex.unlock() 48 | break 49 | self.stop_mutex.unlock() 50 | 51 | # save capture time 52 | self.processing_time = self.clock.elapsed() 53 | # start timer (used to calculate capture rate) 54 | self.clock.start() 55 | 56 | # synchronize with other streams (if enabled for this stream) 57 | self.buffer_manager.sync(self.device_id) 58 | 59 | if not self.cap.grab(): 60 | continue 61 | 62 | # retrieve frame and add it to buffer 63 | _, frame = self.cap.retrieve() 64 | img_frame = ImageFrame(self.clock.msecsSinceStartOfDay(), frame) 65 | self.buffer_manager.get_device(self.device_id).add(img_frame, self.drop_if_full) 66 | 67 | # update statistics 68 | self.update_fps(self.processing_time) 69 | self.stat_data.frames_processed_count += 1 70 | # inform GUI of updated statistics 71 | self.update_statistics_gui.emit(self.stat_data) 72 | 73 | qDebug("Stopping capture thread...") 74 | 75 | def connect_camera(self): 76 | if self.use_gst: 77 | options = gstreamer_pipeline(cam_id=self.device_id, flip_method=self.flip_method) 78 | self.cap.open(options, self.api_preference) 79 | else: 80 | self.cap.open(self.device_id) 81 | # return false if failed to open camera 82 | if not self.cap.isOpened(): 83 | qDebug("Cannot open camera {}".format(self.device_id)) 84 | return False 85 | else: 86 | # try to set camera resolution 87 | if self.resolution is not None: 88 | width, height = self.resolution 89 | self.cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"MJPG")) 90 | self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, width) 91 | self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height) 92 | # some camera may become closed if the resolution is not supported 93 | if not self.cap.isOpened(): 94 | qDebug("Resolution not supported by camera device: {}".format(self.resolution)) 95 | return False 96 | # use the default resolution 97 | else: 98 | self.cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"MJPG")) 99 | width = int(self.cap.get(cv2.CAP_PROP_FRAME_WIDTH)) 100 | height = int(self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) 101 | self.resolution = (width, height) 102 | 103 | return True 104 | 105 | def disconnect_camera(self): 106 | # disconnect camera if it's already opened. 107 | if self.cap.isOpened(): 108 | self.cap.release() 109 | return True 110 | # else do nothing and return 111 | else: 112 | return False 113 | 114 | def is_camera_connected(self): 115 | return self.cap.isOpened() 116 | -------------------------------------------------------------------------------- /surround_view/fisheye_camera.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import cv2 4 | 5 | from . import param_settings as settings 6 | 7 | 8 | class FisheyeCameraModel(object): 9 | 10 | """ 11 | Fisheye camera model, for undistorting, projecting and flipping camera frames. 12 | """ 13 | 14 | def __init__(self, camera_param_file, camera_name): 15 | if not os.path.isfile(camera_param_file): 16 | raise ValueError("Cannot find camera param file") 17 | 18 | if camera_name not in settings.camera_names: 19 | raise ValueError("Unknown camera name: {}".format(camera_name)) 20 | 21 | self.camera_file = camera_param_file 22 | self.camera_name = camera_name 23 | self.scale_xy = (1.0, 1.0) 24 | self.shift_xy = (0, 0) 25 | self.undistort_maps = None 26 | self.project_matrix = None 27 | self.project_shape = settings.project_shapes[self.camera_name] 28 | self.load_camera_params() 29 | 30 | def load_camera_params(self): 31 | fs = cv2.FileStorage(self.camera_file, cv2.FILE_STORAGE_READ) 32 | self.camera_matrix = fs.getNode("camera_matrix").mat() 33 | self.dist_coeffs = fs.getNode("dist_coeffs").mat() 34 | self.resolution = fs.getNode("resolution").mat().flatten() 35 | 36 | scale_xy = fs.getNode("scale_xy").mat() 37 | if scale_xy is not None: 38 | self.scale_xy = scale_xy 39 | 40 | shift_xy = fs.getNode("shift_xy").mat() 41 | if shift_xy is not None: 42 | self.shift_xy = shift_xy 43 | 44 | project_matrix = fs.getNode("project_matrix").mat() 45 | if project_matrix is not None: 46 | self.project_matrix = project_matrix 47 | 48 | fs.release() 49 | self.update_undistort_maps() 50 | 51 | def update_undistort_maps(self): 52 | new_matrix = self.camera_matrix.copy() 53 | new_matrix[0, 0] *= self.scale_xy[0] 54 | new_matrix[1, 1] *= self.scale_xy[1] 55 | new_matrix[0, 2] += self.shift_xy[0] 56 | new_matrix[1, 2] += self.shift_xy[1] 57 | width, height = self.resolution 58 | 59 | self.undistort_maps = cv2.fisheye.initUndistortRectifyMap( 60 | self.camera_matrix, 61 | self.dist_coeffs, 62 | np.eye(3), 63 | new_matrix, 64 | (width, height), 65 | cv2.CV_16SC2 66 | ) 67 | return self 68 | 69 | def set_scale_and_shift(self, scale_xy=(1.0, 1.0), shift_xy=(0, 0)): 70 | self.scale_xy = scale_xy 71 | self.shift_xy = shift_xy 72 | self.update_undistort_maps() 73 | return self 74 | 75 | def undistort(self, image): 76 | result = cv2.remap(image, *self.undistort_maps, interpolation=cv2.INTER_LINEAR, 77 | borderMode=cv2.BORDER_CONSTANT) 78 | return result 79 | 80 | def project(self, image): 81 | result = cv2.warpPerspective(image, self.project_matrix, self.project_shape) 82 | return result 83 | 84 | def flip(self, image): 85 | if self.camera_name == "front": 86 | return image.copy() 87 | 88 | elif self.camera_name == "back": 89 | return image.copy()[::-1, ::-1, :] 90 | 91 | elif self.camera_name == "left": 92 | return cv2.transpose(image)[::-1] 93 | 94 | else: 95 | return np.flip(cv2.transpose(image), 1) 96 | 97 | def save_data(self): 98 | fs = cv2.FileStorage(self.camera_file, cv2.FILE_STORAGE_WRITE) 99 | fs.write("camera_matrix", self.camera_matrix) 100 | fs.write("dist_coeffs", self.dist_coeffs) 101 | fs.write("resolution", self.resolution) 102 | fs.write("project_matrix", self.project_matrix) 103 | fs.write("scale_xy", np.float32(self.scale_xy)) 104 | fs.write("shift_xy", np.float32(self.shift_xy)) 105 | fs.release() 106 | -------------------------------------------------------------------------------- /surround_view/imagebuffer.py: -------------------------------------------------------------------------------- 1 | from PyQt5.QtCore import QSemaphore, QMutex 2 | from PyQt5.QtCore import QMutexLocker, QWaitCondition 3 | from queue import Queue 4 | 5 | 6 | class Buffer(object): 7 | 8 | def __init__(self, buffer_size=5): 9 | self.buffer_size = buffer_size 10 | self.free_slots = QSemaphore(self.buffer_size) 11 | self.used_slots = QSemaphore(0) 12 | self.clear_buffer_add = QSemaphore(1) 13 | self.clear_buffer_get = QSemaphore(1) 14 | self.queue_mutex = QMutex() 15 | self.queue = Queue(self.buffer_size) 16 | 17 | def add(self, data, drop_if_full=False): 18 | self.clear_buffer_add.acquire() 19 | if drop_if_full: 20 | if self.free_slots.tryAcquire(): 21 | self.queue_mutex.lock() 22 | self.queue.put(data) 23 | self.queue_mutex.unlock() 24 | self.used_slots.release() 25 | else: 26 | self.free_slots.acquire() 27 | self.queue_mutex.lock() 28 | self.queue.put(data) 29 | self.queue_mutex.unlock() 30 | self.used_slots.release() 31 | 32 | self.clear_buffer_add.release() 33 | 34 | def get(self): 35 | # acquire semaphores 36 | self.clear_buffer_get.acquire() 37 | self.used_slots.acquire() 38 | self.queue_mutex.lock() 39 | data = self.queue.get() 40 | self.queue_mutex.unlock() 41 | # release semaphores 42 | self.free_slots.release() 43 | self.clear_buffer_get.release() 44 | # return item to caller 45 | return data 46 | 47 | def clear(self): 48 | # check if buffer contains items 49 | if self.queue.qsize() > 0: 50 | # stop adding items to buffer (will return false if an item is currently being added to the buffer) 51 | if self.clear_buffer_add.tryAcquire(): 52 | # stop taking items from buffer (will return false if an item is currently being taken from the buffer) 53 | if self.clear_buffer_get.tryAcquire(): 54 | # release all remaining slots in queue 55 | self.free_slots.release(self.queue.qsize()) 56 | # acquire all queue slots 57 | self.free_slots.acquire(self.buffer_size) 58 | # reset used_slots to zero 59 | self.used_slots.acquire(self.queue.qsize()) 60 | # clear buffer 61 | for _ in range(self.queue.qsize()): 62 | self.queue.get() 63 | # release all slots 64 | self.free_slots.release(self.buffer_size) 65 | # allow get method to resume 66 | self.clear_buffer_get.release() 67 | else: 68 | return False 69 | # allow add method to resume 70 | self.clear_buffer_add.release() 71 | return True 72 | else: 73 | return False 74 | else: 75 | return False 76 | 77 | def size(self): 78 | return self.queue.qsize() 79 | 80 | def maxsize(self): 81 | return self.buffer_size 82 | 83 | def isfull(self): 84 | return self.queue.qsize() == self.buffer_size 85 | 86 | def isempty(self): 87 | return self.queue.qsize() == 0 88 | 89 | 90 | class MultiBufferManager(object): 91 | 92 | """ 93 | Class for synchronizing capture threads from different cameras. 94 | """ 95 | 96 | def __init__(self, do_sync=True): 97 | self.sync_devices = set() 98 | self.do_sync = do_sync 99 | self.wc = QWaitCondition() 100 | self.mutex = QMutex() 101 | self.arrived = 0 102 | self.buffer_maps = dict() 103 | 104 | def bind_thread(self, thread, buffer_size, sync=True): 105 | self.create_buffer_for_device(thread.device_id, buffer_size, sync) 106 | thread.buffer_manager = self 107 | 108 | def create_buffer_for_device(self, device_id, buffer_size, sync=True): 109 | if sync: 110 | with QMutexLocker(self.mutex): 111 | self.sync_devices.add(device_id) 112 | 113 | self.buffer_maps[device_id] = Buffer(buffer_size) 114 | 115 | def get_device(self, device_id): 116 | return self.buffer_maps[device_id] 117 | 118 | def remove_device(self, device_id): 119 | self.buffer_maps.pop(device_id) 120 | with QMutexLocker(self.mutex): 121 | if device_id in self.sync_devices: 122 | self.sync_devices.remove(device_id) 123 | self.wc.wakeAll() 124 | 125 | def sync(self, device_id): 126 | # only perform sync if enabled for specified device/stream 127 | self.mutex.lock() 128 | if device_id in self.sync_devices: 129 | # increment arrived count 130 | self.arrived += 1 131 | # we are the last to arrive: wake all waiting threads 132 | if self.do_sync and self.arrived == len(self.sync_devices): 133 | self.wc.wakeAll() 134 | # still waiting for other streams to arrive: wait 135 | else: 136 | self.wc.wait(self.mutex) 137 | # decrement arrived count 138 | self.arrived -= 1 139 | self.mutex.unlock() 140 | 141 | def wake_all(self): 142 | with QMutexLocker(self.mutex): 143 | self.wc.wakeAll() 144 | 145 | def set_sync(self, enable): 146 | self.do_sync = enable 147 | 148 | def sync_enabled(self): 149 | return self.do_sync 150 | 151 | def sync_enabled_for_device(self, device_id): 152 | return device_id in self.sync_devices 153 | 154 | def __contains__(self, device_id): 155 | return device_id in self.buffer_maps 156 | 157 | def __str__(self): 158 | return (self.__class__.__name__ + ":\n" + \ 159 | "sync: {}\n".format(self.do_sync) + \ 160 | "devices: {}\n".format(tuple(self.buffer_maps.keys())) + \ 161 | "sync enabled devices: {}".format(self.sync_devices)) 162 | -------------------------------------------------------------------------------- /surround_view/param_settings.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | 4 | 5 | camera_names = ["front", "back", "left", "right"] 6 | 7 | # -------------------------------------------------------------------- 8 | # (shift_width, shift_height): how far away the birdview looks outside 9 | # of the calibration pattern in horizontal and vertical directions 10 | shift_w = 300 11 | shift_h = 300 12 | 13 | # size of the gap between the calibration pattern and the car 14 | # in horizontal and vertical directions 15 | inn_shift_w = 20 16 | inn_shift_h = 50 17 | 18 | # total width/height of the stitched image 19 | total_w = 600 + 2 * shift_w 20 | total_h = 1000 + 2 * shift_h 21 | 22 | # four corners of the rectangular region occupied by the car 23 | # top-left (x_left, y_top), bottom-right (x_right, y_bottom) 24 | xl = shift_w + 180 + inn_shift_w 25 | xr = total_w - xl 26 | yt = shift_h + 200 + inn_shift_h 27 | yb = total_h - yt 28 | # -------------------------------------------------------------------- 29 | 30 | project_shapes = { 31 | "front": (total_w, yt), 32 | "back": (total_w, yt), 33 | "left": (total_h, xl), 34 | "right": (total_h, xl) 35 | } 36 | 37 | # pixel locations of the four points to be chosen. 38 | # you must click these pixels in the same order when running 39 | # the get_projection_map.py script 40 | project_keypoints = { 41 | "front": [(shift_w + 120, shift_h), 42 | (shift_w + 480, shift_h), 43 | (shift_w + 120, shift_h + 160), 44 | (shift_w + 480, shift_h + 160)], 45 | 46 | "back": [(shift_w + 120, shift_h), 47 | (shift_w + 480, shift_h), 48 | (shift_w + 120, shift_h + 160), 49 | (shift_w + 480, shift_h + 160)], 50 | 51 | "left": [(shift_h + 280, shift_w), 52 | (shift_h + 840, shift_w), 53 | (shift_h + 280, shift_w + 160), 54 | (shift_h + 840, shift_w + 160)], 55 | 56 | "right": [(shift_h + 160, shift_w), 57 | (shift_h + 720, shift_w), 58 | (shift_h + 160, shift_w + 160), 59 | (shift_h + 720, shift_w + 160)] 60 | } 61 | 62 | car_image = cv2.imread(os.path.join(os.getcwd(), "images", "car.png")) 63 | car_image = cv2.resize(car_image, (xr - xl, yb - yt)) 64 | -------------------------------------------------------------------------------- /surround_view/process_thread.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | from PyQt5.QtCore import qDebug, QMutex 3 | 4 | from .base_thread import BaseThread 5 | 6 | 7 | class CameraProcessingThread(BaseThread): 8 | 9 | """ 10 | Thread for processing individual camera images, i.e. undistort, project and flip. 11 | """ 12 | 13 | def __init__(self, 14 | capture_buffer_manager, 15 | device_id, 16 | camera_model, 17 | drop_if_full=True, 18 | parent=None): 19 | """ 20 | capture_buffer_manager: an instance of the `MultiBufferManager` object. 21 | device_id: device number of the camera to be processed. 22 | camera_model: an instance of the `FisheyeCameraModel` object. 23 | drop_if_full: drop if the buffer is full. 24 | """ 25 | super(CameraProcessingThread, self).__init__(parent) 26 | self.capture_buffer_manager = capture_buffer_manager 27 | self.device_id = device_id 28 | self.camera_model = camera_model 29 | self.drop_if_full = drop_if_full 30 | # an instance of the `ProjectedImageBuffer` object 31 | self.proc_buffer_manager = None 32 | 33 | def run(self): 34 | if self.proc_buffer_manager is None: 35 | raise ValueError("This thread has not been binded to any processing thread yet") 36 | 37 | while True: 38 | self.stop_mutex.lock() 39 | if self.stopped: 40 | self.stopped = False 41 | self.stop_mutex.unlock() 42 | break 43 | self.stop_mutex.unlock() 44 | 45 | self.processing_time = self.clock.elapsed() 46 | self.clock.start() 47 | 48 | self.processing_mutex.lock() 49 | raw_frame = self.capture_buffer_manager.get_device(self.device_id).get() 50 | und_frame = self.camera_model.undistort(raw_frame.image) 51 | pro_frame = self.camera_model.project(und_frame) 52 | flip_frame = self.camera_model.flip(pro_frame) 53 | self.processing_mutex.unlock() 54 | 55 | self.proc_buffer_manager.sync(self.device_id) 56 | self.proc_buffer_manager.set_frame_for_device(self.device_id, flip_frame) 57 | 58 | # update statistics 59 | self.update_fps(self.processing_time) 60 | self.stat_data.frames_processed_count += 1 61 | # inform GUI of updated statistics 62 | self.update_statistics_gui.emit(self.stat_data) 63 | -------------------------------------------------------------------------------- /surround_view/simple_gui.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | 4 | # return -1 if user press 'q'. return 1 if user press 'Enter'. 5 | def display_image(window_title, image): 6 | cv2.imshow(window_title, image) 7 | while True: 8 | click = cv2.getWindowProperty(window_title, cv2.WND_PROP_AUTOSIZE) 9 | if click < 0: 10 | return -1 11 | 12 | key = cv2.waitKey(1) & 0xFF 13 | if key == ord("q"): 14 | return -1 15 | 16 | # 'Enter' key is detected! 17 | if key == 13: 18 | return 1 19 | 20 | 21 | class PointSelector(object): 22 | 23 | """ 24 | --------------------------------------------------- 25 | | A simple gui point selector. | 26 | | Usage: | 27 | | | 28 | | 1. call the `loop` method to show the image. | 29 | | 2. click on the image to select key points, | 30 | | press `d` to delete the last points. | 31 | | 3. press `q` to quit, press `Enter` to confirm. | 32 | --------------------------------------------------- 33 | """ 34 | 35 | POINT_COLOR = (0, 0, 255) 36 | FILL_COLOR = (0, 255, 255) 37 | 38 | def __init__(self, image, title="PointSelector"): 39 | self.image = image 40 | self.title = title 41 | self.keypoints = [] 42 | 43 | def draw_image(self): 44 | """ 45 | Display the selected keypoints and draw the convex hull. 46 | """ 47 | # the trick: draw on another new image 48 | new_image = self.image.copy() 49 | 50 | # draw the selected keypoints 51 | for i, pt in enumerate(self.keypoints): 52 | cv2.circle(new_image, pt, 6, self.POINT_COLOR, -1) 53 | cv2.putText(new_image, str(i), (pt[0], pt[1] - 15), 54 | cv2.FONT_HERSHEY_SIMPLEX, 0.6, self.POINT_COLOR, 2) 55 | 56 | # draw a line if there are two points 57 | if len(self.keypoints) == 2: 58 | p1, p2 = self.keypoints 59 | cv2.line(new_image, p1, p2, self.POINT_COLOR, 2) 60 | 61 | # draw the convex hull if there are more than two points 62 | if len(self.keypoints) > 2: 63 | mask = self.create_mask_from_pixels(self.keypoints, 64 | self.image.shape) 65 | new_image = self.draw_mask_on_image(new_image, mask) 66 | 67 | cv2.imshow(self.title, new_image) 68 | 69 | def onclick(self, event, x, y, flags, param): 70 | """ 71 | Click on a point (x, y) will add this points to the list 72 | and re-draw the image. 73 | """ 74 | if event == cv2.EVENT_LBUTTONDOWN: 75 | print("click ({}, {})".format(x, y)) 76 | self.keypoints.append((x, y)) 77 | self.draw_image() 78 | 79 | def loop(self): 80 | """ 81 | Press "q" will exist the gui and return False 82 | press "d" will delete the last selected point. 83 | Press "Enter" will exist the gui and return True. 84 | """ 85 | cv2.namedWindow(self.title) 86 | cv2.setMouseCallback(self.title, self.onclick, param=()) 87 | cv2.imshow(self.title, self.image) 88 | 89 | while True: 90 | click = cv2.getWindowProperty(self.title, cv2.WND_PROP_AUTOSIZE) 91 | if click < 0: 92 | return False 93 | 94 | key = cv2.waitKey(1) & 0xFF 95 | 96 | # press q to return False 97 | if key == ord("q"): 98 | return False 99 | 100 | # press d to delete the last point 101 | if key == ord("d"): 102 | if len(self.keypoints) > 0: 103 | x, y = self.keypoints.pop() 104 | print("Delete ({}, {})".format(x, y)) 105 | self.draw_image() 106 | 107 | # press Enter to confirm 108 | if key == 13: 109 | return True 110 | 111 | def create_mask_from_pixels(self, pixels, image_shape): 112 | """ 113 | Create mask from the convex hull of a list of pixels. 114 | """ 115 | pixels = np.int32(pixels).reshape(-1, 2) 116 | hull = cv2.convexHull(pixels) 117 | mask = np.zeros(image_shape[:2], np.int8) 118 | cv2.fillConvexPoly(mask, hull, 1, lineType=8, shift=0) 119 | mask = mask.astype(bool) 120 | return mask 121 | 122 | def draw_mask_on_image(self, image, mask): 123 | """ 124 | Paint the region defined by a given mask on an image. 125 | """ 126 | new_image = np.zeros_like(image) 127 | new_image[:, :] = self.FILL_COLOR 128 | mask = np.array(mask, dtype=np.uint8) 129 | new_mask = cv2.bitwise_and(new_image, new_image, mask=mask) 130 | cv2.addWeighted(image, 1.0, new_mask, 0.5, 0.0, image) 131 | return image 132 | -------------------------------------------------------------------------------- /surround_view/structures.py: -------------------------------------------------------------------------------- 1 | class ImageFrame(object): 2 | 3 | def __init__(self, timestamp, image): 4 | self.timestamp = timestamp 5 | self.image = image 6 | 7 | 8 | class ThreadStatisticsData(object): 9 | 10 | def __init__(self): 11 | self.average_fps = 0 12 | self.frames_processed_count = 0 13 | -------------------------------------------------------------------------------- /surround_view/utils.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | 4 | 5 | def gstreamer_pipeline(cam_id=0, 6 | capture_width=960, 7 | capture_height=640, 8 | framerate=60, 9 | flip_method=2): 10 | """ 11 | Use libgstreamer to open csi-cameras. 12 | """ 13 | return ("nvarguscamerasrc sensor-id={} ! ".format(cam_id) + \ 14 | "video/x-raw(memory:NVMM), " 15 | "width=(int)%d, height=(int)%d, " 16 | "format=(string)NV12, framerate=(fraction)%d/1 ! " 17 | "nvvidconv flip-method=%d ! " 18 | "video/x-raw, format=(string)BGRx ! " 19 | "videoconvert ! " 20 | "video/x-raw, format=(string)BGR ! appsink" 21 | % (capture_width, 22 | capture_height, 23 | framerate, 24 | flip_method 25 | ) 26 | ) 27 | 28 | 29 | def convert_binary_to_bool(mask): 30 | """ 31 | Convert a binary image (only one channel and pixels are 0 or 255) to 32 | a bool one (all pixels are 0 or 1). 33 | """ 34 | return (mask.astype(np.float) / 255.0).astype(int) 35 | 36 | 37 | def adjust_luminance(gray, factor): 38 | """ 39 | Adjust the luminance of a grayscale image by a factor. 40 | """ 41 | return np.minimum((gray * factor), 255).astype(np.uint8) 42 | 43 | 44 | def get_mean_statistisc(gray, mask): 45 | """ 46 | Get the total values of a gray image in a region defined by a mask matrix. 47 | The mask matrix must have values either 0 or 1. 48 | """ 49 | return np.sum(gray * mask) 50 | 51 | 52 | def mean_luminance_ratio(grayA, grayB, mask): 53 | return get_mean_statistisc(grayA, mask) / get_mean_statistisc(grayB, mask) 54 | 55 | 56 | def get_mask(img): 57 | """ 58 | Convert an image to a mask array. 59 | """ 60 | gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) 61 | ret, mask = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY) 62 | return mask 63 | 64 | 65 | def get_overlap_region_mask(imA, imB): 66 | """ 67 | Given two images of the save size, get their overlapping region and 68 | convert this region to a mask array. 69 | """ 70 | overlap = cv2.bitwise_and(imA, imB) 71 | mask = get_mask(overlap) 72 | mask = cv2.dilate(mask, np.ones((2, 2), np.uint8), iterations=2) 73 | return mask 74 | 75 | 76 | def get_outmost_polygon_boundary(img): 77 | """ 78 | Given a mask image with the mask describes the overlapping region of 79 | two images, get the outmost contour of this region. 80 | """ 81 | mask = get_mask(img) 82 | mask = cv2.dilate(mask, np.ones((2, 2), np.uint8), iterations=2) 83 | cnts, hierarchy = cv2.findContours( 84 | mask, 85 | cv2.RETR_EXTERNAL, 86 | cv2.CHAIN_APPROX_SIMPLE)[-2:] 87 | 88 | # get the contour with largest aera 89 | C = sorted(cnts, key=lambda x: cv2.contourArea(x), reverse=True)[0] 90 | 91 | # polygon approximation 92 | polygon = cv2.approxPolyDP(C, 0.009 * cv2.arcLength(C, True), True) 93 | 94 | return polygon 95 | 96 | 97 | def get_weight_mask_matrix(imA, imB, dist_threshold=5): 98 | """ 99 | Get the weight matrix G that combines two images imA, imB smoothly. 100 | """ 101 | overlapMask = get_overlap_region_mask(imA, imB) 102 | overlapMaskInv = cv2.bitwise_not(overlapMask) 103 | indices = np.where(overlapMask == 255) 104 | 105 | imA_diff = cv2.bitwise_and(imA, imA, mask=overlapMaskInv) 106 | imB_diff = cv2.bitwise_and(imB, imB, mask=overlapMaskInv) 107 | 108 | G = get_mask(imA).astype(np.float32) / 255.0 109 | 110 | polyA = get_outmost_polygon_boundary(imA_diff) 111 | polyB = get_outmost_polygon_boundary(imB_diff) 112 | 113 | for y, x in zip(*indices): 114 | 115 | #convert this x,y int an INT tuple 116 | xy_tuple = tuple([int(x), int(y)]) 117 | distToB = cv2.pointPolygonTest(polyB, xy_tuple, True) 118 | 119 | if distToB < dist_threshold: 120 | distToA = cv2.pointPolygonTest(polyA, xy_tuple, True) 121 | distToB *= distToB 122 | distToA *= distToA 123 | G[y, x] = distToB / (distToA + distToB) 124 | 125 | return G, overlapMask 126 | 127 | 128 | def make_white_balance(image): 129 | """ 130 | Adjust white balance of an image base on the means of its channels. 131 | """ 132 | B, G, R = cv2.split(image) 133 | m1 = np.mean(B) 134 | m2 = np.mean(G) 135 | m3 = np.mean(R) 136 | K = (m1 + m2 + m3) / 3 137 | c1 = K / m1 138 | c2 = K / m2 139 | c3 = K / m3 140 | B = adjust_luminance(B, c1) 141 | G = adjust_luminance(G, c2) 142 | R = adjust_luminance(R, c3) 143 | return cv2.merge((B, G, R)) 144 | -------------------------------------------------------------------------------- /test_cameras.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import cv2 4 | import time 5 | 6 | # get the installed camera list for initialization. 7 | def get_cam_lst(cam_lst=range(0, 24)): 8 | arr = [] 9 | for iCam in cam_lst: 10 | cap = cv2.VideoCapture(iCam) 11 | if not cap.read()[0]: 12 | continue 13 | else: 14 | arr.append(iCam) 15 | 16 | cap.release() 17 | return arr 18 | 19 | def show_cam_img(caps, cam_list): 20 | print("INFO: Press 'q' to quit! Press 's' to save a picture, 'n' to change to next camera device!") 21 | idx = 0 22 | while True: 23 | cap_device = caps[idx] 24 | ret, frame = cap_device.read() 25 | if ret: 26 | cv2.imshow('video', frame) 27 | else: 28 | print("ERROR: failed read frame!") 29 | 30 | # quit the test 31 | c = cv2.waitKey(1) 32 | if c == ord('q'): 33 | break 34 | 35 | # change to next camera device 36 | if c == ord('n'): 37 | idx += 1 38 | if idx >= len(caps): 39 | idx = 0 40 | continue 41 | 42 | # save the picture 43 | if c == ord('s'): 44 | if ret: 45 | name = 'video{0}_{1}.png'.format(cam_list[idx], 46 | time.strftime("%Y-%m-%d_%H:%M:%S", time.localtime())) 47 | cv2.imwrite(name, frame) 48 | print("saved file: %s!" %name) 49 | 50 | cv2.destroyAllWindows() 51 | 52 | def init_caps(cam_list, resolution=(1280,720)): 53 | caps = [] 54 | for iCam in cam_list: 55 | cap = cv2.VideoCapture(iCam) 56 | cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"MJPG")) 57 | cap.set(3, resolution[0]) 58 | cap.set(4, resolution[1]) 59 | caps.append(cap) 60 | 61 | return caps 62 | 63 | def deinit_caps(cap_list): 64 | for cap in cap_list: 65 | cap.release() 66 | 67 | def show_cameras(video_list=None): 68 | if video_list == None: 69 | print("Start to search all available camera devices, please wait... ") 70 | cam_list = get_cam_lst() 71 | err_msg = "cannot find any video device!" 72 | else: 73 | cam_list = get_cam_lst(video_list) 74 | err_msg = "cannot find available video device in list: {0}!".format(video_list) +\ 75 | "\nPlease check the video devices in /dev/v4l/by-path/ folder!" 76 | 77 | if len(cam_list) < 1: 78 | print("ERROR: " + err_msg) 79 | return 80 | 81 | print("Available video device list is {}".format(cam_list)) 82 | caps = init_caps(cam_list) 83 | show_cam_img(caps, cam_list) 84 | deinit_caps(caps) 85 | 86 | if __name__ == "__main__": 87 | # User can specify the video list here. 88 | #show_cameras([2, 6, 10, 14]) 89 | 90 | # Or search all available video devices automatically. 91 | show_cameras() 92 | -------------------------------------------------------------------------------- /weights.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hynpu/surround-view-system-introduction/8ff4bd95cafb713cf0bfa82115164d02be8894ad/weights.png -------------------------------------------------------------------------------- /yaml/back.yaml: -------------------------------------------------------------------------------- 1 | %YAML:1.0 2 | --- 3 | camera_matrix: !!opencv-matrix 4 | rows: 3 5 | cols: 3 6 | dt: d 7 | data: [ 3.0434907840374234e+02, 0., 4.8133979392511606e+02, 0., 8 | 3.2477726176795460e+02, 3.1646476882040702e+02, 0., 0., 1. ] 9 | dist_coeffs: !!opencv-matrix 10 | rows: 4 11 | cols: 1 12 | dt: d 13 | data: [ -4.1568299226312187e-02, 3.1480645089822291e-03, 14 | -2.3982702848139551e-03, 2.3821781880039081e-05 ] 15 | resolution: !!opencv-matrix 16 | rows: 2 17 | cols: 1 18 | dt: i 19 | data: [ 960, 640 ] 20 | project_matrix: !!opencv-matrix 21 | rows: 3 22 | cols: 3 23 | dt: d 24 | data: [ -9.3070874510219026e-01, -4.1405550648825917e+00, 25 | 1.1216126052183415e+03, 1.3669740891976151e-01, 26 | -4.6651507085268138e+00, 1.0607568570787648e+03, 27 | 2.8474752086329793e-04, -6.8625514942363052e-03, 1. ] 28 | scale_xy: !!opencv-matrix 29 | rows: 2 30 | cols: 1 31 | dt: f 32 | data: [ 8.00000012e-01, 1. ] 33 | shift_xy: !!opencv-matrix 34 | rows: 2 35 | cols: 1 36 | dt: f 37 | data: [ 0., 100. ] 38 | -------------------------------------------------------------------------------- /yaml/front.yaml: -------------------------------------------------------------------------------- 1 | %YAML:1.0 2 | --- 3 | camera_matrix: !!opencv-matrix 4 | rows: 3 5 | cols: 3 6 | dt: d 7 | data: [ 3.0245305983229298e+02, 0., 4.9664001463163459e+02, 0., 8 | 3.2074618594392325e+02, 3.3119980984361649e+02, 0., 0., 1. ] 9 | dist_coeffs: !!opencv-matrix 10 | rows: 4 11 | cols: 1 12 | dt: d 13 | data: [ -4.3735601598704078e-02, 2.1692522970939803e-02, 14 | -2.6388839028513571e-02, 8.4123126605702321e-03 ] 15 | resolution: !!opencv-matrix 16 | rows: 2 17 | cols: 1 18 | dt: i 19 | data: [ 960, 640 ] 20 | project_matrix: !!opencv-matrix 21 | rows: 3 22 | cols: 3 23 | dt: d 24 | data: [ -7.0390891066994388e-01, -2.5544083216952904e+00, 25 | 7.0809808916259806e+02, -2.9600383808093766e-01, 26 | -2.4971504395791286e+00, 6.3578234365104447e+02, 27 | -5.6872782515522376e-04, -4.4482832729892769e-03, 1. ] 28 | scale_xy: !!opencv-matrix 29 | rows: 2 30 | cols: 1 31 | dt: f 32 | data: [ 6.99999988e-01, 8.00000012e-01 ] 33 | shift_xy: !!opencv-matrix 34 | rows: 2 35 | cols: 1 36 | dt: f 37 | data: [ -150., -100. ] 38 | -------------------------------------------------------------------------------- /yaml/left.yaml: -------------------------------------------------------------------------------- 1 | %YAML:1.0 2 | --- 3 | camera_matrix: !!opencv-matrix 4 | rows: 3 5 | cols: 3 6 | dt: d 7 | data: [ 3.0334009006384287e+02, 0., 4.8649280066241465e+02, 0., 8 | 3.2229678244636966e+02, 3.2388095214561167e+02, 0., 0., 1. ] 9 | dist_coeffs: !!opencv-matrix 10 | rows: 4 11 | cols: 1 12 | dt: d 13 | data: [ -3.5510560636666778e-02, -1.9848228876245811e-02, 14 | 2.6080053057044101e-02, -9.7183762742328750e-03 ] 15 | resolution: !!opencv-matrix 16 | rows: 2 17 | cols: 1 18 | dt: i 19 | data: [ 960, 640 ] 20 | project_matrix: !!opencv-matrix 21 | rows: 3 22 | cols: 3 23 | dt: d 24 | data: [ -1.5134401274988541e+01, -4.2169445692489937e+01, 25 | 1.0724381138361938e+04, -6.7381376780590163e-01, 26 | -2.9044610322524363e+01, 4.1490641432000384e+03, 27 | 4.9428170463677217e-04, -4.7489470989931934e-02, 1. ] 28 | scale_xy: !!opencv-matrix 29 | rows: 2 30 | cols: 1 31 | dt: f 32 | data: [ 4.00000006e-01, 8.00000012e-01 ] 33 | shift_xy: !!opencv-matrix 34 | rows: 2 35 | cols: 1 36 | dt: f 37 | data: [ 150., 0. ] 38 | -------------------------------------------------------------------------------- /yaml/right.yaml: -------------------------------------------------------------------------------- 1 | %YAML:1.0 2 | --- 3 | camera_matrix: !!opencv-matrix 4 | rows: 3 5 | cols: 3 6 | dt: d 7 | data: [ 3.0290778983957682e+02, 0., 4.5799765697290070e+02, 0., 8 | 3.2250139109237318e+02, 3.1001321001054703e+02, 0., 0., 1. ] 9 | dist_coeffs: !!opencv-matrix 10 | rows: 4 11 | cols: 1 12 | dt: d 13 | data: [ -4.1177772399310822e-02, 4.6179881138489094e-03, 14 | -4.4499171471619296e-03, 8.2316738506075550e-04 ] 15 | resolution: !!opencv-matrix 16 | rows: 2 17 | cols: 1 18 | dt: i 19 | data: [ 960, 640 ] 20 | project_matrix: !!opencv-matrix 21 | rows: 3 22 | cols: 3 23 | dt: d 24 | data: [ 3.2544137666101460e+00, 7.4286464322815648e+00, 25 | -1.1683387284648961e+03, -5.6169448819470902e-01, 26 | 6.2906359558080212e+00, -2.0726866728764904e+02, 27 | -1.1737215059256397e-03, 1.0223753858614531e-02, 1. ] 28 | scale_xy: !!opencv-matrix 29 | rows: 2 30 | cols: 1 31 | dt: f 32 | data: [ 4.00000006e-01, 1. ] 33 | shift_xy: !!opencv-matrix 34 | rows: 2 35 | cols: 1 36 | dt: f 37 | data: [ 0., 0. ] 38 | --------------------------------------------------------------------------------