├── .gitignore ├── CMakeLists.txt ├── LICENSE ├── img ├── calib_l_01.png ├── closed-up-edge-image.png ├── closed-up-external-contour.png ├── closed-up-inner-contour.png ├── closed-up-raw.png ├── externalContours.png ├── final-00.png ├── final-01.png ├── final-02.png ├── final-03.png ├── innerContours.png ├── line_0deg-second_derivative_gxx.bmp ├── line_0deg-second_derivative_gxy.bmp ├── line_0deg-second_derivative_gyy.bmp ├── line_0deg-second_derivative_line.bmp ├── line_0deg-second_derivative_primary_axis.bmp ├── line_0deg-second_derivative_primary_axis_zoomed_out.bmp ├── line_0deg.bmp ├── outputEdge.png ├── pixel-precise-00.png ├── pixel-precise-01.png ├── pixel-precise-02.png └── pixel-precise-03.png ├── main.cpp └── readme.md /.gitignore: -------------------------------------------------------------------------------- 1 | # JetBrains IDEs: 2 | .idea/ 3 | 4 | # Clion-CMake 5 | cmake-build-*/ 6 | 7 | -------------------------------------------------------------------------------- /CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.21) 2 | project(SubPixEdgeContour) 3 | 4 | set(CMAKE_CXX_STANDARD 20) 5 | 6 | find_package(fmt REQUIRED) 7 | find_package(Eigen3 3.4 REQUIRED NO_MODULE) 8 | find_package(OpenCV 4.5.1 REQUIRED) 9 | include_directories(${OpenCV_INCLUDE_DIRS}) 10 | find_package(TBB REQUIRED) 11 | find_package(Boost REQUIRED COMPONENTS filesystem) 12 | 13 | add_executable(main main.cpp) 14 | target_link_libraries(main 15 | fmt::fmt 16 | Eigen3::Eigen 17 | ${OpenCV_LIBS} 18 | tbb 19 | Boost::boost 20 | ${Boost_LIBRARIES} 21 | ) 22 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Raymond Ngiam 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /img/calib_l_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/calib_l_01.png -------------------------------------------------------------------------------- /img/closed-up-edge-image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/closed-up-edge-image.png -------------------------------------------------------------------------------- /img/closed-up-external-contour.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/closed-up-external-contour.png -------------------------------------------------------------------------------- /img/closed-up-inner-contour.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/closed-up-inner-contour.png -------------------------------------------------------------------------------- /img/closed-up-raw.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/closed-up-raw.png -------------------------------------------------------------------------------- /img/externalContours.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/externalContours.png -------------------------------------------------------------------------------- /img/final-00.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/final-00.png -------------------------------------------------------------------------------- /img/final-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/final-01.png -------------------------------------------------------------------------------- /img/final-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/final-02.png -------------------------------------------------------------------------------- /img/final-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/final-03.png -------------------------------------------------------------------------------- /img/innerContours.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/innerContours.png -------------------------------------------------------------------------------- /img/line_0deg-second_derivative_gxx.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/line_0deg-second_derivative_gxx.bmp -------------------------------------------------------------------------------- /img/line_0deg-second_derivative_gxy.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/line_0deg-second_derivative_gxy.bmp -------------------------------------------------------------------------------- /img/line_0deg-second_derivative_gyy.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/line_0deg-second_derivative_gyy.bmp -------------------------------------------------------------------------------- /img/line_0deg-second_derivative_line.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/line_0deg-second_derivative_line.bmp -------------------------------------------------------------------------------- /img/line_0deg-second_derivative_primary_axis.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/line_0deg-second_derivative_primary_axis.bmp -------------------------------------------------------------------------------- /img/line_0deg-second_derivative_primary_axis_zoomed_out.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/line_0deg-second_derivative_primary_axis_zoomed_out.bmp -------------------------------------------------------------------------------- /img/line_0deg.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/line_0deg.bmp -------------------------------------------------------------------------------- /img/outputEdge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/outputEdge.png -------------------------------------------------------------------------------- /img/pixel-precise-00.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/pixel-precise-00.png -------------------------------------------------------------------------------- /img/pixel-precise-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/pixel-precise-01.png -------------------------------------------------------------------------------- /img/pixel-precise-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/pixel-precise-02.png -------------------------------------------------------------------------------- /img/pixel-precise-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raymondngiam/subpixel-edge-contour-in-opencv/2cd07440844e1ebd599b8494a5dcfc11b15f64e8/img/pixel-precise-03.png -------------------------------------------------------------------------------- /main.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | 13 | double ContourLengthSingle(const std::vector &contour); 14 | void ContourLength(std::vector> &contours, std::vector &lengths); 15 | 16 | std::shared_ptr SubPixelFacet(const cv::Point& p, 17 | cv::Mat& gyMat, 18 | cv::Mat& gxMat, 19 | cv::Mat& gyyMat, 20 | cv::Mat& gxxMat, 21 | cv::Mat& gxyMat); 22 | 23 | std::shared_ptr>> 24 | SubPixelSingle(cv::Mat &gy, 25 | cv::Mat &gx, 26 | cv::Mat &gyy, 27 | cv::Mat &gxx, 28 | cv::Mat &gxy, 29 | const std::vector &cont); 30 | 31 | void SubPixelEdgeContour(const cv::Mat &image_gray, 32 | const std::vector> &filteredCont, 33 | std::vector>>> &contSubPixFull); 34 | 35 | void GetEdgeContourValidIndices(const std::vector &hierarchy, std::vector &validIndices, 36 | std::vector &excludeIndices); 37 | 38 | const std::vector> COLORS{ 39 | cv::Scalar_(0,0,255), 40 | cv::Scalar_(0,255,0), 41 | cv::Scalar_(255,0,0) 42 | }; 43 | 44 | int main() { 45 | auto image_path = "../img"; 46 | auto output_image_path = "./img_out"; 47 | if(!boost::filesystem::exists(output_image_path)){ 48 | boost::filesystem::create_directory(output_image_path); 49 | } 50 | auto filename = "calib_l_01.png"; 51 | cv::Mat image; 52 | image = cv::imread(fmt::format("{}/{}",image_path,filename), cv::IMREAD_COLOR); 53 | cv::Mat image_gray; 54 | cv::cvtColor(image,image_gray,cv::COLOR_BGR2GRAY); 55 | 56 | fmt::print("{},{}\n",image.rows,image.cols); 57 | cv::Mat edgeIm = cv::Mat::zeros(image.rows,image.cols,CV_8UC1); 58 | cv::Canny(image_gray,edgeIm,180,200); 59 | cv::imwrite(fmt::format("{}/{}",output_image_path,"outputEdge.png"),edgeIm); 60 | 61 | std::vector> contours; 62 | std::vector hierarchy; 63 | cv::findContours(edgeIm,contours,hierarchy,cv::RETR_CCOMP,cv::CHAIN_APPROX_NONE); 64 | std::vector validIndices; 65 | std::vector excludeIndices; 66 | GetEdgeContourValidIndices(hierarchy, validIndices, excludeIndices); 67 | 68 | std::vector> innerContours; 69 | innerContours.resize(validIndices.size()); 70 | std::transform(std::execution::par, 71 | validIndices.begin(), 72 | validIndices.end(), 73 | innerContours.begin(), 74 | [contours](int i){return contours[i];} 75 | ); 76 | 77 | cv::Mat contourIm = cv::Mat::zeros(image.rows,image.cols,CV_8UC3); 78 | fmt::print("{}\n",innerContours.size()); 79 | for (int i=0; i> externalContours; 86 | externalContours.resize(excludeIndices.size()); 87 | std::transform(std::execution::par, 88 | excludeIndices.begin(), 89 | excludeIndices.end(), 90 | externalContours.begin(), 91 | [contours](int i){return contours[i];} 92 | ); 93 | 94 | cv::Mat contourExtIm = cv::Mat::zeros(image.rows,image.cols,CV_8UC3); 95 | fmt::print("{}\n",externalContours.size()); 96 | for (int i=0; i contLengths; 103 | ContourLength(innerContours, contLengths); 104 | cv::Mat_ contLengthMat(contLengths); 105 | 106 | // extract properties 107 | std::vector contRa, contAspectRatio; 108 | std::vector contCenter; 109 | for(const auto& c : innerContours){ 110 | auto M = cv::moments(c); 111 | 112 | double area = M.m00; 113 | auto centerX = int(M.m10/area); 114 | auto centerY = int(M.m01/area); 115 | double m20 = M.mu20/area; 116 | double m02 = M.mu02/area; 117 | double m11 = M.mu11/area; 118 | double c1 = m20-m02; 119 | double c2 = c1*c1; 120 | double c3 = 4*m11*m11; 121 | 122 | contCenter.emplace_back(centerX,centerY); 123 | 124 | auto ra = sqrt(2.0*(m20+m02+sqrt(c2+c3))); 125 | auto rb = sqrt(2.0*(m20+m02-sqrt(c2+c3))); 126 | contRa.emplace_back(ra); 127 | contAspectRatio.emplace_back(ra/rb); 128 | } 129 | 130 | cv::Mat_ contRadiusMat(contRa); 131 | cv::Mat_ contAspectRatioMat(contAspectRatio); 132 | 133 | fmt::print("{},{}\n",contRadiusMat.rows,contRadiusMat.cols); 134 | cv::Mat thresAspectRatio, thresRadius, thresContLength; 135 | const double RADIUS_MAX = 5; 136 | const double CONT_LENGTH_MAX = 2*M_PI*RADIUS_MAX; 137 | cv::threshold(contAspectRatioMat,thresAspectRatio,0.8,1.0,cv::ThresholdTypes::THRESH_BINARY); 138 | cv::threshold(contRadiusMat,thresRadius,RADIUS_MAX,1.0,cv::ThresholdTypes::THRESH_BINARY_INV); 139 | cv::threshold(contLengthMat,thresContLength,CONT_LENGTH_MAX,1.0,cv::ThresholdTypes::THRESH_BINARY_INV); 140 | 141 | cv::Mat and1, and2; 142 | cv::bitwise_and(thresAspectRatio,thresRadius,and1); 143 | cv::bitwise_and(and1,thresContLength,and2); 144 | fmt::print("Filtered object count: {}\n",std::accumulate(and2.begin(),and2.end(),0)); 145 | 146 | cv::Mat filteredIdx; 147 | cv::findNonZero(and2,filteredIdx); 148 | 149 | std::vector> filteredCont; 150 | std::vector filteredContCenter; 151 | for (int i=0; i(i).y; 153 | filteredCont.emplace_back(innerContours[index]); 154 | filteredContCenter.emplace_back(contCenter[index].x,contCenter[index].y); 155 | } 156 | 157 | std::vector>>> contSubPixFull; 158 | SubPixelEdgeContour(image_gray, filteredCont, contSubPixFull); 159 | 160 | std::ofstream f("./data.txt",std::ios_base::out); 161 | f<<"contour_id,point_id,x,y"<>(j, i); 192 | } 193 | } 194 | 195 | std::vector displayContour; 196 | for (const auto &p: *contSubPixFull[resultIndex]) { 197 | int x = floor(((p->x - xCropStart) + 0.5) * upScaleFactor); 198 | int y = floor(((p->y - yCropStart) + 0.5) * upScaleFactor); 199 | cv::drawMarker(upScaled, cv::Point(x, y), cv::Scalar(0.0, 255.0, 0.0), cv::MARKER_TILTED_CROSS, 20, 3); 200 | displayContour.emplace_back(x, y); 201 | } 202 | std::vector> displayContourFull{displayContour}; 203 | cv::drawContours(upScaled, displayContourFull, 0, cv::Scalar(255.0, 0.0, 0.0), 3); 204 | 205 | cv::imwrite(fmt::format("{}/final-{:02d}.png", output_image_path, resultIndex), upScaled); 206 | } 207 | return 0; 208 | } 209 | 210 | // For non maximum surpressed edge images, contour lines are single pixel in width. 211 | // For closed contours, there are two possible outcomes from the boundary tracing algorithm, 212 | // namely inner (hole), or external (non-hole) contour. 213 | // OpenCV `findContours` with `RETR_CCOMP` option returns hierarchy list that starts with an external contour. 214 | // Iterate through all external contours in the hierarchy list by following the `NEXT_SAME` indices; 215 | // if the current external contour does have a child, this indicates that it is a false positive that 216 | // corresponds to another inner hole contour in the set. Thus, we add it into the `excludeIndices` list. 217 | void GetEdgeContourValidIndices(const std::vector &hierarchy, std::vector &validIndices, 218 | std::vector &excludeIndices) { 219 | const int NEXT_SAME = 0; 220 | const int PREV_SAME = 1; 221 | const int FIRST_CHILD = 2; 222 | const int PARENT = 3; 223 | 224 | int index=0; 225 | while (index != -1){ 226 | if (hierarchy[index][FIRST_CHILD]!=-1){ 227 | excludeIndices.emplace_back(index); 228 | } 229 | index = hierarchy[index][NEXT_SAME]; 230 | } 231 | 232 | std::vector l(hierarchy.size()); 233 | std::iota(l.begin(),l.end(),0); 234 | std::set setFullIndices(l.begin(),l.end()); 235 | std::set_difference(setFullIndices.begin(), 236 | setFullIndices.end(), 237 | excludeIndices.begin(), 238 | excludeIndices.end(), 239 | std::back_inserter(validIndices) 240 | ); 241 | } 242 | 243 | void SubPixelEdgeContour(const cv::Mat &image_gray, 244 | const std::vector> &filteredCont, 245 | std::vector>>> &contSubPixFull) { 246 | // 7-tap interpolant and 1st and 2nd derivative coefficients according to 247 | // H. Farid and E. Simoncelli, "Differentiation of Discrete Multi-Dimensional Signals" 248 | // IEEE Trans. Image Processing. 13(4): pp. 496-508 (2004) 249 | std::vector p_vec{0.004711, 0.069321, 0.245410, 0.361117, 0.245410, 0.069321, 0.004711}; 250 | std::vector d1_vec{-0.018708, -0.125376, -0.193091, 0.000000, 0.193091, 0.125376, 0.018708}; 251 | std::vector d2_vec{0.055336, 0.137778, -0.056554, -0.273118, -0.056554, 0.137778, 0.055336}; 252 | 253 | auto p = cv::Mat_(p_vec); 254 | auto d1 = cv::Mat_(d1_vec); 255 | auto d2 = cv::Mat_(d2_vec); 256 | 257 | cv::Mat dx, dy, grad; 258 | cv::sepFilter2D(image_gray,dy,CV_64F,p,d1); 259 | cv::sepFilter2D(image_gray,dx,CV_64F,d1,p); 260 | cv::pow(dy.mul(dy,1.0) + dx.mul(dx,1.0),0.5,grad); 261 | 262 | cv::Mat gy, gx, gyy, gxx, gxy; 263 | cv::sepFilter2D(grad,gy,CV_64F,p,d1); 264 | cv::sepFilter2D(grad,gx,CV_64F,d1,p); 265 | cv::sepFilter2D(grad,gyy,CV_64F,p,d2); 266 | cv::sepFilter2D(grad,gxx,CV_64F,d2,p); 267 | cv::sepFilter2D(grad,gxy,CV_64F,d1,d1); 268 | 269 | contSubPixFull.resize(filteredCont.size()); 270 | std::transform(std::execution::par, 271 | filteredCont.cbegin(), 272 | filteredCont.cend(), 273 | contSubPixFull.begin(), 274 | [&gy,&gx,&gyy,&gxx,&gxy](const std::vector& cont){ 275 | return SubPixelSingle(gy,gx,gyy,gxx,gxy,cont);} 276 | ); 277 | } 278 | 279 | std::shared_ptr>> 280 | SubPixelSingle(cv::Mat &gy, 281 | cv::Mat &gx, 282 | cv::Mat &gyy, 283 | cv::Mat &gxx, 284 | cv::Mat &gxy, 285 | const std::vector &cont) { 286 | auto contSubPix = std::make_shared>>(); 287 | contSubPix->resize(cont.size()); 288 | std::transform(std::execution::seq, 289 | cont.cbegin(), 290 | cont.cend(), 291 | contSubPix->begin(), 292 | [&gy,&gx,&gyy,&gxx,&gxy](const cv::Point& p){ return SubPixelFacet(p,gy,gx,gyy,gxx,gxy); } 293 | ); 294 | return contSubPix; 295 | } 296 | 297 | // Subpixel edge extraction method according to 298 | // C. Steger, "An unbiased detector of curvilinear structures", 299 | // IEEE Transactions on Pattern Analysis and Machine Intelligence, 300 | // 20(2): pp. 113-125, (1998) 301 | std::shared_ptr SubPixelFacet(const cv::Point& p, 302 | cv::Mat& gyMat, 303 | cv::Mat& gxMat, 304 | cv::Mat& gyyMat, 305 | cv::Mat& gxxMat, 306 | cv::Mat& gxyMat){ 307 | auto row = p.y; 308 | auto col = p.x; 309 | auto gy = gyMat.at(row,col); 310 | auto gx = gxMat.at(row,col); 311 | auto gyy = gyyMat.at(row,col); 312 | auto gxx = gxxMat.at(row,col); 313 | auto gxy = gxyMat.at(row,col); 314 | 315 | Eigen::Matrixhessian; 316 | hessian << gyy,gxy,gxy,gxx; 317 | Eigen::JacobiSVD svd(hessian, Eigen::ComputeFullV); 318 | auto v = svd.matrixV(); 319 | // first column vector of v, corresponding to largest eigen value 320 | // is the direction perpendicular to the line 321 | auto ny = v(0,0); 322 | auto nx = v(1,0); 323 | auto t=-(gx*nx + gy*ny)/(gxx*nx*nx + 2*gxy*nx*ny + gyy*ny*ny); 324 | auto px=t*nx; 325 | auto py=t*ny; 326 | 327 | return std::make_shared(col+px,row+py); 328 | } 329 | 330 | void ContourLength(std::vector> &contours, std::vector &lengths) { 331 | lengths.resize(std::distance(contours.begin(), contours.end())); 332 | std::transform(std::execution::par, 333 | contours.begin(), 334 | contours.end(), 335 | lengths.begin(), 336 | ContourLengthSingle 337 | ); 338 | } 339 | 340 | // contour length 341 | double ContourLengthSingle(const std::vector &contour) { 342 | std::vector lengths; 343 | for (int i =1; i 12 | 13 | 14 | 15 | 16 |
17 | 18 |
19 | Pixel precise edge contours of circular marks. 20 |
21 | 22 |
23 | 24 | The subpixel edge extraction implementation in this repo is based on the following two papers: 25 | 26 | - C. Steger, "An unbiased detector of curvilinear structures", IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(2): pp. 113-125, (1998) 27 | - H. Farid and E. Simoncelli, "Differentiation of Discrete Multi-Dimensional Signals" IEEE Trans. Image Processing. 13(4): pp. 496-508 (2004) 28 | 29 | Output results: 30 | 31 | 32 | 33 | 34 | 35 | 36 |
37 | 38 |
39 | Subpixel precise edge contours of circular marks. 40 |
41 | 42 | ### Dependencies 43 | 44 | ||Version| 45 | |-|:-:| 46 | |cmake | 3.21| 47 | |OpenCV | 4.5.1| 48 | |Eigen | 3.4| 49 | |libtbb-dev | 2020.1-2| 50 | |Boost | 1.78.0| 51 | |fmt | 8.1.1| 52 | 53 | ### How to run 54 | 55 | ```bash 56 | $ git clone https://github.com/raymondngiam/subpixel-edge-contour-in-opencv.git 57 | $ cd subpixel-edge-contour-in-opencv 58 | $ mkdir build && cd build 59 | $ cmake .. 60 | $ make 61 | $ ./main 62 | ``` 63 | 64 | ### Code Walkthrough 65 | 66 | **Pixel-precise edge contour extraction** 67 | 68 | A typical workflow for edge detection in OpenCV starts with the `cv::Canny` operator. 69 | 70 | The outcome of the `cv::Canny` operator is a binary edge image with non-maximum suppression algorithm applied. Thus, the edges are typically thinned to a single pixel width, where the edge magnitude response is maximum. 71 | 72 | Line 51-58 in main.cpp: 73 | 74 | ```cpp {.line-numbers} 75 | cv::Mat image; 76 | image = cv::imread(fmt::format("{}/{}",image_path,filename), cv::IMREAD_COLOR); 77 | cv::Mat image_gray; 78 | cv::cvtColor(image,image_gray,cv::COLOR_BGR2GRAY); 79 | 80 | fmt::print("{},{}\n",image.rows,image.cols); 81 | cv::Mat edgeIm = cv::Mat::zeros(image.rows,image.cols,CV_8UC1); 82 | cv::Canny(image_gray,edgeIm,180,200); 83 | ``` 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 |
92 | Left: Gray image. 93 | Right: Binary edge image. 94 |
95 | 96 |
97 | 98 | Subsequently, we will apply `cv::findContours` operator to the binary edge image, in order to obtain a list of *boundary-traced* contours. 99 | 100 | There is a slight complication when using the `cv::findContours` operator out-of-the-box with a *non-maximum suppressed* binary edge image. Any closed contour will return not one, but two outputs (one traced from the exterior, another traced from the interior, or hole). 101 | 102 | Thus, we implement a custom method, `GetEdgeContourValidIndices` which takes in the `hierarchy` output from the `cv::findContours` operator (running in `RETR_CCOMP` mode), to filter the closed contours *traced from the exterior*. 103 | 104 | Line 61-66 in main.cpp: 105 | 106 | ```cpp {.line-numbers} 107 | std::vector> contours; 108 | std::vector hierarchy; 109 | cv::findContours(edgeIm,contours,hierarchy,cv::RETR_CCOMP,cv::CHAIN_APPROX_NONE); 110 | std::vector validIndices; 111 | std::vector excludeIndices; 112 | GetEdgeContourValidIndices(hierarchy, validIndices, excludeIndices); 113 | ``` 114 | 115 | The outcome of the `GetEdgeContourValidIndices` method is as visualized below: 116 | 117 | 118 | 119 | 120 | 121 |
122 | 123 | 124 | 125 | 126 | 127 |
128 | 129 | 130 |
131 | Left: Binary edge image.
132 | Center: Interiorly traced closed contours.
133 | Right: Exteriorly traced closed contours. 134 |
135 | 136 |
137 | 138 | From visual inspection, we noticed that the interiorly traced closed contours are the more accuracte, and concise representation of the original binary edges. 139 | 140 | The implementation of the `GetEdgeContourValidIndices` method (line 217-241 in main.cpp) is actually pretty straightforward, as below: 141 | 142 | ```cpp {.line-numbers} 143 | // For non maximum surpressed edge images, contour lines are single pixel in width. 144 | // For closed contours, there are two possible outcomes from the boundary tracing algorithm, 145 | // namely inner (hole), or external (non-hole) contour. 146 | // OpenCV `findContours` with `RETR_CCOMP` option returns hierarchy list that starts with an external contour. 147 | // Iterate through all external contours in the hierarchy list by following the `NEXT_SAME` indices; 148 | // if the current external contour does have a child, this indicates that it is a false positive that 149 | // corresponds to another inner hole contour in the set. Thus, we add it into the `excludeIndices` list. 150 | void GetEdgeContourValidIndices(const std::vector &hierarchy, std::vector &validIndices, 151 | std::vector &excludeIndices) { 152 | const int NEXT_SAME = 0; 153 | const int PREV_SAME = 1; 154 | const int FIRST_CHILD = 2; 155 | const int PARENT = 3; 156 | 157 | int index=0; 158 | while (index != -1){ 159 | if (hierarchy[index][NEXT_CHILD]!=-1){ 160 | excludeIndices.emplace_back(index); 161 | } 162 | index = hierarchy[index][NEXT_SAME]; 163 | } 164 | 165 | std::vector l(hierarchy.size()); 166 | std::iota(l.begin(),l.end(),0); 167 | std::set setFullIndices(l.begin(),l.end()); 168 | std::set_difference(setFullIndices.begin(), 169 | setFullIndices.end(), 170 | excludeIndices.begin(), 171 | excludeIndices.end(), 172 | std::back_inserter(validIndices) 173 | ); 174 | } 175 | ``` 176 | 177 | After we have acquired the pixel-precise contour coordinates, we can move into the subpixel-precise edge point extraction. 178 | 179 | **Subpixel-precise edge contour extraction** 180 | 181 | The subpixel edge extraction function, `SubPixelEdgeContour` takes in the gray image, and a vector of pixel precise contours. 182 | 183 | It then returns a vector of subpixel precise contours of double precision. 184 | 185 | ```cpp 186 | void SubPixelEdgeContour(const cv::Mat &image_gray, 187 | const std::vector> &filteredCont, 188 | std::vector>>> &contSubPixFull); 189 | ``` 190 | 191 | Line 246-267 in `SubPixelEdgeContour`, main.cpp. 192 | 193 | ```cpp {.line-numbers} 194 | // 7-tap interpolant and 1st and 2nd derivative coefficients according to 195 | // H. Farid and E. Simoncelli, "Differentiation of Discrete Multi-Dimensional Signals" 196 | // IEEE Trans. Image Processing. 13(4): pp. 496-508 (2004) 197 | std::vector p_vec{0.004711, 0.069321, 0.245410, 0.361117, 0.245410, 0.069321, 0.004711}; 198 | std::vector d1_vec{-0.018708, -0.125376, -0.193091, 0.000000, 0.193091, 0.125376, 0.018708}; 199 | std::vector d2_vec{0.055336, 0.137778, -0.056554, -0.273118, -0.056554, 0.137778, 0.055336}; 200 | 201 | auto p = cv::Mat_(p_vec); 202 | auto d1 = cv::Mat_(d1_vec); 203 | auto d2 = cv::Mat_(d2_vec); 204 | cv::Mat dx, dy, grad; 205 | cv::sepFilter2D(image_gray,dy,CV_64F,p,d1); 206 | cv::sepFilter2D(image_gray,dx,CV_64F,d1,p); 207 | cv::pow(dy.mul(dy,1.0) + dx.mul(dx,1.0),0.5,grad); 208 | 209 | cv::Mat gy, gx, gyy, gxx, gxy; 210 | cv::sepFilter2D(grad,gy,CV_64F,p,d1); 211 | cv::sepFilter2D(grad,gx,CV_64F,d1,p); 212 | cv::sepFilter2D(grad,gyy,CV_64F,p,d2); 213 | cv::sepFilter2D(grad,gxx,CV_64F,d2,p); 214 | cv::sepFilter2D(grad,gxy,CV_64F,d1,d1); 215 | ``` 216 | 217 | From the input gray image, its gradient amplitude image, `grad` is computed; and subsequently, the first and second order derivatives of the `grad` image are also computed. 218 | 219 | 220 | Lastly, let's drill down to the core implementation details in the function `SubPixelFacet` (line 301-328 in main.cpp) as shown below: 221 | 222 | ```cpp {.line-numbers} 223 | // Subpixel edge extraction method according to 224 | // C. Steger, "An unbiased detector of curvilinear structures", 225 | // IEEE Transactions on Pattern Analysis and Machine Intelligence, 226 | // 20(2): pp. 113-125, (1998) 227 | std::shared_ptr SubPixelFacet(const cv::Point& p, 228 | cv::Mat& gyMat, 229 | cv::Mat& gxMat, 230 | cv::Mat& gyyMat, 231 | cv::Mat& gxxMat, 232 | cv::Mat& gxyMat){ 233 | auto row = p.y; 234 | auto col = p.x; 235 | auto gy = gyMat.at(row,col); 236 | auto gx = gxMat.at(row,col); 237 | auto gyy = gyyMat.at(row,col); 238 | auto gxx = gxxMat.at(row,col); 239 | auto gxy = gxyMat.at(row,col); 240 | 241 | Eigen::Matrixhessian; 242 | hessian << gyy,gxy,gxy,gxx; 243 | Eigen::JacobiSVD svd(hessian, Eigen::ComputeFullV); 244 | auto v = svd.matrixV(); 245 | // first column vector of v, corresponding to largest eigen value 246 | // is the direction perpendicular to the line 247 | auto ny = v(0,0); 248 | auto nx = v(1,0); 249 | auto t=-(gx*nx + gy*ny)/(gxx*nx*nx + 2*gxy*nx*ny + gyy*ny*ny); 250 | auto px=t*nx; 251 | auto py=t*ny; 252 | 253 | return std::make_shared(col+px,row+py); 254 | } 255 | ``` 256 | 257 | Let $r$ be the function value at point $(y_0,x_0)$ of the image where the Taylor series approximation took place; $r_x',r_y',r_{xx}'',r_{yy}'',r_{xy}''$ be the locally estimated first and second order derivatives at the point, that are obtained by convolution with derivative filters. Then the Taylor polynomial is given by 258 | 259 | $p(y,x) = r + r_x'x + r_y'y + \frac{1}{2}r_x''x^2 + \frac{1}{2}r_y''y^2 + \frac{1}{2}r_{xy}''xy$ 260 | 261 | Curvilinear structures in 2D can be modeled as 1D line profile in the direction perpendicular to the line. Let this 262 | direction be a unit vector $[n_y,n_x]^\text{T}$. 263 | 264 | With the current pixel center as origin, any point along the direction perpendicular to the line, can be then defined as $[tn_y,tn_x]^\text{T}$, where $t\in \mathbb{R}$. 265 | 266 | $p(tn_y,tn_x) = r + r_x'tn_x + r_y'tn_y + \frac{1}{2}r_x''t^2n_x^2 + \frac{1}{2}r_y''t^2n_y^2 + \frac{1}{2}r_{xy}''t^2n_xn_y$ 267 | 268 | Reframing the function $p$ to be parameterized by $t$ alone. 269 | 270 | $p(t) = r + r_x'tn_x + r_y'tn_y + \frac{1}{2}r_x''t^2n_x^2 + \frac{1}{2}r_y''t^2n_y^2 + \frac{1}{2}r_{xy}''t^2n_xn_y$ 271 | 272 | We are interested to determine the value of $t$ whereby the function $p(t)$ is stationary. 273 | 274 | Taking the derivative of $p(t)$ with respect to $t$, 275 | 276 | $p'(t) = r_x'n_x + r_y'n_y + r_x''n_x^2t + r_y''n_y^2t + r_{xy}''n_xn_yt$ 277 | 278 | Setting $p'(t)=0$, 279 | 280 | $0 = r_x'n_x + r_y'n_y + r_x''n_x^2t + r_y''n_y^2t + r_{xy}''n_xn_yt$ 281 | 282 | $r_x''n_x^2t + r_y''n_y^2t + r_{xy}''n_xn_yt = - r_x'n_x - r_y'n_y$ 283 | 284 | $t = \frac{- r_x'n_x - r_y'n_y}{r_x''n_x^2 + r_y''n_y^2 + r_{xy}''n_xn_y}$ 285 | 286 | The subpixel edge point where the gradient amplitude is maximum is thus given by, 287 | 288 | $y = y_0 + tn_y$ 289 | 290 | $x = x_0 + tn_x$ 291 | 292 | **Visual explanation on determining the direction perpendicular to the line** 293 | 294 | 295 | 296 | 297 |
298 | 299 |
300 | Left: Gray image of a horizontal line.
301 | Right: Highlighted with red arrow is the direction perpendicular to the line. 302 |
303 | 304 |
305 | 306 |
307 | 308 |
309 | 310 |
311 | Zoomed in view of a particular pixel point (y0,x0) on the line in the gray image. The pixel of interest is marked with a red cross. 312 |
313 | 314 |
315 | 316 | 317 | 318 | 319 |
320 | 321 |
322 | Left: Second order derivative image along x direction, gxx.
323 | Center: Second order derivative image along y direction, gyy.
324 | Right: Second order derivative image along xy direction, gxy. 325 |
326 | 327 |
328 | 329 | $g_{xx}(y_0,x_0) = -3.33786e^{-06}$ 330 | 331 | $g_{yy}(y_0,x_0) = -68.2977$ 332 | 333 | $g_{xy}(y_0,x_0) = 0.0$ 334 | 335 | The Hessian matrix, $\mathbf{H}$ is defined as: 336 | 337 | $\mathbf{H} = \begin{bmatrix}g_{yy}(y_0,x_0) & g_{xy}(y_0,x_0) \\ 338 | g_{xy}(y_0,x_0) & g_{xx}(y_0,x_0) 339 | \end{bmatrix}=\begin{bmatrix}-68.2977 & 0.0 \\ 340 | 0.0 & -3.33786e^{-06} 341 | \end{bmatrix}$ 342 | 343 | Computing SVD of the Hessian: 344 | 345 | $\mathbf{H = USV^{\text{T}}}$ 346 | 347 | $\mathbf{U} = \begin{bmatrix}1.0 & 0.0 \\ 348 | 0.0 & 1.0 349 | \end{bmatrix}$ 350 | 351 | $\mathbf{S} = \begin{bmatrix}68.2977 & 0.0 \\ 352 | 0.0 & 3.33786e^{-06} 353 | \end{bmatrix}$ 354 | 355 | $\mathbf{V} = \begin{bmatrix}-1.0 & 0.0 \\ 356 | 0.0 & -1.0 357 | \end{bmatrix}$ 358 | 359 | First column vector of $\mathbf{V}$ corresponds to the maximum eigen value, which is equal to **the direction perpendicular to the line**. 360 | 361 | $\mathbf{\hat{n}_{primary}} = \begin{bmatrix}-1.0 \\ 362 | 0.0 363 | \end{bmatrix}$ 364 | 365 |
366 | 367 |
368 | 369 |
370 | Zoomed in view of a particular pixel point (y0,x0) on the line in the gray image. Highlighted with red arrow is the direction of (ny,nx)T. 371 |
372 | 373 |
374 | 375 | One way to think about this is that **for a horizontal line, the y axis second order derivative filter will have a maximum response, vice versa. Thus, principal axis with largest eigen value will be the one perpendicular to the line.** 376 | 377 | ### References 378 | 379 | - C. Steger, "An unbiased detector of curvilinear structures", IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(2): pp. 113-125, (1998) 380 | - H. Farid and E. Simoncelli, "Differentiation of Discrete Multi-Dimensional Signals" IEEE Trans. Image Processing. 13(4): pp. 496-508 (2004) --------------------------------------------------------------------------------