├── .gitignore ├── README.md ├── demo.py ├── gen_samples.py ├── pyheal.py └── samples ├── lena_in.png ├── lena_mask.png ├── lena_opencv.png ├── lena_out.png ├── peppers_in.png ├── peppers_mask.png ├── peppers_opencv.png ├── peppers_out.png ├── tulips_in.png ├── tulips_mask.png ├── tulips_opencv.png └── tulips_out.png /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/ 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Fast marching inpainting 2 | 3 | Following the idea that known image information has to be propagated from the contour of the area to inpaint towards its innermost parts, [Alexander Telea's inpainting algorithm][1] uses Sethian's [fast marching method][2] (FFM) to construct and maintain a list of the pixels forming the contour. The area delimited by this band is progressively shrunk while pixels are processed until none remain to be inpainted. 4 | 5 | FFM only helps with the order in which pixels are processed, but does not determine how each pixel is going to be actually inpainted. Telea performs a weighted average of all pixels in the neighborhood of the inpainted pixel. The neighborhood is determined by a radius, which value should be close to the thickness of the area to inpainted. The weight function depends on the following factors: 6 | - the distance between a pixel and it neighbors, ie closers neighbors contribute more; 7 | - the level set distance to the original contour, ie neighbors on the same level set (or iso line) contribute more; 8 | - the collinearity of the vector from a pixel to its neighbors and the FFM direction of propagation. This factor will have the effect of extending isophotes (ie lines) reaching the area to inpaint, by giving more weight to neighbors when they are in the axis going from the inpainting pixel in the direction of propagation of the FFM. 9 | 10 | [1]: https://www.rug.nl/research/portal/files/14404904/2004JGraphToolsTelea.pdf 11 | [2]: https://math.berkeley.edu/~sethian/2006/Explanations/fast_marching_explain.html 12 | 13 | # Python implementation 14 | 15 | This implementation borrows from several sources, including the [OpenCV C++ implementation][3] and [Telea's implementation][4] itself. As advised in the original paper, we first run a FFM in order to compute distances between pixels outside of the mask and the initial mask contour, before running the main FFM that performs the actual inpainting. 16 | 17 | Despite closely following the same algorithm, this Python implementation is considerably slower than the mentioned implementations. Indeed FFM inpainting is not a vectorized algorithm but rather an iterative one, and therefore doesn't fully benefit from using NumPy. In order to keep the processing time under a reasonable amount, we have chosen to only compute the weighted average previously described, dropping the average gradient that is also mentioned in the article and applied in most implementations. This allows for a x6 speed gain while maintaining "good-enough" results, albeit not as smooth. 18 | 19 | [3]: https://github.com/opencv/opencv/blob/master/modules/photo/src/inpaint.cpp 20 | [4]: https://github.com/erich666/jgt-code/tree/master/Volume_09/Number_1/Telea2004/AFMM_Inpainting 21 | 22 | # Results 23 | 24 | *Click for full-scale image* 25 | 26 | | Initial image | Pyheal | OpenCV | 27 | | :-------------------------: | :---------------------------: | :-------------------------: | 28 | | [![][im1_in_thumb]][im1_in] | [![][im1_out_thumb]][im1_out] | [![][im1_cv_thumb]][im1_cv] | 29 | 30 | [im1_in]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/tulips_in.png 31 | [im1_in_thumb]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/tulips_in.png 32 | [im1_out]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/tulips_out.png 33 | [im1_out_thumb]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/tulips_out.png 34 | [im1_cv]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/tulips_opencv.png 35 | [im1_cv_thumb]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/tulips_opencv.png 36 | 37 | | Initial image | Pyheal | OpenCV | 38 | | :-------------------------: | :---------------------------: | :-------------------------: | 39 | | [![][im2_in_thumb]][im2_in] | [![][im2_out_thumb]][im2_out] | [![][im2_cv_thumb]][im2_cv] | 40 | 41 | [im2_in]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/lena_in.png 42 | [im2_in_thumb]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/lena_in.png 43 | [im2_out]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/lena_out.png 44 | [im2_out_thumb]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/lena_out.png 45 | [im2_cv]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/lena_opencv.png 46 | [im2_cv_thumb]: https://raw.githubusercontent.com/olvb/pyheal/master/samples/lena_opencv.png 47 | 48 | *Samples images from https://homepages.cae.wisc.edu/~ece533/images/* 49 | 50 | The Telea algorithm gives satisfying results for narrow masks. One of its niceties is that it can be directly applied to masks containing non-contiguous regions, without any additional code. When used with larger masks or on textured or patterned images, its half-blurring half-stretching effect will however become apparent. 51 | -------------------------------------------------------------------------------- /demo.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import pyheal 4 | import imageio 5 | 6 | parser = argparse.ArgumentParser() 7 | parser.add_argument('in_path', metavar='input_img', type=str, 8 | help='path to input image') 9 | parser.add_argument('mask_path', metavar='mask_img', type=str, 10 | help='path to mask image') 11 | parser.add_argument('out_path', metavar='ouput_img', type=str, 12 | help='path to output image') 13 | parser.add_argument('-r', '--radius', metavar='R', nargs=1, type=int, default=[5], 14 | help='neighborhood radius') 15 | 16 | args = parser.parse_args() 17 | 18 | img = imageio.imread(args.in_path) 19 | mask_img = imageio.imread(args.mask_path) 20 | mask = mask_img[:, :, 0].astype(bool, copy=False) 21 | pyheal.inpaint(img, mask, args.radius[0]) 22 | imageio.imwrite(args.out_path, img) 23 | -------------------------------------------------------------------------------- /gen_samples.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os 4 | import pyheal 5 | import cv2 6 | 7 | for file in os.scandir('samples/'): 8 | if not file.name.endswith('_in.png'): 9 | continue 10 | 11 | img_name = file.name[:-len('_in.png')] 12 | 13 | in_path = 'samples/' + img_name + '_in.png' 14 | mask_path = 'samples/' + img_name + '_mask.png' 15 | out_path = 'samples/' + img_name + '_out.png' 16 | cv_path = 'samples/' + img_name + '_opencv.png' 17 | 18 | in_img = cv2.imread(in_path) 19 | mask_img = cv2.imread(mask_path) 20 | mask = mask_img[:, :, 0].astype(bool, copy=False) 21 | out_img = in_img.copy() 22 | 23 | pyheal.inpaint(out_img, mask, 5) 24 | cv2.imwrite(out_path, out_img) 25 | 26 | cv_img = cv2.inpaint(in_img, mask_img[:, :, 0], 5, cv2.INPAINT_TELEA) 27 | cv2.imwrite(cv_path, cv_img) 28 | -------------------------------------------------------------------------------- /pyheal.py: -------------------------------------------------------------------------------- 1 | from math import sqrt as sqrt 2 | import heapq 3 | import numpy as np 4 | 5 | # flags 6 | KNOWN = 0 7 | BAND = 1 8 | UNKNOWN = 2 9 | # extremity values 10 | INF = 1e6 # dont use np.inf to avoid inf * 0 11 | EPS = 1e-6 12 | 13 | # solves a step of the eikonal equation in order to find closest quadrant 14 | def _solve_eikonal(y1, x1, y2, x2, height, width, dists, flags): 15 | # check image frame 16 | if y1 < 0 or y1 >= height or x1 < 0 or x1 >= width: 17 | return INF 18 | 19 | if y2 < 0 or y2 >= height or x2 < 0 or x2 >= width: 20 | return INF 21 | 22 | flag1 = flags[y1, x1] 23 | flag2 = flags[y2, x2] 24 | 25 | # both pixels are known 26 | if flag1 == KNOWN and flag2 == KNOWN: 27 | dist1 = dists[y1, x1] 28 | dist2 = dists[y2, x2] 29 | d = 2.0 - (dist1 - dist2) ** 2 30 | if d > 0.0: 31 | r = sqrt(d) 32 | s = (dist1 + dist2 - r) / 2.0 33 | if s >= dist1 and s >= dist2: 34 | return s 35 | s += r 36 | if s >= dist1 and s >= dist2: 37 | return s 38 | # unsolvable 39 | return INF 40 | 41 | # only 1st pixel is known 42 | if flag1 == KNOWN: 43 | dist1 = dists[y1, x1] 44 | return 1.0 + dist1 45 | 46 | # only 2d pixel is known 47 | if flag2 == KNOWN: 48 | dist2 = dists[y2, x2] 49 | return 1.0 + dist2 50 | 51 | # no pixel is known 52 | return INF 53 | 54 | # returns gradient for one pixel, computed on 2 pixel range if possible 55 | def _pixel_gradient(y, x, height, width, vals, flags): 56 | val = vals[y, x] 57 | 58 | # compute grad_y 59 | prev_y = y - 1 60 | next_y = y + 1 61 | if prev_y < 0 or next_y >= height: 62 | grad_y = INF 63 | else: 64 | flag_prev_y = flags[prev_y, x] 65 | flag_next_y = flags[next_y, x] 66 | 67 | if flag_prev_y != UNKNOWN and flag_next_y != UNKNOWN: 68 | grad_y = (vals[next_y, x] - vals[prev_y, x]) / 2.0 69 | elif flag_prev_y != UNKNOWN: 70 | grad_y = val - vals[prev_y, x] 71 | elif flag_next_y != UNKNOWN: 72 | grad_y = vals[next_y, x] - val 73 | else: 74 | grad_y = 0.0 75 | 76 | # compute grad_x 77 | prev_x = x - 1 78 | next_x = x + 1 79 | if prev_x < 0 or next_x >= width: 80 | grad_x = INF 81 | else: 82 | flag_prev_x = flags[y, prev_x] 83 | flag_next_x = flags[y, next_x] 84 | 85 | if flag_prev_x != UNKNOWN and flag_next_x != UNKNOWN: 86 | grad_x = (vals[y, next_x] - vals[y, prev_x]) / 2.0 87 | elif flag_prev_x != UNKNOWN: 88 | grad_x = val - vals[y, prev_x] 89 | elif flag_next_x != UNKNOWN: 90 | grad_x = vals[y, next_x] - val 91 | else: 92 | grad_x = 0.0 93 | 94 | return grad_y, grad_x 95 | 96 | # compute distances between initial mask contour and pixels outside mask, using FMM (Fast Marching Method) 97 | def _compute_outside_dists(height, width, dists, flags, band, radius): 98 | band = band.copy() 99 | orig_flags = flags 100 | flags = orig_flags.copy() 101 | # swap INSIDE / OUTSIDE 102 | flags[orig_flags == KNOWN] = UNKNOWN 103 | flags[orig_flags == UNKNOWN] = KNOWN 104 | 105 | last_dist = 0.0 106 | double_radius = radius * 2 107 | while band: 108 | # reached radius limit, stop FFM 109 | if last_dist >= double_radius: 110 | break 111 | 112 | # pop BAND pixel closest to initial mask contour and flag it as KNOWN 113 | _, y, x = heapq.heappop(band) 114 | flags[y, x] = KNOWN 115 | 116 | # process immediate neighbors (top/bottom/left/right) 117 | neighbors = [(y - 1, x), (y, x - 1), (y + 1, x), (y, x + 1)] 118 | for nb_y, nb_x in neighbors: 119 | # skip out of frame 120 | if nb_y < 0 or nb_y >= height or nb_x < 0 or nb_x >= width: 121 | continue 122 | 123 | # neighbor already processed, nothing to do 124 | if flags[nb_y, nb_x] != UNKNOWN: 125 | continue 126 | 127 | # compute neighbor distance to inital mask contour 128 | last_dist = min([ 129 | _solve_eikonal(nb_y - 1, nb_x, nb_y, nb_x - 1, height, width, dists, flags), 130 | _solve_eikonal(nb_y + 1, nb_x, nb_y, nb_x + 1, height, width, dists, flags), 131 | _solve_eikonal(nb_y - 1, nb_x, nb_y, nb_x + 1, height, width, dists, flags), 132 | _solve_eikonal(nb_y + 1, nb_x, nb_y, nb_x - 1, height, width, dists, flags) 133 | ]) 134 | dists[nb_y, nb_x] = last_dist 135 | 136 | # add neighbor to narrow band 137 | flags[nb_y, nb_x] = BAND 138 | heapq.heappush(band, (last_dist, nb_y, nb_x)) 139 | 140 | # distances are opposite to actual FFM propagation direction, fix it 141 | dists *= -1.0 142 | 143 | # computes pixels distances to initial mask contour, flags, and narrow band queue 144 | def _init(height, width, mask, radius): 145 | # init all distances to infinity 146 | dists = np.full((height, width), INF, dtype=float) 147 | # status of each pixel, ie KNOWN, BAND or UNKNOWN 148 | flags = mask.astype(int) * UNKNOWN 149 | # narrow band, queue of contour pixels 150 | band = [] 151 | 152 | mask_y, mask_x = mask.nonzero() 153 | for y, x in zip(mask_y, mask_x): 154 | # look for BAND pixels in neighbors (top/bottom/left/right) 155 | neighbors = [(y - 1, x), (y, x - 1), (y + 1, x), (y, x + 1)] 156 | for nb_y, nb_x in neighbors: 157 | # neighbor out of frame 158 | if nb_y < 0 or nb_y >= height or nb_x < 0 or nb_x >= width: 159 | continue 160 | 161 | # neighbor already flagged as BAND 162 | if flags[nb_y, nb_x] == BAND: 163 | continue 164 | 165 | # neighbor out of mask => mask contour 166 | if mask[nb_y, nb_x] == 0: 167 | flags[nb_y, nb_x] = BAND 168 | dists[nb_y, nb_x] = 0.0 169 | heapq.heappush(band, (0.0, nb_y, nb_x)) 170 | 171 | 172 | # compute distance to inital mask contour for KNOWN pixels 173 | # (by inverting mask/flags and running FFM) 174 | _compute_outside_dists(height, width, dists, flags, band, radius) 175 | 176 | return dists, flags, band 177 | 178 | # returns RGB values for pixel to by inpainted, computed for its neighborhood 179 | def _inpaint_pixel(y, x, img, height, width, dists, flags, radius): 180 | dist = dists[y, x] 181 | # normal to pixel, ie direction of propagation of the FFM 182 | dist_grad_y, dist_grad_x = _pixel_gradient(y, x, height, width, dists, flags) 183 | pixel_sum = np.zeros((3), dtype=float) 184 | weight_sum = 0.0 185 | 186 | # iterate on each pixel in neighborhood (nb stands for neighbor) 187 | for nb_y in range(y - radius, y + radius + 1): 188 | # pixel out of frame 189 | if nb_y < 0 or nb_y >= height: 190 | continue 191 | 192 | for nb_x in range(x - radius, x + radius + 1): 193 | # pixel out of frame 194 | if nb_x < 0 or nb_x >= width: 195 | continue 196 | 197 | # skip unknown pixels (including pixel being inpainted) 198 | if flags[nb_y, nb_x] == UNKNOWN: 199 | continue 200 | 201 | # vector from point to neighbor 202 | dir_y = y - nb_y 203 | dir_x = x - nb_x 204 | dir_length_square = dir_y ** 2 + dir_x ** 2 205 | dir_length = sqrt(dir_length_square) 206 | # pixel out of neighborhood 207 | if dir_length > radius: 208 | continue 209 | 210 | # compute weight 211 | # neighbor has same direction gradient => contributes more 212 | dir_factor = abs(dir_y * dist_grad_y + dir_x * dist_grad_x) 213 | if dir_factor == 0.0: 214 | dir_factor = EPS 215 | 216 | # neighbor has same contour distance => contributes more 217 | nb_dist = dists[nb_y, nb_x] 218 | level_factor = 1.0 / (1.0 + abs(nb_dist - dist)) 219 | 220 | # neighbor is distant => contributes less 221 | dist_factor = 1.0 / (dir_length * dir_length_square) 222 | 223 | weight = abs(dir_factor * dist_factor * level_factor) 224 | 225 | pixel_sum[0] += weight * img[nb_y, nb_x, 0] 226 | pixel_sum[1] += weight * img[nb_y, nb_x, 1] 227 | pixel_sum[2] += weight * img[nb_y, nb_x, 2] 228 | 229 | weight_sum += weight 230 | 231 | return pixel_sum / weight_sum 232 | 233 | # main inpainting function 234 | def inpaint(img, mask, radius=5): 235 | if img.shape[0:2] != mask.shape[0:2]: 236 | raise ValueError("Image and mask dimensions do not match") 237 | 238 | height, width = img.shape[0:2] 239 | dists, flags, band = _init(height, width, mask, radius) 240 | 241 | # find next pixel to inpaint with FFM (Fast Marching Method) 242 | # FFM advances the band of the mask towards its center, 243 | # by sorting the area pixels by their distance to the initial contour 244 | while band: 245 | # pop band pixel closest to initial mask contour 246 | _, y, x = heapq.heappop(band) 247 | # flag it as KNOWN 248 | flags[y, x] = KNOWN 249 | 250 | # process his immediate neighbors (top/bottom/left/right) 251 | neighbors = [(y - 1, x), (y, x - 1), (y + 1, x), (y, x + 1)] 252 | for nb_y, nb_x in neighbors: 253 | # pixel out of frame 254 | if nb_y < 0 or nb_y >= height or nb_x < 0 or nb_x >= width: 255 | continue 256 | 257 | # neighbor outside of initial mask or already processed, nothing to do 258 | if flags[nb_y, nb_x] != UNKNOWN: 259 | continue 260 | 261 | # compute neighbor distance to inital mask contour 262 | nb_dist = min([ 263 | _solve_eikonal(nb_y - 1, nb_x, nb_y, nb_x - 1, height, width, dists, flags), 264 | _solve_eikonal(nb_y + 1, nb_x, nb_y, nb_x + 1, height, width, dists, flags), 265 | _solve_eikonal(nb_y - 1, nb_x, nb_y, nb_x + 1, height, width, dists, flags), 266 | _solve_eikonal(nb_y + 1, nb_x, nb_y, nb_x - 1, height, width, dists, flags) 267 | ]) 268 | dists[nb_y, nb_x] = nb_dist 269 | 270 | # inpaint neighbor 271 | pixel_vals = _inpaint_pixel(nb_y, nb_x, img, height, width, dists, flags, radius) 272 | 273 | img[nb_y, nb_x, 0] = pixel_vals[0] 274 | img[nb_y, nb_x, 1] = pixel_vals[1] 275 | img[nb_y, nb_x, 2] = pixel_vals[2] 276 | 277 | # add neighbor to narrow band 278 | flags[nb_y, nb_x] = BAND 279 | # push neighbor on band 280 | heapq.heappush(band, (nb_dist, nb_y, nb_x)) 281 | -------------------------------------------------------------------------------- /samples/lena_in.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/lena_in.png -------------------------------------------------------------------------------- /samples/lena_mask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/lena_mask.png -------------------------------------------------------------------------------- /samples/lena_opencv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/lena_opencv.png -------------------------------------------------------------------------------- /samples/lena_out.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/lena_out.png -------------------------------------------------------------------------------- /samples/peppers_in.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/peppers_in.png -------------------------------------------------------------------------------- /samples/peppers_mask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/peppers_mask.png -------------------------------------------------------------------------------- /samples/peppers_opencv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/peppers_opencv.png -------------------------------------------------------------------------------- /samples/peppers_out.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/peppers_out.png -------------------------------------------------------------------------------- /samples/tulips_in.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/tulips_in.png -------------------------------------------------------------------------------- /samples/tulips_mask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/tulips_mask.png -------------------------------------------------------------------------------- /samples/tulips_opencv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/tulips_opencv.png -------------------------------------------------------------------------------- /samples/tulips_out.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olvb/pyheal/cae7831b531a24b3ce6ca5d4f6493e4ac25a122b/samples/tulips_out.png --------------------------------------------------------------------------------