├── LICENSE ├── README.md ├── download_model.sh ├── example.png ├── example_mask.png ├── inpaint.lua ├── teaser.png └── utils.lua /LICENSE: -------------------------------------------------------------------------------- 1 | Attribution-NonCommercial-ShareAlike 4.0 International 2 | 3 | ======================================================================= 4 | 5 | Creative Commons Corporation ("Creative Commons") is not a law firm and 6 | does not provide legal services or legal advice. Distribution of 7 | Creative Commons public licenses does not create a lawyer-client or 8 | other relationship. Creative Commons makes its licenses and related 9 | information available on an "as-is" basis. Creative Commons gives no 10 | warranties regarding its licenses, any material licensed under their 11 | terms and conditions, or any related information. Creative Commons 12 | disclaims all liability for damages resulting from their use to the 13 | fullest extent possible. 14 | 15 | Using Creative Commons Public Licenses 16 | 17 | Creative Commons public licenses provide a standard set of terms and 18 | conditions that creators and other rights holders may use to share 19 | original works of authorship and other material subject to copyright 20 | and certain other rights specified in the public license below. The 21 | following considerations are for informational purposes only, are not 22 | exhaustive, and do not form part of our licenses. 23 | 24 | Considerations for licensors: Our public licenses are 25 | intended for use by those authorized to give the public 26 | permission to use material in ways otherwise restricted by 27 | copyright and certain other rights. Our licenses are 28 | irrevocable. Licensors should read and understand the terms 29 | and conditions of the license they choose before applying it. 30 | Licensors should also secure all rights necessary before 31 | applying our licenses so that the public can reuse the 32 | material as expected. Licensors should clearly mark any 33 | material not subject to the license. This includes other CC- 34 | licensed material, or material used under an exception or 35 | limitation to copyright. More considerations for licensors: 36 | wiki.creativecommons.org/Considerations_for_licensors 37 | 38 | Considerations for the public: By using one of our public 39 | licenses, a licensor grants the public permission to use the 40 | licensed material under specified terms and conditions. If 41 | the licensor's permission is not necessary for any reason--for 42 | example, because of any applicable exception or limitation to 43 | copyright--then that use is not regulated by the license. Our 44 | licenses grant only permissions under copyright and certain 45 | other rights that a licensor has authority to grant. Use of 46 | the licensed material may still be restricted for other 47 | reasons, including because others have copyright or other 48 | rights in the material. A licensor may make special requests, 49 | such as asking that all changes be marked or described. 50 | Although not required by our licenses, you are encouraged to 51 | respect those requests where reasonable. More_considerations 52 | for the public: 53 | wiki.creativecommons.org/Considerations_for_licensees 54 | 55 | ======================================================================= 56 | 57 | Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International 58 | Public License 59 | 60 | By exercising the Licensed Rights (defined below), You accept and agree 61 | to be bound by the terms and conditions of this Creative Commons 62 | Attribution-NonCommercial-ShareAlike 4.0 International Public License 63 | ("Public License"). To the extent this Public License may be 64 | interpreted as a contract, You are granted the Licensed Rights in 65 | consideration of Your acceptance of these terms and conditions, and the 66 | Licensor grants You such rights in consideration of benefits the 67 | Licensor receives from making the Licensed Material available under 68 | these terms and conditions. 69 | 70 | 71 | Section 1 -- Definitions. 72 | 73 | a. Adapted Material means material subject to Copyright and Similar 74 | Rights that is derived from or based upon the Licensed Material 75 | and in which the Licensed Material is translated, altered, 76 | arranged, transformed, or otherwise modified in a manner requiring 77 | permission under the Copyright and Similar Rights held by the 78 | Licensor. For purposes of this Public License, where the Licensed 79 | Material is a musical work, performance, or sound recording, 80 | Adapted Material is always produced where the Licensed Material is 81 | synched in timed relation with a moving image. 82 | 83 | b. Adapter's License means the license You apply to Your Copyright 84 | and Similar Rights in Your contributions to Adapted Material in 85 | accordance with the terms and conditions of this Public License. 86 | 87 | c. BY-NC-SA Compatible License means a license listed at 88 | creativecommons.org/compatiblelicenses, approved by Creative 89 | Commons as essentially the equivalent of this Public License. 90 | 91 | d. Copyright and Similar Rights means copyright and/or similar rights 92 | closely related to copyright including, without limitation, 93 | performance, broadcast, sound recording, and Sui Generis Database 94 | Rights, without regard to how the rights are labeled or 95 | categorized. For purposes of this Public License, the rights 96 | specified in Section 2(b)(1)-(2) are not Copyright and Similar 97 | Rights. 98 | 99 | e. Effective Technological Measures means those measures that, in the 100 | absence of proper authority, may not be circumvented under laws 101 | fulfilling obligations under Article 11 of the WIPO Copyright 102 | Treaty adopted on December 20, 1996, and/or similar international 103 | agreements. 104 | 105 | f. Exceptions and Limitations means fair use, fair dealing, and/or 106 | any other exception or limitation to Copyright and Similar Rights 107 | that applies to Your use of the Licensed Material. 108 | 109 | g. License Elements means the license attributes listed in the name 110 | of a Creative Commons Public License. The License Elements of this 111 | Public License are Attribution, NonCommercial, and ShareAlike. 112 | 113 | h. Licensed Material means the artistic or literary work, database, 114 | or other material to which the Licensor applied this Public 115 | License. 116 | 117 | i. Licensed Rights means the rights granted to You subject to the 118 | terms and conditions of this Public License, which are limited to 119 | all Copyright and Similar Rights that apply to Your use of the 120 | Licensed Material and that the Licensor has authority to license. 121 | 122 | j. Licensor means the individual(s) or entity(ies) granting rights 123 | under this Public License. 124 | 125 | k. NonCommercial means not primarily intended for or directed towards 126 | commercial advantage or monetary compensation. For purposes of 127 | this Public License, the exchange of the Licensed Material for 128 | other material subject to Copyright and Similar Rights by digital 129 | file-sharing or similar means is NonCommercial provided there is 130 | no payment of monetary compensation in connection with the 131 | exchange. 132 | 133 | l. Share means to provide material to the public by any means or 134 | process that requires permission under the Licensed Rights, such 135 | as reproduction, public display, public performance, distribution, 136 | dissemination, communication, or importation, and to make material 137 | available to the public including in ways that members of the 138 | public may access the material from a place and at a time 139 | individually chosen by them. 140 | 141 | m. Sui Generis Database Rights means rights other than copyright 142 | resulting from Directive 96/9/EC of the European Parliament and of 143 | the Council of 11 March 1996 on the legal protection of databases, 144 | as amended and/or succeeded, as well as other essentially 145 | equivalent rights anywhere in the world. 146 | 147 | n. You means the individual or entity exercising the Licensed Rights 148 | under this Public License. Your has a corresponding meaning. 149 | 150 | 151 | Section 2 -- Scope. 152 | 153 | a. License grant. 154 | 155 | 1. Subject to the terms and conditions of this Public License, 156 | the Licensor hereby grants You a worldwide, royalty-free, 157 | non-sublicensable, non-exclusive, irrevocable license to 158 | exercise the Licensed Rights in the Licensed Material to: 159 | 160 | a. reproduce and Share the Licensed Material, in whole or 161 | in part, for NonCommercial purposes only; and 162 | 163 | b. produce, reproduce, and Share Adapted Material for 164 | NonCommercial purposes only. 165 | 166 | 2. Exceptions and Limitations. For the avoidance of doubt, where 167 | Exceptions and Limitations apply to Your use, this Public 168 | License does not apply, and You do not need to comply with 169 | its terms and conditions. 170 | 171 | 3. Term. The term of this Public License is specified in Section 172 | 6(a). 173 | 174 | 4. Media and formats; technical modifications allowed. The 175 | Licensor authorizes You to exercise the Licensed Rights in 176 | all media and formats whether now known or hereafter created, 177 | and to make technical modifications necessary to do so. The 178 | Licensor waives and/or agrees not to assert any right or 179 | authority to forbid You from making technical modifications 180 | necessary to exercise the Licensed Rights, including 181 | technical modifications necessary to circumvent Effective 182 | Technological Measures. For purposes of this Public License, 183 | simply making modifications authorized by this Section 2(a) 184 | (4) never produces Adapted Material. 185 | 186 | 5. Downstream recipients. 187 | 188 | a. Offer from the Licensor -- Licensed Material. Every 189 | recipient of the Licensed Material automatically 190 | receives an offer from the Licensor to exercise the 191 | Licensed Rights under the terms and conditions of this 192 | Public License. 193 | 194 | b. Additional offer from the Licensor -- Adapted Material. 195 | Every recipient of Adapted Material from You 196 | automatically receives an offer from the Licensor to 197 | exercise the Licensed Rights in the Adapted Material 198 | under the conditions of the Adapter's License You apply. 199 | 200 | c. No downstream restrictions. You may not offer or impose 201 | any additional or different terms or conditions on, or 202 | apply any Effective Technological Measures to, the 203 | Licensed Material if doing so restricts exercise of the 204 | Licensed Rights by any recipient of the Licensed 205 | Material. 206 | 207 | 6. No endorsement. Nothing in this Public License constitutes or 208 | may be construed as permission to assert or imply that You 209 | are, or that Your use of the Licensed Material is, connected 210 | with, or sponsored, endorsed, or granted official status by, 211 | the Licensor or others designated to receive attribution as 212 | provided in Section 3(a)(1)(A)(i). 213 | 214 | b. Other rights. 215 | 216 | 1. Moral rights, such as the right of integrity, are not 217 | licensed under this Public License, nor are publicity, 218 | privacy, and/or other similar personality rights; however, to 219 | the extent possible, the Licensor waives and/or agrees not to 220 | assert any such rights held by the Licensor to the limited 221 | extent necessary to allow You to exercise the Licensed 222 | Rights, but not otherwise. 223 | 224 | 2. Patent and trademark rights are not licensed under this 225 | Public License. 226 | 227 | 3. To the extent possible, the Licensor waives any right to 228 | collect royalties from You for the exercise of the Licensed 229 | Rights, whether directly or through a collecting society 230 | under any voluntary or waivable statutory or compulsory 231 | licensing scheme. In all other cases the Licensor expressly 232 | reserves any right to collect such royalties, including when 233 | the Licensed Material is used other than for NonCommercial 234 | purposes. 235 | 236 | 237 | Section 3 -- License Conditions. 238 | 239 | Your exercise of the Licensed Rights is expressly made subject to the 240 | following conditions. 241 | 242 | a. Attribution. 243 | 244 | 1. If You Share the Licensed Material (including in modified 245 | form), You must: 246 | 247 | a. retain the following if it is supplied by the Licensor 248 | with the Licensed Material: 249 | 250 | i. identification of the creator(s) of the Licensed 251 | Material and any others designated to receive 252 | attribution, in any reasonable manner requested by 253 | the Licensor (including by pseudonym if 254 | designated); 255 | 256 | ii. a copyright notice; 257 | 258 | iii. a notice that refers to this Public License; 259 | 260 | iv. a notice that refers to the disclaimer of 261 | warranties; 262 | 263 | v. a URI or hyperlink to the Licensed Material to the 264 | extent reasonably practicable; 265 | 266 | b. indicate if You modified the Licensed Material and 267 | retain an indication of any previous modifications; and 268 | 269 | c. indicate the Licensed Material is licensed under this 270 | Public License, and include the text of, or the URI or 271 | hyperlink to, this Public License. 272 | 273 | 2. You may satisfy the conditions in Section 3(a)(1) in any 274 | reasonable manner based on the medium, means, and context in 275 | which You Share the Licensed Material. For example, it may be 276 | reasonable to satisfy the conditions by providing a URI or 277 | hyperlink to a resource that includes the required 278 | information. 279 | 3. If requested by the Licensor, You must remove any of the 280 | information required by Section 3(a)(1)(A) to the extent 281 | reasonably practicable. 282 | 283 | b. ShareAlike. 284 | 285 | In addition to the conditions in Section 3(a), if You Share 286 | Adapted Material You produce, the following conditions also apply. 287 | 288 | 1. The Adapter's License You apply must be a Creative Commons 289 | license with the same License Elements, this version or 290 | later, or a BY-NC-SA Compatible License. 291 | 292 | 2. You must include the text of, or the URI or hyperlink to, the 293 | Adapter's License You apply. You may satisfy this condition 294 | in any reasonable manner based on the medium, means, and 295 | context in which You Share Adapted Material. 296 | 297 | 3. You may not offer or impose any additional or different terms 298 | or conditions on, or apply any Effective Technological 299 | Measures to, Adapted Material that restrict exercise of the 300 | rights granted under the Adapter's License You apply. 301 | 302 | 303 | Section 4 -- Sui Generis Database Rights. 304 | 305 | Where the Licensed Rights include Sui Generis Database Rights that 306 | apply to Your use of the Licensed Material: 307 | 308 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right 309 | to extract, reuse, reproduce, and Share all or a substantial 310 | portion of the contents of the database for NonCommercial purposes 311 | only; 312 | 313 | b. if You include all or a substantial portion of the database 314 | contents in a database in which You have Sui Generis Database 315 | Rights, then the database in which You have Sui Generis Database 316 | Rights (but not its individual contents) is Adapted Material, 317 | including for purposes of Section 3(b); and 318 | 319 | c. You must comply with the conditions in Section 3(a) if You Share 320 | all or a substantial portion of the contents of the database. 321 | 322 | For the avoidance of doubt, this Section 4 supplements and does not 323 | replace Your obligations under this Public License where the Licensed 324 | Rights include other Copyright and Similar Rights. 325 | 326 | 327 | Section 5 -- Disclaimer of Warranties and Limitation of Liability. 328 | 329 | a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE 330 | EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS 331 | AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF 332 | ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, 333 | IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, 334 | WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR 335 | PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, 336 | ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT 337 | KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT 338 | ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. 339 | 340 | b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE 341 | TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, 342 | NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, 343 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, 344 | COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR 345 | USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN 346 | ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR 347 | DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR 348 | IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. 349 | 350 | c. The disclaimer of warranties and limitation of liability provided 351 | above shall be interpreted in a manner that, to the extent 352 | possible, most closely approximates an absolute disclaimer and 353 | waiver of all liability. 354 | 355 | 356 | Section 6 -- Term and Termination. 357 | 358 | a. This Public License applies for the term of the Copyright and 359 | Similar Rights licensed here. However, if You fail to comply with 360 | this Public License, then Your rights under this Public License 361 | terminate automatically. 362 | 363 | b. Where Your right to use the Licensed Material has terminated under 364 | Section 6(a), it reinstates: 365 | 366 | 1. automatically as of the date the violation is cured, provided 367 | it is cured within 30 days of Your discovery of the 368 | violation; or 369 | 370 | 2. upon express reinstatement by the Licensor. 371 | 372 | For the avoidance of doubt, this Section 6(b) does not affect any 373 | right the Licensor may have to seek remedies for Your violations 374 | of this Public License. 375 | 376 | c. For the avoidance of doubt, the Licensor may also offer the 377 | Licensed Material under separate terms or conditions or stop 378 | distributing the Licensed Material at any time; however, doing so 379 | will not terminate this Public License. 380 | 381 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 382 | License. 383 | 384 | 385 | Section 7 -- Other Terms and Conditions. 386 | 387 | a. The Licensor shall not be bound by any additional or different 388 | terms or conditions communicated by You unless expressly agreed. 389 | 390 | b. Any arrangements, understandings, or agreements regarding the 391 | Licensed Material not stated herein are separate from and 392 | independent of the terms and conditions of this Public License. 393 | 394 | 395 | Section 8 -- Interpretation. 396 | 397 | a. For the avoidance of doubt, this Public License does not, and 398 | shall not be interpreted to, reduce, limit, restrict, or impose 399 | conditions on any use of the Licensed Material that could lawfully 400 | be made without permission under this Public License. 401 | 402 | b. To the extent possible, if any provision of this Public License is 403 | deemed unenforceable, it shall be automatically reformed to the 404 | minimum extent necessary to make it enforceable. If the provision 405 | cannot be reformed, it shall be severed from this Public License 406 | without affecting the enforceability of the remaining terms and 407 | conditions. 408 | 409 | c. No term or condition of this Public License will be waived and no 410 | failure to comply consented to unless expressly agreed to by the 411 | Licensor. 412 | 413 | d. Nothing in this Public License constitutes or may be interpreted 414 | as a limitation upon, or waiver of, any privileges and immunities 415 | that apply to the Licensor or You, including from the legal 416 | processes of any jurisdiction or authority. 417 | 418 | ======================================================================= 419 | 420 | Creative Commons is not a party to its public licenses. 421 | Notwithstanding, Creative Commons may elect to apply one of its public 422 | licenses to material it publishes and in those instances will be 423 | considered the "Licensor." Except for the limited purpose of indicating 424 | that material is shared under a Creative Commons public license or as 425 | otherwise permitted by the Creative Commons policies published at 426 | creativecommons.org/policies, Creative Commons does not authorize the 427 | use of the trademark "Creative Commons" or any other trademark or logo 428 | of Creative Commons without its prior written consent including, 429 | without limitation, in connection with any unauthorized modifications 430 | to any of its public licenses or any other arrangements, 431 | understandings, or agreements concerning use of licensed material. For 432 | the avoidance of doubt, this paragraph does not form part of the public 433 | licenses. 434 | 435 | Creative Commons may be contacted at creativecommons.org. 436 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # [Globally and Locally Consistent Image Completion](http://iizuka.cs.tsukuba.ac.jp/projects/completion/) 2 | 3 | [Satoshi Iizuka](http://iizuka.cs.tsukuba.ac.jp/index_eng.html), [Edgar Simo-Serra](https://esslab.jp/~ess/), [Hiroshi Ishikawa](http://www.f.waseda.jp/hfs/indexE.html) 4 | 5 | ![Teaser Image](teaser.png) 6 | 7 | ## News 8 | **09/17/2020 Update:** We have released the following two models: 9 | - `completionnet_places2_freeform.t7`: An image completion model trained with free-form holes on the [Places2 dataset](http://places2.csail.mit.edu/), which will work better than the model trained with rectangular holes, even without post-processing. We used a part of the [context encoder [Pathak et al. 2016]](https://github.com/pathak22/context-encoder) implementation to generate the random free-form holes for training. 10 | - `completionnet_celeba.t7`: A face completion model trained with rectangular holes on the [CelebA dataset](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). This model was trained on face images with the smallest edges in the [160, 178], and thus it will work best on images of similar sizes. 11 | 12 | These models can be downloaded via `download_model.sh`. 13 | 14 | ## Overview 15 | 16 | This code provides an implementation of the research paper: 17 | 18 | ``` 19 | "Globally and Locally Consistent Image Completion" 20 | Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa 21 | ACM Transaction on Graphics (Proc. of SIGGRAPH 2017), 2017 22 | ``` 23 | We learn to inpaint missing regions with a deep convolutional network. 24 | Our network completes images of arbitrary resolutions by filling in 25 | missing regions of any shape. We use global and local context discriminators 26 | to train the completion network to provide both locally and globally consistent results. 27 | See our [project page](http://iizuka.cs.tsukuba.ac.jp/projects/completion/) for more detailed information. 28 | 29 | ## License 30 | 31 | ``` 32 | Copyright (C) <2017> 33 | 34 | This work is licensed under the Creative Commons 35 | Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy 36 | of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or 37 | send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 38 | 39 | Satoshi Iizuka, University of Tsukuba 40 | iizuka@cs.tsukuba.ac.jp, http://iizuka.cs.tsukuba.ac.jp/index_eng.html 41 | 42 | Edgar Simo-Serra, Waseda University 43 | ess@waseda.jp, https://esslab.jp/~ess/ 44 | ``` 45 | 46 | 47 | ## Dependencies 48 | 49 | - [Torch7](http://torch.ch/docs/getting-started.html) 50 | - [nn](https://github.com/torch/nn) 51 | - [image](https://github.com/torch/image) 52 | - [nngraph](https://github.com/torch/nngraph) 53 | - [torch-opencv](https://github.com/VisionLabs/torch-opencv) (optional, required for post-processing) 54 | 55 | The packages of `nn`, `image`, and `nngraph` should be a part of a standard Torch7 install. 56 | For information on how to install Torch7 please see the [official torch documentation](http://torch.ch/docs/getting-started.html) 57 | on the subject. The `torch-opencv` is OpenCV bindings for LuaJit+Torch, which can be installed via 58 | `luarocks install cv` after installing OpenCV 3.1. Please see the [instruction page](https://github.com/VisionLabs/torch-opencv/wiki/Installation) for more detailed information. 59 | 60 | **17/09/2020 Note:** If you fail to install Torch7 with current GPUs, please try the [self-contained Torch installation repository (unofficial)]( https://github.com/nagadomi/distro) by [@nadadomi](https://github.com/nagadomi), which supports CUDA10.1, Volta, and Turing. 61 | 62 | ## Usage 63 | 64 | First, download the models by running the download script: 65 | 66 | ``` 67 | bash download_model.sh 68 | ``` 69 | 70 | Basic usage is: 71 | 72 | ``` 73 | th inpaint.lua --input --mask 74 | ``` 75 | The mask is a binary image (1 for pixels to be completed, 0 otherwise) and should be the same size as the input image. If the mask is not specified, a mask with randomly generated holes will be used. 76 | 77 | Other options: 78 | 79 | - `--model`: Model to be used. Defaults to 'completionnet_places2_freeform.t7'. 80 | - `--gpu`: Use GPU for the computation. [cunn](https://github.com/torch/cunn) is required to use this option. Defaults to false. 81 | - `--maxdim`: Long edge dimension of the input image. Defaults to 600. 82 | - `--postproc`: Perform the post-processing. Defaults to false. If you fail to install the `torch-opencv`, do not use this option to avoid using the package. 83 | 84 | For example: 85 | 86 | ``` 87 | th inpaint.lua --input example.png --mask example_mask.png 88 | ``` 89 | 90 | ### Best Performance 91 | 92 | - The Places models were trained on the [Places2 dataset](http://places2.csail.mit.edu/) and thus best performance is for natural outdoor images. 93 | - While the Places2 models work on images of any size with arbitrary holes, we trained them on images with the smallest edges in the [256, 384] pixel range and random holes in the [96, 128] pixel range. Our models will work best on images with holes of those sizes. 94 | - Significantly large holes or extrapolation when the holes are at the border of images may fail to be filled in due to limited spatial support of the model. 95 | 96 | ### Notes 97 | 98 | - This is developed on a Linux machine running Ubuntu 16.04 during late 2016. 99 | - Provided model and sample code is under a non-commercial creative commons license. 100 | 101 | ## Citing 102 | 103 | If you use this code please cite: 104 | 105 | ``` 106 | @Article{IizukaSIGGRAPH2017, 107 | author = {Satoshi Iizuka and Edgar Simo-Serra and Hiroshi Ishikawa}, 108 | title = {{Globally and Locally Consistent Image Completion}}, 109 | journal = "ACM Transactions on Graphics (Proc. of SIGGRAPH)", 110 | year = 2017, 111 | volume = 36, 112 | number = 4, 113 | pages = 107:1--107:14, 114 | articleno = 107, 115 | } 116 | ``` 117 | 118 | 119 | 120 | 121 | -------------------------------------------------------------------------------- /download_model.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "Downloading the completion models..." 4 | 5 | wget http://iizuka.cs.tsukuba.ac.jp/data/completionnet_places2.t7 -O completionnet_places2.t7 6 | wget http://iizuka.cs.tsukuba.ac.jp/data/completionnet_places2_freeform.t7 -O completionnet_places2_freeform.t7 7 | wget http://iizuka.cs.tsukuba.ac.jp/data/completionnet_celeba.t7 -O completionnet_celeba.t7 8 | 9 | echo -e "Download finished!" 10 | -------------------------------------------------------------------------------- /example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/satoshiiizuka/siggraph2017_inpainting/15c55197720a8579086cf4c58aa301b4b5ec9aa8/example.png -------------------------------------------------------------------------------- /example_mask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/satoshiiizuka/siggraph2017_inpainting/15c55197720a8579086cf4c58aa301b4b5ec9aa8/example_mask.png -------------------------------------------------------------------------------- /inpaint.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (C) <2017> 3 | 4 | This work is licensed under the Creative Commons 5 | Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy 6 | of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or 7 | send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 8 | 9 | Satoshi Iizuka, University of Tsukuba 10 | iizuka@cs.tsukuba.ac.jp, http://iizuka.cs.tsukuba.ac.jp/index_eng.html 11 | 12 | Edgar Simo-Serra, Waseda University 13 | ess@waseda.jp, https://esslab.jp/~ess/ 14 | --]] 15 | 16 | require 'nn' 17 | require 'nngraph' 18 | require 'image' 19 | require 'utils' 20 | torch.setdefaulttensortype( 'torch.FloatTensor' ) 21 | 22 | -- commandline options 23 | cmd = torch.CmdLine() 24 | cmd:addTime() 25 | cmd:text() 26 | cmd:option( '--input', 'none', 'Input image' ) 27 | cmd:option( '--mask', 'none', 'Mask image') 28 | cmd:option( '--model', 'completionnet_places2_freeform.t7', 'Model to be used') 29 | cmd:option( '--maxdim', 600, 'Long edge dimension of an input image') 30 | cmd:option( '--gpu', false, 'Use GPU' ) 31 | cmd:option( '--postproc', false, 'Perform post-processing' ) 32 | local opt = cmd:parse(arg or {}) 33 | print(opt) 34 | assert( opt.input ~= 'none' ) 35 | print( 'Loading model...' ) 36 | 37 | -- load Completion Network 38 | local data = torch.load( opt.model ) 39 | local model = data.model 40 | local datamean = data.mean 41 | model:evaluate() 42 | if opt.gpu then 43 | require 'cunn' 44 | model:cuda() 45 | end 46 | 47 | -- load data 48 | local I = image.load( opt.input ) 49 | local M = torch.Tensor() 50 | if opt.mask~='none' then 51 | M = load_image_gray( opt.mask ) 52 | assert( I:size(2) == M:size(2) and I:size(3) == M:size(3) ) 53 | else 54 | -- generate random holes 55 | M = torch.Tensor( 1, I:size(2), I:size(3) ):fill(0) 56 | local nHoles = torch.random( 2, 4 ) 57 | for i=1,nHoles do 58 | local mask_w = torch.random( 32, 128 ) 59 | local mask_h = torch.random( 32, 128 ) 60 | local px = torch.random(1, I:size(3)-mask_w-1) 61 | local py = torch.random(1, I:size(2)-mask_h-1) 62 | local R = {{},{py,py+mask_h},{px,px+mask_w}} 63 | M[R]:fill(1) 64 | end 65 | end 66 | 67 | local hwmax = math.max( I:size(2), I:size(3) ) 68 | if hwmax > opt.maxdim then 69 | I = image.scale( I, string.format('*%d/%d',opt.maxdim,hwmax) ) 70 | M = image.scale( M, string.format('*%d/%d',opt.maxdim,hwmax) ) 71 | end 72 | 73 | --Set up input 74 | I = image.scale( I, torch.round(I:size(3)/4)*4, torch.round(I:size(2)/4)*4 ) 75 | M = image.scale( M, torch.round(M:size(3)/4)*4, torch.round(M:size(2)/4)*4 ):ge(0.2):float() 76 | local Ip = I:clone() 77 | for j=1,3 do I[j]:add( -datamean[j] ) end 78 | I:maskedFill( torch.repeatTensor(M:byte(),3,1,1), 0 ) 79 | 80 | -- inpaint target holes 81 | print('Inpainting...') 82 | local input = torch.cat(I, M, 1) 83 | input = input:reshape( 1, input:size(1), input:size(2), input:size(3) ) 84 | if opt.gpu then input = input:cuda() end 85 | local res = model:forward( input ):float()[1] 86 | local out = Ip:cmul(torch.repeatTensor((1-M),3,1,1)) + res:cmul(torch.repeatTensor(M,3,1,1)) 87 | 88 | -- perform post-processing 89 | if opt.postproc then 90 | print('Performing post-processing...') 91 | local cv = require 'cv' 92 | require 'cv.photo' 93 | local pflag = false 94 | local minx = 1e5 95 | local maxx = 1 96 | local miny = 1e5 97 | local maxy = 1 98 | for y=1,M:size(3) do 99 | for x=1,M:size(2) do 100 | if M[1][x][y] == 1 then 101 | minx = math.min(minx,x) 102 | maxx = math.max(maxx,x) 103 | miny = math.min(miny,y) 104 | maxy = math.max(maxy,y) 105 | end 106 | end 107 | end 108 | 109 | local p_i = {torch.floor(miny+(maxy-miny)/2)-1,torch.floor(minx+(maxx-minx)/2)-1} 110 | local src_i = tensor2cvimg( out ) 111 | local mask_i = M:clone():permute(2,3,1):mul(255):byte() 112 | local dst_i = cv.inpaint{src_i, mask_i, dst=nil, inpaintRadius=1, flags=cv.INPAINT_TELEA} 113 | local out_i = dst_i:clone() 114 | cv.seamlessClone{ src=src_i, dst=dst_i, mask=mask_i, p=p_i, blend=out_i, flags=cv.NORMAL_CLONE } 115 | out = out_i 116 | out = cvimg2tensor( out ) 117 | end 118 | 119 | -- save output 120 | for j=1,3 do I[j]:add( datamean[j] ) end 121 | image.save('input.png', I) 122 | image.save('out.png', out) 123 | 124 | print('Done.') 125 | -------------------------------------------------------------------------------- /teaser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/satoshiiizuka/siggraph2017_inpainting/15c55197720a8579086cf4c58aa301b4b5ec9aa8/teaser.png -------------------------------------------------------------------------------- /utils.lua: -------------------------------------------------------------------------------- 1 | function load_image_gray ( path ) 2 | local img = image.load( path ) 3 | if img:size(1) > 3 then -- rgb + alpha 4 | img = img[{{1,3},{},{}}] 5 | end 6 | if img:size(1) == 2 then -- grayscale + alpha 7 | img = img[{{1},{},{}}] 8 | end 9 | if img:size(1) > 1 then -- rgb -> grayscale 10 | img = image.rgb2y( img ) 11 | end 12 | return img 13 | end 14 | 15 | function tensor2cvimg( src ) --return byte tensor [0, 255] 16 | local out = src:clone():permute(2,3,1):mul(255):byte() 17 | local outc = out[{{},{},{1}}]:clone() 18 | out[{{},{},{1}}] = out[{{},{},{3}}] 19 | out[{{},{},{3}}] = outc 20 | return out 21 | end 22 | 23 | function cvimg2tensor( src ) --return float tensor [0, 1] 24 | local out = src:clone():permute(3,1,2):float():div(255) 25 | local outc = out[{{1},{},{}}]:clone() 26 | out[{{1},{},{}}] = out[{{3},{},{}}] 27 | out[{{3},{},{}}] = outc 28 | return out 29 | end 30 | --------------------------------------------------------------------------------