├── .gitignore ├── .gitmodules ├── LICENSE ├── README.md ├── assets ├── image_1.png ├── image_2.jpg └── image_3.png ├── script └── predict_and_inpaint.sh └── src ├── drawMask.py └── reapplyLama.py /.gitignore: -------------------------------------------------------------------------------- 1 | detection_results 2 | results 3 | outputs 4 | each_step_results 5 | images -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "CRAFT-pytorch"] 2 | path = CRAFT-pytorch 3 | url = https://github.com/clovaai/CRAFT-pytorch.git 4 | [submodule "lama"] 5 | path = lama 6 | url = https://github.com/advimman/lama.git 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Felix Liawi 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Auto Text Removal 2 | Auto text removal using CRAFT to detect the text and LaMa to remove the text. Highly motivated by [`auto-lama`](https://github.com/andy971022/auto-lama) which do object detection + LaMa. 3 | 4 | ## Implementation pipeline 5 | ![pipeline](./assets/image_1.png) 6 | Applying LaMa multiple times with bigger mask creates a better result. 7 | 8 | ## How to use it 9 | 1. Clone this repository 10 | 2. Setup and download CRAFT model. Please see the official repo for the detail. Please put the downloaded model under [`CRAFT-pytorch`](https://github.com/clovaai/CRAFT-pytorch) folder. 11 | 3. Setup and download LaMa model. Please see the official repo for the detail. Please put the downloaded model under [`LaMa`](https://github.com/advimman/lama) folder. 12 | 4. Put your input images to `images` folder 13 | 5. Run the program 14 | ``` 15 | bash script/predict_and_inpaint.sh 16 | ``` 17 | or 18 | ``` 19 | bash script/predict_and_inpaint.sh 5 20 | ``` 21 | indicating repeat LaMa 5 times (default 3 times) 22 | 23 | > Output will be located at `results` folder and result from every repetition will located at `each_step_results` folder 24 | 25 | ## Result 26 | Before ([Photo source](https://unsplash.com/photos/z1d-LP8sjuI)) 27 | ![before](./assets/image_2.jpg) 28 | After 29 | ![after](./assets/image_3.png) 30 | 31 | ## Note 32 | I do a little modification to [`CRAFT-pytorch`](https://github.com/clovaai/CRAFT-pytorch) original code to fit my usage requirement. 33 | 34 | ## Contributor 35 | Anyone who want to help is welcomed :D. Just open an issue or a PR. 36 | 37 | ## Acknowledgements 38 | - [`CRAFT-pytorch`](https://github.com/clovaai/CRAFT-pytorch) for text detection 39 | - [`LaMa`](https://github.com/advimman/lama) for object(text) removal 40 | 41 | ## License 42 | I am not really clear about the license I should use for this repo. https://github.com/advimman/lama use Apache-2.0 license while https://github.com/clovaai/CRAFT-pytorch use MIT license. Feel free to open an issue if you think the license that I use (MIT license) to label this repo is incorrect. 43 | -------------------------------------------------------------------------------- /assets/image_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/liawifelix/auto-text-removal/b55a372f6cb654e5c21ee40e551fa3b4f16e1230/assets/image_1.png -------------------------------------------------------------------------------- /assets/image_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/liawifelix/auto-text-removal/b55a372f6cb654e5c21ee40e551fa3b4f16e1230/assets/image_2.jpg -------------------------------------------------------------------------------- /assets/image_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/liawifelix/auto-text-removal/b55a372f6cb654e5c21ee40e551fa3b4f16e1230/assets/image_3.png -------------------------------------------------------------------------------- /script/predict_and_inpaint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | mkdir -p detection_results 4 | rm detection_results/* 5 | 6 | mkdir -p outputs 7 | rm outputs/* 8 | 9 | mkdir -p results 10 | rm results/* 11 | 12 | mkdir -p each_step_results 13 | rm each_step_results/* 14 | 15 | python CRAFT-pytorch/test.py --trained_model CRAFT-pytorch/craft_mlt_25k.pth --test_folder=images --result_folder=detection_results 16 | 17 | python src/drawMask.py --coor_input_folder detection_results --input_folder images --output_real_image True --mask_output_folder outputs 18 | 19 | bash lama/docker/2_predict.sh $(pwd)/lama/big-lama $(pwd)/outputs $(pwd)/results device=cpu 20 | 21 | lama_n=3 22 | if [ ! -z $1 ] 23 | then 24 | lama_n=$1 25 | fi 26 | 27 | for (( i=1; i<$lama_n; i++ )) 28 | do 29 | python src/drawMask.py --coor_input_folder detection_results --input_folder images --output_real_image False --radius $((i*5)) --mask_output_folder outputs 30 | python3 src/reapplyLama.py --lama_output_folder=results --lama_input_folder=outputs --each_step_result_folder=each_step_results --n $((i)) 31 | bash lama/docker/2_predict.sh $(pwd)/lama/big-lama $(pwd)/outputs $(pwd)/results device=cpu 32 | done 33 | -------------------------------------------------------------------------------- /src/drawMask.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | import argparse 4 | import os 5 | import glob 6 | 7 | 8 | def drawMask(points, image): 9 | points = np.array(points) 10 | cv2.fillPoly(image, [points], color=(255, 255, 255)) 11 | return image 12 | 13 | 14 | if __name__ == '__main__': 15 | parser = argparse.ArgumentParser(description='Draw Mask') 16 | parser.add_argument('--coor_input_folder', type=str) 17 | parser.add_argument('--input_folder', type=str) 18 | parser.add_argument('--mask_output_folder', type=str) 19 | parser.add_argument('--radius', type=int, default=0) 20 | parser.add_argument('--output_real_image', type=bool, default=False) 21 | args = parser.parse_args() 22 | 23 | folder_path = args.input_folder 24 | file_pattern = "*" 25 | files = glob.glob(os.path.join(folder_path, file_pattern)) 26 | for file in files: 27 | img = cv2.imread(file) 28 | img_mask = np.zeros((img.shape[0], img.shape[1]), dtype=np.uint8) 29 | file_name_ext = file.split("/")[1] 30 | 31 | file_name = file_name_ext.split(".")[0] 32 | ext = file_name_ext.split(".")[1] 33 | gt_file = f"{args.coor_input_folder}/res_{file_name}.txt" 34 | 35 | radius = args.radius 36 | row, col, channel = img.shape 37 | 38 | with open(gt_file, "r", encoding="utf-8") as f: 39 | for line in f: 40 | points = list(map(lambda x: int(x), line.strip().split(","))) 41 | xy1 = (max(points[0] - radius, 0), max(points[1] - radius, 0)) 42 | xy2 = (min(points[2] + radius, col-1), max(points[3] - radius, 0)) 43 | xy3 = (min(points[4] + radius, col-1), min(points[5] + radius, row-1)) 44 | xy4 = (max(points[6] - radius, 0), min(points[7] + radius, row-1)) 45 | points = np.array([xy1, xy2, xy3, xy4]) 46 | 47 | img_mask = drawMask(points, img_mask) 48 | 49 | ext = 'png' 50 | cv2.imwrite(f"{args.mask_output_folder}/{file_name}_mask.{ext}", img_mask) 51 | 52 | if args.output_real_image: 53 | cv2.imwrite(f"{args.mask_output_folder}/{file_name}.{ext}", img) -------------------------------------------------------------------------------- /src/reapplyLama.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import os 3 | import argparse 4 | import shutil 5 | 6 | 7 | if __name__ == '__main__': 8 | parser = argparse.ArgumentParser(description='Reapply lama') 9 | parser.add_argument('--lama_output_folder', type=str) 10 | parser.add_argument('--lama_input_folder', type=str) 11 | parser.add_argument('--each_step_result_folder', type=str) 12 | parser.add_argument('--n', type=int, default=0) 13 | args = parser.parse_args() 14 | 15 | lamaOutputFiles = glob.glob(os.path.join(args.lama_output_folder, "*_mask.*")) 16 | for file in lamaOutputFiles: 17 | file_name = ''.join(file.split("/")[-1].split("_mask")) 18 | shutil.copy(file, os.path.join(args.lama_input_folder, file_name)) 19 | if args.n: 20 | shutil.copy(file, os.path.join(args.each_step_result_folder, f"{args.n}_{file_name}")) 21 | os.remove(file) --------------------------------------------------------------------------------