├── README.md ├── linear-color-transfer.py └── lum-transfer.py /README.md: -------------------------------------------------------------------------------- 1 | # Neural-Tools 2 | Tools made for usage alongside artistic style transfer projects based on the [Controlling Perceptual Factors in Neural Style Transfer](https://arxiv.org/abs/1611.07865) research paper by Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. 3 | 4 | **In-depth information about how to perform Scale Control and Color Control, including the Neural-Style parameters used in the examples, can be found on the [wiki](https://github.com/ProGamerGov/Neural-Tools/wiki).** The Color Control feature is broken down into two different features known as Luminance-Only Style Transfer, and Color Matching. The Scale Control feature focuses on separating style image content/shapes, and style image textures. 5 | 6 | # Linear Color Transfer 7 | 8 | The `match_color` function's code comes from the very talented Leon Gatys' code [here](https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ScaleControl.ipynb). This script was developed to help enable Scale Control in [Neural-Style](https://github.com/jcjohnson/neural-style), but it can be used for anything else that requires linear color transfer. Supported image formats include: `jpg`, `jpeg`, `png`, `tiff`, etc... 9 | 10 | Scale Control examples made with [Neural-Style](https://github.com/jcjohnson/neural-style), can be viewed [here](https://github.com/ProGamerGov/Neural-Tools/wiki/Scale-Control-Examples). 11 | 12 | ### Dependencies: 13 | 14 | `sudo pip install scikit-image` 15 | 16 | `sudo pip install imageio` 17 | 18 | `sudo pip install numpy` 19 | 20 | `sudo pip install scipy` 21 | 22 | ### Usage: 23 | 24 | Basic usage: 25 | 26 | ``` 27 | python linear-color-transfer.py --target_image target.png --source_image source.png 28 | ``` 29 | 30 | Advanced usage: 31 | 32 | ``` 33 | python linear-color-transfer.py --target_image target.png --source_image source.png --output_image output.png --mode pca --eps 1e-5 34 | ``` 35 | 36 | ### Parameters: 37 | 38 | * `--target_image`: The image you are transfering color to. Ex: `target.png` 39 | 40 | * `--source_image`: The image you are transfering color from. Ex: `source.png` 41 | 42 | * `--output_image`: The name of your output image. Ex: `output.png` 43 | 44 | * `--mode`: The color transfer mode. Options are `pca`, `chol`, or `sym`. 45 | 46 | * `--eps`: Your epsilon value in scientific notation or normal notation. Ex: `1e-5` or `0.00001`. 47 | 48 | ### Examples: 49 | 50 | **Source Image:** 51 | 52 | ![](https://i.imgur.com/eoX7f3Il.jpg) 53 | 54 | **Target Image:** 55 | 56 | ![](https://i.imgur.com/7FPCSril.jpg) 57 | 58 | **Output Image:** 59 | 60 | ![](https://i.imgur.com/STZ0Mspl.png) 61 | 62 | **[See here for how to use this script for Scale Control](https://github.com/ProGamerGov/Neural-Tools/wiki/Scale-Control-Examples)**. 63 | 64 | ![](https://i.imgur.com/fsqGmJfl.png) 65 | 66 | 67 | ### Linear Color Transfer is also used for Color Matching Style Transfer: 68 | 69 | ![](https://i.imgur.com/6xf5c9yl.jpg) 70 | 71 | 72 | See [here on the wiki](https://github.com/ProGamerGov/Neural-Tools/wiki/Color-Matching), for more details on Color Matching Style Transfer. 73 | 74 | --- 75 | 76 | --- 77 | 78 | # Luminance Transfer 79 | 80 | This script was developed to help enable colour control in [Neural-Style](https://github.com/jcjohnson/neural-style), also known as "Luminance Transfer". This script uses code from Leon Gatys' code [here](https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ColourControl.ipynb). Supported image formats include: `jpg`, `jpeg`, `png`, `tiff`, etc... 81 | 82 | Luminance transfer/Color Control examples made with [Neural-Style](https://github.com/jcjohnson/neural-style), can be found [here](https://github.com/ProGamerGov/Neural-Tools/wiki/Color-Control-Examples). 83 | 84 | ### How It Works: 85 | 86 | Currently, all available models are trained on images with the RGB/BGR color space. An image's luminance can represented in the form of gray scale color space image, which can be converted to RGB format for Neural-Style. After the gray scale images are run through Neural-Style, re-applying the color to your output is done with the use of the LUV color space. 87 | 88 | Basically due to pre-trained model limitations, gray scale images are used to transfer luminance, and a color space supporting luminance is used to transfer the colors back to the finished output. 89 | 90 | ### Dependencies: 91 | 92 | `sudo pip install scikit-image` 93 | 94 | `sudo pip install imageio` 95 | 96 | `sudo pip install numpy` 97 | 98 | `sudo pip install scipy` 99 | 100 | ### Usage: 101 | 102 | Basic usage: 103 | 104 | ``` 105 | python lum-transfer.py --content_image content.png --style_image style.png 106 | ``` 107 | 108 | Advanced usage: 109 | 110 | ``` 111 | python lum-transfer.py --cp_mode lum --content_image content.png --style_image style.png --org_content content.png --output_style_image output_style.png --output_content_image output_content.png 112 | ``` 113 | 114 | ### Parameters: 115 | 116 | The required input images and the output images, are dependent on the `--cp_mode` option that you choose: 117 | 118 | * `--cp_mode`: The script's mode. Options are `lum`, `lum2`, `match`, `match_style`. 119 | 120 | 121 | **Mode: `lum`** 122 | 123 | 124 | * `--content_image`: Your content image. Ex: `content.png` 125 | 126 | * `--style_image`: Your style image. Ex: `style.png` 127 | 128 | * `--org_content`: Your original unmodified content image. Ex: `original_content.png` 129 | 130 | * `--output_content_image`: The name of your output content image. Ex: `content_output.png` 131 | 132 | * `--output_style_image`: The name of your output style image. Ex: `style_output.png` 133 | 134 | **Mode: `match`** 135 | 136 | * `--content_image`: Your content image. Ex: `content.png` 137 | 138 | * `--style_image`: Your style image. Ex: `style.png` 139 | 140 | * `--output_style_image`: The name of your output style image. Ex: `style_output.png` 141 | 142 | **Mode: `match_style`** 143 | 144 | * `--content_image`: Your content image. Ex: `content.png` 145 | 146 | * `--style_image`: Your style image. Ex: `style.png` 147 | 148 | * `--output_content_image`: The name of your output content image. Ex: `content_output.png` 149 | 150 | **Mode: `lum2`** 151 | 152 | * `--output_lum2`: The name of your output image from Neural-Style. Ex: `out.png` 153 | 154 | * `--org_content`: Your original unmodified content image. Ex: `original_content.png` 155 | 156 | * `--output_image`: The name of your output image. Ex: `output.png` 157 | 158 | ### Examples: 159 | 160 | **The style image is adjusted to match the content image:** 161 | 162 | ![](https://i.imgur.com/Q7phTmel.png) 163 | 164 | ![](https://i.imgur.com/dRf3yZHl.png) 165 | 166 | **After Neural-Style:** 167 | 168 | ![](https://i.imgur.com/hpW8zufl.png) 169 | 170 | **Final ouput image:** 171 | 172 | ![](https://i.imgur.com/o5HDDtDl.png) 173 | 174 | --- 175 | 176 | -------------------------------------------------------------------------------- /linear-color-transfer.py: -------------------------------------------------------------------------------- 1 | # This script performs the linear color transfer step that 2 | # leongatys/NeuralImageSynthesis' Scale Control code performs. 3 | # https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ScaleControl.ipynb 4 | # Standalone script by github.com/htoyryla, and github.com/ProGamerGov 5 | 6 | import numpy as np 7 | import argparse 8 | import imageio 9 | from skimage import io,transform,img_as_float 10 | from skimage.io import imread,imsave 11 | from PIL import Image 12 | from numpy import eye 13 | 14 | parser = argparse.ArgumentParser() 15 | parser.add_argument('-t', '--target_image', type=str, help="The image you are transfering color to. Ex: target.png", required=True) 16 | parser.add_argument('-s', '--source_image', type=str, help="The image you are transfering color from. Ex: source.png", required=True) 17 | parser.add_argument('-o', '--output_image', default='output.png', help="The name of your output image. Ex: output.png", type=str) 18 | parser.add_argument('-m', '--mode', default='pca', help="The color transfer mode. Options are pca, chol, or sym.", type=str) 19 | parser.add_argument('-e', '--eps', default='1e-5', help="Your epsilon value in scientific notation or normal notation. Ex: 1e-5 or 0.00001", type=float) 20 | parser.parse_args() 21 | args = parser.parse_args() 22 | 23 | 24 | Image.MAX_IMAGE_PIXELS = 1000000000 # Support gigapixel images 25 | 26 | 27 | def main(): 28 | 29 | target_img = imageio.imread(args.target_image, pilmode="RGB").astype(float)/256 30 | source_img = imageio.imread(args.source_image, pilmode="RGB").astype(float)/256 31 | 32 | output_img = match_color(target_img, source_img, mode=args.mode, eps=args.eps) 33 | imsave(args.output_image, output_img) 34 | 35 | 36 | def match_color(target_img, source_img, mode='pca', eps=1e-5): 37 | ''' 38 | Matches the colour distribution of the target image to that of the source image 39 | using a linear transform. 40 | Images are expected to be of form (w,h,c) and float in [0,1]. 41 | Modes are chol, pca or sym for different choices of basis. 42 | ''' 43 | mu_t = target_img.mean(0).mean(0) 44 | t = target_img - mu_t 45 | t = t.transpose(2,0,1).reshape(3,-1) 46 | Ct = t.dot(t.T) / t.shape[1] + eps * eye(t.shape[0]) 47 | mu_s = source_img.mean(0).mean(0) 48 | s = source_img - mu_s 49 | s = s.transpose(2,0,1).reshape(3,-1) 50 | Cs = s.dot(s.T) / s.shape[1] + eps * eye(s.shape[0]) 51 | if mode == 'chol': 52 | chol_t = np.linalg.cholesky(Ct) 53 | chol_s = np.linalg.cholesky(Cs) 54 | ts = chol_s.dot(np.linalg.inv(chol_t)).dot(t) 55 | if mode == 'pca': 56 | eva_t, eve_t = np.linalg.eigh(Ct) 57 | Qt = eve_t.dot(np.sqrt(np.diag(eva_t))).dot(eve_t.T) 58 | eva_s, eve_s = np.linalg.eigh(Cs) 59 | Qs = eve_s.dot(np.sqrt(np.diag(eva_s))).dot(eve_s.T) 60 | ts = Qs.dot(np.linalg.inv(Qt)).dot(t) 61 | if mode == 'sym': 62 | eva_t, eve_t = np.linalg.eigh(Ct) 63 | Qt = eve_t.dot(np.sqrt(np.diag(eva_t))).dot(eve_t.T) 64 | Qt_Cs_Qt = Qt.dot(Cs).dot(Qt) 65 | eva_QtCsQt, eve_QtCsQt = np.linalg.eigh(Qt_Cs_Qt) 66 | QtCsQt = eve_QtCsQt.dot(np.sqrt(np.diag(eva_QtCsQt))).dot(eve_QtCsQt.T) 67 | ts = np.linalg.inv(Qt).dot(QtCsQt).dot(np.linalg.inv(Qt)).dot(t) 68 | matched_img = ts.reshape(*target_img.transpose(2,0,1).shape).transpose(1,2,0) 69 | matched_img += mu_s 70 | matched_img[matched_img>1] = 1 71 | matched_img[matched_img<0] = 0 72 | return matched_img 73 | 74 | 75 | if __name__ == "__main__": 76 | main() 77 | -------------------------------------------------------------------------------- /lum-transfer.py: -------------------------------------------------------------------------------- 1 | #This script performs the functions required for lumin transfer that 2 | #leongatys/NeuralImageSynthesis' Color Control code performs. 3 | #https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ColourControl.ipynb 4 | #Standalone script by github.com/ProGamerGov 5 | 6 | import skimage 7 | import numpy as np 8 | import argparse 9 | import imageio 10 | from skimage import io,transform,img_as_float 11 | from skimage.io import imread,imsave 12 | from PIL import Image 13 | from numpy import eye 14 | 15 | 16 | parser = argparse.ArgumentParser() 17 | parser.add_argument('--content_image', type=str, help="The content image. Ex: content.png") 18 | parser.add_argument('--style_image', type=str, help="The style image. Ex: style.png") 19 | parser.add_argument('--output_content_image', default='output_content.png', help="The name of your output content image. Ex: content_output.png", type=str) 20 | parser.add_argument('--output_style_image', default='output_style.png', help="The name of your output style image. Ex: style_output.png", type=str) 21 | parser.add_argument('--output_image', default='output_style.png', help="The name of your output image. Ex: output.png", type=str) 22 | parser.add_argument('--org_content', default='original_content_image.png', help="The name of your original content image. Ex: org_content.png", type=str) 23 | parser.add_argument('--output_lum2', default='out.png', help="The name of your output image from Neural-Style. Ex: out.png", type=str) 24 | parser.add_argument('--cp_mode', default='lum', help="The script's mode. Options are: lum, lum2, match, and match_style", type=str) 25 | parser.parse_args() 26 | args = parser.parse_args() 27 | cp_mode = args.cp_mode 28 | output_content_name = args.output_content_image 29 | output_style_name = args.output_style_image 30 | output_a_name = args.output_image 31 | 32 | Image.MAX_IMAGE_PIXELS = 1000000000 # Support gigapixel images 33 | 34 | def lum_transform(image): 35 | """ 36 | Returns the projection of a colour image onto the luminance channel 37 | Images are expected to be of form (w,h,c) and float in [0,1]. 38 | """ 39 | img = image.transpose(2,0,1).reshape(3,-1) 40 | lum = np.array([.299, .587, .114]).dot(img).squeeze() 41 | img = np.tile(lum[None,:],(3,1)).reshape((3,image.shape[0],image.shape[1])) 42 | return img.transpose(1,2,0) 43 | 44 | def rgb2luv(image): 45 | img = image.transpose(2,0,1).reshape(3,-1) 46 | luv = np.array([[.299, .587, .114],[-.147, -.288, .436],[.615, -.515, -.1]]).dot(img).reshape((3,image.shape[0],image.shape[1])) 47 | return luv.transpose(1,2,0) 48 | def luv2rgb(image): 49 | img = image.transpose(2,0,1).reshape(3,-1) 50 | rgb = np.array([[1, 0, 1.139],[1, -.395, -.580],[1, 2.03, 0]]).dot(img).reshape((3,image.shape[0],image.shape[1])) 51 | return rgb.transpose(1,2,0) 52 | 53 | def match_color(target_img, source_img, mode='pca', eps=1e-5): 54 | ''' 55 | Matches the colour distribution of the target image to that of the source image 56 | using a linear transform. 57 | Images are expected to be of form (w,h,c) and float in [0,1]. 58 | Modes are chol, pca or sym for different choices of basis. 59 | ''' 60 | mu_t = target_img.mean(0).mean(0) 61 | t = target_img - mu_t 62 | t = t.transpose(2,0,1).reshape(3,-1) 63 | Ct = t.dot(t.T) / t.shape[1] + eps * np.eye(t.shape[0]) 64 | mu_s = source_img.mean(0).mean(0) 65 | s = source_img - mu_s 66 | s = s.transpose(2,0,1).reshape(3,-1) 67 | Cs = s.dot(s.T) / s.shape[1] + eps * np.eye(s.shape[0]) 68 | if mode == 'chol': 69 | chol_t = np.linalg.cholesky(Ct) 70 | chol_s = np.linalg.cholesky(Cs) 71 | ts = chol_s.dot(np.linalg.inv(chol_t)).dot(t) 72 | if mode == 'pca': 73 | eva_t, eve_t = np.linalg.eigh(Ct) 74 | Qt = eve_t.dot(np.sqrt(np.diag(eva_t))).dot(eve_t.T) 75 | eva_s, eve_s = np.linalg.eigh(Cs) 76 | Qs = eve_s.dot(np.sqrt(np.diag(eva_s))).dot(eve_s.T) 77 | ts = Qs.dot(np.linalg.inv(Qt)).dot(t) 78 | if mode == 'sym': 79 | eva_t, eve_t = np.linalg.eigh(Ct) 80 | Qt = eve_t.dot(np.sqrt(np.diag(eva_t))).dot(eve_t.T) 81 | Qt_Cs_Qt = Qt.dot(Cs).dot(Qt) 82 | eva_QtCsQt, eve_QtCsQt = np.linalg.eigh(Qt_Cs_Qt) 83 | QtCsQt = eve_QtCsQt.dot(np.sqrt(np.diag(eva_QtCsQt))).dot(eve_QtCsQt.T) 84 | ts = np.linalg.inv(Qt).dot(QtCsQt).dot(np.linalg.inv(Qt)).dot(t) 85 | matched_img = ts.reshape(*target_img.transpose(2,0,1).shape).transpose(1,2,0) 86 | matched_img += mu_s 87 | matched_img[matched_img>1] = 1 88 | matched_img[matched_img<0] = 0 89 | return matched_img 90 | 91 | if cp_mode == 'lum': 92 | style_img = args.style_image 93 | content_img = args.content_image 94 | org_content = args.org_content 95 | org_content = imageio.imread(org_content, pilmode="RGB").astype(float)/256 96 | style_img = imageio.imread(style_img, pilmode="RGB").astype(float)/256 97 | content_img = imageio.imread(content_img, pilmode="RGB").astype(float)/256 98 | 99 | org_content = content_img.copy() 100 | style_img = lum_transform(style_img) 101 | content_img = lum_transform(content_img) 102 | style_img -= style_img.mean(0).mean(0) 103 | style_img += content_img.mean(0).mean(0) 104 | 105 | style_img [style_img < 0 ] = 0 106 | style_img [style_img > 1 ] = 1 107 | 108 | content_img [content_img < 0 ] = 0 109 | content_img [content_img > 1 ] = 1 110 | 111 | imsave(output_content_name, content_img) 112 | imsave(output_style_name, style_img) 113 | elif cp_mode =='match': 114 | style_img = args.style_image 115 | content_img = args.content_image 116 | style_img = imageio.imread(style_img, pilmode="RGB").astype(float)/256 117 | content_img = imageio.imread(content_img, pilmode="RGB").astype(float)/256 118 | 119 | style_img = match_color(style_img, content_img, mode='pca') 120 | 121 | imsave(output_style_name, style_img) 122 | elif cp_mode == 'match_style': 123 | style_img = args.style_image 124 | content_img = args.content_image 125 | style_img = imageio.imread(style_img, pilmode="RGB").astype(float)/256 126 | content_img = imageio.imread(content_img, pilmode="RGB").astype(float)/256 127 | 128 | content_img = match_color(content_img, style_img, mode='pca') 129 | 130 | imsave(output_content_name, content_img) 131 | elif cp_mode == 'lum2': 132 | output = args.output_lum2 133 | org_content = args.org_content 134 | org_content = imageio.imread(org_content, pilmode="RGB").astype(float)/256 135 | output = imageio.imread(output, pilmode="RGB").astype(float)/256 136 | 137 | org_content = skimage.transform.resize(org_content, output.shape) 138 | 139 | org_content = rgb2luv(org_content) 140 | org_content[:,:,0] = output.mean(2) 141 | output = luv2rgb(org_content) 142 | output[output<0] = 0 143 | output[output>1]=1 144 | imsave(output_a_name, output) 145 | else: 146 | raise NameError('Unknown colour preservation mode') 147 | 148 | --------------------------------------------------------------------------------