├── LICENSE ├── README.md ├── content ├── content.jpg ├── content1.jpg └── content2.jpg ├── deepdream ├── 1.jpg ├── 2.jpg └── target.jpg ├── images ├── content0.jpg ├── result.jpg ├── style0.jpg └── target0.jpg ├── method └── method.jpg ├── style ├── style.jpg ├── style1.jpg └── style2.jpg ├── styleTransfer.py └── vgg_para └── README.txt /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 MarTinGuo 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Style-transfer-with-neural-algorithm 2 | 3 | The above code is a simple implementation with TensorFlow of the paper， this paper [Image Style Transfer Using Convolutional Neural Networks](http://openaccess.thecvf.com/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf), which is published in CVPR2016. The idea of one image's style transfer to another image is really cool, and it is very easy to implement with TensorFlow. Meanwhile, training time for one image just cost a few minutes. 4 | 5 | How to use the code 6 | --------------------- 7 | 8 | Firstly, you need these package of python: tensorflow, numpy, scipy, pillow 9 | 10 | just use these commands: pip install tensorflow, pip install numpy, pip install scipy, pip install pillow 11 | 12 | Secondly, you need the pre-trained model of vgg-19, you can download the model from this [address](https://pan.baidu.com/s/1CO-A2GOoym7eCw0hQsvEyw), whose model had removed the fully connected layer to reduce parameters.After you download the model, you should put it in this folder named "vgg_para". 13 | 14 | Method 15 | ------- 16 | 17 | ![algorithm](https://github.com/MingtaoGuo/Style-transfer-with-neural-algorithm/raw/master/method/method.jpg) 18 | 19 | We can see the image above that is from the paper, it shows a simple way to synthesize an image from other style.In this method, x is the variable which we want to update, and it is also an synthesized image as the final result. The squared error is used to control the content which makes the synthesized image is similar to the original content image, and the Gram matrix is used to control the style which makes the synthesized image has the similar style with original style image. 20 | 21 | Result 22 | ----------- 23 | 24 | ![content0](https://github.com/MingtaoGuo/Style-transfer-with-neural-algorithm/raw/master/images/result.jpg) 25 | 26 | This result's parameter: alpha 1e-5, beta 1.0, width 512, height 512, optimizer: L-BFGS, iteration of L-BFGS 500, the result of Adam is not very well. 27 | -------------------------------------------------------------------------------- /content/content.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/content/content.jpg -------------------------------------------------------------------------------- /content/content1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/content/content1.jpg -------------------------------------------------------------------------------- /content/content2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/content/content2.jpg -------------------------------------------------------------------------------- /deepdream/1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/deepdream/1.jpg -------------------------------------------------------------------------------- /deepdream/2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/deepdream/2.jpg -------------------------------------------------------------------------------- /deepdream/target.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/deepdream/target.jpg -------------------------------------------------------------------------------- /images/content0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/images/content0.jpg -------------------------------------------------------------------------------- /images/result.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/images/result.jpg -------------------------------------------------------------------------------- /images/style0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/images/style0.jpg -------------------------------------------------------------------------------- /images/target0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/images/target0.jpg -------------------------------------------------------------------------------- /method/method.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/method/method.jpg -------------------------------------------------------------------------------- /style/style.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/style/style.jpg -------------------------------------------------------------------------------- /style/style1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/style/style1.jpg -------------------------------------------------------------------------------- /style/style2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MingtaoGuo/Style-transfer-with-neural-algorithm/1311e0c69fedf89943790d2d29cda003879e1d28/style/style2.jpg -------------------------------------------------------------------------------- /styleTransfer.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import tensorflow.contrib as contrib 3 | import numpy as np 4 | import scipy.io as sio 5 | import scipy.misc as misc 6 | from PIL import Image 7 | 8 | 9 | def conv(inputs, w, b): 10 | w = tf.constant(w) 11 | b = tf.constant(b) 12 | return tf.nn.conv2d(inputs, w, [1, 1, 1, 1], "SAME") + b 13 | 14 | def mapping(img): 15 | return 255.0 * (img - np.min(img)) / (np.max(img) - np.min(img)) 16 | 17 | class StyleTransfer: 18 | 19 | def __init__(self, H=256, W=256, C=3, alpha=1e-3, beta=1.0, iteration=500, content_path="./content//content.jpg", style_path="./style//style.jpg"): 20 | self.content_img = tf.placeholder("float", [1, H, W, C]) 21 | self.style_img = tf.placeholder("float", [1, H, W, C]) 22 | self.target_img = tf.get_variable("target", shape=[1, H, W, C], initializer=tf.truncated_normal_initializer(stddev=0.02)) 23 | feature_bank_x = self.Network_vgg(self.target_img) 24 | feature_bank_style = self.Network_vgg(self.style_img) 25 | feature_bank_content = self.Network_vgg(self.content_img) 26 | self.L_content = self.content_loss(feature_bank_x, feature_bank_content) 27 | self.L_style = self.style_loss(feature_bank_x, feature_bank_style) 28 | self.total_loss = alpha * self.L_content + beta * self.L_style 29 | # self.Opt = tf.train.AdamOptimizer(0.0002).minimize(self.total_loss) 30 | #L-BFGS 31 | self.optimizer = tf.contrib.opt.ScipyOptimizerInterface(self.total_loss, method='L-BFGS-B',options={'maxiter': iteration, 'disp': 0}) 32 | self.sess = tf.Session() 33 | self.sess.run(tf.global_variables_initializer()) 34 | self.train(H, W, C, content_path, style_path) 35 | 36 | def train(self, H, W, C, content_path, style_path): 37 | content_img = np.reshape(misc.imresize(np.array(Image.open(content_path)), [H, W], mode="RGB"), [1, H, W, C]) 38 | style_img = np.reshape(misc.imresize(np.array(Image.open(style_path)), [H, W], mode="RGB"), [1, H, W, C]) 39 | self.sess.run(tf.assign(self.target_img, content_img), feed_dict={self.content_img: content_img, self.style_img: style_img}) 40 | self.optimizer.minimize(self.sess, feed_dict={self.content_img: content_img, self.style_img: style_img}) 41 | L_content = self.sess.run(self.L_content, feed_dict={self.content_img: content_img, self.style_img: style_img}) 42 | L_style = self.sess.run(self.L_style, feed_dict={self.content_img: content_img, self.style_img: style_img}) 43 | L_total = self.sess.run(self.total_loss, feed_dict={self.content_img: content_img, self.style_img: style_img}) 44 | print("L_content: %g, L_style: %g, L_total: %g" % (L_content, L_style, L_total)) 45 | target_img = self.sess.run(self.target_img,feed_dict={self.content_img: content_img, self.style_img: style_img}) 46 | Image.fromarray(np.uint8(mapping(np.reshape(target_img, [H, W, C])))).save("./deepdream/target.jpg") 47 | 48 | 49 | 50 | 51 | def content_loss(self, feature_bank_x, feature_bank_content): 52 | #content loss 53 | #squared-error 54 | return tf.reduce_sum(tf.square(feature_bank_x["relu4_2"] - feature_bank_content["relu4_2"])) / 2.0 55 | 56 | def style_loss(self, feature_bank_x, feature_bank_style): 57 | #style loss 58 | E = 0 59 | for layer in feature_bank_style.keys(): 60 | if layer == "relu1_1" or layer=="relu2_1" or layer=="relu3_1" or layer=="relu4_1" or layer=="relu5_1": 61 | w = 0.2 62 | else: 63 | w = 0 64 | C = int(feature_bank_x[layer].shape[-1]) 65 | H = int(feature_bank_x[layer].shape[1]) 66 | W = int(feature_bank_x[layer].shape[2]) 67 | F = tf.reshape(tf.transpose(feature_bank_x[layer], [0, 3, 1, 2]), shape=[C, -1]) 68 | #Gram matrix of x 69 | G_x = tf.matmul(F, tf.transpose(F)) 70 | C = int(feature_bank_style[layer].shape[-1]) 71 | F = tf.reshape(tf.transpose(feature_bank_style[layer], [0, 3, 1, 2]), shape=[C, -1]) 72 | #Gram matrix of style 73 | G_s = tf.matmul(F, tf.transpose(F)) 74 | E += w * tf.reduce_sum(tf.square(G_x - G_s)) / (4 * C**2 * H**2 * W**2) 75 | return E 76 | 77 | def Network_vgg(self, inputs): 78 | vgg_para = sio.loadmat("./vgg_para//vgg.mat") 79 | layers = vgg_para["layers"] 80 | feature_bank = {} 81 | with tf.variable_scope("vgg"): 82 | for i in range(37): 83 | if layers[0, i][0, 0]["type"] == "conv": 84 | w = layers[0, i][0, 0]["weights"][0, 0] 85 | b = layers[0, i][0, 0]["weights"][0, 1] 86 | with tf.variable_scope(str(i)): 87 | inputs = conv(inputs, w, b) 88 | elif layers[0, i][0, 0]["type"] == "relu": 89 | inputs = tf.nn.relu(inputs) 90 | feature_bank[layers[0, i][0, 0]["name"][0]] = inputs 91 | else: 92 | inputs = tf.nn.max_pool(inputs, [1, 2, 2, 1], [1, 2, 2, 1], "SAME") 93 | return feature_bank 94 | 95 | if __name__ == "__main__": 96 | st = StyleTransfer(H=512, W=512, C=3, alpha=1e-5, beta=1.0, iteration=500, content_path="./content//content.jpg", style_path="./style//style.jpg") -------------------------------------------------------------------------------- /vgg_para/README.txt: -------------------------------------------------------------------------------- 1 | Please put vgg-19 in this folder. --------------------------------------------------------------------------------