├── LICENSE ├── README.md ├── examples └── inputs │ ├── fast_chicago_starry.gif │ ├── ns_tubingen_scream_seated_picasso.gif │ ├── ns_tubingen_starry_crop_128.gif │ ├── ns_tubingen_starry_crop_256.gif │ ├── ns_tubingen_starry_crop_64.gif │ ├── ns_tubingen_starry_picasso_crop_16-128.gif │ ├── pt_tubingen_scream_crop_32-256.gif │ └── pt_tubingen_seated-nude.gif └── neural_zoom.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 ProGamerGov 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # neural-zoom 2 | 3 | Neural-Zoom works by cropping the output from a style transfer script, and then using the cropped output as either a content image, or an initialization image for the next frame. 4 | 5 | If you are looking for the old version of Neural-Zoom, it can be found [here](https://github.com/ProGamerGov/Neural-Zoom-Legacy). 6 | 7 | 8 | ### Different Crop (Zoom) Values 9 | 10 | By changing the crop value, you can control the "speed" at which the zoom will be. 11 | 12 |

13 |

14 |

15 |

16 |

17 | 18 | ### Different X And Y Axis Crop Values 19 | 20 | You can control the "zoom" for the x axis and y axis separately. 21 | 22 | From the left: "The Scream", and "The Starry Night" + "Picasso Self-Portrait" 23 | 24 |

25 |

26 |

27 |

28 | 29 | 30 | ### Parameters: 31 | 32 | In addition to all the parameters from your chosen style transfer script, neural-zoom has it's own parameters: 33 | 34 | **Basic Options**: 35 | * `-script`: Path to the Lua or Python style transfer script. Currently [Neural-Style](https://github.com/jcjohnson/neural-style), [Fast-Neural-Style](https://github.com/jcjohnson/fast-neural-style), and [Neural-Style-PT](https://github.com/ProGamerGov/neural-style-pt) are supported. 36 | * `-num_frames`: The total number of frames to create. Default is `30`. 37 | 38 | **Zoom Options**: 39 | * `-crop`: How much to crop each frame from all sides. Default is `64`. If both the `-crop_height` and `-crop_width` parameters are greater than zero, this parameter will be ignored. 40 | * `-crop_width`: How much to crop the left and right sides of each frame. 41 | * `-crop_height`: How much to crop the top and bottom sides of each frame. 42 | 43 | **Starting Options**: 44 | * `-starting_image`: Optionally skip creating frame zero, and use your own image instead. 45 | * `-start_num`: The number to start counting from for frame names. Default is `0`. 46 | 47 | **Output options**: 48 | * `-num_zeros`: How many zeros to use for the number portion of each frame name. Default is `0000`. 49 | * `-num_mode`: Whether to use trailing or leading numbers in frame names. If you set this to `0`, leading numbers will be used instead of trailing numbers. 50 | * `-output_dir`: Name of the output image directory. Default is `output_dir`. 51 | * `-verbose`: If this flag is present, then the full set of style transfer parameters used, will be printed for each frame. 52 | 53 | ## Setup 54 | 55 | * Something to convert your frames into the desired media format, like FFMPEG, or Imagemagick. 56 | 57 | * A supported style transfer project. 58 | 59 | You can download Neural-Zoom directly to a style transfer project directory, with: 60 | 61 | ``` 62 | wget -c https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/master/neural_zoom.py 63 | ``` 64 | 65 | ## Usage 66 | 67 | The Neural-Zoom specific parameters will remain the same with each style transfer script, but each script will have it's own different required parameters. 68 | 69 | Below are some basic examples of how to use Neural-Zoom (with either Python 2.7 or Python 3): 70 | 71 | **Neural-Style:** 72 | 73 | ``` 74 | python neural_zoom.py -script neural_style.lua -style_image -content_image -model_file models/VGG_ILSVRC_19_layers.caffemodel -proto_file models/VGG_ILSVRC_19_layers_deploy.prototxt 75 | ``` 76 | 77 | **Fast-Neural-Style:** 78 | 79 | ``` 80 | neural_zoom.py -script fast_neural_style.lua -model -input_image 81 | ``` 82 | 83 | **Neural-Style-PT:** 84 | 85 | ``` 86 | python neural_zoom.py -script neural_style.py -style_image -content_image -model_file models/models/vgg19-d01eb7cb.pth 87 | ``` 88 | 89 | From the left, the style transfer project used is Neural-Style, Fast-Neural-Style, and Neural-Style-PT. 90 | 91 |

92 |

93 |

94 |

95 |

96 | -------------------------------------------------------------------------------- /examples/inputs/fast_chicago_starry.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/fast_chicago_starry.gif -------------------------------------------------------------------------------- /examples/inputs/ns_tubingen_scream_seated_picasso.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/ns_tubingen_scream_seated_picasso.gif -------------------------------------------------------------------------------- /examples/inputs/ns_tubingen_starry_crop_128.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/ns_tubingen_starry_crop_128.gif -------------------------------------------------------------------------------- /examples/inputs/ns_tubingen_starry_crop_256.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/ns_tubingen_starry_crop_256.gif -------------------------------------------------------------------------------- /examples/inputs/ns_tubingen_starry_crop_64.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/ns_tubingen_starry_crop_64.gif -------------------------------------------------------------------------------- /examples/inputs/ns_tubingen_starry_picasso_crop_16-128.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/ns_tubingen_starry_picasso_crop_16-128.gif -------------------------------------------------------------------------------- /examples/inputs/pt_tubingen_scream_crop_32-256.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/pt_tubingen_scream_crop_32-256.gif -------------------------------------------------------------------------------- /examples/inputs/pt_tubingen_seated-nude.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ProGamerGov/Neural-Zoom/0723512e9bc0454aa316081b6d5f4af5f42b5576/examples/inputs/pt_tubingen_seated-nude.gif -------------------------------------------------------------------------------- /neural_zoom.py: -------------------------------------------------------------------------------- 1 | import os 2 | from PIL import Image 3 | import subprocess 4 | 5 | 6 | import argparse 7 | parser = argparse.ArgumentParser() 8 | # Neural-Zoom Options 9 | parser.add_argument("-script", help="the style transfer script", type=str, default='') 10 | parser.add_argument("-num_frames", help="number of frames to produce", type=int, default=30) 11 | 12 | parser.add_argument("-crop", type=int, default=64) 13 | parser.add_argument("-crop_width", type=int, default=0) 14 | parser.add_argument("-crop_height", type=int, default=0) 15 | 16 | parser.add_argument("-starting_image", help="Image to start zooming into", type=str, default='') 17 | parser.add_argument("-start_num", help="Number to start naming from", type=int, default=0) 18 | 19 | parser.add_argument("-num_zeros", help="Number of zeros", type=str, default='0000') 20 | parser.add_argument("-num_mode", help="Whether to use trailing or leading numbers in output image names", type=int, choices=[0, 1], default=1) 21 | parser.add_argument("-output_dir", default='output_dir') 22 | parser.add_argument("-verbose", action='store_true') 23 | 24 | # Basic options 25 | parser.add_argument("-style_image", help="Style target image", default='') 26 | parser.add_argument("-style_blend_weights", default=None) 27 | parser.add_argument("-content_image", help="Content target image", default='') 28 | parser.add_argument("-image_size", help="Maximum height / width of generated image", type=int, default=512) 29 | parser.add_argument("-gpu", help="Zero-indexed ID of the GPU to use; for CPU mode set -gpu = -1 (neural-style) or 'c' for neural-style-pt", default=0) 30 | parser.add_argument("-multigpu_strategy", help="Index of layers to split the network across GPUs", default='') 31 | parser.add_argument("-multidevice_strategy", help="Index of layers to split the network across devices", default='') 32 | 33 | # Optimization options 34 | parser.add_argument("-content_weight", type=float, default=5e0) 35 | parser.add_argument("-style_weight", type=float, default=1e2) 36 | parser.add_argument("-normalize_weights", action='store_true') 37 | parser.add_argument("-tv_weight", type=float, default=1e-3) 38 | parser.add_argument("-num_iterations", type=int, default=1000) 39 | parser.add_argument("-normalize_gradients", action='store_true') 40 | parser.add_argument("-init", choices=['random', 'image'], default='random') 41 | parser.add_argument("-init_image", default=None) 42 | parser.add_argument("-optimizer", choices=['lbfgs', 'adam'], default='lbfgs') 43 | parser.add_argument("-learning_rate", type=float, default=1e0) 44 | parser.add_argument("-lbfgs_num_correction", type=int, default=0) 45 | 46 | # Output options 47 | parser.add_argument("-print_iter", type=int, default=50) 48 | parser.add_argument("-save_iter", type=int, default=0) 49 | parser.add_argument("-output_image", default='out.png') 50 | 51 | # Other options 52 | parser.add_argument("-style_scale", type=float, default=1.0) 53 | parser.add_argument("-original_colors", type=int, default=0) 54 | parser.add_argument("-pooling", choices=['avg', 'max'], default='max') 55 | parser.add_argument("-model_file", type=str, default='models/vgg19-d01eb7cb.pth') 56 | parser.add_argument("-proto_file", type=str, default='') 57 | parser.add_argument("-disable_check", action='store_true') 58 | parser.add_argument("-backend", choices=['nn', 'cuda', 'cudnn', 'clnn', 'mkl', 'cudnn,mkl', 'mkl,cudnn'], default='nn') 59 | parser.add_argument("-cudnn_autotune", action='store_true') 60 | parser.add_argument("-seed", type=int, default=-1) 61 | 62 | parser.add_argument("-content_layers", help="layers for content", default='relu4_2') 63 | parser.add_argument("-style_layers", help="layers for style", default='relu1_1,relu2_1,relu3_1,relu4_1,relu5_1') 64 | 65 | # Fast-Neural-Style options 66 | parser.add_argument("-model", type=str, default='') 67 | parser.add_argument("-median_filter", type=str, default=3) 68 | parser.add_argument("-timing", type=str, default=0) 69 | parser.add_argument("-input_image", type=str, default=1) 70 | parser.add_argument("-use_cudnn", type=str, default=0) 71 | parser.add_argument("-cudnn_benchmark", type=int, default=0) 72 | 73 | params = parser.parse_args() 74 | 75 | 76 | Image.MAX_IMAGE_PIXELS = 1000000000 # Support gigapixel images 77 | 78 | 79 | def main(): 80 | global params_string, tmp_dir 81 | params_string = parameters() 82 | 83 | try: 84 | os.makedirs(params.output_dir) 85 | except OSError: 86 | pass 87 | try: 88 | os.makedirs(params.output_dir + '/tmp') 89 | except OSError: 90 | pass 91 | tmp_dir = params.output_dir + '/tmp/' 92 | 93 | filename = params.output_image 94 | run = howto_run() 95 | 96 | h, w = get_dim() 97 | 98 | # Maybe crop the same relative amount from each side 99 | if params.crop_width == 0 or params.crop_height == 0: 100 | if w != h: 101 | s_size = float(min(h, w)) / float(max(h, w)) 102 | s_size = int(s_size * float(params.crop)) 103 | if h > w: 104 | crop_w, crop_h = s_size, params.crop 105 | elif w > h: 106 | crop_w, crop_h = params.crop, s_size 107 | else: 108 | crop_w, crop_h = params.crop, params.crop 109 | else: 110 | crop_w, crop_h = params.crop_width, params.crop_height 111 | 112 | first_run(run, params.output_image) 113 | 114 | # Process the frames one by one 115 | num = params.start_num + 1 116 | for i in range(params.num_frames - 1): 117 | print('Creating frame number ' + str(num)) 118 | new_image = Image.open(params.output_dir + '/' + zeros(num-1, filename)) 119 | new_image = crop(new_image, w - crop_w, h - crop_h) 120 | new_image.save(tmp_dir + zeros(num-1, filename)) 121 | stylize(run, tmp_dir + zeros(num-1, filename), params.output_dir + '/' + zeros(num, filename)) 122 | os.remove(tmp_dir + zeros(num-1, filename)) 123 | num +=1 124 | 125 | # Final clean up and and correct frame zero 126 | os.rmdir(tmp_dir) 127 | if params.starting_image == '': 128 | image_zero = Image.open(params.output_dir + '/' + zeros(params.start_num, filename)) 129 | image_one = Image.open(params.output_dir + '/' + zeros(params.start_num+1, filename)) 130 | image_zero = image_zero.resize(image_one.size) 131 | os.remove(params.output_dir + '/' + zeros(params.start_num, filename)) 132 | image_zero.save(params.output_dir + '/' + zeros(params.start_num, filename)) 133 | 134 | # Get target image dimensions 135 | def get_dim(): 136 | image = Image.open(params.content_image).convert('RGB') 137 | return [int((float(params.image_size) / max(image.size))*x) for x in (image.height, image.width)] 138 | 139 | # Maybe create the first frame 140 | def first_run(run, output): 141 | if params.starting_image != '': 142 | first_image = Image.open(params.starting_image).convert('RGB') 143 | first_image.save(tmp_dir + zeros(params.start_num -1, output)) 144 | else: 145 | stylize(run, params.content_image, params.output_dir + '/' + zeros(params.start_num, output)) 146 | first_image = Image.open(params.output_dir + '/' + zeros(params.start_num, output)) 147 | first_image.save(tmp_dir + zeros(params.start_num, output)) 148 | 149 | # Run style transfer 150 | def stylize(run, input, output): 151 | if params.init_image != None: 152 | content_param = ' -content_image ' + params.content_image + ' -init_image ' + input 153 | elif params.script == 'fast_neural_style.lua': 154 | content_param = ' -input_image ' + input 155 | else: 156 | content_param = ' -content_image ' + input 157 | cmd = run + content_param + ' -output_image ' + output + params_string 158 | if params.verbose: 159 | print(cmd) 160 | p = subprocess.Popen(cmd, shell=True) 161 | p.wait() 162 | return 163 | 164 | # Add zeros to output image name 165 | def zeros(num, filename): 166 | if params.num_mode == 0: 167 | filename = params.num_zeros[:-len(str(num))] + str(num) + "_" + filename 168 | elif params.num_mode == 1: 169 | filename, ext = os.path.splitext(filename) 170 | filename = filename + "_" + params.num_zeros[:-len(str(num))] + str(num) + ext 171 | return filename 172 | 173 | # Crop the image based on specified values 174 | def crop(image, crop_w, crop_h): 175 | w, h = image.size 176 | crop_dim = (w//2 - crop_w//2, h//2 - crop_h//2, w//2 + crop_w//2, h//2 + crop_h//2) 177 | return image.crop(crop_dim) 178 | 179 | # Figure out how to run the specified style transfer script 180 | def howto_run(): 181 | scriptname, script_ext = os.path.splitext(params.script) 182 | if script_ext == '.py': 183 | from sys import version_info 184 | if version_info[0] < 3: 185 | run = 'python ' 186 | else: 187 | run = 'python3 ' 188 | elif script_ext == '.lua': 189 | run = 'th ' 190 | return run + params.script 191 | 192 | # Remove neural-zoom parameters and ensure that the appropriate style transfer parameters are used 193 | def parameters(): 194 | fast_ns_list = ['model', 'median_filter', 'timing', 'use_cudnn', 'cudnn_benchmark'] 195 | remove_list = ['output_dir', 'num_zeros', 'num_frames', 'script', 'crop', 'crop_width', 'crop_height', 'content_image', 'init_image', 'output_image', 'starting_image', 'start_num', 'num_mode', 'input_image', 'verbose'] 196 | if not params.normalize_gradients: 197 | remove_list.append('normalize_gradients') 198 | if not params.cudnn_autotune: 199 | remove_list.append('cudnn_autotune') 200 | if not params.normalize_weights: 201 | remove_list.append('normalize_weights') 202 | if params.multigpu_strategy == '': 203 | remove_list.append('multigpu_strategy') 204 | if params.multidevice_strategy == '': 205 | remove_list.append('multidevice_strategy') 206 | if params.proto_file == '': 207 | remove_list.append('proto_file') 208 | if params.original_colors == 0: 209 | remove_list.append('original_colors') 210 | if not params.disable_check: 211 | remove_list.append('disable_check') 212 | params_dict = dict(vars(params)) 213 | new_string = '' 214 | 215 | if params.script != 'fast_neural_style.lua': 216 | for arg, value in params_dict.items(): 217 | if arg not in remove_list and arg not in fast_ns_list and value != None: 218 | new_string = new_string + str("-" + arg) + " " + str(value) + " " 219 | elif params.script == 'fast_neural_style.lua': 220 | params.content_image = params.input_image 221 | for arg, value in params_dict.items(): 222 | if arg in fast_ns_list and arg not in remove_list: 223 | if value != None or value !=0: 224 | new_string = new_string + str("-" + arg) + " " + str(value) + " " 225 | return ' ' + new_string 226 | 227 | 228 | if __name__ == "__main__": 229 | main() 230 | --------------------------------------------------------------------------------