├── .gitignore ├── LICENSE ├── README.md ├── faces.py ├── faces ├── __init__.py ├── generate.py ├── instance.py ├── model.py └── train.py ├── img └── example.gif ├── jaffe └── semantic-ratings.csv ├── params ├── drunk.yaml ├── interpolate.yaml ├── random.yaml └── single.yaml └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | RaFD/ 2 | output/ 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2016 Flynn, Michael D. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Generating Faces with Deconvolution Networks 2 | 3 | ![Example generations](img/example.gif) 4 | 5 | This repo contains code to train and interface with a deconvolution network adapted from [this paper][Chairs] to generate faces using data from the [Radboud Faces Database][RaFD]. Requires [Keras][Keras], [NumPy][NumPy], [SciPy][SciPy], and [tqdm][tqdm] with Python 3 to use. 6 | 7 | ## Training New Models 8 | 9 | To train a new model, simply run: 10 | 11 | python3 faces.py train path/to/data 12 | 13 | You can specify the number of deconvolution layers with `-d` to generate larger images, assuming your GPU has the memory for it. You can play with the batch size and the number of kernels per layer (using `-b` and `-k` respectively) until it fits in memory, although this may result in worse results or longer training. 14 | 15 | Using 6 deconvolution layers with a batch size of 8 and the default number of kernels per layer, a model was trained on an Nvidia Titan X card (12 GB) to generate 512x640 images in a little over a day. 16 | 17 | ## Generating Images 18 | 19 | To generate images using a trained model, you can specify parameters in a yaml file and run: 20 | 21 | python3 faces.py generate -m path/to/model -o output/directory -f path/to/params.yaml 22 | 23 | There are four different modes you can use to generate images: 24 | 25 | * `single`, produce a single image. 26 | * `random`, produce a set of random images. 27 | * `drunk`, similar to random, but produces a more contiguous sequence of images. 28 | * `interpolate`, animate between a set of specified keyframes. 29 | 30 | You can find examples of these files in the `params` directory, which should give you a good idea of how to format these and what's available. 31 | 32 | ## Examples 33 | 34 | Interpolating between identities and emotions: 35 | 36 | [![Interpolating between identities and emotions](http://img.youtube.com/vi/UdTq_Q-WgTs/0.jpg)](https://www.youtube.com/watch?v=UdTq_Q-WgTs) 37 | 38 | Interpolating between orientations: (which the model is unable to learn) 39 | 40 | [![Interpolating between orientation](http://img.youtube.com/vi/F4OFkN3EURk/0.jpg)](https://www.youtube.com/watch?v=F4OFkN3EURk) 41 | 42 | Random generations (using "drunk" mode): 43 | 44 | [![Random generations](http://img.youtube.com/vi/vt8zNvJNjSo/0.jpg)](https://www.youtube.com/watch?v=vt8zNvJNjSo) 45 | 46 | [Chairs]: https://arxiv.org/abs/1411.5928 47 | [RaFD]: http://www.socsci.ru.nl:8180/RaFD2/RaFD?p=main 48 | [Keras]: https://keras.io/ 49 | [NumPy]: http://www.numpy.org/ 50 | [SciPy]: https://www.scipy.org/ 51 | [tqdm]: https://github.com/noamraph/tqdm 52 | -------------------------------------------------------------------------------- /faces.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | fg 4 | 5 | CLI for training and interfacing with face generating models. 6 | 7 | """ 8 | 9 | import argparse 10 | import sys 11 | import types 12 | 13 | 14 | # ---- Available commands 15 | 16 | def train(): 17 | """ 18 | Command to train a model. 19 | """ 20 | 21 | parser = argparse.ArgumentParser( 22 | description = "Trains a model using the Radboud Face Database", 23 | usage = "faces []", 24 | ) 25 | parser.add_argument('data', type=str, help= 26 | "Directory where RaFD data lives.") 27 | parser.add_argument('-o', '--output', type=str, default='output', help= 28 | "Directory to output results to.") 29 | 30 | parser.add_argument('-m', '--model', type=str, default='', help= 31 | "The model to load. If none specified, a new model will be made instead.") 32 | parser.add_argument('-b', '--batch-size', type=int, default=16, help= 33 | "Batch size to use while training.") 34 | parser.add_argument('-e', '--num-epochs', type=int, default=100, help= 35 | "The number of epochs to train.") 36 | parser.add_argument('-opt', '--optimizer', type=str, default='adam', help= 37 | "Optimizer to use, must be a valid optimizer included in Keras.") 38 | parser.add_argument('-d', '--deconv-layers', type=int, default=5, help= 39 | "The number of deconvolution layers to include in the model.") 40 | parser.add_argument('-k', '--kernels-per-layer', type=int, nargs='+', help= 41 | "The number of kernels to include in each layer.") 42 | 43 | parser.add_argument('-v', '--visualize', action='store_true', help= 44 | "Output intermediate results after each epoch.") 45 | parser.add_argument('--use-yalefaces', action='store_true', help= 46 | "Use YaleFaces data instead of RaFD") 47 | parser.add_argument('--use-jaffe', action='store_true', help= 48 | "Use JAFFE data instead of RaFD") 49 | 50 | args = parser.parse_args(sys.argv[2:]) 51 | 52 | 53 | import faces.train 54 | 55 | if args.deconv_layers > 6: 56 | print("Warning: Having more than 6 deconv layers will create images " 57 | "larger than the original data! (and may not fit in memory)") 58 | 59 | faces.train.train_model(args.data, args.output, args.model, 60 | batch_size = args.batch_size, 61 | num_epochs = args.num_epochs, 62 | optimizer = args.optimizer, 63 | deconv_layers = args.deconv_layers, 64 | kernels_per_layer = args.kernels_per_layer, 65 | generate_intermediate = args.visualize, 66 | use_yale = args.use_yalefaces, 67 | use_jaffe = args.use_jaffe, 68 | verbose = True, 69 | ) 70 | 71 | 72 | def generate(): 73 | """ 74 | Command to generate faces with a trained model. 75 | """ 76 | 77 | parser = argparse.ArgumentParser( 78 | description = "Generate faces using a trained model.", 79 | usage = "faces []", 80 | ) 81 | parser.add_argument('-m', '--model', type=str, required=True, help= 82 | "Model definition file to use.") 83 | parser.add_argument('-o', '--output', type=str, required=True, help= 84 | "Directory to output results to.") 85 | parser.add_argument('-f', '--gen-file', type=str, required=True, help= 86 | "YAML file that specifies the parameters to generate.") 87 | parser.add_argument('-b', '--batch_size', type=int, default=64, help= 88 | "Batch size to use while generating images.") 89 | parser.add_argument('-ext', '--extension', type=str, default='jpg', help= 90 | "Image file extension to use when saving images.") 91 | 92 | args = parser.parse_args(sys.argv[2:]) 93 | 94 | import faces.generate 95 | 96 | faces.generate.generate_from_yaml(args.gen_file, args.model, args.output, 97 | batch_size=args.batch_size, extension=args.extension) 98 | 99 | 100 | # ---- Command-line invocation 101 | 102 | if __name__ == '__main__': 103 | 104 | # Use all functions defined in this file as possible commands to run 105 | cmd_fns = [x for x in locals().values() if isinstance(x, types.FunctionType)] 106 | cmd_names = sorted([fn.__name__ for fn in cmd_fns]) 107 | cmd_dict = {fn.__name__: fn for fn in cmd_fns} 108 | 109 | parser = argparse.ArgumentParser( 110 | description = "Generate faces using a deconvolution network.", 111 | usage = "faces []" 112 | ) 113 | parser.add_argument('command', type=str, help= 114 | "Command to run. Available commands: {}.".format(cmd_names)) 115 | 116 | args = parser.parse_args([sys.argv[1]]) 117 | 118 | cmd = None 119 | try: 120 | cmd = cmd_dict[args.command] 121 | except KeyError: 122 | sys.stderr.write('\033[91m') 123 | sys.stderr.write("\nInvalid command {}!\n\n".format(args.command)) 124 | sys.stderr.write('\033[0m') 125 | sys.stderr.flush() 126 | 127 | parser.print_help() 128 | 129 | if cmd is not None: 130 | cmd() 131 | -------------------------------------------------------------------------------- /faces/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | faces/__init__.py 3 | 4 | Initialize the facegen package. 5 | 6 | """ 7 | 8 | -------------------------------------------------------------------------------- /faces/generate.py: -------------------------------------------------------------------------------- 1 | """ 2 | faces/generate.py 3 | 4 | Methods for generating faces. 5 | 6 | """ 7 | 8 | import os 9 | import yaml 10 | 11 | import numpy as np 12 | from scipy import interpolate 13 | import scipy.misc 14 | from tqdm import tqdm 15 | 16 | from .instance import Emotion, NUM_YALE_POSES 17 | 18 | class GenParser: 19 | """ 20 | Class to parse and create inputs based on the parameters in a yaml file. 21 | """ 22 | 23 | # Default parameters to use 24 | DefaultParams = { 25 | 'mode' : 'single', 26 | 'constrained' : True, 27 | 'id' : None, 28 | 'em' : None, 29 | 'or' : None, 30 | 'ps' : None, 31 | 'lt' : None, 32 | 'id_scale' : 1.0, 33 | 'id_step' : 0.1, 34 | 'id_min' : None, 35 | 'id_max' : None, 36 | 'em_scale' : 1.0, 37 | 'em_step' : 0.1, 38 | 'em_min' : None, 39 | 'em_max' : None, 40 | 'or_scale' : 1.0, 41 | 'or_step' : 0.1, 42 | 'or_min' : None, 43 | 'or_max' : None, 44 | 'ps_scale' : 1.0, 45 | 'ps_step' : 0.1, 46 | 'ps_min' : None, 47 | 'ps_max' : None, 48 | 'lt_scale' : 1.0, 49 | 'lt_step' : 0.1, 50 | 'lt_min' : None, 51 | 'lt_max' : None, 52 | 'num_images' : '1s', 53 | 'fps' : 30, 54 | 'keyframes' : None, 55 | } 56 | 57 | def __init__(self, yaml_path): 58 | self.yaml_file = open(yaml_path, 'r') 59 | 60 | self.modes = { 61 | 'single' : self.mode_single, 62 | 'random' : self.mode_random, 63 | 'drunk' : self.mode_drunk, 64 | 'interpolate': self.mode_interpolate, 65 | } 66 | 67 | def __del__(self): 68 | self.yaml_file.close() 69 | 70 | 71 | # Methods for generating inputs by mode 72 | 73 | def mode_single(self, params): 74 | """ 75 | Generate network inputs for a single image. 76 | """ 77 | 78 | if params['id'] is None: 79 | params['id'] = 0 80 | if params['em'] is None: 81 | params['em'] = 'neutral' 82 | if params['or'] is None: 83 | params['or'] = 0 84 | if params['ps'] is None: 85 | params['ps'] = 0 86 | if params['lt'] is None: 87 | params['lt'] = 0 88 | 89 | if params['dataset'] == 'YALE': 90 | inputs = { 91 | 'identity': np.empty((1, params['num_ids'])), 92 | 'pose': np.empty((1, NUM_YALE_POSES)), 93 | 'lighting': np.empty((1, 4)), 94 | } 95 | inputs['identity'][0,:] = self.identity_vector(params['id'], params) 96 | inputs['pose'][0,:] = self.pose_vector(params['ps'], params) 97 | inputs['lighting'][0,:] = self.lighting_vector(params['lt'], params) 98 | else: 99 | inputs = { 100 | 'identity': np.empty((1, params['num_ids'])), 101 | 'emotion': np.empty((1, Emotion.length())), 102 | 'orientation': np.empty((1, 2)), 103 | } 104 | 105 | inputs['identity'][0,:] = self.identity_vector(params['id'], params) 106 | inputs['emotion'][0,:] = self.emotion_vector(params['em'], params) 107 | inputs['orientation'][0,:] = self.orientation_vector(params['or'], params) 108 | 109 | return inputs 110 | 111 | 112 | def mode_random(self, params): 113 | """ 114 | Generate random network inputs. 115 | """ 116 | 117 | num_images = self.num_frames(params['num_images'], params) 118 | 119 | if params['dataset'] == 'YALE': 120 | inputs = { 121 | 'identity': np.empty((num_images, params['num_ids'])), 122 | 'pose': np.empty((num_images, NUM_YALE_POSES)), 123 | 'lighting': np.empty((num_images, 4)), 124 | } 125 | else: 126 | inputs = { 127 | 'identity': np.empty((num_images, params['num_ids'])), 128 | 'emotion': np.empty((num_images, Emotion.length())), 129 | 'orientation': np.empty((num_images, 2)), 130 | } 131 | 132 | for i in range(0, num_images): 133 | if params['id'] is None: 134 | inputs['identity'][i,:] = self.random_identity(params) 135 | else: 136 | inputs['identity'][i,:] = self.identity_vector(params['id'], params) 137 | 138 | if params['dataset'] == "YALE": 139 | if params['ps'] is None: 140 | inputs['pose'][i,:] = self.random_pose(params) 141 | else: 142 | inputs['pose'][i,:] = self.pose_vector(params['ps'], params) 143 | 144 | if params['lt'] is None: 145 | inputs['lighting'][i,:], _ = self.random_lighting(params) 146 | else: 147 | inputs['lighting'][i,:] = self.lighting_vector(params['lt'], params) 148 | else: 149 | if params['em'] is None: 150 | inputs['emotion'][i,:] = self.random_emotion(params) 151 | else: 152 | inputs['emotion'][i,:] = self.emotion_vector(params['em'], params) 153 | 154 | if params['or'] is None: 155 | inputs['orientation'][i,:], _ = self.random_orientation(params) 156 | else: 157 | inputs['orientation'][i,:] = self.orientation_vector(params['or'], params) 158 | 159 | return inputs 160 | 161 | 162 | def mode_drunk(self, params): 163 | """ 164 | Generate "drunk" network inputs, random vectors created by randomly 165 | shifting the last vector. 166 | """ 167 | 168 | num_images = self.num_frames(params['num_images'], params) 169 | 170 | if params['dataset'] == "YALE": 171 | inputs = { 172 | 'identity': np.empty((num_images, params['num_ids'])), 173 | 'pose': np.empty((num_images, NUM_YALE_POSES)), 174 | 'lighting': np.empty((num_images, 4)), 175 | } 176 | else: 177 | inputs = { 178 | 'identity': np.empty((num_images, params['num_ids'])), 179 | 'emotion': np.empty((num_images, Emotion.length())), 180 | 'orientation': np.empty((num_images, 2)), 181 | } 182 | 183 | last_id, last_em, last_or, last_ps, last_lt = None, None, None, None, None 184 | 185 | for i in range(0, num_images): 186 | if params['id'] is None: 187 | inputs['identity'][i,:] = self.random_identity(params, last_id) 188 | last_id = inputs['identity'][i,:] 189 | else: 190 | inputs['identity'][i,:] = self.identity_vector(params['id'], params) 191 | 192 | if params['dataset'] == "YALE": 193 | if params['ps'] is None: 194 | inputs['pose'][i,:] = self.random_pose(params, last_ps) 195 | last_ps = inputs['pose'][i,:] 196 | else: 197 | inputs['pose'][i,:] = self.pose_vector(params['ps'], params) 198 | 199 | if params['lt'] is None: 200 | inputs['lighting'][i,:], last_lt = self.random_lighting(params, last_lt) 201 | else: 202 | inputs['lighting'][i,:] = self.lighting_vector(params['lt'], params) 203 | else: 204 | if params['em'] is None: 205 | inputs['emotion'][i,:] = self.random_emotion(params, last_em) 206 | last_em = inputs['emotion'][i,:] 207 | else: 208 | inputs['emotion'][i,:] = self.emotion_vector(params['em'], params) 209 | 210 | if params['or'] is None: 211 | inputs['orientation'][i,:], last_or = self.random_orientation(params, last_or) 212 | else: 213 | inputs['orientation'][i,:] = self.orientation_vector(params['or'], params) 214 | 215 | return inputs 216 | 217 | 218 | def mode_interpolate(self, params): 219 | """ 220 | Generate network inputs that interpolate between keyframes. 221 | """ 222 | 223 | use_yale = params['dataset'] == "YALE" 224 | 225 | # Set starting/default values 226 | id_val = params['id'] if params['id'] is not None else 0 227 | if use_yale: 228 | ps_val = params['ps'] if params['ps'] is not None else 0 229 | lt_val = params['lt'] if params['lt'] is not None else 0 230 | else: 231 | em_val = params['em'] if params['em'] is not None else 0 232 | or_val = params['or'] if params['or'] is not None else 0 233 | 234 | # List of all id/em/or vectors for each keyframe 235 | id_keyframes = list() 236 | if use_yale: 237 | ps_keyframes = list() 238 | lt_keyframes = list() 239 | else: 240 | em_keyframes = list() 241 | or_keyframes = list() 242 | 243 | keyframe_indicies = list() 244 | 245 | 246 | frame_index = None 247 | 248 | for keyframe_params in params['keyframes']: 249 | 250 | # Get new parameters, otherwise use values from the last keyframe 251 | if 'id' in keyframe_params: id_val = keyframe_params['id'] 252 | if use_yale: 253 | if 'ps' in keyframe_params: ps_val = keyframe_params['ps'] 254 | if 'lt' in keyframe_params: lt_val = keyframe_params['lt'] 255 | else: 256 | if 'em' in keyframe_params: em_val = keyframe_params['em'] 257 | if 'or' in keyframe_params: or_val = keyframe_params['or'] 258 | 259 | # Determine which frame index this is in the animation 260 | if frame_index is None: 261 | frame_index = 0 262 | else: 263 | if 'length' not in keyframe_params: 264 | raise RuntimeError("A length must be specified for every " 265 | "keyframe except the first") 266 | frame_index += self.num_frames(keyframe_params['length'], params) 267 | 268 | # Create input vectors for this keyframe 269 | id_keyframes.append( self.identity_vector(id_val, params) ) 270 | if use_yale: 271 | ps_keyframes.append( self.pose_vector(ps_val, params) ) 272 | lt_keyframes.append( self.lighting_vector(lt_val, params) ) 273 | else: 274 | em_keyframes.append( self.emotion_vector(em_val, params) ) 275 | or_keyframes.append( self.orientation_vector(or_val, params) ) 276 | 277 | keyframe_indicies.append( frame_index ) 278 | 279 | # Convert python lists to numpy arrays 280 | id_keyframes = np.vstack(id_keyframes) 281 | if use_yale: 282 | ps_keyframes = np.vstack(ps_keyframes) 283 | lt_keyframes = np.vstack(lt_keyframes) 284 | else: 285 | em_keyframes = np.vstack(em_keyframes) 286 | or_keyframes = np.vstack(or_keyframes) 287 | 288 | keyframe_indicies = np.array(keyframe_indicies) 289 | 290 | num_frames = keyframe_indicies[-1]+1 291 | 292 | # Interpolate 293 | if use_yale: 294 | id_idx = np.arange(0, params['num_ids']) 295 | ps_idx = np.arange(0, NUM_YALE_POSES) 296 | lt_idx = np.arange(0, 4) 297 | else: 298 | id_idx = np.arange(0, params['num_ids']) 299 | em_idx = np.arange(0, Emotion.length()) 300 | or_idx = np.arange(0, 2) 301 | 302 | f_id = interpolate.interp2d(id_idx, keyframe_indicies, id_keyframes) 303 | if use_yale: 304 | f_ps = interpolate.interp2d(ps_idx, keyframe_indicies, ps_keyframes) 305 | f_lt = interpolate.interp2d(lt_idx, keyframe_indicies, lt_keyframes) 306 | else: 307 | f_em = interpolate.interp2d(em_idx, keyframe_indicies, em_keyframes) 308 | f_or = interpolate.interp2d(or_idx, keyframe_indicies, or_keyframes) 309 | 310 | if use_yale: 311 | return { 312 | 'identity': f_id(id_idx, np.arange(0, num_frames)), 313 | 'pose': f_ps(ps_idx, np.arange(0, num_frames)), 314 | 'lighting': f_lt(lt_idx, np.arange(0, num_frames)), 315 | } 316 | else: 317 | return { 318 | 'identity': f_id(id_idx, np.arange(0, num_frames)), 319 | 'emotion': f_em(em_idx, np.arange(0, num_frames)), 320 | 'orientation': f_or(or_idx, np.arange(0, num_frames)), 321 | } 322 | 323 | 324 | # Helper methods 325 | 326 | def num_frames(self, val, params): 327 | """ Gets the number of frames for a value. """ 328 | 329 | if isinstance(val, int): 330 | return val 331 | elif isinstance(val, str): 332 | if val.endswith('s'): 333 | return int( float(val[:-1]) * params['fps'] ) 334 | else: 335 | raise RuntimeError("Length '{}' not understood".format(val)) 336 | else: 337 | raise RuntimeError("Length '{}' not understood".format(val)) 338 | 339 | 340 | def identity_vector(self, value, params): 341 | """ Create an identity vector for a provided value. """ 342 | 343 | if isinstance(value, str): 344 | if '+' not in value: 345 | raise RuntimeError("Identity '{}' not understood".format(value)) 346 | 347 | try: 348 | values = [int(x) for x in value.split('+')] 349 | except: 350 | raise RuntimeError("Identity '{}' not understood".format(value)) 351 | elif isinstance(value, int): 352 | values = [value] 353 | else: 354 | raise RuntimeError("Identity '{}' not understood".format(value)) 355 | 356 | vec = np.zeros((params['num_ids'],)) 357 | for val in values: 358 | if val < 0 or params['num_ids'] <= val: 359 | raise RuntimeError("Identity '{}' invalid".format(val)) 360 | vec[val] += 1.0 361 | 362 | return self.constrain(vec, params['constrained'], params['id_scale'], 363 | params['id_min'], params['id_max']) 364 | 365 | 366 | def emotion_vector(self, value, params): 367 | """ Create an emotion vector for a provided value. """ 368 | 369 | if not isinstance(value, str): 370 | raise RuntimeError("Emotion '{}' not understood".format(value)) 371 | 372 | if '+' in value: 373 | values = value.split('+') 374 | else: 375 | values = [value] 376 | 377 | vec = np.zeros((Emotion.length(),)) 378 | for emotion in values: 379 | try: 380 | vec += getattr(Emotion, emotion) 381 | except AttributeError: 382 | raise RuntimeError("Emotion '{}' is invalid".format(emotion)) 383 | 384 | return self.constrain(vec, params['constrained'], params['em_scale'], 385 | params['em_min'], params['em_max']) 386 | 387 | 388 | def orientation_vector(self, value, params): 389 | """ Create an orientation vector for a provided value. """ 390 | 391 | if isinstance(value, int) or isinstance(value, float): 392 | value = np.deg2rad(value) 393 | return np.array([np.sin(value), np.cos(value)]) 394 | 395 | elif isinstance(value, str): 396 | if params['constrained']: 397 | raise RuntimeError("Cannot manually set orientation vector " 398 | "values when constrained is set to True") 399 | 400 | values = value.split() 401 | if len(values) != 2: 402 | raise RuntimeError("Orientation '{}' not understood".format(value)) 403 | 404 | vec = np.empty((2,)) 405 | try: 406 | vec[0] = float(values[0]) 407 | vec[1] = float(values[1]) 408 | except ValueError: 409 | raise RuntimeError("Orientation '{}' not understood".format(value)) 410 | 411 | return vec 412 | else: 413 | raise RuntimeError("Orientation '{}' not understood".format(value)) 414 | 415 | 416 | def pose_vector(self, value, params): 417 | """ Create an pose vector for a provided value. """ 418 | 419 | if isinstance(value, str): 420 | if '+' not in value: 421 | raise RuntimeError("Pose '{}' not understood".format(value)) 422 | 423 | try: 424 | values = [int(x) for x in value.split('+')] 425 | except: 426 | raise RuntimeError("Pose '{}' not understood".format(value)) 427 | elif isinstance(value, int): 428 | values = [value] 429 | else: 430 | raise RuntimeError("Pose '{}' not understood".format(value)) 431 | 432 | vec = np.zeros((NUM_YALE_POSES,)) 433 | for val in values: 434 | if val < 0 or NUM_YALE_POSES <= val: 435 | raise RuntimeError("Pose '{}' invalid".format(val)) 436 | vec[val] += 1.0 437 | 438 | return self.constrain(vec, params['constrained'], params['ps_scale'], 439 | params['ps_min'], params['ps_max']) 440 | 441 | 442 | def lighting_vector(self, value, params): 443 | """ Create a lighting vector for a provided value. """ 444 | 445 | if isinstance(value, int) or isinstance(value, float): 446 | value = np.deg2rad(value) 447 | return np.array([np.sin(value), np.cos(value), np.sin(value), np.cos(value)]) 448 | 449 | elif isinstance(value, str): 450 | 451 | values = value.split() 452 | if len(values) != 2: 453 | raise RuntimeError("Lighting '{}' not understood".format(value)) 454 | 455 | vec = np.empty((4,)) 456 | try: 457 | # First element is azimuth 458 | vec[0] = np.sin(float(values[0])) 459 | vec[1] = np.cos(float(values[0])) 460 | # Second element is elevation 461 | vec[2] = np.sin(float(values[1])) 462 | vec[3] = np.cos(float(values[1])) 463 | except ValueError: 464 | raise RuntimeError("Lighting '{}' not understood".format(value)) 465 | 466 | return vec 467 | else: 468 | raise RuntimeError("Lighting '{}' not understood".format(value)) 469 | 470 | 471 | def random_identity(self, params, start=None): 472 | """ Create a random identity vector. """ 473 | 474 | step = params['id_step'] 475 | 476 | if start is None: 477 | vec = 2*(np.random.rand(params['num_ids'])-0.5) 478 | else: 479 | vec = start + (2*step*np.random.rand(params['num_ids'])-step) 480 | 481 | return self.constrain(vec, params['constrained'], params['id_scale'], 482 | params['id_min'], params['id_max']) 483 | 484 | 485 | def random_emotion(self, params, start=None): 486 | """ Create a random emotion vector. """ 487 | 488 | step = params['em_step'] 489 | 490 | if start is None: 491 | vec = 2*(np.random.rand(Emotion.length())-0.5) 492 | else: 493 | vec = start + (2*step*np.random.rand(Emotion.length())-step) 494 | 495 | return self.constrain(vec, params['constrained'], params['em_scale'], 496 | params['em_min'], params['em_max']) 497 | 498 | 499 | def random_orientation(self, params, start=None): 500 | """ Create a random orientation vector. """ 501 | 502 | step = params['or_step'] 503 | 504 | if params['constrained']: 505 | if start is None: 506 | angle = 180*np.random.rand() - 90 507 | else: 508 | angle = start + step * (180*np.random.rand()-90) 509 | rad = np.deg2rad(angle) 510 | 511 | # Return the angle as a second argument so the caller can grab it 512 | # in case it's in the drunk mode 513 | return np.array([np.sin(rad), np.cos(rad)]), angle 514 | else: 515 | if start is None: 516 | vec = 2*np.random.rand(2) - 1 517 | else: 518 | vec = start + (2*step*np.random.rand(2)-step) 519 | 520 | vec = self.constrain(vec, params['constrained'], params['or_scale'], 521 | params['or_min'], params['or_max']) 522 | 523 | # Return the vector twice so it behaves the same as constrained 524 | return vec, vec 525 | 526 | 527 | def random_pose(self, params, start=None): 528 | """ Create a random pose vector. """ 529 | 530 | step = params['ps_step'] 531 | 532 | if start is None: 533 | vec = 2*(np.random.rand(NUM_YALE_POSES)-0.5) 534 | else: 535 | vec = start + (2*step*np.random.rand(NUM_YALE_POSES)-step) 536 | 537 | return self.constrain(vec, params['constrained'], params['ps_scale'], 538 | params['ps_min'], params['ps_max']) 539 | 540 | 541 | def random_lighting(self, params, start=None): 542 | """ Create a random lighting vector. """ 543 | 544 | step = params['lt_step'] 545 | 546 | if params['constrained']: 547 | if start is None: 548 | azimuth = 180*np.random.rand() - 90 549 | elevation = 180*np.random.rand() - 90 550 | else: 551 | azimuth = start[0] + step * (180*np.random.rand()-90) 552 | elevation = start[1] + step * (180*np.random.rand()-90) 553 | azrad = np.deg2rad(azimuth) 554 | elrad = np.deg2rad(elevation) 555 | 556 | # Return the angle as a second argument so the caller can grab it 557 | # in case it's in the drunk mode 558 | return np.array([np.sin(azrad), np.cos(azrad), np.sin(elrad), 559 | np.cos(elrad)]), (azimuth, elevation) 560 | else: 561 | if start is None: 562 | vec = 2*np.random.rand(4) - 1 563 | else: 564 | vec = start + (2*step*np.random.rand(4)-step) 565 | 566 | vec = self.constrain(vec, params['constrained'], params['lt_scale'], 567 | params['lt_min'], params['lt_max']) 568 | 569 | # Return the vector twice so it behaves the same as constrained 570 | return vec, vec 571 | 572 | 573 | def constrain(self, vec, constrained, scale, vec_min, vec_max): 574 | """ Constrains the emotion vector based on params. """ 575 | 576 | if constrained: 577 | vec = vec / np.linalg.norm(vec) 578 | 579 | if scale is not None: 580 | vec = vec * scale 581 | 582 | if vec_min is not None and vec_max is not None: 583 | vec = np.clip(vec, vec_min, vec_max) 584 | 585 | return vec 586 | 587 | 588 | # Main parsing method 589 | 590 | def parse_params(self): 591 | """ 592 | Parses the yaml file and creates input vectors to use with the model. 593 | """ 594 | 595 | self.yaml_file.seek(0) 596 | 597 | yaml_params = yaml.load(self.yaml_file) 598 | 599 | params = GenParser.DefaultParams 600 | 601 | for field in params.keys(): 602 | if field in yaml_params: 603 | params[field] = yaml_params[field] 604 | 605 | return params 606 | 607 | def gen_inputs(self, params): 608 | """ 609 | creates input vectors to use with the model. 610 | """ 611 | 612 | fn = None 613 | try: 614 | fn = self.modes[ params['mode'] ] 615 | except KeyError: 616 | raise RuntimeError("Mode '{}' is invalid".format(params['mode'])) 617 | 618 | return fn(params) 619 | 620 | 621 | 622 | def generate_from_yaml(yaml_path, model_path, output_dir, batch_size=32, 623 | extension='jpg'): 624 | """ 625 | Generate images based on parameters specified in a yaml file. 626 | """ 627 | 628 | from keras import backend as K 629 | from keras.models import load_model 630 | 631 | print("Loading model...") 632 | 633 | model = load_model(model_path) 634 | num_ids = model.input_shape[0][1] # XXX is there a nicer way to get this? 635 | dataset = os.path.basename(model_path).split('.')[1] 636 | 637 | parser = GenParser(yaml_path) 638 | 639 | try: 640 | params = parser.parse_params() 641 | except RuntimeError as e: 642 | print("Error: Unable to parse '{}'. Encountered exception:".format(yaml_path)) 643 | print(e) 644 | return 645 | params['dataset'] = dataset 646 | params['num_ids'] = num_ids 647 | inputs = parser.gen_inputs(params) 648 | 649 | if not os.path.exists(output_dir): 650 | os.makedirs(output_dir) 651 | else: 652 | raise RuntimeError( 653 | "Directory '{}' exists. Cowardly refusing to continue." 654 | .format(output_dir)) 655 | 656 | print("Generating images...") 657 | 658 | num_images = inputs['identity'].shape[0] 659 | count = 0 660 | 661 | for idx in tqdm(range(0, num_images, batch_size)): 662 | 663 | if dataset == "YALE": 664 | batch = { 665 | 'identity': inputs['identity'][idx:idx+batch_size,:], 666 | 'pose': inputs['pose'] [idx:idx+batch_size,:], 667 | 'lighting': inputs['lighting'][idx:idx+batch_size,:], 668 | } 669 | else: 670 | batch = { 671 | 'identity': inputs['identity'] [idx:idx+batch_size,:], 672 | 'emotion': inputs['emotion'] [idx:idx+batch_size,:], 673 | 'orientation': inputs['orientation'][idx:idx+batch_size,:], 674 | } 675 | 676 | gen = model.predict_on_batch(batch) 677 | 678 | for i in range(0, gen.shape[0]): 679 | if K.image_dim_ordering() == 'th': 680 | if dataset == "YALE": 681 | image[:,:] = gen[i,0,:,:] 682 | else: 683 | image = np.empty(gen.shape[2:]+(3,)) 684 | for x in range(0, 3): 685 | image[:,:,x] = gen[i,x,:,:] 686 | else: 687 | if dataset == "YALE" or dataset == "JAFFE": 688 | image = gen[i,:,:,0] 689 | else: 690 | image = gen[i,:,:,:] 691 | image = np.array(255*np.clip(image,0,1), dtype=np.uint8) 692 | file_path = os.path.join(output_dir, '{:05}.{}'.format(count, extension)) 693 | scipy.misc.imsave(file_path, image) 694 | count += 1 695 | -------------------------------------------------------------------------------- /faces/instance.py: -------------------------------------------------------------------------------- 1 | """ 2 | faces/instance.py 3 | 4 | Instance class to hold data for each example. 5 | 6 | """ 7 | 8 | import os 9 | import csv 10 | 11 | from keras import backend as K 12 | 13 | import numpy as np 14 | import scipy.misc as misc 15 | from tqdm import tqdm 16 | 17 | 18 | NUM_YALE_POSES = 10 19 | 20 | 21 | # ---- Enum classes for vector descriptions 22 | 23 | class Emotion: 24 | angry = [1., 0., 0., 0., 0., 0., 0., 0.] 25 | contemptuous = [0., 1., 0., 0., 0., 0., 0., 0.] 26 | disgusted = [0., 0., 1., 0., 0., 0., 0., 0.] 27 | fearful = [0., 0., 0., 1., 0., 0., 0., 0.] 28 | happy = [0., 0., 0., 0., 1., 0., 0., 0.] 29 | neutral = [0., 0., 0., 0., 0., 1., 0., 0.] 30 | sad = [0., 0., 0., 0., 0., 0., 1., 0.] 31 | surprised = [0., 0., 0., 0., 0., 0., 0., 1.] 32 | 33 | @classmethod 34 | def length(cls): 35 | return len(Emotion.neutral) 36 | 37 | 38 | # ---- Loading functions 39 | 40 | class RaFDInstances: 41 | 42 | def __init__(self, directory): 43 | """ 44 | Constructor for a RaFDInstances object. 45 | 46 | Args: 47 | directory (str): Directory where the data lives. 48 | """ 49 | 50 | self.directory = directory 51 | 52 | # A list of all files in the current directory (no kids, only frontal gaze) 53 | self.filenames = [x for x in os.listdir(directory) 54 | if 'Kid' not in x and 'frontal' in x] 55 | 56 | # The number of times the directory has been read over 57 | self.num_iterations = 0 58 | 59 | # Count identities and map each identity present to a contiguous value 60 | identities = list() 61 | for filename in self.filenames: 62 | identity = int(filename.split('_')[1])-1 # Identities are 1-indexed 63 | if identity not in identities: 64 | identities.append(identity) 65 | self.identity_map = dict() 66 | for idx, identity in enumerate(identities): 67 | self.identity_map[identity] = idx 68 | 69 | self.num_identities = len(self.identity_map) 70 | self.num_instances = len(self.filenames) 71 | 72 | 73 | def load_data(self, image_size, verbose=False): 74 | """ 75 | Loads RaFD data for training. 76 | 77 | Args: 78 | image_size (tuple): Size images should be resized to. 79 | Returns: 80 | numpy.ndarray, training data (face parameters). 81 | numpy.ndarray, output data (the actual images to generate). 82 | """ 83 | 84 | inputs = { 85 | 'emotion' : np.empty((self.num_instances, len(Emotion.neutral))), 86 | 'identity' : np.empty((self.num_instances, self.num_identities)), 87 | 'orientation': np.empty((self.num_instances, 2)), 88 | } 89 | 90 | if K.image_dim_ordering() == 'th': 91 | outputs = np.empty((self.num_instances, 3)+image_size) 92 | else: 93 | outputs = np.empty((self.num_instances,)+image_size+(3,)) 94 | 95 | all_instances = range(0, len(self.filenames)) 96 | if verbose: 97 | all_instances = tqdm(all_instances) 98 | 99 | for i in all_instances: 100 | instance = RaFDInstance(self.directory, self.filenames[i], image_size) 101 | 102 | inputs['emotion'][i,:] = instance.emotion 103 | inputs['identity'][i,:] = instance.identity_vector(self.identity_map) 104 | inputs['orientation'][i,:] = instance.orientation 105 | 106 | if K.image_dim_ordering() == 'th': 107 | outputs[i,:,:,:] = instance.th_image() 108 | else: 109 | outputs[i,:,:,:] = instance.tf_image() 110 | 111 | return inputs, outputs 112 | 113 | 114 | class YaleInstances: 115 | 116 | def __init__(self, directory): 117 | """ 118 | Constructor for a YaleInstances object. 119 | 120 | Args: 121 | directory (str): Directory where the data lives. 122 | """ 123 | 124 | self.directory = directory 125 | 126 | subdirs = [x for x in os.listdir(directory) if 'yaleB' in x] 127 | 128 | self.num_identities = len(subdirs) 129 | self.identity_map = dict() 130 | for idx, subdir in enumerate(sorted(subdirs)): 131 | identity = int(subdir[5:7]) 132 | self.identity_map[identity] = idx 133 | 134 | self.filenames = list() 135 | 136 | for subdir in subdirs: 137 | path = os.path.join(directory, subdir) 138 | self.filenames.extend( 139 | [os.path.join(subdir,x) for x in os.listdir(path) 140 | if 'pgm' in x 141 | and 'Ambient' not in x] 142 | ) 143 | 144 | self.num_instances = len(self.filenames) 145 | 146 | 147 | def load_data(self, image_size, verbose=False): 148 | """ 149 | Loads YaleFaces data for training. 150 | 151 | Args: 152 | image_size (tuple): Size images should be resized to. 153 | Returns: 154 | numpy.ndarray, training data (face parameters). 155 | numpy.ndarray, output data (the actual images to generate). 156 | """ 157 | 158 | inputs = { 159 | 'identity' : np.empty((self.num_instances, self.num_identities)), 160 | 'pose' : np.empty((self.num_instances, NUM_YALE_POSES)), 161 | 'lighting' : np.empty((self.num_instances, 4)), 162 | } 163 | 164 | if K.image_dim_ordering() == 'th': 165 | outputs = np.empty((self.num_instances, 1)+image_size) 166 | else: 167 | outputs = np.empty((self.num_instances,)+image_size+(1,)) 168 | 169 | all_instances = range(0, len(self.filenames)) 170 | if verbose: 171 | all_instances = tqdm(all_instances) 172 | 173 | for i in all_instances: 174 | instance = YaleInstance(self.directory, self.filenames[i], image_size) 175 | 176 | inputs['identity'][i,:] = instance.identity_vector(self.identity_map) 177 | inputs['pose'][i,:] = instance.pose 178 | inputs['lighting'][i,:] = instance.lighting 179 | 180 | if K.image_dim_ordering() == 'th': 181 | outputs[i,:,:,:] = instance.th_image() 182 | else: 183 | outputs[i,:,:,:] = instance.tf_image() 184 | 185 | return inputs, outputs 186 | 187 | 188 | class JAFFEInstances: 189 | """ 190 | This is a reader for the JAFFE dataset of Japanese female faces 191 | acting out varying expressions, scored by a panel of FACS evaluators. 192 | Image download link at http://www.kasrl.org/jaffe_info.html 193 | The unpacked directory structure is flat, with filenames like KA.AN1.39.tiff 194 | You should add to this directory a CSV version of the semantic ratings 195 | table appearing on the download page, as semantic-ratings.csv 196 | You'll have to make this yourself. First two lines will look like: 197 | N,HAP,SAD,SUR,ANG,DIS,FEA,PIC 198 | 1,2.87,2.52,2.10,1.97,1.97,2.06,KM-NE1 199 | """ 200 | 201 | def __init__(self, directory): 202 | """ 203 | Constructor for a JAFFEInstances object. 204 | 205 | Args: 206 | directory (str): Directory where the data lives 207 | """ 208 | 209 | self.directory = directory 210 | self.filenames = [x for x in os.listdir(directory) if x.endswith('tiff')] 211 | self.num_instances = len(self.filenames) 212 | identity_map = {} 213 | for fname in self.filenames: 214 | ident, emotion = fname.split('.')[:2] 215 | # assign identity strings contiguous indices 216 | identity_map[ident] = identity_map.get(ident, len(identity_map)) 217 | self.identity_map = identity_map 218 | self.num_identities = len(identity_map) 219 | 220 | def load_semantic_ratings(self): 221 | """ 222 | Loads semantic ratings for each instance. These assign 223 | human-evaluated levels for each emotion in a given face 224 | (a face will generally have nonzero score on multiple emotions). 225 | 226 | Returns: 227 | dict, ratings (vectors of emotion scores keyed by inst#) 228 | """ 229 | 230 | # map JAFFE emotion labels to local Emotion indices 231 | # note that there is no explicit JAFFE neutral, it is implied when 232 | # no specific emotion dominates. 233 | emotions = ('ANG', '_', 'DIS', 'FEA', 'HAP', 'NEU', 'SAD', 'SUR') 234 | emotion_map = {emotion:idx for idx, emotion in enumerate(emotions)} 235 | ratings = {} 236 | with open(os.path.join(self.directory, 'semantic-ratings.csv')) as rows: 237 | reader = csv.DictReader(rows) 238 | for row in reader: 239 | rates = np.array([float(row.get(emotion, 1)) 240 | for emotion in emotions]) 241 | # emotions are scored 1..5, make them 0-1 242 | rates = (rates - 1.0) / 4.0 243 | # synthesize 'neutral' score as the complement of the strongest 244 | # emotion present. 245 | rates[emotion_map['NEU']] = 1.0 - np.max(rates) 246 | # input convention is for the emotion vector to sum to 1. 247 | rates = rates / np.linalg.norm(rates) 248 | N = int(row['N']) - 1 249 | ratings[N] = rates 250 | return ratings 251 | 252 | def load_data(self, image_size, verbose=False): 253 | """ 254 | Loads JAFFE data for training. 255 | 256 | Args: 257 | image_size (tuple): Size images should be resized to. 258 | Returns: 259 | numpy.ndarray, training data (face parameters). 260 | numpy.ndarray, output data (the actual images to generate). 261 | """ 262 | 263 | instances = [JAFFEInstance(self.directory, fname, image_size) 264 | for fname in self.filenames] 265 | inst_idents = np.zeros((self.num_instances, self.num_identities)) 266 | for idx, inst in enumerate(instances): 267 | # each row in inst_idents is a one-hot encoding of identity idx 268 | inst_idents[idx, self.identity_map[inst.identity]] = 1 269 | 270 | inst_orient = np.tile((0, 1), self.num_instances).reshape(-1,2) 271 | ratings = self.load_semantic_ratings() 272 | # Note: there are some scored instance N's with no instance file! 273 | inst_emotion = np.array([ratings[inst.N] for inst in instances]) 274 | 275 | inputs = { 276 | 'identity': inst_idents, # 1-hot 277 | 'orientation': inst_orient, 278 | 'emotion': inst_emotion, 279 | } 280 | print("JAFFE: found %d identities, %instances" % ( 281 | self.num_identities, self.num_instances)) 282 | if K.image_dim_ordering() == 'th': 283 | inst_image = [inst.th_image() for inst in instances] 284 | outputs = np.empty((self.num_instances, 1)+image_size) 285 | else: 286 | inst_image = [inst.tf_image() for inst in instances] 287 | outputs = np.empty((self.num_instances,)+image_size+(1,)) 288 | outputs[np.arange(self.num_instances)] = inst_image 289 | 290 | return inputs, outputs 291 | 292 | 293 | # ---- Instance class definition 294 | 295 | class RaFDInstance: 296 | """ 297 | Holds information about each RaFD example. 298 | """ 299 | 300 | def __init__(self, directory, filename, image_size, trim=24, top=24): 301 | """ 302 | Constructor for an RaFDInstance object. 303 | 304 | Args: 305 | directory (str): Base directory where the example lives. 306 | filename (str): The name of the file of the example. 307 | image_size (tuple): Size to resize the image to. 308 | Args (optional): 309 | trim (int): How many pixels from the edge to trim off the top and sides. 310 | top (int): How much extra to trim off the top. 311 | """ 312 | 313 | self.image = misc.imread( os.path.join(directory, filename) ) 314 | 315 | # Trim the image to get more of the face 316 | 317 | height, width, d = self.image.shape 318 | 319 | width = int(width-2*trim) 320 | height = int(width*image_size[0]/image_size[1]) 321 | 322 | self.image = self.image[trim+top:trim+height,trim:trim+width,:] 323 | 324 | # Resize and fit between 0-1 325 | self.image = misc.imresize( self.image, image_size ) 326 | self.image = self.image / 255.0 327 | 328 | #self.mask = misc.imread( os.path.join(directory, 'mask', filename) ) 329 | #self.mask = misc.imresize( self.mask, image_size ) 330 | #self.mask = self.mask / 255.0 331 | 332 | # Parse filename to get parameters 333 | 334 | items = filename.split('_') 335 | 336 | # Represent orientation as sin/cos vector 337 | angle = np.deg2rad(float(items[0][-3:])-90) 338 | self.orientation = np.array([np.sin(angle), np.cos(angle)]) 339 | 340 | self.identity_index = int(items[1])-1 # Identities are 1-indexed 341 | 342 | self.emotion = np.array(getattr(Emotion, items[4])) 343 | 344 | 345 | def identity_vector(self, identity_map): 346 | """ 347 | Creates a one-in-k encoding of the instance's identity. 348 | 349 | Args: 350 | identity_map (dict): Mapping from identity to a unique index. 351 | Returns: 352 | numpy.ndarray, the identity vector. 353 | """ 354 | 355 | identity_vec = np.zeros(len(identity_map), dtype=np.float32) 356 | identity_vec[ identity_map[self.identity_index] ] = 1. 357 | 358 | return identity_vec 359 | 360 | 361 | def th_image(self): 362 | """ 363 | Returns a Theano-ordered representation of the image. 364 | """ 365 | 366 | image = np.empty((3,)+self.image.shape[0:2]) 367 | for i in range(0, 3): 368 | image[i,:,:] = self.image[:,:,i] 369 | return image 370 | 371 | 372 | def tf_image(self): 373 | """ 374 | Returns a TensorFlow-ordered representation of the image. 375 | """ 376 | 377 | # As-is 378 | return self.image 379 | 380 | 381 | class YaleInstance: 382 | """ 383 | Holds information about each YaleFaces example. 384 | """ 385 | 386 | def __init__(self, directory, filepath, image_size): 387 | """ 388 | Constructor for an YaleInstance object. 389 | 390 | Args: 391 | directory (str): Base directory where the example lives. 392 | filename (str): The name of the file of the example. 393 | image_size (tuple): Size to resize the image to. 394 | """ 395 | 396 | filename = filepath.split('/')[-1] 397 | 398 | self.image = misc.imread( os.path.join(directory, filepath) ) 399 | 400 | # Resize and scale values to [0 1] 401 | self.image = misc.imresize( self.image, image_size ) 402 | self.image = self.image / 255.0 403 | 404 | self.identity_index = int(filename[5:7]) 405 | 406 | pose_idx = int(filename[9:11]) 407 | self.pose = np.zeros(NUM_YALE_POSES, dtype=np.float32) 408 | self.pose[pose_idx] = 1 409 | 410 | # Light azimuth and elevation 411 | az = np.deg2rad(float(filename[12:16])) 412 | el = np.deg2rad(float(filename[17:20])) 413 | 414 | self.lighting = np.array([np.sin(az), np.cos(az), np.sin(el), np.cos(el)]) 415 | 416 | 417 | def identity_vector(self, identity_map): 418 | """ 419 | Creates a one-in-k encoding of the instance's identity. 420 | 421 | Args: 422 | identity_map (dict): Mapping from identity to a unique index. 423 | Returns: 424 | numpy.ndarray, the identity vector. 425 | """ 426 | 427 | identity_vec = np.zeros(len(identity_map), dtype=np.float32) 428 | identity_vec[ identity_map[self.identity_index] ] = 1. 429 | 430 | return identity_vec 431 | 432 | 433 | def th_image(self): 434 | """ 435 | Returns a Theano-ordered representation of the image. 436 | """ 437 | 438 | return np.expand_dims(self.image, 0) 439 | 440 | 441 | def tf_image(self): 442 | """ 443 | Returns a TensorFlow-ordered representation of the image. 444 | """ 445 | 446 | return np.expand_dims(self.image, 2) 447 | 448 | 449 | class JAFFEInstance: 450 | """ 451 | Holds information about each JAFFE example. 452 | """ 453 | 454 | def __init__(self, directory, filepath, image_size): 455 | """ 456 | Constructor for an JAFFEInstance object. 457 | 458 | Args: 459 | directory (str): Base directory where the example lives. 460 | filename (str): The name of the file of the example. 461 | image_size (tuple): Size to resize the image to. 462 | """ 463 | 464 | filename = filepath.split('/')[-1] 465 | 466 | self.image = misc.imread( os.path.join(directory, filepath) ) 467 | # some of the jaffe images are 3-channel greyscale, some are 1-channel! 468 | self.image = np.atleast_3d(self.image)[...,0] # make image 2d for sure 469 | # Resize and scale values to [0 1] 470 | self.image = misc.imresize( self.image, image_size ) 471 | self.image = self.image / 255.0 472 | ident, _, N, _ = filename.split('.') 473 | # Note: the emotion encoded in the filename is the dominant 474 | # scoring emotion, but we ignore this and use precise emotion scores 475 | # from the semantic ratings table 476 | self.identity, self.N = ident, int(N) - 1 # 0-based instance numbering 477 | 478 | def th_image(self): 479 | """ 480 | Returns a Theano-ordered representation of the image. 481 | """ 482 | 483 | return np.expand_dims(self.image, 0) 484 | 485 | 486 | def tf_image(self): 487 | """ 488 | Returns a TensorFlow-ordered representation of the image. 489 | """ 490 | 491 | return np.expand_dims(self.image, 2) 492 | -------------------------------------------------------------------------------- /faces/model.py: -------------------------------------------------------------------------------- 1 | """ 2 | faces/model.py 3 | 4 | Methods to build FaceGen models. 5 | 6 | """ 7 | 8 | from keras import backend as K 9 | from keras.layers import BatchNormalization, Convolution2D, Dense, LeakyReLU, \ 10 | Input, MaxPooling2D, merge, Reshape, UpSampling2D 11 | from keras.models import Model 12 | 13 | from .instance import Emotion, NUM_YALE_POSES 14 | 15 | 16 | def build_model(identity_len=57, orientation_len=2, lighting_len=4, 17 | emotion_len=Emotion.length(), pose_len=NUM_YALE_POSES, 18 | initial_shape=(5,4), deconv_layers=5, num_kernels=None, 19 | optimizer='adam', use_yale=False, use_jaffe=False): 20 | """ 21 | Builds a deconvolution FaceGen model. 22 | 23 | Args (optional): 24 | identity_len (int): Length of the identity input vector. 25 | orientation_len (int): Length of the orientation input vector. 26 | emotion_len (int): Length of the emotion input vector. 27 | initial_shape (tuple): The starting shape of the deconv. network. 28 | deconv_layers (int): How many deconv. layers to use. More layers 29 | gives better resolution, although requires more GPU memory. 30 | num_kernels (list): Number of convolution kernels for each layer. 31 | optimizer (str): The optimizer to use. Will only use default values. 32 | Returns: 33 | keras.Model, the constructed model. 34 | """ 35 | 36 | print(initial_shape) 37 | 38 | if num_kernels is None: 39 | num_kernels = [128, 128, 96, 96, 32, 32, 16] 40 | 41 | # TODO: Parameter validation 42 | 43 | identity_input = Input(shape=(identity_len,), name='identity') 44 | 45 | if use_yale: 46 | lighting_input = Input(shape=(lighting_len,), name='lighting') 47 | pose_input = Input(shape=(pose_len,), name='pose') 48 | else: 49 | orientation_input = Input(shape=(orientation_len,), name='orientation') 50 | emotion_input = Input(shape=(emotion_len,), name='emotion') 51 | 52 | # Hidden representation for input parameters 53 | 54 | fc1 = LeakyReLU()( Dense(512)(identity_input) ) 55 | fc2 = LeakyReLU()( Dense(512)(lighting_input if use_yale else orientation_input) ) 56 | fc3 = LeakyReLU()( Dense(512)(pose_input if use_yale else emotion_input) ) 57 | 58 | params = merge([fc1, fc2, fc3], mode='concat') 59 | params = LeakyReLU()( Dense(1024)(params) ) 60 | 61 | # Apply deconvolution layers 62 | 63 | height, width = initial_shape 64 | 65 | print('height:', height, 'width:', width) 66 | 67 | x = LeakyReLU()( Dense(height*width*num_kernels[0])(params) ) 68 | if K.image_dim_ordering() == 'th': 69 | x = Reshape((num_kernels[0], height, width))(x) 70 | else: 71 | x = Reshape((height, width, num_kernels[0]))(x) 72 | 73 | for i in range(0, deconv_layers): 74 | # Upsample input 75 | x = UpSampling2D((2,2))(x) 76 | 77 | # Apply 5x5 and 3x3 convolutions 78 | 79 | # If we didn't specify the number of kernels to use for this many 80 | # layers, just repeat the last one in the list. 81 | idx = i if i < len(num_kernels) else -1 82 | x = LeakyReLU()( Convolution2D(num_kernels[idx], 5, 5, border_mode='same')(x) ) 83 | x = LeakyReLU()( Convolution2D(num_kernels[idx], 3, 3, border_mode='same')(x) ) 84 | x = BatchNormalization()(x) 85 | 86 | # Last deconvolution layer: Create 3-channel image. 87 | x = MaxPooling2D((1,1))(x) 88 | x = UpSampling2D((2,2))(x) 89 | x = LeakyReLU()( Convolution2D(8, 5, 5, border_mode='same')(x) ) 90 | x = LeakyReLU()( Convolution2D(8, 3, 3, border_mode='same')(x) ) 91 | x = Convolution2D(1 if use_yale or use_jaffe else 3, 3, 3, 92 | border_mode='same', activation='sigmoid')(x) 93 | 94 | # Compile the model 95 | 96 | if use_yale: 97 | model = Model(input=[identity_input, pose_input, lighting_input], output=x) 98 | else: 99 | model = Model(input=[identity_input, orientation_input, emotion_input], output=x) 100 | 101 | # TODO: Optimizer options 102 | model.compile(optimizer=optimizer, loss='msle') 103 | 104 | return model 105 | -------------------------------------------------------------------------------- /faces/train.py: -------------------------------------------------------------------------------- 1 | """ 2 | faces/train.py 3 | """ 4 | 5 | import os 6 | 7 | from keras import backend as K 8 | from keras.callbacks import Callback, EarlyStopping, ModelCheckpoint 9 | from keras.models import load_model 10 | 11 | import numpy as np 12 | import scipy.misc 13 | 14 | from .instance import ( 15 | Emotion, RaFDInstances, YaleInstances, JAFFEInstances, NUM_YALE_POSES) 16 | from .model import build_model 17 | 18 | 19 | class GenerateIntermediate(Callback): 20 | """ Callback to generate intermediate images after each epoch. """ 21 | 22 | def __init__(self, output_dir, num_identities, batch_size=32, use_yale=False, 23 | use_jaffe=False): 24 | """ 25 | Constructor for a GenerateIntermediate object. 26 | 27 | Args: 28 | output_dir (str): Directory to save intermediate results in. 29 | num_identities (int): Number of identities in the training set. 30 | Args: (optional) 31 | batch_size (int): Batch size to use when generating images. 32 | """ 33 | super(Callback, self).__init__() 34 | 35 | self.output_dir = output_dir 36 | self.num_identities = num_identities 37 | self.batch_size = batch_size 38 | self.use_yale = use_yale 39 | self.use_jaffe = use_jaffe 40 | 41 | self.parameters = dict() 42 | 43 | # Sweep through identities 44 | self.parameters['identity'] = np.eye(num_identities) 45 | 46 | if use_yale: 47 | # Use pose 0, lighting at 0deg azimuth and elevation 48 | self.parameters['pose'] = np.zeros((num_identities, NUM_YALE_POSES)) 49 | self.parameters['lighting'] = np.zeros((num_identities, 4)) 50 | for i in range(0, num_identities): 51 | self.parameters['pose'][i,0] = 0 52 | self.parameters['lighting'][i,1] = 1 53 | self.parameters['lighting'][i,3] = 1 54 | else: 55 | # Make all have neutral expressions, front-facing 56 | self.parameters['emotion'] = np.empty((num_identities, Emotion.length())) 57 | self.parameters['orientation'] = np.zeros((num_identities, 2)) 58 | for i in range(0, num_identities): 59 | self.parameters['emotion'][i,:] = Emotion.neutral 60 | self.parameters['orientation'][i,1] = 1 61 | 62 | 63 | def on_train_begin(self, logs={}): 64 | """ Create directories. """ 65 | 66 | if not os.path.exists(self.output_dir): 67 | os.makedirs(self.output_dir) 68 | 69 | 70 | def on_epoch_end(self, epoch, logs={}): 71 | """ Generate and save results to the output directory. """ 72 | 73 | dest_dir = os.path.join(self.output_dir, 'e{:04}'.format(epoch)) 74 | if not os.path.exists(dest_dir): 75 | os.makedirs(dest_dir) 76 | 77 | gen = self.model.predict(self.parameters, batch_size=self.batch_size) 78 | 79 | for i in range(0, gen.shape[0]): 80 | if K.image_dim_ordering() == 'th': 81 | if self.use_yale or self.use_jaffe: 82 | image = np.empty(gen.shape[2:]) 83 | image[:,:] = gen[i,0,:,:] 84 | else: 85 | image = np.empty(gen.shape[2:]+(3,)) 86 | for x in range(0, 3): 87 | image[:,:,x] = gen[i,x,:,:] 88 | else: 89 | if self.use_yale or self.use_jaffe: 90 | image = gen[i,:,:,0] 91 | else: 92 | image = gen[i,:,:,:] 93 | image = np.array(255*np.clip(image,0,1), dtype=np.uint8) 94 | file_path = os.path.join(dest_dir, '{:02}.png'.format(i)) 95 | scipy.misc.imsave(file_path, image) 96 | 97 | 98 | def train_model(data_dir, output_dir, model_file='', batch_size=32, 99 | num_epochs=100, optimizer='adam', deconv_layers=5, 100 | use_yale=False, use_jaffe=False, 101 | kernels_per_layer=None, generate_intermediate=False, 102 | verbose=False): 103 | """ 104 | Trains the model on the data, generating intermediate results every epoch. 105 | 106 | Args: 107 | data_dir (str): Directory where the data lives. 108 | output_dir (str): Directory where outputs should be saved. 109 | model_file (str): Model file to load. If none specified, a new model 110 | will be created. 111 | Args (optional): 112 | batch_size (int): Size of the batch to use. 113 | num_epochs (int): Number of epochs to train for. 114 | optimizer (str): Keras optimizer to use. 115 | deconv_layers (int): The number of deconvolution layers to use. 116 | generate_intermediate (bool): Whether or not to generate intermediate results. 117 | """ 118 | 119 | data_dir = os.path.expanduser(data_dir) 120 | 121 | if not os.path.exists(output_dir): 122 | os.makedirs(output_dir) 123 | 124 | instances = (YaleInstances(data_dir) if use_yale 125 | else JAFFEInstances(data_dir) if use_jaffe 126 | else RaFDInstances(data_dir)) 127 | 128 | if verbose: 129 | print("Found {} instances with {} identities".format( 130 | instances.num_instances, instances.num_identities)) 131 | 132 | 133 | # Create FaceGen model to use 134 | 135 | if model_file: 136 | model = load_model(model_file) 137 | if verbose: 138 | print("Loaded model %d identities from {}".format(model.model_file)) 139 | else: 140 | # TODO: Refactor this to a more elegant way to determine params by dataset 141 | initial_shape = (5,4) 142 | if use_yale: 143 | initial_shape = (6,8) 144 | if use_jaffe: 145 | initial_shape = (4,4) 146 | 147 | model = build_model( 148 | identity_len = instances.num_identities, 149 | deconv_layers = deconv_layers, 150 | num_kernels = kernels_per_layer, 151 | optimizer = optimizer, 152 | initial_shape = initial_shape, 153 | use_yale = use_yale, 154 | use_jaffe = use_jaffe, 155 | ) 156 | if verbose: 157 | print("Built model with:") 158 | print("\tDeconv layers: {}".format(deconv_layers)) 159 | print("\tOutput shape: {}".format(model.output_shape[1:])) 160 | 161 | # Create training callbacks 162 | 163 | callbacks = list() 164 | 165 | if generate_intermediate: 166 | intermediate_dir = os.path.join(output_dir, 'intermediate.d{}.{}'.format(deconv_layers, optimizer)) 167 | callbacks.append( GenerateIntermediate(intermediate_dir, instances.num_identities, 168 | use_yale=use_yale, use_jaffe=use_jaffe) ) 169 | 170 | model_path = os.path.join(output_dir, 'FaceGen.{}.model.d{}.{}.h5' 171 | .format('YaleFaces' if use_yale else 'JAFFE' if use_jaffe else 'RaFD', deconv_layers, optimizer)) 172 | 173 | callbacks.append( 174 | ModelCheckpoint( 175 | model_path, 176 | monitor='loss', verbose=0, save_best_only=True, 177 | ) 178 | ) 179 | callbacks.append( 180 | EarlyStopping(monitor='loss', patience=8) 181 | ) 182 | 183 | # Load data and begin training 184 | 185 | if verbose: 186 | print("Loading data...") 187 | 188 | if K.image_dim_ordering() == 'th': 189 | image_size = model.output_shape[2:4] 190 | else: 191 | image_size = model.output_shape[1:3] 192 | 193 | inputs, outputs = instances.load_data(image_size, verbose=verbose) 194 | 195 | if verbose: 196 | print("Training...") 197 | 198 | model.fit(inputs, outputs, batch_size=batch_size, nb_epoch=num_epochs, 199 | callbacks=callbacks, shuffle=True, verbose=1) 200 | 201 | if verbose: 202 | print("Done!") 203 | -------------------------------------------------------------------------------- /img/example.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/somewacko/deconvfaces/9e9620c52cb1de37e6ae3f52387791ee21dbda67/img/example.gif -------------------------------------------------------------------------------- /jaffe/semantic-ratings.csv: -------------------------------------------------------------------------------- 1 | N,HAP,SAD,SUR,ANG,DIS,FEA,PIC 2 | 1,2.87,2.52,2.10,1.97,1.97,2.06,KM-NE1 3 | 2,2.87,2.42,1.58,1.84,1.77,1.77,KM-NE2 4 | 3,2.50,2.10,1.70,1.50,1.73,1.53,KM-NE3 5 | 4,4.90,1.13,1.26,1.10,1.03,1.10,KM-HA1 6 | 5,4.87,1.20,1.43,1.03,1.07,1.07,KM-HA2 7 | 6,4.61,1.32,1.39,1.23,1.10,1.19,KM-HA3 8 | 7,5.00,1.13,1.26,1.10,1.10,1.06,KM-HA4 9 | 8,4.65,1.29,1.39,1.23,1.16,1.16,KM-HA5 10 | 9,1.42,4.00,1.55,2.39,3.26,3.03,KM-SA1 11 | 10,1.23,4.39,1.45,2.61,3.19,2.71,KM-SA2 12 | 11,1.32,4.00,1.87,2.60,3.77,3.19,KM-SA3 13 | 12,1.26,4.29,1.58,2.26,3.39,2.94,KM-SA4 14 | 13,1.35,4.13,1.61,2.10,3.81,3.10,KM-SA5 15 | 14,2.13,1.55,4.97,1.65,1.45,2.10,KM-SU1 16 | 15,2.16,1.90,4.97,1.74,1.81,3.10,KM-SU2 17 | 16,2.16,1.84,4.94,1.71,1.65,2.32,KM-SU3 18 | 17,1.39,2.03,1.57,4.58,3.77,1.68,KM-AN1 19 | 18,1.45,2.84,1.68,4.06,3.87,1.90,KM-AN2 20 | 19,1.42,2.13,1.55,4.16,3.58,1.77,KM-AN3 21 | 20,1.26,2.10,1.81,3.52,4.71,2.26,KM-DI1 22 | 21,1.55,2.13,1.74,3.16,4.32,2.13,KM-DI2 23 | 22,1.39,1.87,1.94,3.61,4.58,1.94,KM-DI3 24 | 23,2.52,2.58,2.61,1.97,2.90,2.87,KM-FE1 25 | 24,1.50,2.97,3.67,2.83,3.77,3.80,KM-FE2 26 | 25,1.68,2.84,3.61,2.90,3.84,3.19,KM-FE3 27 | 26,3.03,2.16,2.06,1.94,1.84,1.87,KA-NE1 28 | 27,2.84,1.94,2.13,1.77,1.68,1.87,KA-NE2 29 | 28,3.06,1.84,2.29,1.77,1.71,1.84,KA-NE3 30 | 29,4.39,1.35,2.29,1.16,1.23,1.26,KA-HA1 31 | 30,4.77,1.29,2.45,1.26,1.23,1.23,KA-HA2 32 | 31,4.61,1.48,2.19,1.29,1.32,1.39,KA-HA3 33 | 32,4.03,1.68,1.42,1.19,1.13,1.29,KA-HA4 34 | 33,1.39,3.97,1.68,2.19,3.68,3.61,KA-SA1 35 | 34,1.26,4.13,1.77,2.40,3.10,2.74,KA-SA2 36 | 35,1.68,3.71,1.74,2.42,3.16,3.03,KA-SA3 37 | 36,2.87,1.55,4.68,1.52,1.52,1.65,KA-SU1 38 | 37,3.00,1.94,4.35,1.68,1.55,1.58,KA-SU2 39 | 38,2.94,1.97,4.74,1.68,1.71,2.03,KA-SU3 40 | 39,1.55,1.90,2.10,4.32,3.90,1.81,KA-AN1 41 | 40,1.67,1.97,1.70,4.60,4.10,1.77,KA-AN2 42 | 41,1.30,1.70,1.70,4.30,4.43,1.73,KA-AN3 43 | 42,1.42,3.55,2.13,2.65,4.19,3.03,KA-DI1 44 | 43,1.29,2.26,2.13,3.32,4.74,2.87,KA-DI2 45 | 44,1.32,1.94,1.94,2.52,4.68,2.55,KA-DI3 46 | 45,1.32,4.16,3.65,1.97,2.87,4.26,KA-FE1 47 | 46,1.29,3.42,3.16,1.94,4.06,3.87,KA-FE2 48 | 47,1.39,3.00,4.29,2.32,3.74,4.13,KA-FE3 49 | 48,2.06,2.16,4.29,1.61,2.10,2.42,KA-FE4 50 | 49,2.71,2.97,1.45,1.90,2.00,1.94,YM-NE1 51 | 50,2.73,2.67,1.87,1.97,2.00,2.00,YM-NE2 52 | 51,3.03,2.45,1.74,2.00,1.90,1.77,YM-NE3 53 | 52,4.90,1.39,1.29,1.32,1.29,1.35,YM-HA1 54 | 53,4.84,1.42,1.35,1.23,1.23,1.23,YN-HA2 55 | 54,4.87,1.32,1.32,1.19,1.13,1.16,YM-HA3 56 | 55,1.45,4.61,1.52,1.55,2.13,2.39,YM-SA1 57 | 56,1.65,4.26,1.77,1.94,2.61,2.77,YM-SA2 58 | 57,1.55,4.32,1.35,1.71,2.74,2.55,YM-SA3 59 | 58,2.48,1.55,4.87,1.65,1.58,1.81,YM-SU1 60 | 59,2.74,1.48,4.45,1.39,1.42,1.68,YM-SU2 61 | 60,2.35,1.81,4.87,1.81,1.71,2.29,YM-SU3 62 | 61,1.19,2.16,1.55,4.74,2.84,1.39,YM-AN1 63 | 62,1.68,2.26,1.45,4.13,2.90,1.61,YM-AN2 64 | 63,1.55,2.32,1.68,4.77,3.16,1.87,YM-AN3 65 | 64,1.35,2.90,2.45,2.65,4.81,3.32,YM-DI1 66 | 65,1.13,2.13,2.37,3.03,4.63,3.00,YM-DI2 67 | 66,1.23,3.00,2.61,2.74,4.65,3.68,YM-DI3 68 | 67,1.23,4.26,3.16,2.13,3.26,4.03,YM-FE1 69 | 68,1.26,4.19,3.74,2.16,3.42,4.26,YM-FE2 70 | 69,1.29,3.52,3.48,2.03,3.42,3.35,YM-FE3 71 | 70,1.29,3.13,4.52,2.55,3.45,4.19,YM-FE4 72 | 71,2.23,2.06,1.84,2.87,2.35,1.87,KR-NE1 73 | 72,2.32,2.23,1.68,2.68,2.35,1.84,KR-NE2 74 | 73,2.03,2.16,1.68,2.81,2.39,1.81,KR-NE3 75 | 74,4.45,1.29,1.10,1.19,1.26,1.19,KR-HA1 76 | 75,4.19,1.48,1.65,1.29,1.32,1.35,KR-HA2 77 | 76,4.39,1.29,1.23,1.16,1.19,1.16,KR-HA3 78 | 77,1.48,4.23,1.84,2.55,2.74,2.29,KR-SA1 79 | 78,1.42,4.26,1.74,2.52,3.06,2.48,KR-SA2 80 | 79,1.48,3.90,1.77,2.81,3.29,2.42,KR-SA3 81 | 80,2.00,1.77,4.68,1.81,1.65,2.03,KR-SU1 82 | 81,2.10,1.84,4.58,2.19,2.03,2.10,KR-SU2 83 | 82,2.42,1.84,4.74,1.94,1.87,2.06,KR-SU3 84 | 83,1.39,1.87,1.73,4.58,3.19,1.68,KR-AN1 85 | 84,1.52,2.58,1.74,4.58,3.65,1.61,KR-AN2 86 | 85,1.48,2.48,2.19,4.48,3.42,2.16,KR-AN3 87 | 86,1.35,2.19,2.23,3.39,4.55,2.58,KR-DI1 88 | 87,1.35,2.32,1.97,4.03,4.39,2.35,KR-DI2 89 | 88,1.39,2.45,1.67,4.29,4.06,2.10,KR-DI3 90 | 89,1.52,2.26,4.45,3.16,3.03,2.90,KR-FE1 91 | 90,1.61,2.39,3.97,3.19,3.71,2.74,KR-FE2 92 | 91,1.55,2.42,4.00,3.58,3.55,3.06,KR-FE3 93 | 92,3.03,2.32,1.77,1.74,1.77,1.68,NM-NE1 94 | 93,3.29,2.23,1.84,1.68,1.68,1.71,NM-NE2 95 | 94,3.19,2.26,1.71,1.87,1.74,1.71,NM-NE3 96 | 95,4.29,1.55,1.45,1.39,1.26,1.23,NM-HA1 97 | 96,4.39,1.42,2.10,1.32,1.26,1.23,NM-HA2 98 | 97,4.19,1.45,1.84,1.42,1.35,1.39,NM-HA3 99 | 98,2.00,3.77,1.67,1.77,1.77,2.29,NM-SA1 100 | 99,2.48,2.68,1.81,1.97,1.90,1.84,NM-SA2 101 | 100,2.03,3.48,1.94,1.87,2.16,2.77,NM-SA3 102 | 101,2.68,1.65,4.94,1.48,1.45,1.77,NM-SU1 103 | 102,2.90,1.53,4.80,1.43,1.37,1.57,NM-SU2 104 | 103,2.65,1.77,4.94,1.65,1.52,1.90,NM-SU3 105 | 104,2.55,2.16,1.81,2.16,1.94,1.94,NM-AN1 106 | 105,2.03,2.29,1.61,2.71,2.35,1.87,NM-AN2 107 | 106,1.68,3.00,1.84,3.17,3.65,2.58,NM-AN3 108 | 107,2.67,2.67,1.77,2.30,3.10,2.21,NM-DI1 109 | 108,2.61,2.39,1.74,2.00,2.39,1.94,NM-DI2 110 | 109,2.45,1.94,1.52,2.16,2.61,1.65,NM-DI3 111 | 110,1.55,3.61,3.65,1.84,2.71,3.87,NM-FE1 112 | 111,2.29,2.29,3.84,1.94,2.32,2.29,NM-FE2 113 | 112,1.90,2.03,4.52,1.87,1.87,2.61,NM-FE3 114 | 113,2.90,2.65,1.74,1.87,1.90,1.87,MK-NE1 115 | 114,3.48,2.19,1.55,1.68,1.55,1.58,MK-NE2 116 | 115,2.97,2.39,1.71,1.81,1.61,1.84,MK-NE3 117 | 116,4.52,1.32,1.58,1.10,1.26,1.23,MK-HA1 118 | 117,4.32,1.61,1.42,1.48,1.52,1.48,MK-HA2 119 | 118,4.65,1.32,1.45,1.19,1.29,1.29,MK-HA3 120 | 119,1.65,4.23,1.74,2.13,2.74,2.68,MK-SA1 121 | 120,1.39,4.52,1.61,1.77,2.71,2.61,MK-SA2 122 | 121,1.26,4.84,1.65,2.13,2.52,2.23,MK-SA3 123 | 122,1.84,2.35,4.94,2.00,2.52,3.29,MK-SU1 124 | 123,2.32,1.65,4.97,1.81,1.90,2.26,MK-SU2 125 | 124,1.97,2.23,4.90,2.13,2.39,3.03,MK-SU3 126 | 125,1.26,1.87,1.48,4.68,4.00,1.55,MK-AN1 127 | 126,1.26,3.32,1.52,4.13,4.52,2.74,MK-AN2 128 | 127,1.26,3.03,2.03,4.35,4.61,2.97,MK-AN3 129 | 128,1.26,2.48,2.26,3.68,4.61,2.58,MK-DI1 130 | 129,1.32,3.10,2.43,2.26,4.39,3.10,MK-DI2 131 | 130,1.61,2.52,2.29,2.68,4.71,3.19,MK-DI3 132 | 131,1.23,4.06,4.10,2.32,3.26,4.26,MK-FE1 133 | 132,1.35,3.32,4.65,2.42,3.97,4.52,MK-FE2 134 | 133,1.42,3.65,2.84,2.16,3.29,4.06,MK-FE3 135 | 134,2.32,2.29,1.97,2.06,1.97,1.87,UY-NE1 136 | 135,2.10,2.35,1.68,2.23,1.90,1.87,UY-NE2 137 | 136,2.00,2.42,1.71,2.26,2.35,2.06,UY-NE3 138 | 137,3.42,2.52,1.61,1.55,1.55,1.61,UY-HA1 139 | 138,3.84,1.97,1.52,1.42,1.45,1.61,UY-HA2 140 | 139,4.13,1.81,1.42,1.26,1.32,1.55,UY-HA3 141 | 140,1.35,3.65,1.58,3.00,3.55,2.32,UY-SA1 142 | 141,1.52,3.84,1.55,2.61,3.55,2.58,UY-SA2 143 | 142,1.35,3.55,1.35,2.48,3.06,2.26,UY-SA3 144 | 143,2.52,2.29,3.61,1.74,1.94,2.26,UY-SU1 145 | 144,1.84,2.65,4.26,1.77,2.13,2.61,UY-SU2 146 | 145,1.97,2.65,4.45,1.90,1.81,2.42,UY-SU3 147 | 146,1.39,3.23,1.71,4.16,3.90,2.19,UY-AN1 148 | 147,1.45,2.35,1.58,4.23,3.10,1.81,UY-AN2 149 | 148,1.58,2.81,1.74,3.61,3.10,2.03,UY-AN3 150 | 149,1.37,2.93,2.30,2.80,4.20,3.33,UY-DI1 151 | 150,1.45,3.26,1.77,2.84,4.32,2.77,UY-DI2 152 | 151,1.35,3.23,1.52,3.39,4.39,2.84,UY-DI3 153 | 152,1.42,3.45,2.16,2.37,2.48,3.03,UY-FE1 154 | 153,1.74,3.16,2.32,2.42,2.48,2.68,UY-FE2 155 | 154,1.77,3.26,2.55,2.00,2.45,3.06,UY-FE3 156 | 155,1.84,2.29,3.71,2.35,2.32,2.13,KL-NE1 157 | 156,1.90,2.81,2.52,2.39,2.32,2.32,KL-NE2 158 | 157,2.10,2.13,3.61,2.16,2.10,2.06,KL-NE3 159 | 158,4.42,1.55,1.42,1.42,1.45,1.45,KL-HA1 160 | 159,4.48,1.68,1.58,1.65,1.55,1.58,KL-HA2 161 | 160,4.29,1.45,1.48,1.45,1.42,1.32,KL-HA3 162 | 161,1.19,4.35,1.83,2.87,3.77,3.42,KL-SA1 163 | 162,1.26,4.26,1.90,2.87,3.61,3.35,KL-SA2 164 | 163,1.39,4.42,1.94,2.77,3.68,3.81,KL-SA3 165 | 164,1.77,2.55,4.71,2.42,2.58,2.87,KL-SU1 166 | 165,1.84,1.80,4.84,2.16,1.90,2.19,KL-SU2 167 | 166,1.77,1.77,4.81,2.00,2.00,2.52,KL-SU3 168 | 167,1.39,3.00,1.87,3.84,3.81,2.29,KL-AN1 169 | 168,1.68,3.32,2.10,4.10,3.90,2.71,KL-AN2 170 | 169,1.39,3.10,1.84,3.52,3.94,2.35,KL-AN3 171 | 170,1.45,3.06,1.60,3.27,4.48,2.77,KL-DI1 172 | 171,1.16,4.03,2.10,3.52,4.42,2.87,KL-DI2 173 | 172,1.10,3.35,2.32,3.32,4.52,2.74,KL-DI3 174 | 173,1.16,3.00,1.97,3.06,4.35,2.55,KL-DI4 175 | 174,1.26,3.03,4.19,2.94,4.45,4.10,KL-FE1 176 | 175,1.19,3.29,3.03,3.52,4.16,3.45,KL-FE2 177 | 176,1.23,3.16,2.74,3.55,4.29,3.39,KL-FE3 178 | 177,2.84,2.58,1.87,1.84,1.87,1.84,TM-NE1 179 | 178,3.06,2.32,1.52,1.84,1.71,1.68,TM-NE2 180 | 179,2.97,1.90,1.42,1.61,1.52,1.32,TM-NE3 181 | 180,4.32,1.52,1.26,1.13,1.13,1.26,TM-HA1 182 | 181,4.35,1.65,1.13,1.13,1.26,1.55,TM-HA2 183 | 182,4.61,1.68,1.52,1.29,1.32,1.45,TM-HA3 184 | 183,4.81,1.58,1.42,1.29,1.26,1.39,TM-HA4 185 | 184,1.68,4.10,1.45,1.74,2.23,2.29,TM-SA1 186 | 185,1.48,4.23,1.45,1.84,2.52,2.48,TM-SA2 187 | 186,1.61,3.87,1.55,2.35,3.03,2.48,TM-SA3 188 | 187,1.65,2.77,4.61,2.10,2.87,3.74,TM-SU1 189 | 188,2.03,2.03,4.58,1.87,1.94,2.81,TM-SU2 190 | 189,1.77,2.52,4.77,2.03,2.13,3.32,TM-SU3 191 | 190,1.42,3.19,1.77,4.06,4.00,2.58,TM-AN1 192 | 191,1.39,2.58,1.74,4.13,3.48,1.94,TM-AN2 193 | 192,1.55,2.16,1.77,4.16,3.32,1.94,TM-AN3 194 | 193,1.35,1.81,2.19,3.16,4.81,2.39,TM-DI1 195 | 194,1.55,2.39,2.32,3.23,4.65,2.32,TM-DI2 196 | 195,1.35,2.39,1.90,3.10,4.55,2.87,TM-DI3 197 | 196,1.42,3.06,3.29,2.52,4.16,3.45,TM-FE1 198 | 197,1.45,2.77,3.13,2.32,4.39,3.10,TM-FE2 199 | 198,1.35,3.06,3.35,2.39,4.19,3.42,TM-FE3 200 | 199,2.68,2.10,1.61,2.19,2.03,1.68,NA-NE1 201 | 200,2.84,2.29,1.87,2.65,2.45,2.03,NA-NE2 202 | 201,2.77,2.00,1.84,2.32,2.23,2.61,NA-NE3 203 | 202,4.77,1.35,1.42,1.13,1.13,1.16,NA-HA1 204 | 203,4.65,1.47,1.29,1.23,1.32,1.10,NA-HA2 205 | 204,4.48,1.42,1.48,1.32,1.55,1.35,NA-HA3 206 | 205,1.52,4.03,1.77,2.65,3.03,2.94,NA-SA1 207 | 206,1.29,3.74,1.39,3.00,3.23,2.29,NA-SA2 208 | 207,1.71,3.16,1.84,3.48,3.19,2.58,NA-SA3 209 | 208,4.32,1.48,3.32,1.35,1.39,1.39,NA-SU1 210 | 209,2.68,1.45,4.52,1.90,2.03,2.00,NA-SU2 211 | 210,2.39,1.55,4.48,1.97,1.84,1.81,NA-SU3 212 | 211,1.19,2.55,1.61,4.45,3.65,1.71,NA-AN1 213 | 212,1.45,2.39,1.67,4.52,3.32,1.71,NA-AN2 214 | 213,1.32,2.61,1.61,4.48,3.23,1.94,NA-AN3 215 | 214,1.32,3.03,1.58,3.65,4.29,1.87,NA-DI1 216 | 215,1.45,3.19,1.81,3.16,4.19,2.77,NA-DI2 217 | 216,1.43,2.87,1.77,4.33,3.87,2.10,NA-DI3 218 | 217,1.61,2.68,4.10,3.16,3.81,3.90,NA-FE1 219 | 218,1.68,3.10,3.74,3.19,3.58,3.87,NA-FE2 220 | 219,1.48,3.26,3.39,2.71,3.06,3.74,NA-FE3 221 | -------------------------------------------------------------------------------- /params/drunk.yaml: -------------------------------------------------------------------------------- 1 | # A simple gen filt to produce a "drunk" set of randomly generated images. 2 | --- 3 | mode: drunk # This mode is similar to 'random', except each subsequent 4 | # generation is created by taking a random step from the 5 | # previous generation. This produces an erratic, yet 6 | # coherent animation. 7 | 8 | constrained: False 9 | 10 | or: 0 # If we set id/em/or parameters here, they'll stay constant. 11 | 12 | id_step: 0.2 # The amount of random step to move in each generation. 13 | em_step: 0.2 # In this case, a random value between -0.1 and 0.1 will be 14 | # added to each parameter value. 15 | 16 | min: 0 # However, we'll want to set min and max values for our 17 | max: 1 # min and max values so they don't step too far out. 18 | 19 | fps: 30 # Set a frames per second for this set of images. 20 | 21 | num_images: 10s # If an 's' is appended to a number, we'll interpret it as 22 | # seconds and calculate the number of frames based on the 23 | # specified fps. 24 | -------------------------------------------------------------------------------- /params/interpolate.yaml: -------------------------------------------------------------------------------- 1 | # Now let's interpolate between parameters to create an animation! 2 | --- 3 | mode: interpolate # When using mode 'interpolate', we'll have to provide a 4 | # set of keyframes with parameters to interpolate between. 5 | 6 | constrained: True 7 | fps: 30 8 | 9 | # Let's specify an animation: 10 | keyframes: 11 | # Person #1 goes from "happy" to "sad" 12 | - 13 | id: 1 14 | em: happy 15 | - 16 | id: 1 17 | em: sad 18 | length: 1s # We specify how many frames it should take to get to each 19 | # keyframe except for the first one. If a number ends with 20 | # an 's', we'll treat that as seconds and calculate the 21 | # number of frames based on the fps. 22 | 23 | # Now they're going to get jaded 24 | - 25 | id: 1 26 | em: neutral 27 | length: 1s 28 | 29 | # Now let's do something weird 30 | - 31 | id: 1+3+5 # We can use a '+' to add identity vectors. Note that if 32 | # 'constrained' is set to True, these will be kept at unit 33 | # length 34 | em: sad+happy # We can also do the same with emotions 35 | length: 2s 36 | 37 | -------------------------------------------------------------------------------- /params/random.yaml: -------------------------------------------------------------------------------- 1 | # A simple gen file to produce a image with random, unconstrained parameters. 2 | --- 3 | mode: random # Use mode 'random' to produce random images 4 | 5 | constrained: False # Unconstrain parameters, allowing "illegal" inputs. 6 | # Typically the inputs are constrained to unit length (or 7 | # satisfy [sin(x) cos(x)] for orientation), but disabling 8 | # this allows for some wild, unexpected generations. 9 | 10 | id_scale: 2.0 # Scale the identity vector by 2 11 | em_scale: 2.0 # Scale the emotion vector by 2 as well 12 | 13 | num_images: 100 # Produce one hundred random images 14 | -------------------------------------------------------------------------------- /params/single.yaml: -------------------------------------------------------------------------------- 1 | # A simple gen file to produce a single image of a happy person. 2 | --- 3 | mode: single # Use mode 'single' to produce a single image 4 | id: 3 # We want person #3 5 | em: happy # And we want them to be happy 6 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | dask==0.13.0 2 | decorator==4.0.10 3 | h5py==2.6.0 4 | Keras==1.2.0 5 | networkx==1.11 6 | numpy==1.11.3 7 | olefile==0.44 8 | Pillow==4.0.0 9 | PyYAML==3.12 10 | scikit-image==0.12.3 11 | scipy==0.18.1 12 | six==1.10.0 13 | Theano==0.8.2 14 | toolz==0.8.2 15 | tqdm==4.10.0 16 | --------------------------------------------------------------------------------