├── funit ├── __init__.py ├── .gitignore ├── tools │ └── extract_animalfaces.py ├── data.py ├── configs │ └── funit_animals.yaml ├── try_k_shot.py ├── train.py ├── README.md ├── funit_model.py ├── trainer.py ├── blocks.py ├── utils.py ├── networks.py └── LICENSE.md ├── nntools ├── __init__.py └── maybe_cuda.py ├── monster_mirror ├── __init__.py ├── test_integration.py ├── LICENSE ├── localcam.py └── spooky.py ├── pretrained-models ├── .gitkeep └── .gitignore ├── sfd_pytorch ├── .gitignore ├── __init__.py ├── iou.pyx ├── README.md ├── test_integration.py ├── LICENSE ├── livecam.py ├── wider_eval_pytorch.py ├── bbox.py ├── detect_faces.py └── net_s3fd.py ├── samples ├── leo-dog.gif └── ellen-selfie.jpg ├── target-images ├── tiger │ ├── LICENSE │ └── tiger-face.jpg ├── pomeranian │ ├── pomeranian-1.jpg │ ├── pomeranian-2.jpg │ └── LICENSE └── meerkat │ ├── n02138441_1512-174_67_408_267_0.999992.jpg │ ├── n02138441_280-143_11_438_245_0.999972.jpg │ ├── n02138441_390-123_44_362_247_0.999989.jpg │ ├── n02138441_75-167_65_407_273_0.999893.jpg │ └── n02138441_763-141_168_340_352_0.999998.jpg ├── requirements.txt ├── run-spooky.sh ├── run-furry.sh ├── download-pretrained.sh ├── .gitignore ├── README.md └── LICENSE /funit/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /nntools/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /monster_mirror/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /pretrained-models/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /sfd_pytorch/.gitignore: -------------------------------------------------------------------------------- 1 | data/ 2 | *.pth 3 | 4 | -------------------------------------------------------------------------------- /funit/.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | *.tar.gz 3 | *ipynb 4 | *.zip 5 | *.pkl 6 | *.pyc 7 | -------------------------------------------------------------------------------- /pretrained-models/.gitignore: -------------------------------------------------------------------------------- 1 | *.pt 2 | *.pth 3 | *.tgz 4 | *.tar.gz 5 | animal* 6 | s3fd_* 7 | -------------------------------------------------------------------------------- /samples/leo-dog.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/samples/leo-dog.gif -------------------------------------------------------------------------------- /target-images/tiger/LICENSE: -------------------------------------------------------------------------------- 1 | CC BY 2.0 https://www.flickr.com/photos/pokerbrit/10971343685 2 | 3 | -------------------------------------------------------------------------------- /samples/ellen-selfie.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/samples/ellen-selfie.jpg -------------------------------------------------------------------------------- /sfd_pytorch/__init__.py: -------------------------------------------------------------------------------- 1 | from .net_s3fd import S3fd_Model 2 | from .detect_faces import detect_faces 3 | -------------------------------------------------------------------------------- /target-images/tiger/tiger-face.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/tiger/tiger-face.jpg -------------------------------------------------------------------------------- /target-images/pomeranian/pomeranian-1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/pomeranian/pomeranian-1.jpg -------------------------------------------------------------------------------- /target-images/pomeranian/pomeranian-2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/pomeranian/pomeranian-2.jpg -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | torch 2 | torchvision 3 | numpy 4 | timebudget>=0.7 5 | opencv-python 6 | gdown 7 | pillow 8 | pyyaml 9 | pytest 10 | -------------------------------------------------------------------------------- /target-images/meerkat/n02138441_1512-174_67_408_267_0.999992.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/meerkat/n02138441_1512-174_67_408_267_0.999992.jpg -------------------------------------------------------------------------------- /target-images/meerkat/n02138441_280-143_11_438_245_0.999972.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/meerkat/n02138441_280-143_11_438_245_0.999972.jpg -------------------------------------------------------------------------------- /target-images/meerkat/n02138441_390-123_44_362_247_0.999989.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/meerkat/n02138441_390-123_44_362_247_0.999989.jpg -------------------------------------------------------------------------------- /target-images/meerkat/n02138441_75-167_65_407_273_0.999893.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/meerkat/n02138441_75-167_65_407_273_0.999893.jpg -------------------------------------------------------------------------------- /target-images/meerkat/n02138441_763-141_168_340_352_0.999998.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leopd/MonsterMirror/HEAD/target-images/meerkat/n02138441_763-141_168_340_352_0.999998.jpg -------------------------------------------------------------------------------- /target-images/pomeranian/LICENSE: -------------------------------------------------------------------------------- 1 | pomeranian-1.jpg Public domain https://commons.wikimedia.org/wiki/File:Pomeranian.JPG 2 | pomeranian-2.jpg CC BY-SA 3.0 https://en.wikipedia.org/wiki/File:Smiling_Tan_Pomeranian.jpg 3 | -------------------------------------------------------------------------------- /nntools/maybe_cuda.py: -------------------------------------------------------------------------------- 1 | import torch 2 | 3 | def mbcuda(t: torch.Tensor) -> torch.Tensor: 4 | """Short for "maybe cuda" this moves a tensor (or model) to cuda if available 5 | """ 6 | if torch.cuda.is_available(): 7 | return t.cuda() 8 | else: 9 | return t 10 | 11 | -------------------------------------------------------------------------------- /run-spooky.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | python monster_mirror/localcam.py \ 4 | --extra_detail 1 \ 5 | --max_faces 7 \ 6 | --cycle_delay 7 \ 7 | -t meerkat,pomeranian,tiger \ 8 | --color_map 0.5,1,0.7 \ 9 | --scale_embedding 0.15 \ 10 | --grow_facebox 1.0 \ 11 | --noise_mag 3 \ 12 | --noise_speed 2.5 \ 13 | --noise_drift 0.3 \ 14 | $@ 15 | 16 | -------------------------------------------------------------------------------- /run-furry.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | python monster_mirror/localcam.py \ 4 | --extra_detail 1 \ 5 | --max_faces 7 \ 6 | --cycle_delay 15 \ 7 | -t meerkat,pomeranian,tiger \ 8 | --color_map 1,1,1 \ 9 | --scale_embedding 1.0 \ 10 | --grow_facebox 1.0 \ 11 | --noise_mag 5 \ 12 | --noise_speed 2.5 \ 13 | --noise_drift 0.3 \ 14 | --max_alpha 0.85 \ 15 | $@ 16 | 17 | -------------------------------------------------------------------------------- /download-pretrained.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | set -x 5 | cd pretrained-models 6 | 7 | echo "Downloading face detector" 8 | wget https://github.com/leopd/SFD_pytorch/releases/download/0.3/s3fd_model.tgz 9 | tar xvzf s3fd_model.tgz 10 | rm s3fd_model.tgz 11 | 12 | echo "Downloading GANimal model" 13 | pip install gdown 14 | gdown https://drive.google.com/uc?id=1CsmSSWyMngtOLUL5lI-sEHVWc2gdJpF9 15 | tar xvzf pretrained.tar.gz 16 | rm pretrained.tar.gz 17 | rm animal119_gen_00100000.pt # we use the other one. 18 | -------------------------------------------------------------------------------- /sfd_pytorch/iou.pyx: -------------------------------------------------------------------------------- 1 | # cython iou.pyx 2 | # gcc -c -fPIC -I/usr/include/python2.7 -o iou.o iou.c 3 | # gcc -shared -L/usr/lib/x86_64-linux-gnu -lpython2.7 iou.o -o iou.so 4 | 5 | def IOU(float ax1,float ay1,float ax2,float ay2,float bx1,float by1,float bx2,float by2): 6 | cdef float sa,sb,x1,y1,x2,y2,w,h 7 | sa = abs((ax2-ax1)*(ay2-ay1)) 8 | sb = abs((bx2-bx1)*(by2-by1)) 9 | x1,y1 = max(ax1,bx1),max(ay1,by1) 10 | x2,y2 = min(ax2,bx2),min(ay2,by2) 11 | w = x2 - x1 12 | h = y2 - y1 13 | if w<0 or h<0: return 0.0 14 | else: return 1.0*w*h/(sa+sb-w*h) -------------------------------------------------------------------------------- /sfd_pytorch/README.md: -------------------------------------------------------------------------------- 1 | # S³FD: Single Shot Scale-invariant Face Detector 2 | A PyTorch Implementation of Single Shot Scale-invariant Face Detector. 3 | 4 | ## Eval 5 | ``` 6 | python wider_eval_pytorch.py 7 | 8 | cd eval/eval_tools_old-version 9 | octave wider_eval_pytorch.m 10 | ``` 11 | ## Model 12 | [s3fd_convert.7z](https://github.com/clcarwin/SFD_pytorch/releases/tag/v0.1) 13 | 14 | ## Test 15 | ``` 16 | python test.py --model data/s3fd_convert.pth --path data/test01.jpg 17 | ``` 18 | ![output](data/test01_output.png) 19 | 20 | # References 21 | [SFD](https://github.com/sfzhang15/SFD) 22 | -------------------------------------------------------------------------------- /sfd_pytorch/test_integration.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | import os 4 | import pytest 5 | import torch 6 | 7 | from nntools.maybe_cuda import mbcuda 8 | 9 | from .detect_faces import detect_faces 10 | from .net_s3fd import S3fd_Model 11 | 12 | def test_ellen_selfie(): 13 | model = S3fd_Model() 14 | try: 15 | state_dict = torch.load("pretrained-models/s3fd_convert.pth") 16 | model.load_state_dict(state_dict) 17 | except: 18 | print("Failed to load pre-trained model for test") 19 | raise 20 | mbcuda(model) 21 | model.eval() 22 | with torch.no_grad(): 23 | img = cv2.imread('samples/ellen-selfie.jpg') 24 | faces = detect_faces(model, img) 25 | assert len(faces) == 11 26 | -------------------------------------------------------------------------------- /sfd_pytorch/LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 carwin 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /funit/tools/extract_animalfaces.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import os 7 | import argparse 8 | 9 | from PIL import Image 10 | 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('imagenet_folder', type=str) 13 | parser.add_argument('--output_folder', type=str) 14 | parser.add_argument('--coor_file', type=str) 15 | opts = parser.parse_args() 16 | IMAGENET_TRAIN = opts.imagenet_folder 17 | OUT_PUT_FOLDER = opts.output_folder 18 | COOR_FILE = opts.coor_File 19 | 20 | with open(COOR_FILE, 'rt') as f: 21 | lines = f.readlines() 22 | 23 | for l in lines: 24 | ls = l.strip().split(' ') 25 | img_name = os.path.join(IMAGENET_TRAIN, ls[0]) 26 | img = Image.open(img_name) 27 | img = img.convert('RGB') 28 | x = int(ls[1]) 29 | y = int(ls[2]) 30 | w = int(ls[3]) 31 | h = int(ls[4]) 32 | out_name = os.path.join(OUT_PUT_FOLDER, 33 | '%s_%d_%d_%d_%d.jpg' % (ls[0], x, y, w, h)) 34 | crop = img.crop((x, y, w, h)) 35 | os.makedirs(os.path.dirname(out_name), exist_ok=True) 36 | print(out_name) 37 | crop.save(out_name) 38 | -------------------------------------------------------------------------------- /monster_mirror/test_integration.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | import cv2 4 | import numpy as np 5 | import monster_mirror.spooky as spooky 6 | 7 | def test_random_noise(): 8 | spookifier = spooky.RoundRobinSpookifier('target-images','meerkat') 9 | rand_img = np.random.uniform(0,255,(480,640,3)).astype(np.uint8) 10 | out_img = spookifier.process_npimage(rand_img, None) 11 | 12 | assert spookifier.face_transform_cnt == 0 13 | assert out_img.shape[0] == 480 14 | assert out_img.shape[1] == 640 15 | 16 | 17 | def test_ellen_cnn(): 18 | spookifier = spooky.RoundRobinSpookifier('target-images','meerkat', max_faces=6, face_detector_type='cnn') 19 | img = cv2.imread('samples/ellen-selfie.jpg') 20 | out_img = spookifier.process_npimage(img, None) 21 | 22 | assert spookifier.face_transform_cnt == 6 23 | assert out_img.shape[0] == 480 24 | #assert out_img.shape[1] == 640 #TODO figure out why this changes. 25 | 26 | def test_ellen_haar(): 27 | spookifier = spooky.RoundRobinSpookifier('target-images','meerkat', max_faces=6, face_detector_type='haar') 28 | img = cv2.imread('samples/ellen-selfie.jpg') 29 | out_img = spookifier.process_npimage(img, None) 30 | 31 | assert spookifier.face_transform_cnt == 6 32 | assert out_img.shape[0] == 480 33 | #assert out_img.shape[1] == 640 #TODO figure out why this changes. 34 | 35 | -------------------------------------------------------------------------------- /sfd_pytorch/livecam.py: -------------------------------------------------------------------------------- 1 | import time 2 | import torch 3 | import torch.nn as nn 4 | import torch.optim as optim 5 | import torch.nn.functional as F 6 | from torch.autograd import Variable 7 | torch.backends.cudnn.benchmark = True 8 | 9 | import os,sys,cv2,random,datetime,time,math 10 | import argparse 11 | import numpy as np 12 | 13 | from nntools.maybe_cuda import mbcuda 14 | 15 | import net_s3fd 16 | from detect_faces import detect_faces 17 | 18 | parser = argparse.ArgumentParser(description='PyTorch face detect') 19 | parser.add_argument('--net','-n', default='s3fd', type=str) 20 | parser.add_argument('--model', required=True, type=str) 21 | parser.add_argument('--path', default='CAMERA', type=str) 22 | 23 | args = parser.parse_args() 24 | use_cuda = torch.cuda.is_available() 25 | 26 | 27 | net = getattr(net_s3fd,args.net)() 28 | net.load_state_dict(torch.load(args.model)) 29 | mbcuda(net) 30 | net.eval() 31 | 32 | 33 | if args.path=='CAMERA': 34 | cap = cv2.VideoCapture(0) 35 | with torch.no_grad(): 36 | while(True): 37 | if args.path=='CAMERA': 38 | ret, img = cap.read() 39 | else: 40 | img = cv2.imread(args.path) 41 | 42 | imgshow = np.copy(img) 43 | start_time = time.time() 44 | bboxlist = detect_faces(net, img, 3) 45 | print(f"Running detect_faces took {1000*(time.time() - start_time):.1f}ms. Found {len(bboxlist)} faces.") 46 | for b in bboxlist: 47 | x1,y1,x2,y2,s = b 48 | cv2.rectangle(imgshow,(int(x1),int(y1)),(int(x2),int(y2)),(0,255,0),1) 49 | cv2.imshow('test',imgshow) 50 | 51 | if args.path=='CAMERA': 52 | if cv2.waitKey(1) & 0xFF == ord('q'): break 53 | else: 54 | cv2.imwrite(args.path[:-4]+'_output.png',imgshow) 55 | if cv2.waitKey(0) or True: break 56 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | 106 | # Mac 107 | .DS_Store 108 | 109 | # vim 110 | *.swp 111 | -------------------------------------------------------------------------------- /monster_mirror/LICENSE: -------------------------------------------------------------------------------- 1 | Modified BSD License for Live Performance Only 2 | 3 | Copyright (c) 2019, Leo Dirac 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. This software and any derivates must be used for live performances only. 17 | Any media content produced by this software may not be stored nor archived 18 | electronically. Any transmission of said media must be limited to immediate 19 | ephemeral display within clear view of any people whose likeness is being 20 | recorded, captured, or modified. Exception to this clause is only allowed 21 | with express written consent of every individual whose likeness is being 22 | recorded, captured, or modified, as well as any individual being portrayed, 23 | represented, or impersonated. 24 | 25 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 26 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 27 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 28 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 29 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 30 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 31 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 32 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 33 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 34 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 35 | -------------------------------------------------------------------------------- /funit/data.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import os.path 7 | from PIL import Image 8 | 9 | import torch.utils.data as data 10 | 11 | 12 | def default_loader(path): 13 | return Image.open(path).convert('RGB') 14 | 15 | 16 | def default_filelist_reader(filelist): 17 | im_list = [] 18 | with open(filelist, 'r') as rf: 19 | for line in rf.readlines(): 20 | im_path = line.strip() 21 | im_list.append(im_path) 22 | return im_list 23 | 24 | 25 | class ImageLabelFilelist(data.Dataset): 26 | def __init__(self, 27 | root, 28 | filelist, 29 | transform=None, 30 | filelist_reader=default_filelist_reader, 31 | loader=default_loader, 32 | return_paths=False): 33 | self.root = root 34 | self.im_list = filelist_reader(os.path.join(filelist)) 35 | self.transform = transform 36 | self.loader = loader 37 | self.classes = sorted( 38 | list(set([path.split('/')[0] for path in self.im_list]))) 39 | self.class_to_idx = {self.classes[i]: i for i in 40 | range(len(self.classes))} 41 | self.imgs = [(im_path, self.class_to_idx[im_path.split('/')[0]]) for 42 | im_path in self.im_list] 43 | self.return_paths = return_paths 44 | print('Data loader') 45 | print("\tRoot: %s" % root) 46 | print("\tList: %s" % filelist) 47 | print("\tNumber of classes: %d" % (len(self.classes))) 48 | 49 | def __getitem__(self, index): 50 | im_path, label = self.imgs[index] 51 | path = os.path.join(self.root, im_path) 52 | img = self.loader(path) 53 | if self.transform is not None: 54 | img = self.transform(img) 55 | if self.return_paths: 56 | return img, label, path 57 | else: 58 | return img, label 59 | 60 | def __len__(self): 61 | return len(self.imgs) 62 | -------------------------------------------------------------------------------- /funit/configs/funit_animals.yaml: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 2 | # Licensed under the CC BY-NC-SA 4.0 license 3 | # (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 4 | 5 | # logger options 6 | image_save_iter: 2500 # How often do you want to save output images during training 7 | image_display_iter: 100 # How often do you want to display output images during training 8 | snapshot_save_iter: 5000 # How often do you want to save trained models 9 | log_iter: 1 # How often do you want to log the training stats 10 | 11 | # optimization options 12 | max_iter: 100000 # maximum number of training iterations 13 | weight_decay: 0.0001 # weight decay 14 | lr_gen: 0.0001 # learning rate for the generator 15 | lr_dis: 0.0001 # learning rate for the discriminator 16 | init: kaiming # initialization [gaussian/kaiming/xavier/orthogonal] 17 | gan_w: 1 # weight of adversarial loss for image translation 18 | fm_w: 1 # weight on distance between gan features of style and translated image 19 | r_w: 0.1 # weight of image reconstruction loss 20 | 21 | # model options 22 | gen: 23 | nf: 64 # number of base filters in the generator 24 | n_res_blks: 2 # number of residual blocks in content encoder/decoder 25 | nf_mlp: 256 # number of base filters in MLP module 26 | latent_dim: 64 # dimension of the latent code for the class model 27 | n_mlp_blks: 3 # number of mlp blocks 28 | n_downs_content: 3 # number of downsampling layers in content encoder 29 | n_downs_class: 4 # number of downsampling layers in class model encoder 30 | dis: 31 | nf: 64 # base number of filters 32 | n_res_blks: 10 # number of residual blocks in the discriminator 33 | num_classes: 119 # number of classes in the training set 34 | 35 | # data options 36 | num_workers: 4 37 | batch_size: 64 38 | new_size: 140 # first resize the shortest image side to this size 39 | crop_image_height: 128 # random crop image of this height 40 | crop_image_width: 128 # random crop image of this width 41 | data_folder_train: ./datasets/animals 42 | data_list_train: ./datasets/animals_list_train.txt 43 | data_folder_test: ./datasets/animals 44 | data_list_test: ./datasets/animals_list_test.txt 45 | -------------------------------------------------------------------------------- /funit/try_k_shot.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import os 7 | import numpy as np 8 | from PIL import Image 9 | 10 | import torch 11 | import torch.backends.cudnn as cudnn 12 | from torchvision import transforms 13 | 14 | from nntools.maybe_cuda import mbcuda 15 | 16 | from .utils import get_config 17 | from .trainer import Trainer 18 | 19 | import argparse 20 | 21 | 22 | parser = argparse.ArgumentParser() 23 | parser.add_argument('--config', 24 | type=str, 25 | default='configs/funit_animals.yaml') 26 | parser.add_argument('--ckpt', 27 | type=str, 28 | default='pretrained/animal119_gen_00200000.pt') 29 | parser.add_argument('--class_image_folder', 30 | type=str, 31 | default='images/n02138411') 32 | parser.add_argument('--input', 33 | type=str, 34 | default='images/input_content.jpg') 35 | parser.add_argument('--output', 36 | type=str, 37 | default='images/output.jpg') 38 | opts = parser.parse_args() 39 | cudnn.benchmark = True 40 | opts.vis = True 41 | config = get_config(opts.config) 42 | config['batch_size'] = 1 43 | config['gpus'] = 1 44 | 45 | trainer = Trainer(config) 46 | mbcuda(trainer) 47 | trainer.load_ckpt(opts.ckpt) 48 | trainer.eval() 49 | 50 | transform_list = [transforms.ToTensor(), 51 | transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))] 52 | transform_list = [transforms.Resize((128, 128))] + transform_list 53 | transform = transforms.Compose(transform_list) 54 | 55 | print('Compute average class codes for images in %s' % opts.class_image_folder) 56 | images = os.listdir(opts.class_image_folder) 57 | for i, f in enumerate(images): 58 | fn = os.path.join(opts.class_image_folder, f) 59 | img = Image.open(fn).convert('RGB') 60 | img_tensor = mbcuda(transform(img).unsqueeze(0)) 61 | with torch.no_grad(): 62 | class_code = trainer.model.compute_k_style(img_tensor, 1) 63 | if i == 0: 64 | new_class_code = class_code 65 | else: 66 | new_class_code += class_code 67 | final_class_code = new_class_code / len(images) 68 | image = Image.open(opts.input) 69 | image = image.convert('RGB') 70 | content_img = transform(image).unsqueeze(0) 71 | 72 | print('Compute translation for %s' % opts.input) 73 | with torch.no_grad(): 74 | output_image = trainer.model.translate_simple(content_img, final_class_code) 75 | image = output_image.detach().cpu().squeeze().numpy() 76 | image = np.transpose(image, (1, 2, 0)) 77 | image = ((image + 1) * 0.5 * 255.0) 78 | output_img = Image.fromarray(np.uint8(image)) 79 | output_img.save(opts.output, 'JPEG', quality=99) 80 | print('Save output to %s' % opts.output) 81 | -------------------------------------------------------------------------------- /sfd_pytorch/wider_eval_pytorch.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | 3 | import torch 4 | import torch.nn as nn 5 | import torch.optim as optim 6 | import torch.nn.functional as F 7 | from torch.autograd import Variable 8 | # torch.backends.cudnn.bencmark = True 9 | 10 | import os,sys,cv2,random,datetime,math 11 | import argparse 12 | import numpy as np 13 | 14 | import scipy.io as sio 15 | import zipfile 16 | from net_s3fd import s3fd 17 | from bbox import * 18 | 19 | from detect_faces import detect 20 | 21 | def flip_detect(net,img): 22 | img = cv2.flip(img, 1) 23 | b = detect(net,img) 24 | 25 | bboxlist = np.zeros(b.shape) 26 | bboxlist[:, 0] = img.shape[1] - b[:, 2] 27 | bboxlist[:, 1] = b[:, 1] 28 | bboxlist[:, 2] = img.shape[1] - b[:, 0] 29 | bboxlist[:, 3] = b[:, 3] 30 | bboxlist[:, 4] = b[:, 4] 31 | return bboxlist 32 | 33 | def scale_detect(net,img,scale=2.0,facesize=None): 34 | img = cv2.resize(img,(0,0),fx=scale,fy=scale) 35 | b = detect(net,img) 36 | 37 | bboxlist = np.zeros(b.shape) 38 | bboxlist[:, 0] = b[:, 0]/scale 39 | bboxlist[:, 1] = b[:, 1]/scale 40 | bboxlist[:, 2] = b[:, 2]/scale 41 | bboxlist[:, 3] = b[:, 3]/scale 42 | bboxlist[:, 4] = b[:, 4] 43 | b = bboxlist 44 | if scale>1: index = np.where(np.minimum(b[:,2]-b[:,0]+1,b[:,3]-b[:,1]+1)facesize)[0] # only detect large face 46 | bboxlist = b[index,:] 47 | if 0==len(bboxlist): bboxlist=np.zeros((1, 5)) 48 | return bboxlist 49 | 50 | wider_face_mat = sio.loadmat('./eval/wider_face_val.mat') 51 | event_list = wider_face_mat['event_list'] 52 | file_list = wider_face_mat['file_list'] 53 | 54 | save_path = './eval/sfd_val_pytorch/' 55 | dataset = '../../dataset/face/WIDER/WIDER_val.zip' 56 | datazip = zipfile.ZipFile(dataset) 57 | 58 | net = s3fd() 59 | net.load_state_dict(torch.load('data/s3fd_convert.pth')) 60 | net.cuda() 61 | net.eval() 62 | 63 | # for i in range(1000): 64 | # size = 1024+64*i; print(size) 65 | # detect(net,np.zeros((size,size,3))) 66 | 67 | for index, event in enumerate(event_list): 68 | filelist = file_list[index][0] 69 | im_dir = event[0][0].encode('utf-8') 70 | if not os.path.exists(save_path + im_dir): os.makedirs(save_path + im_dir) 71 | 72 | for num, file in enumerate(filelist): 73 | im_name = file[0][0].encode('utf-8') 74 | zipname = '%s/%s.jpg' % (im_dir,im_name) 75 | 76 | data = np.frombuffer(datazip.read('WIDER_val/images/'+zipname),np.uint8) 77 | img = cv2.imdecode(data,1) 78 | 79 | imgshow = np.copy(img) 80 | b1 = detect(net,img) 81 | b2 = flip_detect(net,img) 82 | if img.shape[0]*img.shape[1]*4>3000*3000: b3 = np.zeros((1, 5)) 83 | else: b3 = scale_detect(net,img,scale=2,facesize=100) 84 | b4 = scale_detect(net,img,scale=0.5,facesize=100) 85 | bboxlist = np.concatenate((b1,b2,b3,b4)) 86 | 87 | keep = nms(bboxlist,0.3) 88 | keep = keep[0:750] # keep only max 750 boxes 89 | bboxlist = bboxlist[keep,:] 90 | 91 | # for b in bboxlist: 92 | # x1,y1,x2,y2,s = b 93 | # if s<0.5: continue 94 | # cv2.rectangle(imgshow,(int(x1),int(y1)),(int(x2),int(y2)),(0,0,255),1) 95 | # cv2.imshow('',imgshow) 96 | # cv2.waitKey(0) 97 | # continue 98 | 99 | f = open(save_path + im_dir + '/' + im_name + '.txt', 'w') 100 | f.write('{:s}\n'.format('%s/%s.jpg' % (im_dir,im_name))) 101 | f.write('{:d}\n'.format(len(bboxlist))) 102 | for b in bboxlist: 103 | x1,y1,x2,y2,s = b 104 | f.write('{:.1f} {:.1f} {:.1f} {:.1f} {:.3f}\n'.format(x1,y1,(x2-x1+1),(y2-y1+1),s)) 105 | f.close() 106 | print('event:%d num:%d' % (index + 1, num + 1)) 107 | -------------------------------------------------------------------------------- /sfd_pytorch/bbox.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os,sys,cv2,random,datetime,time,math 3 | import argparse 4 | import numpy as np 5 | import torch 6 | from timebudget import timebudget 7 | 8 | try: 9 | from iou import IOU 10 | except: 11 | # IOU cython speedup 10x 12 | def IOU(ax1,ay1,ax2,ay2,bx1,by1,bx2,by2): 13 | sa = abs((ax2-ax1)*(ay2-ay1)) 14 | sb = abs((bx2-bx1)*(by2-by1)) 15 | x1,y1 = max(ax1,bx1),max(ay1,by1) 16 | x2,y2 = min(ax2,bx2),min(ay2,by2) 17 | w = x2 - x1 18 | h = y2 - y1 19 | if w<0 or h<0: return 0.0 20 | else: return 1.0*w*h/(sa+sb-w*h) 21 | 22 | def bboxlog(x1,y1,x2,y2,axc,ayc,aww,ahh): 23 | xc,yc,ww,hh = (x2+x1)/2,(y2+y1)/2,x2-x1,y2-y1 24 | dx,dy = (xc-axc)/aww,(yc-ayc)/ahh 25 | dw,dh = math.log(ww/aww),math.log(hh/ahh) 26 | return dx,dy,dw,dh 27 | 28 | def bboxloginv(dx,dy,dw,dh,axc,ayc,aww,ahh): 29 | xc,yc = dx*aww+axc, dy*ahh+ayc 30 | ww,hh = math.exp(dw)*aww,math.exp(dh)*ahh 31 | x1,x2,y1,y2 = xc-ww/2,xc+ww/2,yc-hh/2,yc+hh/2 32 | return x1,y1,x2,y2 33 | 34 | @timebudget 35 | def nms(bboxlist:torch.Tensor, thresh:float) -> list: 36 | """Given an Nx5 tensor of bounding boxes, and a threshold, 37 | return a list of the indexes of bounding boxes to keep. 38 | """ 39 | if len(bboxlist) == 0: 40 | return [] 41 | x1 = bboxlist[:,0] 42 | y1 = bboxlist[:,1] 43 | x2 = bboxlist[:,2] 44 | y2 = bboxlist[:,3] 45 | scores = bboxlist[:,4] 46 | areas = (x2 - x1 + 1) * (y2 - y1 + 1) 47 | 48 | # Go through the boxes in order of decreasing score... 49 | scores = np.asarray(scores) 50 | order = scores.argsort() 51 | order = np.asarray(list(order[::-1])) 52 | keep = [] 53 | while len(order) > 0: 54 | i = order[0] 55 | keep.append(i) # Keep this one. 56 | 57 | # For all the remaining (lower score) bounding boxes, figure out something about the overlap I think 58 | xx1,yy1 = np.maximum(x1[i], x1[order[1:]]),np.maximum(y1[i], y1[order[1:]]) 59 | xx2,yy2 = np.minimum(x2[i], x2[order[1:]]),np.minimum(y2[i], y2[order[1:]]) 60 | w,h = np.maximum(0.0, xx2 - xx1 + 1),np.maximum(0.0, yy2 - yy1 + 1) 61 | ovr = w*h / (areas[i] + areas[order[1:]] - w*h) # looks like the overlap 62 | inds = np.where(ovr <= thresh)[0] 63 | order = order[inds + 1] # eliminate the ones that don't meet the threshhold 64 | 65 | return keep 66 | 67 | 68 | def encode(matched, priors, variances): 69 | """Encode the variances from the priorbox layers into the ground truth boxes 70 | we have matched (based on jaccard overlap) with the prior boxes. 71 | Args: 72 | matched: (tensor) Coords of ground truth for each prior in point-form 73 | Shape: [num_priors, 4]. 74 | priors: (tensor) Prior boxes in center-offset form 75 | Shape: [num_priors,4]. 76 | variances: (list[float]) Variances of priorboxes 77 | Return: 78 | encoded boxes (tensor), Shape: [num_priors, 4] 79 | """ 80 | 81 | # dist b/t match center and prior's center 82 | g_cxcy = (matched[:, :2] + matched[:, 2:])/2 - priors[:, :2] 83 | # encode variance 84 | g_cxcy /= (variances[0] * priors[:, 2:]) 85 | # match wh / prior wh 86 | g_wh = (matched[:, 2:] - matched[:, :2]) / priors[:, 2:] 87 | g_wh = torch.log(g_wh) / variances[1] 88 | # return target for smooth_l1_loss 89 | return torch.cat([g_cxcy, g_wh], 1) # [num_priors,4] 90 | 91 | def decode(loc, priors, variances): 92 | """Decode locations from predictions using priors to undo 93 | the encoding we did for offset regression at train time. 94 | Args: 95 | loc (tensor): location predictions for loc layers, 96 | Shape: [num_priors,4] 97 | priors (tensor): Prior boxes in center-offset form. 98 | Shape: [num_priors,4]. 99 | variances: (list[float]) Variances of priorboxes 100 | Return: 101 | decoded bounding box predictions 102 | """ 103 | 104 | boxes = torch.cat(( 105 | priors[:, :2] + loc[:, :2] * variances[0] * priors[:, 2:], 106 | priors[:, 2:] * torch.exp(loc[:, 2:] * variances[1])), 1) 107 | boxes[:, :2] -= boxes[:, 2:] / 2 108 | boxes[:, 2:] += boxes[:, :2] 109 | return boxes 110 | -------------------------------------------------------------------------------- /sfd_pytorch/detect_faces.py: -------------------------------------------------------------------------------- 1 | from PIL import Image 2 | import time 3 | import torch 4 | import torch.nn as nn 5 | import torch.optim as optim 6 | import torch.nn.functional as F 7 | from torch.autograd import Variable 8 | from typing import List, Tuple 9 | torch.backends.cudnn.benchmark = True 10 | 11 | import os,sys,cv2,random,datetime,time,math 12 | import argparse 13 | import numpy as np 14 | 15 | from timebudget import timebudget 16 | 17 | from .bbox import decode, nms 18 | from .net_s3fd import S3fd_Model 19 | from nntools.maybe_cuda import mbcuda 20 | 21 | # It's a trade-off. Not really clear which is faster or why. 22 | # On CPU, all the time (~40ms) is spent moving the data from GPU to CPU. 23 | # On GPU, all the time (~40ms) is spent on the *first* torch.nonzero for some reason. 24 | use_cpu_for_decoding_bbox = True 25 | 26 | def detect_faces(net:nn.Module, img:np.ndarray, minscale:int=3, ovr_threshhold:float=0.3, 27 | score_threshhold:float=0.5) -> List[Tuple]: 28 | """returns an list of tuples describing bounding boxes: [x1,y1,x2,y2,score]. 29 | Setting minscale to 0 finds the smallest faces, but takes the longest. 30 | """ 31 | bboxlist = detect(net, img, minscale) 32 | keep_idx = nms(bboxlist, ovr_threshhold) 33 | bboxlist = bboxlist[keep_idx,:] 34 | out = [] 35 | for b in bboxlist: 36 | x1,y1,x2,y2,s = b 37 | if s<0.5: 38 | continue 39 | out.append((int(x1),int(y1),int(x2),int(y2),s)) 40 | return out 41 | 42 | @timebudget 43 | def _process_bbox(i:torch.Tensor, ocls:torch.Tensor, oreg:torch.Tensor, hw:torch.Tensor) -> Tuple[torch.Tensor]: 44 | stride = 2**(i+2) # 4,8,16,32,64,128 45 | anchor = stride*4 46 | hindex = hw[0] 47 | windex = hw[1] 48 | score = ocls[0,1,hindex,windex] 49 | loc = oreg[0,:,hindex,windex].contiguous().view(1,4) 50 | variances = torch.Tensor([0.1,0.2]) 51 | return _process_bbox2(stride, anchor, score, loc, hindex, windex, variances) 52 | 53 | def _process_bbox2(stride, anchor, score, loc, hindex, windex, variances): 54 | axc,ayc = stride/2+windex*stride,stride/2+hindex*stride 55 | priors = torch.cat([axc/1.0,ayc/1.0,stride*4/1.0,stride*4/1.0]).unsqueeze(0) 56 | if not use_cpu_for_decoding_bbox: 57 | priors = mcuda(priors) 58 | variances = mbcuda(variances) 59 | box = decode(loc,priors,variances) 60 | x1,y1,x2,y2 = box[0]*1.0 61 | return (x1,y1,x2,y2,score) 62 | 63 | @timebudget 64 | def bboxlist_from_olist(olist:List[torch.Tensor], minscale:int=3) -> torch.Tensor: 65 | bboxlist = [] 66 | for i in range(minscale, len(olist)//2): 67 | ocls = F.softmax(olist[i*2], dim=1).data 68 | oreg = olist[i*2+1].data 69 | FB,FC,FH,FW = ocls.size() # feature map size 70 | stride = 2**(i+2) # 4,8,16,32,64,128 71 | anchor = stride*4 72 | all_scores = ocls[0,1,:,:] 73 | if use_cpu_for_decoding_bbox: 74 | with timebudget('move-to-cpu'): 75 | all_scores = all_scores.cpu() 76 | oreg = oreg.cpu() 77 | # instead of running a sliding window, first find the places where score is big enough to bother 78 | with timebudget('scan-bigenough'): 79 | # For some reason, this is crazy slow (38ms) on GPU, but only the first time it's called. 80 | bigenough = torch.nonzero(all_scores > 0.05) 81 | for hw in bigenough: 82 | i_t = torch.ones(1) * i 83 | bboxlist.append(_process_bbox(i_t, ocls, oreg, hw)) 84 | if len(bboxlist) == 0: 85 | bboxlist=torch.zeros((1, 5)) 86 | bboxlist = torch.Tensor(bboxlist) 87 | return bboxlist 88 | 89 | 90 | def olist_from_img(net:nn.Module, img:np.ndarray) -> List[torch.Tensor]: 91 | img = img - np.array([104,117,123]) 92 | img = img.transpose(2, 0, 1) 93 | img = img.reshape((1,)+img.shape) 94 | 95 | img = mbcuda(Variable(torch.from_numpy(img).float())) 96 | olist = net(img) 97 | return olist 98 | 99 | def detect(net:nn.Module, img:np.ndarray, minscale:int=3) -> torch.Tensor: 100 | """returns an Nx5 tensor describing bounding boxes: [x1,y1,x2,y2,score]. 101 | This will have LOTS of similar/overlapping regions. Need to call bbox.nms to reconcile them. 102 | Setting minscale to 0 finds the smallest faces, but takes the longest. 103 | """ 104 | olist = olist_from_img(net, img) 105 | return bboxlist_from_olist(olist, minscale) 106 | 107 | 108 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MonsterMirror 2 | ### An electronic funhouse mirror that changes people into animals and monsters 3 | 4 | Set this up for your haunted house at Halloween! Amuse your guests as their faces are dynamically transformed into different animals and monsters in front of their eyes. Watch the animals' faces and expression change as yours does. Feed it different example animals and monsters to see what it can generate. 5 | 6 | ![Leo as a dog](samples/leo-dog.gif) 7 | 8 | 9 | ## How does it work? 10 | 11 | It uses modern computer vision techniques based in deep learning to first locate any faces in the image, and then feed them through an encoder-decoder network to modify them into a new structure. For details see [this video about the FUNIT model](https://www.youtube.com/watch?v=kgPAqsC8PLM&feature=youtu.be) or the [project page](https://nvlabs.github.io/FUNIT/). 12 | 13 | 14 | ## How can I use it? 15 | 16 | Please respect people's privacy when using this system. This software must only be used for live performances. Recording or transmission of the created media is not allowed without express written consent. Moreover, the display must be in clear view of the people being captured. For details, see the [license](LICENSE). 17 | 18 | It will attempt to transform any faces it finds into "target" creatures you specify. It will cycle through different target creatures, using examples in sub-directories of the `target-images` directory. Feel free to play around with these yourself to see what it can and can't do. The base model was trained mostly on dogs, so it will do best with those. Setting the `scale_embedding` option can make it give a monster-like effect for targets it hasn't seen yet. A `color_map` is a blunt but effective tool for making people look more monstery. See `run-spooky.sh` and `run-furry.sh` for example configurations. The "noise" parameters cause the neural network to explore variations on what you've set as the target, and will result in other animal types. 19 | 20 | ### System requirements 21 | 22 | This has been tested on **Ubuntu** 18.04, and on **Mac OS X**. It will run **much faster** with an **NVIDIA GPU** with CUDA to run the neural nets. With a GPU, I see about 10 frames/sec (fps) which is fast enough to look like a mirror. Without a GPU, it will switch to a less accurate face detector model called a Haar cascade, and get maybe 1 fps. 23 | 24 | It _should_ work on **Windows**, but I don't have a windows box to test it on. If you try it, [let me know](https://github.com/leopd/MonsterMirror/issues/1). 25 | 26 | ### Installing 27 | 28 | Setting up the python dependencies can be done lots of ways. I like using conda and pip: 29 | 30 | ``` 31 | conda create -n monster_mirror python=3.6 32 | conda activate monster_mirror 33 | pip install -r requirements.txt 34 | ``` 35 | 36 | You must download the pre-trained models. (Setup python first, since I use the `gdown` python library to download the GANimal model.) To get the dependencies run 37 | 38 | ``` 39 | ./download-pretrained.sh 40 | ``` 41 | 42 | Check that it's working with 43 | 44 | ``` 45 | pytest 46 | ``` 47 | 48 | 49 | ### Running it 50 | 51 | There are lots of (too many) options on the `localcam.py` file. For some nice presets, try just running 52 | 53 | ``` 54 | ./run-furry.sh # animal faces 55 | ``` 56 | 57 | or 58 | 59 | ``` 60 | ./run-spooky.sh # ghoulish monster look 61 | ``` 62 | 63 | You can modify each of these by adding extra arguments like 64 | 65 | ``` 66 | ./run-furry.sh --no_full_screen --target_classes tiger,pomeranian 67 | ``` 68 | 69 | Drop your own images into new folders under the `target-images` directory to create new `target_classes`. 70 | 71 | To really dive in, see the full set of options with 72 | 73 | ``` 74 | ./run-furry.sh --help 75 | ``` 76 | 77 | 78 | ## Acknowledgements 79 | 80 | This project copies the code in from two key dependencies. (Git sub-modules are arguably the "correct" way to do this, but whenever I've tried to use sub-modules I've gotten very confused, and folks I know who have worked with them professionally have told me they're intrinsically broken.) 81 | 82 | The key model in this software is the GANimal / FUNIT model from NVIDIA: [https://github.com/NVLabs/FUNIT](https://github.com/NVLabs/FUNIT) which is licensed [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). 83 | 84 | * `sfd_pytorch` is used for face detection. Originally from [clcarwin](https://github.com/clcarwin/SFD_pytorch) under MIT license. I've cleaned up the code somewhat in [a fork](https://github.com/leopd/SFD_pytorch). 85 | 86 | I'll give a plug for my library [timebudget](https://github.com/leopd/timebudget) which I built while writing MonsterMirror. I found it instrumental in understanding where the time was going, and helped me get this from 0.5 frames/sec up to about 10 fps. 87 | 88 | Other depencies in `requirements.txt`. 89 | -------------------------------------------------------------------------------- /funit/train.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import torch 7 | import os 8 | import sys 9 | import argparse 10 | import shutil 11 | 12 | from tensorboardX import SummaryWriter 13 | 14 | from .utils import get_config, get_train_loaders, make_result_folders 15 | from .utils import write_loss, write_html, write_1images, Timer 16 | from .trainer import Trainer 17 | 18 | import torch.backends.cudnn as cudnn 19 | # Enable auto-tuner to find the best algorithm to use for your hardware. 20 | cudnn.benchmark = True 21 | 22 | parser = argparse.ArgumentParser() 23 | parser.add_argument('--config', 24 | type=str, 25 | default='configs/funit_animals.yaml', 26 | help='configuration file for training and testing') 27 | parser.add_argument('--output_path', 28 | type=str, 29 | default='.', 30 | help="outputs path") 31 | parser.add_argument('--multigpus', 32 | action="store_true") 33 | parser.add_argument('--batch_size', 34 | type=int, 35 | default=0) 36 | parser.add_argument('--test_batch_size', 37 | type=int, 38 | default=4) 39 | parser.add_argument("--resume", 40 | action="store_true") 41 | opts = parser.parse_args() 42 | 43 | # Load experiment setting 44 | config = get_config(opts.config) 45 | max_iter = config['max_iter'] 46 | # Override the batch size if specified. 47 | if opts.batch_size != 0: 48 | config['batch_size'] = opts.batch_size 49 | 50 | trainer = Trainer(config) 51 | trainer.cuda() 52 | if opts.multigpus: 53 | ngpus = torch.cuda.device_count() 54 | config['gpus'] = ngpus 55 | print("Number of GPUs: %d" % ngpus) 56 | trainer.model = torch.nn.DataParallel( 57 | trainer.model, device_ids=range(ngpus)) 58 | else: 59 | config['gpus'] = 1 60 | 61 | loaders = get_train_loaders(config) 62 | train_content_loader = loaders[0] 63 | train_class_loader = loaders[1] 64 | test_content_loader = loaders[2] 65 | test_class_loader = loaders[3] 66 | 67 | # Setup logger and output folders 68 | model_name = os.path.splitext(os.path.basename(opts.config))[0] 69 | train_writer = SummaryWriter( 70 | os.path.join(opts.output_path + "/logs", model_name)) 71 | output_directory = os.path.join(opts.output_path + "/outputs", model_name) 72 | checkpoint_directory, image_directory = make_result_folders(output_directory) 73 | shutil.copy(opts.config, os.path.join(output_directory, 'config.yaml')) 74 | 75 | iterations = trainer.resume(checkpoint_directory, 76 | hp=config, 77 | multigpus=opts.multigpus) if opts.resume else 0 78 | 79 | while True: 80 | for it, (co_data, cl_data) in enumerate( 81 | zip(train_content_loader, train_class_loader)): 82 | with Timer("Elapsed time in update: %f"): 83 | d_acc = trainer.dis_update(co_data, cl_data, config) 84 | g_acc = trainer.gen_update(co_data, cl_data, config, 85 | opts.multigpus) 86 | torch.cuda.synchronize() 87 | print('D acc: %.4f\t G acc: %.4f' % (d_acc, g_acc)) 88 | 89 | if (iterations + 1) % config['log_iter'] == 0: 90 | print("Iteration: %08d/%08d" % (iterations + 1, max_iter)) 91 | write_loss(iterations, trainer, train_writer) 92 | 93 | if ((iterations + 1) % config['image_save_iter'] == 0 or ( 94 | iterations + 1) % config['image_display_iter'] == 0): 95 | if (iterations + 1) % config['image_save_iter'] == 0: 96 | key_str = '%08d' % (iterations + 1) 97 | write_html(output_directory + "/index.html", iterations + 1, 98 | config['image_save_iter'], 'images') 99 | else: 100 | key_str = 'current' 101 | with torch.no_grad(): 102 | for t, (val_co_data, val_cl_data) in enumerate( 103 | zip(train_content_loader, train_class_loader)): 104 | if t >= opts.test_batch_size: 105 | break 106 | val_image_outputs = trainer.test(val_co_data, val_cl_data, 107 | opts.multigpus) 108 | write_1images(val_image_outputs, image_directory, 109 | 'train_%s_%02d' % (key_str, t)) 110 | for t, (test_co_data, test_cl_data) in enumerate( 111 | zip(test_content_loader, test_class_loader)): 112 | if t >= opts.test_batch_size: 113 | break 114 | test_image_outputs = trainer.test(test_co_data, 115 | test_cl_data, 116 | opts.multigpus) 117 | write_1images(test_image_outputs, image_directory, 118 | 'test_%s_%02d' % (key_str, t)) 119 | 120 | if (iterations + 1) % config['snapshot_save_iter'] == 0: 121 | trainer.save(checkpoint_directory, iterations, opts.multigpus) 122 | print('Saved model at iteration %d' % (iterations + 1)) 123 | 124 | iterations += 1 125 | if iterations >= max_iter: 126 | print("Finish Training") 127 | sys.exit(0) 128 | -------------------------------------------------------------------------------- /funit/README.md: -------------------------------------------------------------------------------- 1 | [![License CC BY-NC-SA 4.0](https://img.shields.io/badge/license-CC4.0-blue.svg)](LICENSE.md) 2 | ![Python 3.7](https://img.shields.io/badge/python-3.7-green.svg) 3 | 4 | # FUNIT: Few-Shot Unsupervised Image-to-Image Translation 5 | ![animal swap gif](docs/images/animal.gif) 6 | 7 | ### [Project page](https://nvlabs.github.io/FUNIT/) | [Paper](https://arxiv.org/abs/1905.01723) | [FUNIT Explained](https://youtu.be/kgPAqsC8PLM) | [GANimal Demo Video](https://youtu.be/JTu-U0C4xEU) | [Have fun with GANimal](https://nvlabs.github.io/FUNIT/ganimal.html) 8 | 9 | 10 | Few-shot Unsueprvised Image-to-Image Translation
11 | [Ming-Yu Liu](http://mingyuliu.net/), [Xun Huang](http://www.cs.cornell.edu/~xhuang/), [Arun Mallya](http://arunmallya.com/), [Tero Karras](https://research.nvidia.com/person/tero-karras), [Timo Aila](https://users.aalto.fi/~ailat1/), [Jaakko Lehtinen](https://users.aalto.fi/~lehtinj7/), and [Jan Kautz](http://jankautz.com/).
12 | In arXiv 2019. 13 | 14 | 15 | ### [License](https://raw.githubusercontent.com/nvlabs/FUNIT/master/LICENSE.md) 16 | 17 | Copyright (C) 2019 NVIDIA Corporation. 18 | 19 | All rights reserved. 20 | Licensed under the [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode) (**Attribution-NonCommercial-ShareAlike 4.0 International**) 21 | 22 | The code is released for academic research use only. For commercial use, please contact [researchinquiries@nvidia.com](researchinquiries@nvidia.com). 23 | 24 | ## Installation 25 | 26 | - Clone this repo `git clone https://github.com/NVlabs/FUNIT.git` 27 | - Install [CUDA10.0+](https://developer.nvidia.com/cuda-downloads) 28 | - Install [cuDNN7.5](https://developer.nvidia.com/cudnn) 29 | - Install [Anaconda3](https://www.anaconda.com/distribution/) 30 | - Install required python pakcages 31 | - `conda install -y pytorch torchvision cudatoolkit=10.0 -c pytorch` 32 | - `conda install -y -c anaconda pip` 33 | - `pip install pyyaml tensorboardX` 34 | - `conda install -y -c menpo opencv3` 35 | 36 | To reproduce the results reported in the paper, you would need an **NVIDIA DGX1 machine with 8 V100 GPUs**. 37 | 38 | ## Hardware Requirement 39 | 40 | To reproduce the experiment results reported in our ICCV paper, you would need to have an NVIDIA DGX1 machine with 8 V100 32GB GPUs. The training will use all 8 GPUS and take almost all of the GPU memory. It would take about 2 weeks to finish the training. 41 | 42 | 43 | ## Dataset Preparation 44 | 45 | ### Animal Face Dataset 46 | 47 | We are releasing the Animal Face dataset. **If you use this dataset in your publication, please cite the FUNIT paper.** 48 | 49 | - The dataset consists of image crops of the [ImageNet ILSVRC2012 training set](http://www.image-net.org/challenges/LSVRC/2012/nonpub-downloads). Download the dataset and untar the files 50 | ``` 51 | cd dataset 52 | wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_train.tar 53 | tar xvf ILSVRC2012_img_train.tar 54 | ``` 55 | - The training images should be in `datasets/ILSVRC/Data/CLS-LOC/train`. Now, extract the animal face images by running 56 | ``` 57 | python tools/extract_animalfaces.py datasets/ILSVRC/Data/CLS-LOC/train --output_folder datasets/animals --coor_file datasets/animalface_coordinates.txt 58 | ``` 59 | - The animal face images should be in `datasets/animals`. Note there are 149 folders. Each folder contains images of one animal kind. The number of images of the dataset is 117,484. 60 | - We use 119 animal kinds for training and the ramining 30 animal kinds for evaluation. 61 | 62 | ## Training 63 | 64 | Once the animal face dataset is prepared, you can train an animal face translation model by running. 65 | 66 | ```bash 67 | python train.py --config configs/funit_animals.yaml --multigpus 68 | ``` 69 | 70 | The training results including the checkpoints and intermediate results will be stored in `outputs/funit_animals`. 71 | 72 | For custom dataset, you would need to write an new configuration file. Please create one based on the [example config file](configs/funit_animals.yaml). 73 | 74 | ## Testing pretrained model 75 | 76 | To test the pretrained model, please first create a folder `pretrained` under the root folder. Then, we need to downlowad the pretrained models via the [link](https://drive.google.com/open?id=1CsmSSWyMngtOLUL5lI-sEHVWc2gdJpF9) and save it in `pretrained`. Untar the file `tar xvf pretrained.tar.gz`. 77 | 78 | Now, we can test the translation 79 | ```bash 80 | python test_k_shot.py --config configs/funit_animals.yaml --ckpt pretrained/animal149_gen.pt --input images/input_content.jpg --class_image_folder images/n02138411 --output images/output.jpg 81 | ``` 82 | 83 | The above command with translate the input image 84 | 85 | `images/input_content.jpg` 86 | 87 | ![input image](images/input_content.jpg) 88 | 89 | 90 | to an output meerkat image 91 | 92 | ![output image](images/output.jpg) 93 | 94 | by using a set of 5 example meerkat images 95 | 96 | ![](images/n02138411/n02138441_75-167_65_407_273_0.999893.jpg)![](images/n02138411/n02138441_280-143_11_438_245_0.999972.jpg)![](images/n02138411/n02138441_390-123_44_362_247_0.999989.jpg)![](images/n02138411/n02138441_763-141_168_340_352_0.999998.jpg)![](images/n02138411/n02138441_1512-174_67_408_267_0.999992.jpg) 97 | 98 | 99 | 100 | 101 | ### Citation 102 | If you use this code for your research, please cite our papers. 103 | ``` 104 | @inproceedings{liu2019few, 105 | title={Few-shot Unsueprvised Image-to-Image Translation}, 106 | author={Ming-Yu Liu and Xun Huang and Arun Mallya and Tero Karras and Timo Aila and Jaakko Lehtinen and Jan Kautz.}, 107 | booktitle={arxiv}, 108 | year={2019} 109 | } 110 | ``` 111 | -------------------------------------------------------------------------------- /sfd_pytorch/net_s3fd.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | from torch.autograd import Variable 4 | import torch.nn.functional as F 5 | 6 | class L2Norm(nn.Module): 7 | def __init__(self,n_channels, scale=1.0): 8 | super(L2Norm,self).__init__() 9 | self.n_channels = n_channels 10 | self.scale = scale 11 | self.eps = 1e-10 12 | self.weight = nn.Parameter(torch.Tensor(self.n_channels)) 13 | self.weight.data *= 0.0 14 | self.weight.data += self.scale 15 | 16 | def forward(self, x): 17 | norm = x.pow(2).sum(dim=1, keepdim=True).sqrt()+self.eps 18 | x = x / norm * self.weight.view(1,-1,1,1) 19 | return x 20 | 21 | class S3fd_Model(nn.Module): 22 | def __init__(self): 23 | super(s3fd, self).__init__() 24 | self.conv1_1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1) 25 | self.conv1_2 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1) 26 | 27 | self.conv2_1 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1) 28 | self.conv2_2 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1) 29 | 30 | self.conv3_1 = nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1) 31 | self.conv3_2 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1) 32 | self.conv3_3 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1) 33 | 34 | self.conv4_1 = nn.Conv2d(256, 512, kernel_size=3, stride=1, padding=1) 35 | self.conv4_2 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1) 36 | self.conv4_3 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1) 37 | 38 | self.conv5_1 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1) 39 | self.conv5_2 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1) 40 | self.conv5_3 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1) 41 | 42 | self.fc6 = nn.Conv2d(512, 1024, kernel_size=3, stride=1, padding=3) 43 | self.fc7 = nn.Conv2d(1024, 1024, kernel_size=1, stride=1, padding=0) 44 | 45 | self.conv6_1 = nn.Conv2d(1024, 256, kernel_size=1, stride=1, padding=0) 46 | self.conv6_2 = nn.Conv2d(256, 512, kernel_size=3, stride=2, padding=1) 47 | 48 | self.conv7_1 = nn.Conv2d(512, 128, kernel_size=1, stride=1, padding=0) 49 | self.conv7_2 = nn.Conv2d(128, 256, kernel_size=3, stride=2, padding=1) 50 | 51 | self.conv3_3_norm = L2Norm(256,scale=10) 52 | self.conv4_3_norm = L2Norm(512,scale=8) 53 | self.conv5_3_norm = L2Norm(512,scale=5) 54 | 55 | self.conv3_3_norm_mbox_conf = nn.Conv2d(256, 4, kernel_size=3, stride=1, padding=1) 56 | self.conv3_3_norm_mbox_loc = nn.Conv2d(256, 4, kernel_size=3, stride=1, padding=1) 57 | self.conv4_3_norm_mbox_conf = nn.Conv2d(512, 2, kernel_size=3, stride=1, padding=1) 58 | self.conv4_3_norm_mbox_loc = nn.Conv2d(512, 4, kernel_size=3, stride=1, padding=1) 59 | self.conv5_3_norm_mbox_conf = nn.Conv2d(512, 2, kernel_size=3, stride=1, padding=1) 60 | self.conv5_3_norm_mbox_loc = nn.Conv2d(512, 4, kernel_size=3, stride=1, padding=1) 61 | 62 | self.fc7_mbox_conf = nn.Conv2d(1024, 2, kernel_size=3, stride=1, padding=1) 63 | self.fc7_mbox_loc = nn.Conv2d(1024, 4, kernel_size=3, stride=1, padding=1) 64 | self.conv6_2_mbox_conf = nn.Conv2d(512, 2, kernel_size=3, stride=1, padding=1) 65 | self.conv6_2_mbox_loc = nn.Conv2d(512, 4, kernel_size=3, stride=1, padding=1) 66 | self.conv7_2_mbox_conf = nn.Conv2d(256, 2, kernel_size=3, stride=1, padding=1) 67 | self.conv7_2_mbox_loc = nn.Conv2d(256, 4, kernel_size=3, stride=1, padding=1) 68 | 69 | def forward(self, x): 70 | h = F.relu(self.conv1_1(x)) 71 | h = F.relu(self.conv1_2(h)) 72 | h = F.max_pool2d(h, 2, 2) 73 | 74 | h = F.relu(self.conv2_1(h)) 75 | h = F.relu(self.conv2_2(h)) 76 | h = F.max_pool2d(h, 2, 2) 77 | 78 | h = F.relu(self.conv3_1(h)) 79 | h = F.relu(self.conv3_2(h)) 80 | h = F.relu(self.conv3_3(h)); f3_3 = h 81 | h = F.max_pool2d(h, 2, 2) 82 | 83 | h = F.relu(self.conv4_1(h)) 84 | h = F.relu(self.conv4_2(h)) 85 | h = F.relu(self.conv4_3(h)); f4_3 = h 86 | h = F.max_pool2d(h, 2, 2) 87 | 88 | h = F.relu(self.conv5_1(h)) 89 | h = F.relu(self.conv5_2(h)) 90 | h = F.relu(self.conv5_3(h)); f5_3 = h 91 | h = F.max_pool2d(h, 2, 2) 92 | 93 | h = F.relu(self.fc6(h)) 94 | h = F.relu(self.fc7(h)); ffc7 = h 95 | h = F.relu(self.conv6_1(h)) 96 | h = F.relu(self.conv6_2(h)); f6_2 = h 97 | h = F.relu(self.conv7_1(h)) 98 | h = F.relu(self.conv7_2(h)); f7_2 = h 99 | 100 | f3_3 = self.conv3_3_norm(f3_3) 101 | f4_3 = self.conv4_3_norm(f4_3) 102 | f5_3 = self.conv5_3_norm(f5_3) 103 | 104 | cls1 = self.conv3_3_norm_mbox_conf(f3_3) 105 | reg1 = self.conv3_3_norm_mbox_loc(f3_3) 106 | cls2 = self.conv4_3_norm_mbox_conf(f4_3) 107 | reg2 = self.conv4_3_norm_mbox_loc(f4_3) 108 | cls3 = self.conv5_3_norm_mbox_conf(f5_3) 109 | reg3 = self.conv5_3_norm_mbox_loc(f5_3) 110 | cls4 = self.fc7_mbox_conf(ffc7) 111 | reg4 = self.fc7_mbox_loc(ffc7) 112 | cls5 = self.conv6_2_mbox_conf(f6_2) 113 | reg5 = self.conv6_2_mbox_loc(f6_2) 114 | cls6 = self.conv7_2_mbox_conf(f7_2) 115 | reg6 = self.conv7_2_mbox_loc(f7_2) 116 | 117 | # max-out background label 118 | chunk = torch.chunk(cls1,4,1) 119 | bmax = torch.max(torch.max(chunk[0],chunk[1]),chunk[2]) 120 | cls1 = torch.cat([bmax,chunk[3]],dim=1) 121 | 122 | return [cls1,reg1,cls2,reg2,cls3,reg3,cls4,reg4,cls5,reg5,cls6,reg6] 123 | 124 | s3fd = S3fd_Model 125 | -------------------------------------------------------------------------------- /funit/funit_model.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import copy 7 | 8 | import torch 9 | import torch.nn as nn 10 | 11 | from nntools.maybe_cuda import mbcuda 12 | 13 | from .networks import FewShotGen, GPPatchMcResDis 14 | 15 | 16 | def recon_criterion(predict, target): 17 | return torch.mean(torch.abs(predict - target)) 18 | 19 | 20 | class FUNITModel(nn.Module): 21 | def __init__(self, hp): 22 | super(FUNITModel, self).__init__() 23 | self.gen = FewShotGen(hp['gen']) 24 | self.dis = GPPatchMcResDis(hp['dis']) 25 | self.gen_test = copy.deepcopy(self.gen) 26 | 27 | def forward(self, co_data, cl_data, hp, mode): 28 | xa = mbcuda(co_data[0]) 29 | la = mbcuda(co_data[1]) 30 | xb = mbcuda(cl_data[0]) 31 | lb = mbcuda(cl_data[1]) 32 | if mode == 'gen_update': 33 | c_xa = self.gen.enc_content(xa) 34 | s_xa = self.gen.enc_class_model(xa) 35 | s_xb = self.gen.enc_class_model(xb) 36 | xt = self.gen.decode(c_xa, s_xb) # translation 37 | xr = self.gen.decode(c_xa, s_xa) # reconstruction 38 | l_adv_t, gacc_t, xt_gan_feat = self.dis.calc_gen_loss(xt, lb) 39 | l_adv_r, gacc_r, xr_gan_feat = self.dis.calc_gen_loss(xr, la) 40 | _, xb_gan_feat = self.dis(xb, lb) 41 | _, xa_gan_feat = self.dis(xa, la) 42 | l_c_rec = recon_criterion(xr_gan_feat.mean(3).mean(2), 43 | xa_gan_feat.mean(3).mean(2)) 44 | l_m_rec = recon_criterion(xt_gan_feat.mean(3).mean(2), 45 | xb_gan_feat.mean(3).mean(2)) 46 | l_x_rec = recon_criterion(xr, xa) 47 | l_adv = 0.5 * (l_adv_t + l_adv_r) 48 | acc = 0.5 * (gacc_t + gacc_r) 49 | l_total = (hp['gan_w'] * l_adv + hp['r_w'] * l_x_rec + hp[ 50 | 'fm_w'] * (l_c_rec + l_m_rec)) 51 | l_total.backward() 52 | return l_total, l_adv, l_x_rec, l_c_rec, l_m_rec, acc 53 | elif mode == 'dis_update': 54 | xb.requires_grad_() 55 | l_real_pre, acc_r, resp_r = self.dis.calc_dis_real_loss(xb, lb) 56 | l_real = hp['gan_w'] * l_real_pre 57 | l_real.backward(retain_graph=True) 58 | l_reg_pre = self.dis.calc_grad2(resp_r, xb) 59 | l_reg = 10 * l_reg_pre 60 | l_reg.backward() 61 | with torch.no_grad(): 62 | c_xa = self.gen.enc_content(xa) 63 | s_xb = self.gen.enc_class_model(xb) 64 | xt = self.gen.decode(c_xa, s_xb) 65 | l_fake_p, acc_f, resp_f = self.dis.calc_dis_fake_loss(xt.detach(), 66 | lb) 67 | l_fake = hp['gan_w'] * l_fake_p 68 | l_fake.backward() 69 | l_total = l_fake + l_real + l_reg 70 | acc = 0.5 * (acc_f + acc_r) 71 | return l_total, l_fake_p, l_real_pre, l_reg_pre, acc 72 | else: 73 | assert 0, 'Not support operation' 74 | 75 | def test(self, co_data, cl_data): 76 | self.eval() 77 | self.gen.eval() 78 | self.gen_test.eval() 79 | xa = mbcuda(co_data[0]) 80 | xb = mbcuda(cl_data[0]) 81 | c_xa_current = self.gen.enc_content(xa) 82 | s_xa_current = self.gen.enc_class_model(xa) 83 | s_xb_current = self.gen.enc_class_model(xb) 84 | xt_current = self.gen.decode(c_xa_current, s_xb_current) 85 | xr_current = self.gen.decode(c_xa_current, s_xa_current) 86 | c_xa = self.gen_test.enc_content(xa) 87 | s_xa = self.gen_test.enc_class_model(xa) 88 | s_xb = self.gen_test.enc_class_model(xb) 89 | xt = self.gen_test.decode(c_xa, s_xb) 90 | xr = self.gen_test.decode(c_xa, s_xa) 91 | self.train() 92 | return xa, xr_current, xt_current, xb, xr, xt 93 | 94 | def translate_k_shot(self, co_data, cl_data, k): 95 | self.eval() 96 | xa = mbcuda(co_data[0]) 97 | xb = mbcuda(cl_data[0]) 98 | c_xa_current = self.gen_test.enc_content(xa) 99 | if k == 1: 100 | c_xa_current = self.gen_test.enc_content(xa) 101 | s_xb_current = self.gen_test.enc_class_model(xb) 102 | xt_current = self.gen_test.decode(c_xa_current, s_xb_current) 103 | else: 104 | s_xb_current_before = self.gen_test.enc_class_model(xb) 105 | s_xb_current_after = s_xb_current_before.squeeze(-1).permute(1, 106 | 2, 107 | 0) 108 | s_xb_current_pool = torch.nn.functional.avg_pool1d( 109 | s_xb_current_after, k) 110 | s_xb_current = s_xb_current_pool.permute(2, 0, 1).unsqueeze(-1) 111 | xt_current = self.gen_test.decode(c_xa_current, s_xb_current) 112 | return xt_current 113 | 114 | def compute_k_style(self, style_batch, k): 115 | self.eval() 116 | style_batch = mbcuda(style_batch) 117 | s_xb_before = self.gen_test.enc_class_model(style_batch) 118 | s_xb_after = s_xb_before.squeeze(-1).permute(1, 2, 0) 119 | s_xb_pool = torch.nn.functional.avg_pool1d(s_xb_after, k) 120 | s_xb = s_xb_pool.permute(2, 0, 1).unsqueeze(-1) 121 | return s_xb 122 | 123 | def translate_simple(self, content_image, class_code): 124 | self.eval() 125 | xa = mbcuda(content_image) 126 | s_xb_current = mbcuda(class_code) 127 | c_xa_current = self.gen_test.enc_content(xa) 128 | xt_current = self.gen_test.decode(c_xa_current, s_xb_current) 129 | return xt_current 130 | -------------------------------------------------------------------------------- /funit/trainer.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import copy 7 | import os 8 | import math 9 | 10 | import torch 11 | import torch.nn as nn 12 | import torch.nn.init as init 13 | from torch.optim import lr_scheduler 14 | 15 | from .funit_model import FUNITModel 16 | 17 | 18 | def update_average(model_tgt, model_src, beta=0.999): 19 | with torch.no_grad(): 20 | param_dict_src = dict(model_src.named_parameters()) 21 | for p_name, p_tgt in model_tgt.named_parameters(): 22 | p_src = param_dict_src[p_name] 23 | assert(p_src is not p_tgt) 24 | p_tgt.copy_(beta*p_tgt + (1. - beta)*p_src) 25 | 26 | 27 | class Trainer(nn.Module): 28 | def __init__(self, cfg): 29 | super(Trainer, self).__init__() 30 | self.model = FUNITModel(cfg) 31 | lr_gen = cfg['lr_gen'] 32 | lr_dis = cfg['lr_dis'] 33 | dis_params = list(self.model.dis.parameters()) 34 | gen_params = list(self.model.gen.parameters()) 35 | self.dis_opt = torch.optim.RMSprop( 36 | [p for p in dis_params if p.requires_grad], 37 | lr=lr_gen, weight_decay=cfg['weight_decay']) 38 | self.gen_opt = torch.optim.RMSprop( 39 | [p for p in gen_params if p.requires_grad], 40 | lr=lr_dis, weight_decay=cfg['weight_decay']) 41 | self.dis_scheduler = get_scheduler(self.dis_opt, cfg) 42 | self.gen_scheduler = get_scheduler(self.gen_opt, cfg) 43 | self.apply(weights_init(cfg['init'])) 44 | self.model.gen_test = copy.deepcopy(self.model.gen) 45 | 46 | def gen_update(self, co_data, cl_data, hp, multigpus): 47 | self.gen_opt.zero_grad() 48 | al, ad, xr, cr, sr, ac = self.model(co_data, cl_data, hp, 'gen_update') 49 | self.loss_gen_total = torch.mean(al) 50 | self.loss_gen_recon_x = torch.mean(xr) 51 | self.loss_gen_recon_c = torch.mean(cr) 52 | self.loss_gen_recon_s = torch.mean(sr) 53 | self.loss_gen_adv = torch.mean(ad) 54 | self.accuracy_gen_adv = torch.mean(ac) 55 | self.gen_opt.step() 56 | this_model = self.model.module if multigpus else self.model 57 | update_average(this_model.gen_test, this_model.gen) 58 | return self.accuracy_gen_adv.item() 59 | 60 | def dis_update(self, co_data, cl_data, hp): 61 | self.dis_opt.zero_grad() 62 | al, lfa, lre, reg, acc = self.model(co_data, cl_data, hp, 'dis_update') 63 | self.loss_dis_total = torch.mean(al) 64 | self.loss_dis_fake_adv = torch.mean(lfa) 65 | self.loss_dis_real_adv = torch.mean(lre) 66 | self.loss_dis_reg = torch.mean(reg) 67 | self.accuracy_dis_adv = torch.mean(acc) 68 | self.dis_opt.step() 69 | return self.accuracy_dis_adv.item() 70 | 71 | def test(self, co_data, cl_data, multigpus): 72 | this_model = self.model.module if multigpus else self.model 73 | return this_model.test(co_data, cl_data) 74 | 75 | def resume(self, checkpoint_dir, hp, multigpus): 76 | this_model = self.model.module if multigpus else self.model 77 | 78 | last_model_name = get_model_list(checkpoint_dir, "gen") 79 | state_dict = torch.load(last_model_name) 80 | this_model.gen.load_state_dict(state_dict['gen']) 81 | this_model.gen_test.load_state_dict(state_dict['gen_test']) 82 | iterations = int(last_model_name[-11:-3]) 83 | 84 | last_model_name = get_model_list(checkpoint_dir, "dis") 85 | state_dict = torch.load(last_model_name) 86 | this_model.dis.load_state_dict(state_dict['dis']) 87 | 88 | state_dict = torch.load(os.path.join(checkpoint_dir, 'optimizer.pt')) 89 | self.dis_opt.load_state_dict(state_dict['dis']) 90 | self.gen_opt.load_state_dict(state_dict['gen']) 91 | 92 | self.dis_scheduler = get_scheduler(self.dis_opt, hp, iterations) 93 | self.gen_scheduler = get_scheduler(self.gen_opt, hp, iterations) 94 | print('Resume from iteration %d' % iterations) 95 | return iterations 96 | 97 | def save(self, snapshot_dir, iterations, multigpus): 98 | this_model = self.model.module if multigpus else self.model 99 | # Save generators, discriminators, and optimizers 100 | gen_name = os.path.join(snapshot_dir, 'gen_%08d.pt' % (iterations + 1)) 101 | dis_name = os.path.join(snapshot_dir, 'dis_%08d.pt' % (iterations + 1)) 102 | opt_name = os.path.join(snapshot_dir, 'optimizer.pt') 103 | torch.save({'gen': this_model.gen.state_dict(), 104 | 'gen_test': this_model.gen_test.state_dict()}, gen_name) 105 | torch.save({'dis': this_model.dis.state_dict()}, dis_name) 106 | torch.save({'gen': self.gen_opt.state_dict(), 107 | 'dis': self.dis_opt.state_dict()}, opt_name) 108 | 109 | def load_ckpt(self, ckpt_name): 110 | if torch.cuda.is_available(): 111 | state_dict = torch.load(ckpt_name) 112 | else: 113 | state_dict = torch.load(ckpt_name, map_location=torch.device('cpu')) 114 | self.model.gen.load_state_dict(state_dict['gen']) 115 | self.model.gen_test.load_state_dict(state_dict['gen_test']) 116 | 117 | def translate(self, co_data, cl_data): 118 | return self.model.translate(co_data, cl_data) 119 | 120 | def translate_k_shot(self, co_data, cl_data, k, mode): 121 | return self.model.translate_k_shot(co_data, cl_data, k, mode) 122 | 123 | def forward(self, *inputs): 124 | print('Forward function not implemented.') 125 | pass 126 | 127 | 128 | def get_model_list(dirname, key): 129 | if os.path.exists(dirname) is False: 130 | return None 131 | gen_models = [os.path.join(dirname, f) for f in os.listdir(dirname) if 132 | os.path.isfile(os.path.join(dirname, f)) and 133 | key in f and ".pt" in f] 134 | if gen_models is None: 135 | return None 136 | gen_models.sort() 137 | last_model_name = gen_models[-1] 138 | return last_model_name 139 | 140 | 141 | def get_scheduler(optimizer, hp, it=-1): 142 | if 'lr_policy' not in hp or hp['lr_policy'] == 'constant': 143 | scheduler = None # constant scheduler 144 | elif hp['lr_policy'] == 'step': 145 | scheduler = lr_scheduler.StepLR(optimizer, step_size=hp['step_size'], 146 | gamma=hp['gamma'], last_epoch=it) 147 | else: 148 | return NotImplementedError('%s not implemented', hp['lr_policy']) 149 | return scheduler 150 | 151 | 152 | def weights_init(init_type='gaussian'): 153 | def init_fun(m): 154 | classname = m.__class__.__name__ 155 | if (classname.find('Conv') == 0 or classname.find( 156 | 'Linear') == 0) and hasattr(m, 'weight'): 157 | if init_type == 'gaussian': 158 | init.normal_(m.weight.data, 0.0, 0.02) 159 | elif init_type == 'xavier': 160 | init.xavier_normal_(m.weight.data, gain=math.sqrt(2)) 161 | elif init_type == 'kaiming': 162 | init.kaiming_normal_(m.weight.data, a=0, mode='fan_in') 163 | elif init_type == 'orthogonal': 164 | init.orthogonal_(m.weight.data, gain=math.sqrt(2)) 165 | elif init_type == 'default': 166 | pass 167 | else: 168 | assert 0, "Unsupported initialization: {}".format(init_type) 169 | if hasattr(m, 'bias') and m.bias is not None: 170 | init.constant_(m.bias.data, 0.0) 171 | return init_fun 172 | -------------------------------------------------------------------------------- /funit/blocks.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import torch 7 | import torch.nn.functional as F 8 | from torch import nn 9 | 10 | 11 | class ResBlocks(nn.Module): 12 | def __init__(self, num_blocks, dim, norm, activation, pad_type): 13 | super(ResBlocks, self).__init__() 14 | self.model = [] 15 | for i in range(num_blocks): 16 | self.model += [ResBlock(dim, 17 | norm=norm, 18 | activation=activation, 19 | pad_type=pad_type)] 20 | self.model = nn.Sequential(*self.model) 21 | 22 | def forward(self, x): 23 | return self.model(x) 24 | 25 | 26 | class ResBlock(nn.Module): 27 | def __init__(self, dim, norm='in', activation='relu', pad_type='zero'): 28 | super(ResBlock, self).__init__() 29 | model = [] 30 | model += [Conv2dBlock(dim, dim, 3, 1, 1, 31 | norm=norm, 32 | activation=activation, 33 | pad_type=pad_type)] 34 | model += [Conv2dBlock(dim, dim, 3, 1, 1, 35 | norm=norm, 36 | activation='none', 37 | pad_type=pad_type)] 38 | self.model = nn.Sequential(*model) 39 | 40 | def forward(self, x): 41 | residual = x 42 | out = self.model(x) 43 | out += residual 44 | return out 45 | 46 | 47 | class ActFirstResBlock(nn.Module): 48 | def __init__(self, fin, fout, fhid=None, 49 | activation='lrelu', norm='none'): 50 | super().__init__() 51 | self.learned_shortcut = (fin != fout) 52 | self.fin = fin 53 | self.fout = fout 54 | self.fhid = min(fin, fout) if fhid is None else fhid 55 | self.conv_0 = Conv2dBlock(self.fin, self.fhid, 3, 1, 56 | padding=1, pad_type='reflect', norm=norm, 57 | activation=activation, activation_first=True) 58 | self.conv_1 = Conv2dBlock(self.fhid, self.fout, 3, 1, 59 | padding=1, pad_type='reflect', norm=norm, 60 | activation=activation, activation_first=True) 61 | if self.learned_shortcut: 62 | self.conv_s = Conv2dBlock(self.fin, self.fout, 1, 1, 63 | activation='none', use_bias=False) 64 | 65 | def forward(self, x): 66 | x_s = self.conv_s(x) if self.learned_shortcut else x 67 | dx = self.conv_0(x) 68 | dx = self.conv_1(dx) 69 | out = x_s + dx 70 | return out 71 | 72 | 73 | class LinearBlock(nn.Module): 74 | def __init__(self, in_dim, out_dim, norm='none', activation='relu'): 75 | super(LinearBlock, self).__init__() 76 | use_bias = True 77 | self.fc = nn.Linear(in_dim, out_dim, bias=use_bias) 78 | 79 | # initialize normalization 80 | norm_dim = out_dim 81 | if norm == 'bn': 82 | self.norm = nn.BatchNorm1d(norm_dim) 83 | elif norm == 'in': 84 | self.norm = nn.InstanceNorm1d(norm_dim) 85 | elif norm == 'none': 86 | self.norm = None 87 | else: 88 | assert 0, "Unsupported normalization: {}".format(norm) 89 | 90 | # initialize activation 91 | if activation == 'relu': 92 | self.activation = nn.ReLU(inplace=True) 93 | elif activation == 'lrelu': 94 | self.activation = nn.LeakyReLU(0.2, inplace=True) 95 | elif activation == 'tanh': 96 | self.activation = nn.Tanh() 97 | elif activation == 'none': 98 | self.activation = None 99 | else: 100 | assert 0, "Unsupported activation: {}".format(activation) 101 | 102 | def forward(self, x): 103 | out = self.fc(x) 104 | if self.norm: 105 | out = self.norm(out) 106 | if self.activation: 107 | out = self.activation(out) 108 | return out 109 | 110 | 111 | class Conv2dBlock(nn.Module): 112 | def __init__(self, in_dim, out_dim, ks, st, padding=0, 113 | norm='none', activation='relu', pad_type='zero', 114 | use_bias=True, activation_first=False): 115 | super(Conv2dBlock, self).__init__() 116 | self.use_bias = use_bias 117 | self.activation_first = activation_first 118 | # initialize padding 119 | if pad_type == 'reflect': 120 | self.pad = nn.ReflectionPad2d(padding) 121 | elif pad_type == 'replicate': 122 | self.pad = nn.ReplicationPad2d(padding) 123 | elif pad_type == 'zero': 124 | self.pad = nn.ZeroPad2d(padding) 125 | else: 126 | assert 0, "Unsupported padding type: {}".format(pad_type) 127 | 128 | # initialize normalization 129 | norm_dim = out_dim 130 | if norm == 'bn': 131 | self.norm = nn.BatchNorm2d(norm_dim) 132 | elif norm == 'in': 133 | self.norm = nn.InstanceNorm2d(norm_dim) 134 | elif norm == 'adain': 135 | self.norm = AdaptiveInstanceNorm2d(norm_dim) 136 | elif norm == 'none': 137 | self.norm = None 138 | else: 139 | assert 0, "Unsupported normalization: {}".format(norm) 140 | 141 | # initialize activation 142 | if activation == 'relu': 143 | self.activation = nn.ReLU(inplace=True) 144 | elif activation == 'lrelu': 145 | self.activation = nn.LeakyReLU(0.2, inplace=True) 146 | elif activation == 'tanh': 147 | self.activation = nn.Tanh() 148 | elif activation == 'none': 149 | self.activation = None 150 | else: 151 | assert 0, "Unsupported activation: {}".format(activation) 152 | 153 | self.conv = nn.Conv2d(in_dim, out_dim, ks, st, bias=self.use_bias) 154 | 155 | def forward(self, x): 156 | if self.activation_first: 157 | if self.activation: 158 | x = self.activation(x) 159 | x = self.conv(self.pad(x)) 160 | if self.norm: 161 | x = self.norm(x) 162 | else: 163 | x = self.conv(self.pad(x)) 164 | if self.norm: 165 | x = self.norm(x) 166 | if self.activation: 167 | x = self.activation(x) 168 | return x 169 | 170 | 171 | class AdaptiveInstanceNorm2d(nn.Module): 172 | def __init__(self, num_features, eps=1e-5, momentum=0.1): 173 | super(AdaptiveInstanceNorm2d, self).__init__() 174 | self.num_features = num_features 175 | self.eps = eps 176 | self.momentum = momentum 177 | self.weight = None 178 | self.bias = None 179 | self.register_buffer('running_mean', torch.zeros(num_features)) 180 | self.register_buffer('running_var', torch.ones(num_features)) 181 | 182 | def forward(self, x): 183 | assert self.weight is not None and \ 184 | self.bias is not None, "Please assign AdaIN weight first" 185 | b, c = x.size(0), x.size(1) 186 | running_mean = self.running_mean.repeat(b) 187 | running_var = self.running_var.repeat(b) 188 | x_reshaped = x.contiguous().view(1, b * c, *x.size()[2:]) 189 | out = F.batch_norm( 190 | x_reshaped, running_mean, running_var, self.weight, self.bias, 191 | True, self.momentum, self.eps) 192 | return out.view(b, c, *x.size()[2:]) 193 | 194 | def __repr__(self): 195 | return self.__class__.__name__ + '(' + str(self.num_features) + ')' 196 | -------------------------------------------------------------------------------- /monster_mirror/localcam.py: -------------------------------------------------------------------------------- 1 | """ 2 | Modified BSD License for Live Performance Only 3 | 4 | Copyright (c) 2019, Leo Dirac 5 | All rights reserved. 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are met: 9 | 10 | 1. Redistributions of source code must retain the above copyright notice, this 11 | list of conditions and the following disclaimer. 12 | 13 | 2. Redistributions in binary form must reproduce the above copyright notice, 14 | this list of conditions and the following disclaimer in the documentation 15 | and/or other materials provided with the distribution. 16 | 17 | 3. This software and any derivates must be used for live performances only. 18 | Any media content produced by this software may not be stored nor archived 19 | electronically. Any transmission of said media must be limited to immediate 20 | ephemeral display within clear view of any people whose likeness is being 21 | recorded, captured, or modified. Exception to this clause is only allowed 22 | with express written consent of every individual whose likeness is being 23 | recorded, captured, or modified, as well as any individual being portrayed, 24 | represented, or impersonated. 25 | 26 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 27 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 28 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 29 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 30 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 31 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 32 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 33 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 34 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 35 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 36 | """ 37 | 38 | __doc__ = '''MonsterMirror 39 | 40 | For Live Performances Only. See LICENSE for details. 41 | ''' 42 | 43 | import argparse 44 | import cv2 45 | import os 46 | import numpy as np 47 | import sys 48 | import time 49 | from timebudget import timebudget 50 | import traceback 51 | 52 | # Add the parent directory to the python path to get funit, sfd_pytorch 53 | sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')) 54 | import spooky 55 | 56 | 57 | def flip_img(img:np.ndarray) -> np.ndarray: 58 | """Mirror image left-right 59 | """ 60 | return img[:,::-1,:] 61 | 62 | def clear_screen(): 63 | print("\033[H\033[J") # clears display 64 | 65 | def local_spooky(no_full_screen:bool, camera_resolution:str='640x480', **kwargs): 66 | timebudget.set_quiet() # just print reports. 67 | if not no_full_screen: 68 | for _ in range(10): 69 | print("Will run full screen. Press SPACEBAR key to exit...\n\n") 70 | time.sleep(0.05) 71 | spookifier = spooky.RoundRobinSpookifier(**kwargs) 72 | 73 | try: 74 | camera = cv2.VideoCapture(0) 75 | camera.set(cv2.CAP_PROP_FPS, 15) 76 | cam_width, cam_height = [int(hw) for hw in camera_resolution.split('x')] 77 | camera.set(cv2.CAP_PROP_FRAME_WIDTH, cam_width) 78 | camera.set(cv2.CAP_PROP_FRAME_HEIGHT, cam_height) 79 | start_time = time.time() 80 | frame_cnt = 0 81 | recent_fps_time = None 82 | show_perf = True 83 | if no_full_screen: 84 | display = cv2.namedWindow("Spooky") 85 | else: 86 | display = cv2.namedWindow("Spooky", cv2.WND_PROP_FULLSCREEN) 87 | cv2.setWindowProperty("Spooky", cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN) 88 | while True: 89 | frame_cnt += 1 90 | total_fps = frame_cnt / (time.time() - start_time) 91 | if (frame_cnt % 3 == 0) and (show_perf): 92 | if recent_fps_time: 93 | current_fps = 3 / (time.time() - recent_fps_time) 94 | else: 95 | current_fps = float("nan") 96 | recent_fps_time = time.time() 97 | clear_screen() 98 | print(f"------------ At frame {frame_cnt} getting {current_fps:.1f}fps. Avg is {total_fps:.2f}fps") 99 | timebudget.report('process_npimage', reset=True) 100 | status_code, img = camera.read() 101 | print(f"Captured image of shape {img.shape}") 102 | img = spookifier.process_npimage(img, None) 103 | flipped = flip_img(img) 104 | cv2.imshow('Spooky', flipped) 105 | key = cv2.waitKey(1) 106 | if key == ord('e') or key == ord('E'): # E = embedding capture 107 | e = spookifier.get_target_embedding() 108 | with open("embeddings.txt","at") as f: 109 | f.write(str(e)) 110 | f.write("\n") 111 | print("Recorded embedding") 112 | elif key == ord('d') or key == ord('D'): # D = debug 113 | current_time = time.time() 114 | import pdb; pdb.set_trace() 115 | elif key == ord('p') or key == ord('P'): # P = perf data 116 | show_perf = not show_perf 117 | elif key == 32 or key == ord('Q') or key == ord('q'): 118 | print(f"Quitting") 119 | return 120 | elif key >= 0: 121 | print(f"Key #{key} pressed. No action for this key.") 122 | finally: 123 | del(camera) 124 | 125 | 126 | def parse_args(): 127 | parser = argparse.ArgumentParser(description=__doc__) 128 | parser.add_argument('--target_image_base', 129 | type=str, 130 | default='target-images', 131 | help='Folder where target image subdirs are located') 132 | parser.add_argument('-t', '--target_classes', 133 | type=str, 134 | default='tiger,meerkat', 135 | help='List of target class names. Must be folders under target_image_base') 136 | parser.add_argument('-nfs', '--no_full_screen', 137 | action='store_true', 138 | default=False, 139 | help='Run windowed instead of full screen.') 140 | parser.add_argument('-gf', '--grow_facebox', 141 | type=float, 142 | default=0.3, 143 | help='factor to increase size of facebox by') 144 | parser.add_argument('-mf', '--max_faces', 145 | type=int, 146 | default=4, 147 | help='maximum number of faces to process') 148 | parser.add_argument('-ed', '--extra_detail', 149 | type=int, 150 | default=1, 151 | help='number of extra CNN passes to refine detail') 152 | parser.add_argument('-cyc', '--cycle_delay', 153 | type=float, 154 | default=5.0, 155 | help='Number of seconds between switching classes') 156 | parser.add_argument('-ns', '--noise_speed', 157 | type=float, 158 | default=0.7, 159 | help='Number of noise cycles per class') 160 | parser.add_argument('-nm', '--noise_mag', 161 | type=float, 162 | default=3, 163 | help='How large the noise vector can get') 164 | parser.add_argument('-nd', '--noise_drift', 165 | type=float, 166 | default=0.1, 167 | help='How quickly the noise vector should change') 168 | parser.add_argument('-cm', '--color_map', 169 | type=str, 170 | default='0.7,1,0.5', 171 | help='Color map as RGB multipliers. e.g. 1,1,1 is true-color') 172 | parser.add_argument('-se', '--scale_embedding', 173 | type=float, 174 | default=0.2, 175 | help='Scale embedding. 1.0 for full animal. Lower for freaky') 176 | parser.add_argument('-ma', '--max_alpha', 177 | type=float, 178 | default=0.7, 179 | help='Max alpha to blend generated images back in with') 180 | parser.add_argument('-cr', '--camera_resolution', 181 | type=str, 182 | default='640x480', 183 | help='Resolution to capture images at. Turn this up if your hardware is amazing.') 184 | return parser.parse_args() 185 | 186 | 187 | if __name__ == "__main__": 188 | opts = parse_args() 189 | kwargs = dict(opts.__dict__) 190 | local_spooky(**kwargs) 191 | -------------------------------------------------------------------------------- /funit/utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import os 7 | import yaml 8 | import time 9 | 10 | import torch 11 | from torch.utils.data import DataLoader 12 | from torchvision import transforms 13 | import torchvision.utils as vutils 14 | 15 | from .data import ImageLabelFilelist 16 | 17 | 18 | def update_average(model_tgt, model_src, beta=0.999): 19 | with torch.no_grad(): 20 | param_dict_src = dict(model_src.named_parameters()) 21 | for p_name, p_tgt in model_tgt.named_parameters(): 22 | p_src = param_dict_src[p_name] 23 | assert(p_src is not p_tgt) 24 | p_tgt.copy_(beta*p_tgt + (1. - beta)*p_src) 25 | 26 | 27 | def loader_from_list( 28 | root, 29 | file_list, 30 | batch_size, 31 | new_size=None, 32 | height=128, 33 | width=128, 34 | crop=True, 35 | num_workers=4, 36 | shuffle=True, 37 | center_crop=False, 38 | return_paths=False, 39 | drop_last=True): 40 | transform_list = [transforms.ToTensor(), 41 | transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))] 42 | if center_crop: 43 | transform_list = [transforms.CenterCrop((height, width))] + \ 44 | transform_list if crop else transform_list 45 | else: 46 | transform_list = [transforms.RandomCrop((height, width))] + \ 47 | transform_list if crop else transform_list 48 | transform_list = [transforms.Resize(new_size)] + transform_list \ 49 | if new_size is not None else transform_list 50 | if not center_crop: 51 | transform_list = [transforms.RandomHorizontalFlip()] + transform_list 52 | transform = transforms.Compose(transform_list) 53 | dataset = ImageLabelFilelist(root, 54 | file_list, 55 | transform, 56 | return_paths=return_paths) 57 | loader = DataLoader(dataset, 58 | batch_size, 59 | shuffle=shuffle, 60 | drop_last=drop_last, 61 | num_workers=num_workers) 62 | return loader 63 | 64 | 65 | def get_evaluation_loaders(conf, shuffle_content=False): 66 | batch_size = conf['batch_size'] 67 | num_workers = conf['num_workers'] 68 | new_size = conf['new_size'] 69 | width = conf['crop_image_width'] 70 | height = conf['crop_image_height'] 71 | content_loader = loader_from_list( 72 | root=conf['data_folder_train'], 73 | file_list=conf['data_list_train'], 74 | batch_size=batch_size, 75 | new_size=new_size, 76 | height=height, 77 | width=width, 78 | crop=True, 79 | num_workers=num_workers, 80 | shuffle=shuffle_content, 81 | center_crop=True, 82 | return_paths=True, 83 | drop_last=False) 84 | 85 | class_loader = loader_from_list( 86 | root=conf['data_folder_test'], 87 | file_list=conf['data_list_test'], 88 | batch_size=batch_size * conf['k_shot'], 89 | new_size=new_size, 90 | height=height, 91 | width=width, 92 | crop=True, 93 | num_workers=1, 94 | shuffle=False, 95 | center_crop=True, 96 | return_paths=True, 97 | drop_last=False) 98 | return content_loader, class_loader 99 | 100 | 101 | def get_train_loaders(conf): 102 | batch_size = conf['batch_size'] 103 | num_workers = conf['num_workers'] 104 | new_size = conf['new_size'] 105 | width = conf['crop_image_width'] 106 | height = conf['crop_image_height'] 107 | train_content_loader = loader_from_list( 108 | root=conf['data_folder_train'], 109 | file_list=conf['data_list_train'], 110 | batch_size=batch_size, 111 | new_size=new_size, 112 | height=height, 113 | width=width, 114 | crop=True, 115 | num_workers=num_workers) 116 | train_class_loader = loader_from_list( 117 | root=conf['data_folder_train'], 118 | file_list=conf['data_list_train'], 119 | batch_size=batch_size, 120 | new_size=new_size, 121 | height=height, 122 | width=width, 123 | crop=True, 124 | num_workers=num_workers) 125 | test_content_loader = loader_from_list( 126 | root=conf['data_folder_test'], 127 | file_list=conf['data_list_test'], 128 | batch_size=batch_size, 129 | new_size=new_size, 130 | height=height, 131 | width=width, 132 | crop=True, 133 | num_workers=1) 134 | test_class_loader = loader_from_list( 135 | root=conf['data_folder_test'], 136 | file_list=conf['data_list_test'], 137 | batch_size=batch_size, 138 | new_size=new_size, 139 | height=height, 140 | width=width, 141 | crop=True, 142 | num_workers=1) 143 | 144 | return (train_content_loader, train_class_loader, test_content_loader, 145 | test_class_loader) 146 | 147 | 148 | def get_config(config): 149 | with open(config, 'r') as stream: 150 | return yaml.load(stream, Loader=yaml.FullLoader) 151 | 152 | 153 | def make_result_folders(output_directory): 154 | image_directory = os.path.join(output_directory, 'images') 155 | if not os.path.exists(image_directory): 156 | print("Creating directory: {}".format(image_directory)) 157 | os.makedirs(image_directory) 158 | checkpoint_directory = os.path.join(output_directory, 'checkpoints') 159 | if not os.path.exists(checkpoint_directory): 160 | print("Creating directory: {}".format(checkpoint_directory)) 161 | os.makedirs(checkpoint_directory) 162 | return checkpoint_directory, image_directory 163 | 164 | 165 | def __write_images(im_outs, dis_img_n, file_name): 166 | im_outs = [images.expand(-1, 3, -1, -1) for images in im_outs] 167 | image_tensor = torch.cat([images[:dis_img_n] for images in im_outs], 0) 168 | image_grid = vutils.make_grid(image_tensor.data, 169 | nrow=dis_img_n, padding=0, normalize=True) 170 | vutils.save_image(image_grid, file_name, nrow=1) 171 | 172 | 173 | def write_1images(image_outputs, image_directory, postfix): 174 | display_image_num = image_outputs[0].size(0) 175 | __write_images(image_outputs, display_image_num, 176 | '%s/gen_%s.jpg' % (image_directory, postfix)) 177 | 178 | 179 | def _write_row(html_file, it, fn, all_size): 180 | html_file.write("

iteration [%d] (%s)

" % (it, fn.split('/')[-1])) 181 | html_file.write(""" 182 |

183 | 184 |
185 |

186 | """ % (fn, fn, all_size)) 187 | return 188 | 189 | 190 | def write_html(filename, it, img_save_it, img_dir, all_size=1536): 191 | html_file = open(filename, "w") 192 | html_file.write(''' 193 | 194 | 195 | 196 | Experiment name = %s 197 | 198 | 199 | 200 | ''' % os.path.basename(filename)) 201 | html_file.write("

current

") 202 | _write_row(html_file, it, '%s/gen_train_current.jpg' % img_dir, all_size) 203 | for j in range(it, img_save_it - 1, -1): 204 | _write_row(html_file, j, '%s/gen_train_%08d.jpg' % (img_dir, j), 205 | all_size) 206 | html_file.write("") 207 | html_file.close() 208 | 209 | 210 | def write_loss(iterations, trainer, train_writer): 211 | members = [attr for attr in dir(trainer) 212 | if ((not callable(getattr(trainer, attr)) 213 | and not attr.startswith("__")) 214 | and ('loss' in attr 215 | or 'grad' in attr 216 | or 'nwd' in attr 217 | or 'accuracy' in attr))] 218 | for m in members: 219 | train_writer.add_scalar(m, getattr(trainer, m), iterations + 1) 220 | 221 | 222 | class Timer: 223 | def __init__(self, msg): 224 | self.msg = msg 225 | self.start_time = None 226 | 227 | def __enter__(self): 228 | self.start_time = time.time() 229 | 230 | def __exit__(self, exc_type, exc_value, exc_tb): 231 | print(self.msg % (time.time() - self.start_time)) 232 | -------------------------------------------------------------------------------- /funit/networks.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (C) 2019 NVIDIA Corporation. All rights reserved. 3 | Licensed under the CC BY-NC-SA 4.0 license 4 | (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode). 5 | """ 6 | import numpy as np 7 | 8 | import torch 9 | from torch import nn 10 | from torch import autograd 11 | 12 | from nntools.maybe_cuda import mbcuda 13 | 14 | from .blocks import LinearBlock, Conv2dBlock, ResBlocks, ActFirstResBlock 15 | 16 | 17 | def assign_adain_params(adain_params, model): 18 | # assign the adain_params to the AdaIN layers in model 19 | for m in model.modules(): 20 | if m.__class__.__name__ == "AdaptiveInstanceNorm2d": 21 | mean = adain_params[:, :m.num_features] 22 | std = adain_params[:, m.num_features:2*m.num_features] 23 | m.bias = mean.contiguous().view(-1) 24 | m.weight = std.contiguous().view(-1) 25 | if adain_params.size(1) > 2*m.num_features: 26 | adain_params = adain_params[:, 2*m.num_features:] 27 | 28 | 29 | def get_num_adain_params(model): 30 | # return the number of AdaIN parameters needed by the model 31 | num_adain_params = 0 32 | for m in model.modules(): 33 | if m.__class__.__name__ == "AdaptiveInstanceNorm2d": 34 | num_adain_params += 2*m.num_features 35 | return num_adain_params 36 | 37 | 38 | class GPPatchMcResDis(nn.Module): 39 | def __init__(self, hp): 40 | super(GPPatchMcResDis, self).__init__() 41 | assert hp['n_res_blks'] % 2 == 0, 'n_res_blk must be multiples of 2' 42 | self.n_layers = hp['n_res_blks'] // 2 43 | nf = hp['nf'] 44 | cnn_f = [Conv2dBlock(3, nf, 7, 1, 3, 45 | pad_type='reflect', 46 | norm='none', 47 | activation='none')] 48 | for i in range(self.n_layers - 1): 49 | nf_out = np.min([nf * 2, 1024]) 50 | cnn_f += [ActFirstResBlock(nf, nf, None, 'lrelu', 'none')] 51 | cnn_f += [ActFirstResBlock(nf, nf_out, None, 'lrelu', 'none')] 52 | cnn_f += [nn.ReflectionPad2d(1)] 53 | cnn_f += [nn.AvgPool2d(kernel_size=3, stride=2)] 54 | nf = np.min([nf * 2, 1024]) 55 | nf_out = np.min([nf * 2, 1024]) 56 | cnn_f += [ActFirstResBlock(nf, nf, None, 'lrelu', 'none')] 57 | cnn_f += [ActFirstResBlock(nf, nf_out, None, 'lrelu', 'none')] 58 | cnn_c = [Conv2dBlock(nf_out, hp['num_classes'], 1, 1, 59 | norm='none', 60 | activation='lrelu', 61 | activation_first=True)] 62 | self.cnn_f = nn.Sequential(*cnn_f) 63 | self.cnn_c = nn.Sequential(*cnn_c) 64 | 65 | def forward(self, x, y): 66 | assert(x.size(0) == y.size(0)) 67 | feat = self.cnn_f(x) 68 | out = self.cnn_c(feat) 69 | index = mbcuda(torch.LongTensor(range(out.size(0)))) 70 | out = out[index, y, :, :] 71 | return out, feat 72 | 73 | def calc_dis_fake_loss(self, input_fake, input_label): 74 | resp_fake, gan_feat = self.forward(input_fake, input_label) 75 | total_count = mbcuda(torch.tensor(np.prod(resp_fake.size()), 76 | dtype=torch.float)) 77 | fake_loss = torch.nn.ReLU()(1.0 + resp_fake).mean() 78 | correct_count = (resp_fake < 0).sum() 79 | fake_accuracy = correct_count.type_as(fake_loss) / total_count 80 | return fake_loss, fake_accuracy, resp_fake 81 | 82 | def calc_dis_real_loss(self, input_real, input_label): 83 | resp_real, gan_feat = self.forward(input_real, input_label) 84 | total_count = mbcuda(torch.tensor(np.prod(resp_real.size()), 85 | dtype=torch.float)) 86 | real_loss = torch.nn.ReLU()(1.0 - resp_real).mean() 87 | correct_count = (resp_real >= 0).sum() 88 | real_accuracy = correct_count.type_as(real_loss) / total_count 89 | return real_loss, real_accuracy, resp_real 90 | 91 | def calc_gen_loss(self, input_fake, input_fake_label): 92 | resp_fake, gan_feat = self.forward(input_fake, input_fake_label) 93 | total_count = mbcuda(torch.tensor(np.prod(resp_fake.size()), 94 | dtype=torch.float)) 95 | loss = -resp_fake.mean() 96 | correct_count = (resp_fake >= 0).sum() 97 | accuracy = correct_count.type_as(loss) / total_count 98 | return loss, accuracy, gan_feat 99 | 100 | def calc_grad2(self, d_out, x_in): 101 | batch_size = x_in.size(0) 102 | grad_dout = autograd.grad(outputs=d_out.mean(), 103 | inputs=x_in, 104 | create_graph=True, 105 | retain_graph=True, 106 | only_inputs=True)[0] 107 | grad_dout2 = grad_dout.pow(2) 108 | assert (grad_dout2.size() == x_in.size()) 109 | reg = grad_dout2.sum()/batch_size 110 | return reg 111 | 112 | 113 | class FewShotGen(nn.Module): 114 | def __init__(self, hp): 115 | super(FewShotGen, self).__init__() 116 | nf = hp['nf'] 117 | nf_mlp = hp['nf_mlp'] 118 | down_class = hp['n_downs_class'] 119 | down_content = hp['n_downs_content'] 120 | n_mlp_blks = hp['n_mlp_blks'] 121 | n_res_blks = hp['n_res_blks'] 122 | latent_dim = hp['latent_dim'] 123 | self.enc_class_model = ClassModelEncoder(down_class, 124 | 3, 125 | nf, 126 | latent_dim, 127 | norm='none', 128 | activ='relu', 129 | pad_type='reflect') 130 | 131 | self.enc_content = ContentEncoder(down_content, 132 | n_res_blks, 133 | 3, 134 | nf, 135 | 'in', 136 | activ='relu', 137 | pad_type='reflect') 138 | 139 | self.dec = Decoder(down_content, 140 | n_res_blks, 141 | self.enc_content.output_dim, 142 | 3, 143 | res_norm='adain', 144 | activ='relu', 145 | pad_type='reflect') 146 | 147 | self.mlp = MLP(latent_dim, 148 | get_num_adain_params(self.dec), 149 | nf_mlp, 150 | n_mlp_blks, 151 | norm='none', 152 | activ='relu') 153 | 154 | def forward(self, one_image, model_set): 155 | # reconstruct an image 156 | content, model_codes = self.encode(one_image, model_set) 157 | model_code = torch.mean(model_codes, dim=0).unsqueeze(0) 158 | images_trans = self.decode(content, model_code) 159 | return images_trans 160 | 161 | def encode(self, one_image, model_set): 162 | # extract content code from the input image 163 | content = self.enc_content(one_image) 164 | # extract model code from the images in the model set 165 | class_codes = self.enc_class_model(model_set) 166 | class_code = torch.mean(class_codes, dim=0).unsqueeze(0) 167 | return content, class_code 168 | 169 | def decode(self, content, model_code): 170 | # decode content and style codes to an image 171 | adain_params = self.mlp(model_code) 172 | assign_adain_params(adain_params, self.dec) 173 | images = self.dec(content) 174 | return images 175 | 176 | 177 | class ClassModelEncoder(nn.Module): 178 | def __init__(self, downs, ind_im, dim, latent_dim, norm, activ, pad_type): 179 | super(ClassModelEncoder, self).__init__() 180 | self.model = [] 181 | self.model += [Conv2dBlock(ind_im, dim, 7, 1, 3, 182 | norm=norm, 183 | activation=activ, 184 | pad_type=pad_type)] 185 | for i in range(2): 186 | self.model += [Conv2dBlock(dim, 2 * dim, 4, 2, 1, 187 | norm=norm, 188 | activation=activ, 189 | pad_type=pad_type)] 190 | dim *= 2 191 | for i in range(downs - 2): 192 | self.model += [Conv2dBlock(dim, dim, 4, 2, 1, 193 | norm=norm, 194 | activation=activ, 195 | pad_type=pad_type)] 196 | self.model += [nn.AdaptiveAvgPool2d(1)] 197 | self.model += [nn.Conv2d(dim, latent_dim, 1, 1, 0)] 198 | self.model = nn.Sequential(*self.model) 199 | self.output_dim = dim 200 | 201 | def forward(self, x): 202 | return self.model(x) 203 | 204 | 205 | class ContentEncoder(nn.Module): 206 | def __init__(self, downs, n_res, input_dim, dim, norm, activ, pad_type): 207 | super(ContentEncoder, self).__init__() 208 | self.model = [] 209 | self.model += [Conv2dBlock(input_dim, dim, 7, 1, 3, 210 | norm=norm, 211 | activation=activ, 212 | pad_type=pad_type)] 213 | for i in range(downs): 214 | self.model += [Conv2dBlock(dim, 2 * dim, 4, 2, 1, 215 | norm=norm, 216 | activation=activ, 217 | pad_type=pad_type)] 218 | dim *= 2 219 | self.model += [ResBlocks(n_res, dim, 220 | norm=norm, 221 | activation=activ, 222 | pad_type=pad_type)] 223 | self.model = nn.Sequential(*self.model) 224 | self.output_dim = dim 225 | 226 | def forward(self, x): 227 | return self.model(x) 228 | 229 | 230 | class Decoder(nn.Module): 231 | def __init__(self, ups, n_res, dim, out_dim, res_norm, activ, pad_type): 232 | super(Decoder, self).__init__() 233 | 234 | self.model = [] 235 | self.model += [ResBlocks(n_res, dim, res_norm, 236 | activ, pad_type=pad_type)] 237 | for i in range(ups): 238 | self.model += [nn.Upsample(scale_factor=2), 239 | Conv2dBlock(dim, dim // 2, 5, 1, 2, 240 | norm='in', 241 | activation=activ, 242 | pad_type=pad_type)] 243 | dim //= 2 244 | self.model += [Conv2dBlock(dim, out_dim, 7, 1, 3, 245 | norm='none', 246 | activation='tanh', 247 | pad_type=pad_type)] 248 | self.model = nn.Sequential(*self.model) 249 | 250 | def forward(self, x): 251 | return self.model(x) 252 | 253 | 254 | class MLP(nn.Module): 255 | def __init__(self, in_dim, out_dim, dim, n_blk, norm, activ): 256 | 257 | super(MLP, self).__init__() 258 | self.model = [] 259 | self.model += [LinearBlock(in_dim, dim, norm=norm, activation=activ)] 260 | for i in range(n_blk - 2): 261 | self.model += [LinearBlock(dim, dim, norm=norm, activation=activ)] 262 | self.model += [LinearBlock(dim, out_dim, 263 | norm='none', activation='none')] 264 | self.model = nn.Sequential(*self.model) 265 | 266 | def forward(self, x): 267 | return self.model(x.view(x.size(0), -1)) 268 | -------------------------------------------------------------------------------- /monster_mirror/spooky.py: -------------------------------------------------------------------------------- 1 | """ 2 | Modified BSD License for Live Performance Only 3 | 4 | Copyright (c) 2019, Leo Dirac 5 | All rights reserved. 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are met: 9 | 10 | 1. Redistributions of source code must retain the above copyright notice, this 11 | list of conditions and the following disclaimer. 12 | 13 | 2. Redistributions in binary form must reproduce the above copyright notice, 14 | this list of conditions and the following disclaimer in the documentation 15 | and/or other materials provided with the distribution. 16 | 17 | 3. This software and any derivates must be used for live performances only. 18 | Any media content produced by this software may not be stored nor archived 19 | electronically. Any transmission of said media must be limited to immediate 20 | ephemeral display within clear view of any people whose likeness is being 21 | recorded, captured, or modified. Exception to this clause is only allowed 22 | with express written consent of every individual whose likeness is being 23 | recorded, captured, or modified, as well as any individual being portrayed, 24 | represented, or impersonated. 25 | 26 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 27 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 28 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 29 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 30 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 31 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 32 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 33 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 34 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 35 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 36 | """ 37 | import argparse 38 | import cv2 39 | import functools 40 | import math 41 | import numpy as np 42 | import os 43 | from PIL import Image 44 | import sys 45 | import time 46 | import torch 47 | import torch.backends.cudnn as cudnn 48 | import torchvision 49 | from typing import Tuple 50 | 51 | from nntools.maybe_cuda import mbcuda 52 | from funit.utils import get_config 53 | from funit.trainer import Trainer 54 | from sfd_pytorch import S3fd_Model, detect_faces 55 | from timebudget import timebudget 56 | 57 | 58 | class Spookifier(): 59 | 60 | def __init__(self, 61 | config_file:str='funit/configs/funit_animals.yaml', 62 | face_finder_model:str='pretrained-models/s3fd_convert.pth', 63 | funit_model:str='pretrained-models/animal149_gen.pt', 64 | target_image_folder:str='target-images/meerkat', 65 | grow_facebox:float=0.2, 66 | cycle_delay:float=5.0, 67 | extra_detail:int=2, 68 | min_face_size:int=20, 69 | max_faces:int=5, 70 | color_map:str='1,1,1', 71 | scale_embedding:float=1.0, 72 | max_alpha:float=0.7, 73 | face_detector_type:str='auto', 74 | ): 75 | self.face_transform_cnt = 0 76 | self.grow_facebox = grow_facebox 77 | self.extra_detail = extra_detail 78 | self.cycle_delay = cycle_delay 79 | self.min_face_size = min_face_size 80 | self.max_faces = max_faces 81 | self.set_color(*[float(n) for n in color_map.split(',')]) 82 | self.scale_embedding = scale_embedding 83 | self.max_alpha = max_alpha 84 | 85 | self.load_face_detector(face_detector_type, face_finder_model) 86 | 87 | print("Loading trainer...") 88 | config = get_config(config_file) 89 | self.trainer = Trainer(config) 90 | mbcuda(self.trainer) 91 | self.trainer.load_ckpt(funit_model) 92 | self.trainer.eval() 93 | 94 | print("Loading transfomer...") 95 | transform_list = [torchvision.transforms.ToTensor(), 96 | torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))] 97 | transform_list = [torchvision.transforms.Resize((128, 128))] + transform_list 98 | self.transform = torchvision.transforms.Compose(transform_list) 99 | 100 | self.target_embedding = self.target_embedding_from_images(target_image_folder) 101 | 102 | def load_face_detector(self, face_detector_type:str, face_finder_model:str): 103 | if face_detector_type == 'auto': 104 | if torch.cuda.is_available(): 105 | face_detector_type = 'cnn' 106 | else: 107 | face_detector_type = 'haar' 108 | 109 | if face_detector_type == 'cnn': 110 | print("Loading CNN face detector...") 111 | self.face_cnn = True 112 | self.face_detect_model = S3fd_Model() 113 | self.face_detect_model.load_state_dict(torch.load(face_finder_model)) 114 | mbcuda(self.face_detect_model) 115 | self.face_detect_model.eval() 116 | 117 | elif face_detector_type == 'haar': 118 | cv2_base_dir = os.path.dirname(os.path.abspath(cv2.__file__)) 119 | haar_model = os.path.join(cv2_base_dir, 'data/haarcascade_frontalface_default.xml') 120 | print(f"Loading Haar face detector {haar_model}...") 121 | self.face_cnn = False 122 | self.haar_face_finder = cv2.CascadeClassifier(haar_model) 123 | 124 | else: 125 | raise RuntimeError(f"Unknown face_detector type {face_detector_type}. Must be auto, cnn, or haar.") 126 | 127 | 128 | def set_color(self, R:float, G:float, B:float): 129 | self.colorshift = mbcuda(torch.Tensor([[[R,G,B]]])) 130 | 131 | def target_embedding_from_images(self, target_image_folder:str) -> torch.Tensor: 132 | images = os.listdir(target_image_folder) 133 | print(f"Found {len(images)} target images in {target_image_folder}") 134 | new_class_code = None 135 | for i, f in enumerate(images): 136 | if f.startswith('.') or f=="LICENSE": 137 | continue # .DS_Store or ._whatever 138 | fn = os.path.join(target_image_folder, f) 139 | img = Image.open(fn).convert('RGB') 140 | img_tensor = mbcuda(self.transform(img).unsqueeze(0)) 141 | with torch.no_grad(): 142 | class_code = self.trainer.model.compute_k_style(img_tensor, 1) 143 | if new_class_code is None: 144 | new_class_code = class_code 145 | else: 146 | new_class_code += class_code 147 | return new_class_code / len(images) 148 | 149 | 150 | def transform_face_multi(self, input_img:np.ndarray) -> torch.Tensor: 151 | # This totally doesn't work. 152 | w = input_img.shape[0] 153 | if w < 256: 154 | return self.transform_face_1(input_img) 155 | else: 156 | split = w // 2 157 | tile = [[None, None],[None,None]] 158 | for i in range(2): 159 | for j in range(2): 160 | sub = input_img[ 161 | i*split : i*split+split, 162 | j*split : j*split+split 163 | ] 164 | tile[i][j] = self.transform_face_1(sub) 165 | # rebuild 166 | for i in range(2): 167 | tile[i] = torch.cat(tile[i], dim=1) 168 | tiled = torch.cat(tile, dim=2) 169 | return tiled 170 | 171 | 172 | def transform_face(self, input_img:np.ndarray) -> torch.Tensor: 173 | self.face_transform_cnt += 1 174 | return self.transform_face_1(input_img) 175 | 176 | @timebudget 177 | def transform_face_1(self, input_img:np.ndarray) -> torch.Tensor: 178 | image = Image.fromarray(input_img) 179 | content_img = self.transform(image).unsqueeze(0) 180 | with torch.no_grad(): 181 | output_image = self.trainer.model.translate_simple(content_img, self.get_target_embedding()) 182 | image = output_image.squeeze() 183 | image = (image + 1) / 2 # from (-1,1) to (0,1) 184 | return image 185 | 186 | def get_target_embedding(self) -> torch.Tensor: 187 | return self.target_embedding * self.scale_embedding 188 | 189 | 190 | def add_extra_faces(self, faces:np.ndarray) -> np.ndarray: 191 | """If "extra_detail" is configured, then it adds extra face box(es) so 192 | the CNN does a second (refiner) pass on the middle of the face. 193 | This adds amazing detail to tiger, but maybe not so good for others. 194 | """ 195 | if not self.extra_detail: 196 | return faces 197 | assert isinstance(faces, np.ndarray) 198 | out = [] 199 | for n in range(faces.shape[0]): 200 | f = faces[n,:] 201 | out.append(f) 202 | #TODO: generalize 640x480 203 | detail = list(self.resize_facebox(*f, (480,640), -0.2)) # smaller box 204 | out.append(list(detail)) 205 | if self.extra_detail == 2: 206 | # add a second shifted down a titc. 207 | detail[1] += int(detail[3] * 0.5) 208 | out.append(detail) 209 | return out 210 | 211 | 212 | def resize_facebox(self, x:int, y:int, w:int, h:int, shape:Tuple, growth:float) -> Tuple: 213 | """Adjusts the facebox to be bigger (growth>0) or smaller (growth<0), and ensures it's a square 214 | that fits in the frame. 215 | """ 216 | grow = int(w * growth) 217 | up_grow = int(grow * 0.7) 218 | nx = max(0, x-grow) 219 | ny = max(0, y-up_grow) 220 | nw = min(nx+w+2*grow, shape[1]) - nx 221 | nh = min(ny+h+2*grow, shape[0]) - ny 222 | #print(f"Old max is {x+w},{y+h}. New is {nx+nw},{ny+nh}. Grown by {grow}") 223 | nh = nw = min(nw,nh) 224 | return (nx, ny, nw, nh) 225 | 226 | 227 | def process_image(self, jpeg:bytes, save_file:str='output/face.jpg') -> np.ndarray: 228 | nparr = np.fromstring(jpeg, np.uint8) 229 | img_np = cv2.imdecode(nparr, cv2.IMREAD_COLOR) 230 | return self.process_npimage(img_np, save_file) 231 | 232 | @timebudget 233 | def find_faces(self, img_np:np.ndarray) -> np.ndarray: 234 | if self.face_cnn: 235 | bboxes = detect_faces(self.face_detect_model, img_np, minscale=2, ovr_threshhold=0.3, score_threshhold=0.5) 236 | out = [] 237 | for bb in bboxes: 238 | x1, y1, x2, y2, _score = bb 239 | out.append([x1, y1, (x2-x1), (y2-y1)]) 240 | return np.asarray(out) 241 | else: 242 | with timebudget('haar face detector'): 243 | gray = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY) 244 | return self.haar_face_finder.detectMultiScale(gray, 1.1, 4) 245 | 246 | @timebudget 247 | def process_npimage(self, img_np:np.ndarray, save_file:str='output/face.jpg') -> np.ndarray: 248 | # or in torch... 249 | #pil_img = PILImage.open(io.BytesIO(jpeg)) 250 | #img_tensor = ToTensor()(pil_img) 251 | faces = self.find_faces(img_np) 252 | if isinstance(faces,tuple): 253 | assert len(faces) == 0 254 | print("No faces found") 255 | else: 256 | faces = self.add_extra_faces(faces) 257 | num_faces = len(faces) 258 | if num_faces > self.max_faces: 259 | print(f"Too many faces {len(faces)}. Pruning") 260 | faces = sorted(faces, key=lambda f: f[2], reverse=True) 261 | faces = np.asarray(faces[:self.max_faces]) 262 | num_faces = len(faces) 263 | for face_num in range(num_faces): 264 | ox, oy, ow, oh = faces[face_num] 265 | if ow < self.min_face_size: 266 | print(f"face at {ox},{oy} is {ow}x{oh} too small. skipping") 267 | continue 268 | x, y, w, h = self.resize_facebox(ox,oy,ow,oh, img_np.shape, self.grow_facebox) 269 | print(f"Found {w}x{h}px face at {x},{y}", end=" ") 270 | sub_img = img_np[y:y+h, x:x+w] 271 | xformed128 = self.transform_face(sub_img) 272 | self.blend_merge(img_np, xformed128, x, y, w, h) 273 | if save_file: 274 | cv2.imwrite(save_file, img_np) 275 | print(f"Saved to {save_file}") 276 | return img_np 277 | 278 | @timebudget 279 | def blend_merge(self, base:np.ndarray, face128:torch.Tensor, x:int, y:int, w:int, h:int): 280 | """Take the 128x128 transformed image, and resize it and blend it back into the 281 | original in place.""" 282 | xforms = torchvision.transforms.Compose([ 283 | torchvision.transforms.ToPILImage(), 284 | torchvision.transforms.Resize((h,w)), 285 | torchvision.transforms.ToTensor(), 286 | ]) 287 | face = xforms(face128.cpu()) 288 | face = mbcuda(face) 289 | face = face.permute(1,2,0) # CHW -> HWC 290 | face *= 255 291 | face = face[:, :, [2,1,0]] # BGR to RGB 292 | face = self.mod_colors(face) 293 | alpha = self.prepare_alpha_mask_pt(h) 294 | old = mbcuda(torch.Tensor(base[y:y+h, x:x+w])) 295 | blended = old * (1-alpha) + face * alpha 296 | base[y:y+h, x:x+w] = blended.cpu().numpy() 297 | 298 | def mod_colors(self, face:torch.Tensor) -> torch.Tensor: 299 | return face * self.colorshift 300 | 301 | @functools.lru_cache(maxsize=1000) 302 | @timebudget 303 | def prepare_alpha_mask_pt(self, h:int, alpha_clamp:float=0.5) -> torch.Tensor: 304 | """alpha_clamp # Smaller numbers mean harsher boarders, but using more of the generated image 305 | """ 306 | # Some heuristic math to come up with an alpha mask to apply to the image before pasting it back in 307 | line = mbcuda(torch.arange(-1, 1, 2/h, dtype=torch.float32).unsqueeze(0)) 308 | assert len(line.shape) == 2 # see https://github.com/pytorch/pytorch/issues/28347 309 | line = line[:,0:h] 310 | assert line.shape == (1,h) 311 | alpha = line.T + line 312 | assert len(alpha.shape) == 2 313 | alpha = torch.abs(alpha) + torch.abs(torch.rot90(alpha)) 314 | # Pretty much all of these constants can be tweaked to change how blending looks. 315 | alpha = torch.exp( - ((alpha/3)**2) * 5) 316 | alpha = (alpha - alpha.min()) ** 0.8 317 | alpha = torch.clamp(alpha, 0, alpha_clamp) / alpha_clamp * self.max_alpha 318 | alpha = alpha.unsqueeze(2).repeat(1,1,3) 319 | return alpha 320 | 321 | 322 | class RoundRobinSpookifier(Spookifier): 323 | 324 | def __init__(self, target_image_base:str, target_classes:[str], noise_drift:float=0.1, 325 | noise_speed:float=0.4, noise_mag:float=5, **kwargs): 326 | target_dirs = target_classes.split(',') 327 | target_dir = os.path.join(target_image_base, target_dirs[0]) # Any to init base class 328 | super().__init__(target_image_folder=target_dir, **kwargs) 329 | self._targets = [] 330 | self._target_names = [] 331 | for target in target_dirs: 332 | target_dir = os.path.join(target_image_base, target) 333 | embed = self.target_embedding_from_images(target_dir) 334 | print(f"Embedding range for {target} is {float(embed.min()):.2f} to {float(embed.max()):.2f}") 335 | self._targets.append(embed) 336 | self._target_names.append(target) 337 | self.noise_drift = noise_drift 338 | self.noise_speed = noise_speed 339 | self.noise_mag = noise_mag 340 | self.noise = self.noise_mag * torch.randn_like(self._targets[0]) 341 | 342 | def get_target_embedding(self, current_time:float=None) -> torch.Tensor: 343 | if current_time is None: 344 | current_time = time.time() 345 | num = len(self._targets) 346 | turns = (current_time - 1.57e9) / self.cycle_delay # subtract a base to reduce rounding errors 347 | alpha = turns - int(turns) # for blending 348 | n = int(turns) 349 | # Do a simple randomish shuffle so target class order is pseudo-random 350 | first = ((n * 859) % 241) % num 351 | second = (((n+1) * 859) % 241) % num 352 | embed = self._targets[first] 353 | noise_mag = math.sin( turns * 6.28 * self.noise_speed ) 354 | if alpha > 0: 355 | embed2 = self._targets[ second ] 356 | embed = embed2 * alpha + embed * (1-alpha) 357 | print(f"{100*alpha:.0f}% {self._target_names[second]} +{100*(1-alpha):.0f}% {self._target_names[first]} {noise_mag:+.2f}noise") 358 | else: 359 | print(f"Embedding is 100% {self._target_names[first]} with {noise_mag:.3f} noise") 360 | self.noise += self.noise_drift * torch.randn_like(self.noise) 361 | self.noise = torch.clamp(self.noise, -self.noise_mag, self.noise_mag) 362 | embed += self.noise * noise_mag 363 | embed *= self.scale_embedding 364 | return embed 365 | 366 | 367 | def parse_args(): 368 | parser = argparse.ArgumentParser(description=__doc__) 369 | parser.add_argument('input_file', 370 | type=str, 371 | help='Image file to process') 372 | parser.add_argument('-t', '--target_image_folder', 373 | type=str, 374 | default='target-images/meerkat', 375 | help='Folder with examples of the target class') 376 | parser.add_argument('-b', '--box_expand', 377 | type=float, 378 | default=0.3, 379 | help='factor to increase size of facebox by') 380 | parser.add_argument('-t2', '--target2_image_folder', 381 | type=str, 382 | default=None, 383 | help='Optional folder with second target class to blend to') 384 | return parser.parse_args() 385 | 386 | 387 | if __name__ == "__main__": 388 | args = parse_args() 389 | jpeg = open(args.input_file, "rb").read() 390 | spook = Spookifier(target_image_folder=args.target_image_folder, grow_facebox=args.box_expand) 391 | if args.target2_image_folder: 392 | target_embed = spook.target_embedding 393 | target2_embed = spook.target_embedding_from_images(args.target2_image_folder) 394 | for n, blend in enumerate(list(np.arange(0,1.01,0.1))): 395 | spook.target_embedding = target2_embed * blend + target_embed * (1-blend) 396 | spook.process_image(jpeg, f"output/blend-{n}.jpg") 397 | else: 398 | spook.process_image(jpeg, "output/spooky.jpg") 399 | 400 | -------------------------------------------------------------------------------- /funit/LICENSE.md: -------------------------------------------------------------------------------- 1 | ## creative commons 2 | 3 | # Attribution-NonCommercial-ShareAlike 4.0 International 4 | 5 | Creative Commons Corporation (“Creative Commons”) is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an “as-is” basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible. 6 | 7 | ### Using Creative Commons Public Licenses 8 | 9 | Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses. 10 | 11 | * __Considerations for licensors:__ Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC-licensed material, or material used under an exception or limitation to copyright. [More considerations for licensors](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensors). 12 | 13 | * __Considerations for the public:__ By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor’s permission is not necessary for any reason–for example, because of any applicable exception or limitation to copyright–then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. [More considerations for the public](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensees). 14 | 15 | ## Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License 16 | 17 | By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. 18 | 19 | ### Section 1 – Definitions. 20 | 21 | a. __Adapted Material__ means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. 22 | 23 | b. __Adapter's License__ means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. 24 | 25 | c. __BY-NC-SA Compatible License__ means a license listed at [creativecommons.org/compatiblelicenses](http://creativecommons.org/compatiblelicenses), approved by Creative Commons as essentially the equivalent of this Public License. 26 | 27 | d. __Copyright and Similar Rights__ means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. 28 | 29 | e. __Effective Technological Measures__ means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. 30 | 31 | f. __Exceptions and Limitations__ means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. 32 | 33 | g. __License Elements__ means the license attributes listed in the name of a Creative Commons Public License. The License Elements of this Public License are Attribution, NonCommercial, and ShareAlike. 34 | 35 | h. __Licensed Material__ means the artistic or literary work, database, or other material to which the Licensor applied this Public License. 36 | 37 | i. __Licensed Rights__ means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. 38 | 39 | h. __Licensor__ means the individual(s) or entity(ies) granting rights under this Public License. 40 | 41 | i. __NonCommercial__ means not primarily intended for or directed towards commercial advantage or monetary compensation. For purposes of this Public License, the exchange of the Licensed Material for other material subject to Copyright and Similar Rights by digital file-sharing or similar means is NonCommercial provided there is no payment of monetary compensation in connection with the exchange. 42 | 43 | j. __Share__ means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. 44 | 45 | k. __Sui Generis Database Rights__ means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. 46 | 47 | l. __You__ means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. 48 | 49 | ### Section 2 – Scope. 50 | 51 | a. ___License grant.___ 52 | 53 | 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: 54 | 55 | A. reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only; and 56 | 57 | B. produce, reproduce, and Share Adapted Material for NonCommercial purposes only. 58 | 59 | 2. __Exceptions and Limitations.__ For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. 60 | 61 | 3. __Term.__ The term of this Public License is specified in Section 6(a). 62 | 63 | 4. __Media and formats; technical modifications allowed.__ The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material. 64 | 65 | 5. __Downstream recipients.__ 66 | 67 | A. __Offer from the Licensor – Licensed Material.__ Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. 68 | 69 | B. __Additional offer from the Licensor – Adapted Material.__ Every recipient of Adapted Material from You automatically receives an offer from the Licensor to exercise the Licensed Rights in the Adapted Material under the conditions of the Adapter’s License You apply. 70 | 71 | C. __No downstream restrictions.__ You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. 72 | 73 | 6. __No endorsement.__ Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). 74 | 75 | b. ___Other rights.___ 76 | 77 | 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. 78 | 79 | 2. Patent and trademark rights are not licensed under this Public License. 80 | 81 | 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties, including when the Licensed Material is used other than for NonCommercial purposes. 82 | 83 | ### Section 3 – License Conditions. 84 | 85 | Your exercise of the Licensed Rights is expressly made subject to the following conditions. 86 | 87 | a. ___Attribution.___ 88 | 89 | 1. If You Share the Licensed Material (including in modified form), You must: 90 | 91 | A. retain the following if it is supplied by the Licensor with the Licensed Material: 92 | 93 | i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); 94 | 95 | ii. a copyright notice; 96 | 97 | iii. a notice that refers to this Public License; 98 | 99 | iv. a notice that refers to the disclaimer of warranties; 100 | 101 | v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; 102 | 103 | B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and 104 | 105 | C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. 106 | 107 | 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. 108 | 109 | 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. 110 | 111 | b. ___ShareAlike.___ 112 | 113 | In addition to the conditions in Section 3(a), if You Share Adapted Material You produce, the following conditions also apply. 114 | 115 | 1. The Adapter’s License You apply must be a Creative Commons license with the same License Elements, this version or later, or a BY-NC-SA Compatible License. 116 | 117 | 2. You must include the text of, or the URI or hyperlink to, the Adapter's License You apply. You may satisfy this condition in any reasonable manner based on the medium, means, and context in which You Share Adapted Material. 118 | 119 | 3. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, Adapted Material that restrict exercise of the rights granted under the Adapter's License You apply. 120 | 121 | ### Section 4 – Sui Generis Database Rights. 122 | 123 | Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: 124 | 125 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database for NonCommercial purposes only; 126 | 127 | b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material, including for purposes of Section 3(b); and 128 | 129 | c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. 130 | 131 | For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. 132 | 133 | ### Section 5 – Disclaimer of Warranties and Limitation of Liability. 134 | 135 | a. __Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.__ 136 | 137 | b. __To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.__ 138 | 139 | c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. 140 | 141 | ### Section 6 – Term and Termination. 142 | 143 | a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. 144 | 145 | b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: 146 | 147 | 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 148 | 149 | 2. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 150 | 151 | For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. 152 | 153 | c. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. 154 | 155 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. 156 | 157 | ### Section 7 – Other Terms and Conditions. 158 | 159 | a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. 160 | 161 | b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. 162 | 163 | ### Section 8 – Interpretation. 164 | 165 | a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. 166 | 167 | b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. 168 | 169 | c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. 170 | 171 | d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. 172 | 173 | ``` 174 | Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at [creativecommons.org/policies](http://creativecommons.org/policies), Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. 175 | 176 | Creative Commons may be contacted at [creativecommons.org](http://creativecommons.org/). 177 | ``` 178 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MonsterMirror License 2 | --------------------- 3 | 4 | Modified BSD License for Live Performance Only 5 | 6 | Copyright (c) 2019, Leo Dirac 7 | All rights reserved. 8 | 9 | Redistribution and use in source and binary forms, with or without 10 | modification, are permitted provided that the following conditions are met: 11 | 12 | 1. Redistributions of source code must retain the above copyright notice, this 13 | list of conditions and the following disclaimer. 14 | 15 | 2. Redistributions in binary form must reproduce the above copyright notice, 16 | this list of conditions and the following disclaimer in the documentation 17 | and/or other materials provided with the distribution. 18 | 19 | 3. This software and any derivates must be used for live performances only. 20 | Any media content produced by this software may not be stored nor archived 21 | electronically. Any transmission of said media must be limited to immediate 22 | ephemeral display within clear view of any people whose likeness is being 23 | recorded, captured, or modified. Exception to this clause is only allowed 24 | with express written consent of every individual whose likeness is being 25 | recorded, captured, or modified, as well as any individual being portrayed, 26 | represented, or impersonated. 27 | 28 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 29 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 30 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 31 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 32 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 33 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 34 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 35 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 36 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 37 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 38 | 39 | 40 | 41 | FUNIT License 42 | ------------- 43 | 44 | ## creative commons 45 | 46 | # Attribution-NonCommercial-ShareAlike 4.0 International 47 | 48 | Creative Commons Corporation (“Creative Commons”) is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an “as-is” basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible. 49 | 50 | ### Using Creative Commons Public Licenses 51 | 52 | Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses. 53 | 54 | * __Considerations for licensors:__ Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC-licensed material, or material used under an exception or limitation to copyright. [More considerations for licensors](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensors). 55 | 56 | * __Considerations for the public:__ By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor’s permission is not necessary for any reason–for example, because of any applicable exception or limitation to copyright–then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. [More considerations for the public](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensees). 57 | 58 | ## Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License 59 | 60 | By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. 61 | 62 | ### Section 1 – Definitions. 63 | 64 | a. __Adapted Material__ means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. 65 | 66 | b. __Adapter's License__ means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. 67 | 68 | c. __BY-NC-SA Compatible License__ means a license listed at [creativecommons.org/compatiblelicenses](http://creativecommons.org/compatiblelicenses), approved by Creative Commons as essentially the equivalent of this Public License. 69 | 70 | d. __Copyright and Similar Rights__ means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. 71 | 72 | e. __Effective Technological Measures__ means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. 73 | 74 | f. __Exceptions and Limitations__ means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. 75 | 76 | g. __License Elements__ means the license attributes listed in the name of a Creative Commons Public License. The License Elements of this Public License are Attribution, NonCommercial, and ShareAlike. 77 | 78 | h. __Licensed Material__ means the artistic or literary work, database, or other material to which the Licensor applied this Public License. 79 | 80 | i. __Licensed Rights__ means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. 81 | 82 | h. __Licensor__ means the individual(s) or entity(ies) granting rights under this Public License. 83 | 84 | i. __NonCommercial__ means not primarily intended for or directed towards commercial advantage or monetary compensation. For purposes of this Public License, the exchange of the Licensed Material for other material subject to Copyright and Similar Rights by digital file-sharing or similar means is NonCommercial provided there is no payment of monetary compensation in connection with the exchange. 85 | 86 | j. __Share__ means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. 87 | 88 | k. __Sui Generis Database Rights__ means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. 89 | 90 | l. __You__ means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. 91 | 92 | ### Section 2 – Scope. 93 | 94 | a. ___License grant.___ 95 | 96 | 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: 97 | 98 | A. reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only; and 99 | 100 | B. produce, reproduce, and Share Adapted Material for NonCommercial purposes only. 101 | 102 | 2. __Exceptions and Limitations.__ For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. 103 | 104 | 3. __Term.__ The term of this Public License is specified in Section 6(a). 105 | 106 | 4. __Media and formats; technical modifications allowed.__ The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material. 107 | 108 | 5. __Downstream recipients.__ 109 | 110 | A. __Offer from the Licensor – Licensed Material.__ Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. 111 | 112 | B. __Additional offer from the Licensor – Adapted Material.__ Every recipient of Adapted Material from You automatically receives an offer from the Licensor to exercise the Licensed Rights in the Adapted Material under the conditions of the Adapter’s License You apply. 113 | 114 | C. __No downstream restrictions.__ You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. 115 | 116 | 6. __No endorsement.__ Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). 117 | 118 | b. ___Other rights.___ 119 | 120 | 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. 121 | 122 | 2. Patent and trademark rights are not licensed under this Public License. 123 | 124 | 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties, including when the Licensed Material is used other than for NonCommercial purposes. 125 | 126 | ### Section 3 – License Conditions. 127 | 128 | Your exercise of the Licensed Rights is expressly made subject to the following conditions. 129 | 130 | a. ___Attribution.___ 131 | 132 | 1. If You Share the Licensed Material (including in modified form), You must: 133 | 134 | A. retain the following if it is supplied by the Licensor with the Licensed Material: 135 | 136 | i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); 137 | 138 | ii. a copyright notice; 139 | 140 | iii. a notice that refers to this Public License; 141 | 142 | iv. a notice that refers to the disclaimer of warranties; 143 | 144 | v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; 145 | 146 | B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and 147 | 148 | C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. 149 | 150 | 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. 151 | 152 | 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. 153 | 154 | b. ___ShareAlike.___ 155 | 156 | In addition to the conditions in Section 3(a), if You Share Adapted Material You produce, the following conditions also apply. 157 | 158 | 1. The Adapter’s License You apply must be a Creative Commons license with the same License Elements, this version or later, or a BY-NC-SA Compatible License. 159 | 160 | 2. You must include the text of, or the URI or hyperlink to, the Adapter's License You apply. You may satisfy this condition in any reasonable manner based on the medium, means, and context in which You Share Adapted Material. 161 | 162 | 3. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, Adapted Material that restrict exercise of the rights granted under the Adapter's License You apply. 163 | 164 | ### Section 4 – Sui Generis Database Rights. 165 | 166 | Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: 167 | 168 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database for NonCommercial purposes only; 169 | 170 | b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material, including for purposes of Section 3(b); and 171 | 172 | c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. 173 | 174 | For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. 175 | 176 | ### Section 5 – Disclaimer of Warranties and Limitation of Liability. 177 | 178 | a. __Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.__ 179 | 180 | b. __To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.__ 181 | 182 | c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. 183 | 184 | ### Section 6 – Term and Termination. 185 | 186 | a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. 187 | 188 | b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: 189 | 190 | 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 191 | 192 | 2. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 193 | 194 | For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. 195 | 196 | c. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. 197 | 198 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. 199 | 200 | ### Section 7 – Other Terms and Conditions. 201 | 202 | a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. 203 | 204 | b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. 205 | 206 | ### Section 8 – Interpretation. 207 | 208 | a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. 209 | 210 | b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. 211 | 212 | c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. 213 | 214 | d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. 215 | 216 | ``` 217 | Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at [creativecommons.org/policies](http://creativecommons.org/policies), Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. 218 | 219 | Creative Commons may be contacted at [creativecommons.org](http://creativecommons.org/). 220 | ``` 221 | 222 | 223 | 224 | SFD_Pytorch license 225 | ------------------- 226 | 227 | MIT License 228 | 229 | Copyright (c) 2017 carwin 230 | 231 | Permission is hereby granted, free of charge, to any person obtaining a copy 232 | of this software and associated documentation files (the "Software"), to deal 233 | in the Software without restriction, including without limitation the rights 234 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 235 | copies of the Software, and to permit persons to whom the Software is 236 | furnished to do so, subject to the following conditions: 237 | 238 | The above copyright notice and this permission notice shall be included in all 239 | copies or substantial portions of the Software. 240 | 241 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 242 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 243 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 244 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 245 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 246 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 247 | SOFTWARE. 248 | --------------------------------------------------------------------------------