；
7 |
8 | We provide PyTorch implementations for both unpaired and paired image-to-image translation.
9 |
10 | The code was written by [Jun-Yan Zhu](https://github.com/junyanz) and [Taesung Park](https://github.com/taesung), and supported by [Tongzhou Wang](https://ssnl.github.io/).
11 |
12 | This PyTorch implementation produces results comparable to or better than our original Torch software. If you would like to reproduce the same results as in the papers, check out the original [CycleGAN Torch](https://github.com/junyanz/CycleGAN) and [pix2pix Torch](https://github.com/phillipi/pix2pix) code
13 |
14 | **Note**: The current software works well with PyTorch 0.41+. Check out the older [branch](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/tree/pytorch0.3.1) that supports PyTorch 0.1-0.3.
15 |
16 | You may find useful information in [training/test tips](docs/tips.md) and [frequently asked questions](docs/qa.md). To implement custom models and datasets, check out our [templates](#custom-model-and-dataset). To help users better understand and adapt our codebase, we provide an [overview](docs/overview.md) of the code structure of this repository.
17 |
18 | **CycleGAN: [Project](https://junyanz.github.io/CycleGAN/) | [Paper](https://arxiv.org/pdf/1703.10593.pdf) | [Torch](https://github.com/junyanz/CycleGAN)**
19 | 
20 |
21 |
22 | **Pix2pix: [Project](https://phillipi.github.io/pix2pix/) | [Paper](https://arxiv.org/pdf/1611.07004.pdf) | [Torch](https://github.com/phillipi/pix2pix)**
23 |
24 | 
25 |
26 |
27 | **[EdgesCats Demo](https://affinelayer.com/pixsrv/) | [pix2pix-tensorflow](https://github.com/affinelayer/pix2pix-tensorflow) | by [Christopher Hesse](https://twitter.com/christophrhesse)**
28 |
29 | 
30 |
31 | If you use this code for your research, please cite:
32 |
33 | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks.
34 | [Jun-Yan Zhu](https://people.eecs.berkeley.edu/~junyanz/)\*, [Taesung Park](https://taesung.me/)\*, [Phillip Isola](https://people.eecs.berkeley.edu/~isola/), [Alexei A. Efros](https://people.eecs.berkeley.edu/~efros). In ICCV 2017. (* equal contributions) [[Bibtex]](https://junyanz.github.io/CycleGAN/CycleGAN.txt)
35 |
36 |
37 | Image-to-Image Translation with Conditional Adversarial Networks.
38 | [Phillip Isola](https://people.eecs.berkeley.edu/~isola), [Jun-Yan Zhu](https://people.eecs.berkeley.edu/~junyanz), [Tinghui Zhou](https://people.eecs.berkeley.edu/~tinghuiz), [Alexei A. Efros](https://people.eecs.berkeley.edu/~efros). In CVPR 2017. [[Bibtex]](http://people.csail.mit.edu/junyanz/projects/pix2pix/pix2pix.bib)
39 |
40 | ## Talks and Course
41 | pix2pix slides: [keynote](http://efrosgans.eecs.berkeley.edu/CVPR18_slides/pix2pix.key) | [pdf](http://efrosgans.eecs.berkeley.edu/CVPR18_slides/pix2pix.pdf),
42 | CycleGAN slides: [pptx](http://efrosgans.eecs.berkeley.edu/CVPR18_slides/CycleGAN.pptx) | [pdf](http://efrosgans.eecs.berkeley.edu/CVPR18_slides/CycleGAN.pdf)
43 |
44 | CycleGAN course assignment [code](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/assignments/a4-code.zip) and [handout](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/assignments/a4-handout.pdf) designed by Prof. [Roger Grosse](http://www.cs.toronto.edu/~rgrosse/) for [CSC321](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/) "Intro to Neural Networks and Machine Learning" at University of Toronto. Please contact the instructor if you would like to adopt it in your course.
45 |
46 | ## Other implementations
47 | ### CycleGAN
48 | [Tensorflow] (by Harry Yang),
49 | [Tensorflow] (by Archit Rathore),
50 | [Tensorflow] (by Van Huy),
51 | [Tensorflow] (by Xiaowei Hu),
52 | [Tensorflow-simple] (by Zhenliang He),
53 | [TensorLayer] (by luoxier),
54 | [Chainer] (by Yanghua Jin),
55 | [Minimal PyTorch] (by yunjey),
56 | [Mxnet] (by Ldpe2G),
57 | [lasagne/keras] (by tjwei)
58 |
59 |
60 | ### pix2pix
61 | [Tensorflow] (by Christopher Hesse),
62 | [Tensorflow] (by Eyyüb Sariu),
63 | [Tensorflow (face2face)] (by Dat Tran),
64 | [Tensorflow (film)] (by Arthur Juliani),
65 | [Tensorflow (zi2zi)] (by Yuchen Tian),
66 | [Chainer] (by mattya),
67 | [tf/torch/keras/lasagne] (by tjwei),
68 | [Pytorch] (by taey16)
69 |
70 |
71 |
72 | ## Prerequisites
73 | - Linux or macOS
74 | - Python 3
75 | - CPU or NVIDIA GPU + CUDA CuDNN
76 |
77 | ## Getting Started
78 | ### Installation
79 |
80 | - Clone this repo:
81 | ```bash
82 | git clone https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
83 | cd pytorch-CycleGAN-and-pix2pix
84 | ```
85 |
86 | - Install [PyTorch](http://pytorch.org and) 0.4+ and other dependencies (e.g., torchvision, [visdom](https://github.com/facebookresearch/visdom) and [dominate](https://github.com/Knio/dominate)).
87 | - For pip users, please type the command `pip install -r requirements.txt`.
88 | - For Conda users, we provide a installation script `./scripts/conda_deps.sh`. Alternatively, you can create a new Conda environment using `conda env create -f environment.yml`.
89 | - For Docker users, we provide the pre-built Docker image and Dockerfile. Please refer to our [Docker](docs/docker.md) page.
90 |
91 | ### CycleGAN train/test
92 | - Download a CycleGAN dataset (e.g. maps):
93 | ```bash
94 | bash ./datasets/download_cyclegan_dataset.sh maps
95 | ```
96 | - To view training results and loss plots, run `python -m visdom.server` and click the URL http://localhost:8097.
97 | - Train a model:
98 | ```bash
99 | #!./scripts/train_cyclegan.sh
100 | python train.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan
101 | ```
102 | To see more intermediate results, check out `./checkpoints/maps_cyclegan/web/index.html`.
103 | - Test the model:
104 | ```bash
105 | #!./scripts/test_cyclegan.sh
106 | python test.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan
107 | ```
108 | - The test results will be saved to a html file here: `./results/maps_cyclegan/latest_test/index.html`.
109 |
110 | ### pix2pix train/test
111 | - Download a pix2pix dataset (e.g.facades):
112 | ```bash
113 | bash ./datasets/download_pix2pix_dataset.sh facades
114 | ```
115 | - Train a model:
116 | ```bash
117 | #!./scripts/train_pix2pix.sh
118 | python train.py --dataroot ./datasets/facades --name facades_pix2pix --model pix2pix --direction BtoA
119 | ```
120 | - To view training results and loss plots, run `python -m visdom.server` and click the URL http://localhost:8097. To see more intermediate results, check out `./checkpoints/facades_pix2pix/web/index.html`.
121 |
122 | - Test the model (`bash ./scripts/test_pix2pix.sh`):
123 | ```bash
124 | #!./scripts/test_pix2pix.sh
125 | python test.py --dataroot ./datasets/facades --name facades_pix2pix --model pix2pix --direction BtoA
126 | ```
127 | - The test results will be saved to a html file here: `./results/facades_pix2pix/test_latest/index.html`. You can find more scripts at `scripts` directory.
128 | - To train and test pix2pix-based colorization models, please add `--model colorization` and `--dataset_mode colorization`. See our training [tips](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md#notes-on-colorization) for more details.
129 |
130 | ### Apply a pre-trained model (CycleGAN)
131 | - You can download a pretrained model (e.g. horse2zebra) with the following script:
132 | ```bash
133 | bash ./scripts/download_cyclegan_model.sh horse2zebra
134 | ```
135 | - The pretrained model is saved at `./checkpoints/{name}_pretrained/latest_net_G.pth`. Check [here](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/scripts/download_cyclegan_model.sh#L3) for all the available CycleGAN models.
136 | - To test the model, you also need to download the horse2zebra dataset:
137 | ```bash
138 | bash ./datasets/download_cyclegan_dataset.sh horse2zebra
139 | ```
140 |
141 | - Then generate the results using
142 | ```bash
143 | python test.py --dataroot datasets/horse2zebra/testA --name horse2zebra_pretrained --model test --no_dropout
144 | ```
145 | - The option `--model test` is used for generating results of CycleGAN only for one side. This option will automatically set `--dataset_mode single`, which only loads the images from one set. On the contrary, using `--model cycle_gan` requires loading and generating results in both directions, which is sometimes unnecessary. The results will be saved at `./results/`. Use `--results_dir {directory_path_to_save_result}` to specify the results directory.
146 |
147 | - For your own experiments, you might want to specify `--netG`, `--norm`, `--no_dropout` to match the generator architecture of the trained model.
148 |
149 | ### Apply a pre-trained model (pix2pix)
150 | Download a pre-trained model with `./scripts/download_pix2pix_model.sh`.
151 |
152 | - Check [here](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/scripts/download_pix2pix_model.sh#L3) for all the available pix2pix models. For example, if you would like to download label2photo model on the Facades dataset,
153 | ```bash
154 | bash ./scripts/download_pix2pix_model.sh facades_label2photo
155 | ```
156 | - Download the pix2pix facades datasets:
157 | ```bash
158 | bash ./datasets/download_pix2pix_dataset.sh facades
159 | ```
160 | - Then generate the results using
161 | ```bash
162 | python test.py --dataroot ./datasets/facades/ --direction BtoA --model pix2pix --name facades_label2photo_pretrained
163 | ```
164 | - Note that we specified `--direction BtoA` as Facades dataset's A to B direction is photos to labels.
165 |
166 | - If you would like to apply a pre-trained model to a collection of input images (rather than image pairs), please use `--model test` option. See `./scripts/test_single.sh` for how to apply a model to Facade label maps (stored in the directory `facades/testB`).
167 |
168 | - See a list of currently available models at `./scripts/download_pix2pix_model.sh`
169 |
170 | ## [Docker](docs/docker.md)
171 | We provide the pre-built Docker image and Dockerfile that can run this code repo. See [docker](docs/docker.md).
172 |
173 | ## [Datasets](docs/datasets.md)
174 | Download pix2pix/CycleGAN datasets and create your own datasets.
175 |
176 | ## [Training/Test Tips](docs/tips.md)
177 | Best practice for training and testing your models.
178 |
179 | ## [Frequently Asked Questions](docs/qa.md)
180 | Before you post a new question, please first look at the above Q & A and existing GitHub issues.
181 |
182 | ## Custom Model and Dataset
183 | If you plan to implement custom models and dataset for your new applications, we provide a dataset [template](data/template_dataset.py) and a model [template](models/template_model.py) as a starting point.
184 |
185 | ## [Code structure](docs/overview.md)
186 | To help users better understand and use our code, we briefly overview the functionality and implementation of each package and each module.
187 |
188 | ## Pull Request
189 | You are always welcome to contribute to this repository by sending a [pull request](https://help.github.com/articles/about-pull-requests/).
190 | Please run `flake8 --ignore E501 .` and `python ./scripts/test_before_push.py` before you commit the code. Please also update the code structure [overview](docs/overview.md) accordingly if you add or remove files.
191 |
192 | ## Citation
193 | If you use this code for your research, please cite our papers.
194 | ```
195 | @inproceedings{CycleGAN2017,
196 | title={Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkss},
197 | author={Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A},
198 | booktitle={Computer Vision (ICCV), 2017 IEEE International Conference on},
199 | year={2017}
200 | }
201 |
202 |
203 | @inproceedings{isola2017image,
204 | title={Image-to-Image Translation with Conditional Adversarial Networks},
205 | author={Isola, Phillip and Zhu, Jun-Yan and Zhou, Tinghui and Efros, Alexei A},
206 | booktitle={Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on},
207 | year={2017}
208 | }
209 | ```
210 |
211 |
212 |
213 | ## Related Projects
214 | **[CycleGAN-Torch](https://github.com/junyanz/CycleGAN) |
215 | [pix2pix-Torch](https://github.com/phillipi/pix2pix) | [pix2pixHD](https://github.com/NVIDIA/pix2pixHD) |
216 | [iGAN](https://github.com/junyanz/iGAN) |
217 | [BicycleGAN](https://github.com/junyanz/BicycleGAN) | [vid2vid](https://tcwang0509.github.io/vid2vid/)**
218 |
219 | ## Cat Paper Collection
220 | If you love cats, and love reading cool graphics, vision, and learning papers, please check out the Cat Paper [Collection](https://github.com/junyanz/CatPapers).
221 |
222 | ## Acknowledgments
223 | Our code is inspired by [pytorch-DCGAN](https://github.com/pytorch/examples/tree/master/dcgan).
224 |
--------------------------------------------------------------------------------
/Untitled Diagram.drawio:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |
--------------------------------------------------------------------------------
/data/__init__.py:
--------------------------------------------------------------------------------
1 | """This package includes all the modules related to data loading and preprocessing
2 |
3 | To add a custom dataset class called 'dummy', you need to add a file called 'dummy_dataset.py' and define a subclass 'DummyDataset' inherited from BaseDataset.
4 | You need to implement four functions:
5 | -- <__init__>: initialize the class, first call BaseDataset.__init__(self, opt).
6 | -- <__len__>: return the size of dataset.
7 | -- <__getitem__>: get a data point from data loader.
8 | -- : (optionally) add dataset-specific options and set default options.
9 |
10 | Now you can use the dataset class by specifying flag '--dataset_mode dummy'.
11 | See our template dataset class 'template_dataset.py' for more details.
12 | """
13 | import importlib
14 | import torch.utils.data
15 | from data.base_dataset import BaseDataset
16 |
17 |
18 | def find_dataset_using_name(dataset_name):
19 | """Import the module "data/[dataset_name]_dataset.py".
20 |
21 | In the file, the class called DatasetNameDataset() will
22 | be instantiated. It has to be a subclass of BaseDataset,
23 | and it is case-insensitive.
24 | """
25 | dataset_filename = "data." + dataset_name + "_dataset"
26 | datasetlib = importlib.import_module(dataset_filename)
27 |
28 | dataset = None
29 | target_dataset_name = dataset_name.replace('_', '') + 'dataset'
30 | for name, cls in datasetlib.__dict__.items():
31 | if name.lower() == target_dataset_name.lower() \
32 | and issubclass(cls, BaseDataset):
33 | dataset = cls
34 |
35 | if dataset is None:
36 | raise NotImplementedError("In %s.py, there should be a subclass of BaseDataset with class name that matches %s in lowercase." % (dataset_filename, target_dataset_name))
37 |
38 | return dataset
39 |
40 |
41 | def get_option_setter(dataset_name):
42 | """Return the static method of the dataset class."""
43 | dataset_class = find_dataset_using_name(dataset_name)
44 | return dataset_class.modify_commandline_options
45 |
46 |
47 | def create_dataset(opt):
48 | """Create a dataset given the option.
49 |
50 | This function wraps the class CustomDatasetDataLoader.
51 | This is the main interface between this package and 'train.py'/'test.py'
52 |
53 | Example:
54 | >>> from data import create_dataset
55 | >>> dataset = create_dataset(opt)
56 | """
57 | data_loader = CustomDatasetDataLoader(opt)
58 | dataset = data_loader.load_data()
59 | return dataset
60 |
61 |
62 | class CustomDatasetDataLoader():
63 | """Wrapper class of Dataset class that performs multi-threaded data loading"""
64 |
65 | def __init__(self, opt):
66 | """Initialize this class
67 |
68 | Step 1: create a dataset instance given the name [dataset_mode]
69 | Step 2: create a multi-threaded data loader.
70 | """
71 | self.opt = opt
72 | dataset_class = find_dataset_using_name(opt.dataset_mode)
73 | self.dataset = dataset_class(opt)
74 | print("dataset [%s] was created" % type(self.dataset).__name__)
75 | self.dataloader = torch.utils.data.DataLoader(
76 | self.dataset,
77 | batch_size=opt.batch_size,
78 | shuffle=not opt.serial_batches,
79 | num_workers=int(opt.num_threads))
80 |
81 | def load_data(self):
82 | return self
83 |
84 | def __len__(self):
85 | """Return the number of data in the dataset"""
86 | return min(len(self.dataset), self.opt.max_dataset_size)
87 |
88 | def __iter__(self):
89 | """Return a batch of data"""
90 | for i, data in enumerate(self.dataloader):
91 | if i * self.opt.batch_size >= self.opt.max_dataset_size:
92 | break
93 | yield data
94 |
--------------------------------------------------------------------------------
/data/aligned_dataset.py:
--------------------------------------------------------------------------------
1 | import os.path
2 | from data.base_dataset import BaseDataset, get_params, get_transform
3 | from data.image_folder import make_dataset
4 | from PIL import Image
5 |
6 |
7 | class AlignedDataset(BaseDataset):
8 | """A dataset class for paired image dataset.
9 |
10 | It assumes that the directory '/path/to/data/train' contains image pairs in the form of {A,B}.
11 | During test time, you need to prepare a directory '/path/to/data/test'.
12 | """
13 |
14 | def __init__(self, opt):
15 | """Initialize this dataset class.
16 |
17 | Parameters:
18 | opt (Option class) -- stores all the experiment flags; needs to be a subclass of BaseOptions
19 | """
20 | BaseDataset.__init__(self, opt)
21 | self.dir_AB = os.path.join(opt.dataroot, opt.phase) # get the image directory
22 | self.AB_paths = sorted(make_dataset(self.dir_AB, opt.max_dataset_size)) # get image paths
23 | assert(self.opt.load_size >= self.opt.crop_size) # crop_size should be smaller than the size of loaded image
24 | self.input_nc = self.opt.output_nc if self.opt.direction == 'BtoA' else self.opt.input_nc
25 | self.output_nc = self.opt.input_nc if self.opt.direction == 'BtoA' else self.opt.output_nc
26 |
27 | def __getitem__(self, index):
28 | """Return a data point and its metadata information.
29 |
30 | Parameters:
31 | index - - a random integer for data indexing
32 |
33 | Returns a dictionary that contains A, B, A_paths and B_paths
34 | A (tensor) - - an image in the input domain
35 | B (tensor) - - its corresponding image in the target domain
36 | A_paths (str) - - image paths
37 | B_paths (str) - - image paths (same as A_paths)
38 | """
39 | # read a image given a random integer index
40 | AB_path = self.AB_paths[index]
41 | AB = Image.open(AB_path).convert('RGB')
42 | # split AB image into A and B
43 | w, h = AB.size
44 | w2 = int(w / 2)
45 | A = AB.crop((0, 0, w2, h))
46 | B = AB.crop((w2, 0, w, h))
47 |
48 | # apply the same transform to both A and B
49 | transform_params = get_params(self.opt, A.size)
50 | A_transform = get_transform(self.opt, transform_params, grayscale=(self.input_nc == 1))
51 | B_transform = get_transform(self.opt, transform_params, grayscale=(self.output_nc == 1))
52 |
53 | A = A_transform(A)
54 | B = B_transform(B)
55 |
56 | return {'A': A, 'B': B, 'A_paths': AB_path, 'B_paths': AB_path}
57 |
58 | def __len__(self):
59 | """Return the total number of images in the dataset."""
60 | return len(self.AB_paths)
61 |
--------------------------------------------------------------------------------
/data/base_dataset.py:
--------------------------------------------------------------------------------
1 | """This module implements an abstract base class (ABC) 'BaseDataset' for datasets.
2 |
3 | It also includes common transformation functions (e.g., get_transform, __scale_width), which can be later used in subclasses.
4 | """
5 | import random
6 | import numpy as np
7 | import torch.utils.data as data
8 | from PIL import Image
9 | import torchvision.transforms as transforms
10 | from abc import ABC, abstractmethod
11 |
12 |
13 | class BaseDataset(data.Dataset, ABC):
14 | """This class is an abstract base class (ABC) for datasets.
15 |
16 | To create a subclass, you need to implement the following four functions:
17 | -- <__init__>: initialize the class, first call BaseDataset.__init__(self, opt).
18 | -- <__len__>: return the size of dataset.
19 | -- <__getitem__>: get a data point.
20 | -- : (optionally) add dataset-specific options and set default options.
21 | """
22 |
23 | def __init__(self, opt):
24 | """Initialize the class; save the options in the class
25 |
26 | Parameters:
27 | opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
28 | """
29 | self.opt = opt
30 | self.root = opt.dataroot
31 |
32 | @staticmethod
33 | def modify_commandline_options(parser, is_train):
34 | """Add new dataset-specific options, and rewrite default values for existing options.
35 |
36 | Parameters:
37 | parser -- original option parser
38 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
39 |
40 | Returns:
41 | the modified parser.
42 | """
43 | return parser
44 |
45 | @abstractmethod
46 | def __len__(self):
47 | """Return the total number of images in the dataset."""
48 | return 0
49 |
50 | @abstractmethod
51 | def __getitem__(self, index):
52 | """Return a data point and its metadata information.
53 |
54 | Parameters:
55 | index - - a random integer for data indexing
56 |
57 | Returns:
58 | a dictionary of data with their names. It ususally contains the data itself and its metadata information.
59 | """
60 | pass
61 |
62 |
63 | def get_params(opt, size):
64 | w, h = size
65 | new_h = h
66 | new_w = w
67 | if opt.preprocess == 'resize_and_crop':
68 | new_h = new_w = opt.load_size
69 | elif opt.preprocess == 'scale_width_and_crop':
70 | new_w = opt.load_size
71 | new_h = opt.load_size * h // w
72 |
73 | x = random.randint(0, np.maximum(0, new_w - opt.crop_size))
74 | y = random.randint(0, np.maximum(0, new_h - opt.crop_size))
75 |
76 | flip = random.random() > 0.5
77 |
78 | return {'crop_pos': (x, y), 'flip': flip}
79 |
80 |
81 | def get_transform(opt, params=None, grayscale=False, method=Image.BICUBIC, convert=True):
82 | transform_list = []
83 | if grayscale:
84 | transform_list.append(transforms.Grayscale(1))
85 | if 'resize' in opt.preprocess:
86 | osize = [opt.load_size, opt.load_size]
87 | transform_list.append(transforms.Resize(osize, method))
88 | elif 'scale_width' in opt.preprocess:
89 | transform_list.append(transforms.Lambda(lambda img: __scale_width(img, opt.load_size, method)))
90 |
91 | if 'crop' in opt.preprocess:
92 | if params is None:
93 | transform_list.append(transforms.RandomCrop(opt.crop_size))
94 | else:
95 | transform_list.append(transforms.Lambda(lambda img: __crop(img, params['crop_pos'], opt.crop_size)))
96 |
97 | if opt.preprocess == 'none':
98 | transform_list.append(transforms.Lambda(lambda img: __make_power_2(img, base=4, method=method)))
99 |
100 | if not opt.no_flip:
101 | if params is None:
102 | transform_list.append(transforms.RandomHorizontalFlip())
103 | elif params['flip']:
104 | transform_list.append(transforms.Lambda(lambda img: __flip(img, params['flip'])))
105 |
106 | if convert:
107 | transform_list += [transforms.ToTensor()]
108 | if grayscale:
109 | transform_list += [transforms.Normalize((0.5,), (0.5,))]
110 | else:
111 | transform_list += [transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]
112 | return transforms.Compose(transform_list)
113 |
114 |
115 | def __make_power_2(img, base, method=Image.BICUBIC):
116 | ow, oh = img.size
117 | h = int(round(oh / base) * base)
118 | w = int(round(ow / base) * base)
119 | if (h == oh) and (w == ow):
120 | return img
121 |
122 | __print_size_warning(ow, oh, w, h)
123 | return img.resize((w, h), method)
124 |
125 |
126 | def __scale_width(img, target_width, method=Image.BICUBIC):
127 | ow, oh = img.size
128 | if (ow == target_width):
129 | return img
130 | w = target_width
131 | h = int(target_width * oh / ow)
132 | return img.resize((w, h), method)
133 |
134 |
135 | def __crop(img, pos, size):
136 | ow, oh = img.size
137 | x1, y1 = pos
138 | tw = th = size
139 | if (ow > tw or oh > th):
140 | return img.crop((x1, y1, x1 + tw, y1 + th))
141 | return img
142 |
143 |
144 | def __flip(img, flip):
145 | if flip:
146 | return img.transpose(Image.FLIP_LEFT_RIGHT)
147 | return img
148 |
149 |
150 | def __print_size_warning(ow, oh, w, h):
151 | """Print warning information about image size(only print once)"""
152 | if not hasattr(__print_size_warning, 'has_printed'):
153 | print("The image size needs to be a multiple of 4. "
154 | "The loaded image size was (%d, %d), so it was adjusted to "
155 | "(%d, %d). This adjustment will be done to all images "
156 | "whose sizes are not multiples of 4" % (ow, oh, w, h))
157 | __print_size_warning.has_printed = True
158 |
--------------------------------------------------------------------------------
/data/colorization_dataset.py:
--------------------------------------------------------------------------------
1 | import os.path
2 | from data.base_dataset import BaseDataset, get_transform
3 | from data.image_folder import make_dataset
4 | from skimage import color # require skimage
5 | from PIL import Image
6 | import numpy as np
7 | import torchvision.transforms as transforms
8 |
9 |
10 | class ColorizationDataset(BaseDataset):
11 | """This dataset class can load a set of natural images in RGB, and convert RGB format into (L, ab) pairs in Lab color space.
12 |
13 | This dataset is required by pix2pix-based colorization model ('--model colorization')
14 | """
15 | @staticmethod
16 | def modify_commandline_options(parser, is_train):
17 | """Add new dataset-specific options, and rewrite default values for existing options.
18 |
19 | Parameters:
20 | parser -- original option parser
21 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
22 |
23 | Returns:
24 | the modified parser.
25 |
26 | By default, the number of channels for input image is 1 (L) and
27 | the nubmer of channels for output image is 2 (ab). The direction is from A to B
28 | """
29 | parser.set_defaults(input_nc=1, output_nc=2, direction='AtoB')
30 | return parser
31 |
32 | def __init__(self, opt):
33 | """Initialize this dataset class.
34 |
35 | Parameters:
36 | opt (Option class) -- stores all the experiment flags; needs to be a subclass of BaseOptions
37 | """
38 | BaseDataset.__init__(self, opt)
39 | self.dir = os.path.join(opt.dataroot)
40 | self.AB_paths = sorted(make_dataset(self.dir, opt.max_dataset_size))
41 | assert(opt.input_nc == 1 and opt.output_nc == 2 and opt.direction == 'AtoB')
42 | self.transform = get_transform(self.opt, convert=False)
43 |
44 | def __getitem__(self, index):
45 | """Return a data point and its metadata information.
46 |
47 | Parameters:
48 | index - - a random integer for data indexing
49 |
50 | Returns a dictionary that contains A, B, A_paths and B_paths
51 | A (tensor) - - the L channel of an image
52 | B (tensor) - - the ab channels of the same image
53 | A_paths (str) - - image paths
54 | B_paths (str) - - image paths (same as A_paths)
55 | """
56 | path = self.AB_paths[index]
57 | im = Image.open(path).convert('RGB')
58 | im = self.transform(im)
59 | im = np.array(im)
60 | lab = color.rgb2lab(im).astype(np.float32)
61 | lab_t = transforms.ToTensor()(lab)
62 | A = lab_t[[0], ...] / 50.0 - 1.0
63 | B = lab_t[[1, 2], ...] / 110.0
64 | return {'A': A, 'B': B, 'A_paths': path, 'B_paths': path}
65 |
66 | def __len__(self):
67 | """Return the total number of images in the dataset."""
68 | return len(self.AB_paths)
69 |
--------------------------------------------------------------------------------
/data/image_folder.py:
--------------------------------------------------------------------------------
1 | """A modified image folder class
2 |
3 | We modify the official PyTorch image folder (https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py)
4 | so that this class can load images from both current directory and its subdirectories.
5 | """
6 |
7 | import torch.utils.data as data
8 |
9 | from PIL import Image
10 | import os
11 | import os.path
12 |
13 | IMG_EXTENSIONS = [
14 | '.jpg', '.JPG', '.jpeg', '.JPEG',
15 | '.png', '.PNG', '.ppm', '.PPM', '.bmp', '.BMP',
16 | ]
17 |
18 |
19 | def is_image_file(filename):
20 | return any(filename.endswith(extension) for extension in IMG_EXTENSIONS)
21 |
22 |
23 | def make_dataset(dir, max_dataset_size=float("inf")):
24 | images = []
25 | assert os.path.isdir(dir), '%s is not a valid directory' % dir
26 |
27 | for root, _, fnames in sorted(os.walk(dir)):
28 | for fname in fnames:
29 | if is_image_file(fname):
30 | path = os.path.join(root, fname)
31 | images.append(path)
32 | return images[:min(max_dataset_size, len(images))]
33 |
34 |
35 | def default_loader(path):
36 | return Image.open(path).convert('RGB')
37 |
38 |
39 | class ImageFolder(data.Dataset):
40 |
41 | def __init__(self, root, transform=None, return_paths=False,
42 | loader=default_loader):
43 | imgs = make_dataset(root)
44 | if len(imgs) == 0:
45 | raise(RuntimeError("Found 0 images in: " + root + "\n"
46 | "Supported image extensions are: " +
47 | ",".join(IMG_EXTENSIONS)))
48 |
49 | self.root = root
50 | self.imgs = imgs
51 | self.transform = transform
52 | self.return_paths = return_paths
53 | self.loader = loader
54 |
55 | def __getitem__(self, index):
56 | path = self.imgs[index]
57 | img = self.loader(path)
58 | if self.transform is not None:
59 | img = self.transform(img)
60 | if self.return_paths:
61 | return img, path
62 | else:
63 | return img
64 |
65 | def __len__(self):
66 | return len(self.imgs)
67 |
--------------------------------------------------------------------------------
/data/single_dataset.py:
--------------------------------------------------------------------------------
1 | from data.base_dataset import BaseDataset, get_transform
2 | from data.image_folder import make_dataset
3 | from PIL import Image
4 |
5 |
6 | class SingleDataset(BaseDataset):
7 | """This dataset class can load a set of images specified by the path --dataroot /path/to/data.
8 |
9 | It can be used for generating CycleGAN results only for one side with the model option '-model test'.
10 | """
11 |
12 | def __init__(self, opt):
13 | """Initialize this dataset class.
14 |
15 | Parameters:
16 | opt (Option class) -- stores all the experiment flags; needs to be a subclass of BaseOptions
17 | """
18 | BaseDataset.__init__(self, opt)
19 | self.A_paths = sorted(make_dataset(opt.dataroot, opt.max_dataset_size))
20 | input_nc = self.opt.output_nc if self.opt.direction == 'BtoA' else self.opt.input_nc
21 | self.transform = get_transform(opt, grayscale=(input_nc == 1))
22 |
23 | def __getitem__(self, index):
24 | """Return a data point and its metadata information.
25 |
26 | Parameters:
27 | index - - a random integer for data indexing
28 |
29 | Returns a dictionary that contains A and A_paths
30 | A(tensor) - - an image in one domain
31 | A_paths(str) - - the path of the image
32 | """
33 | A_path = self.A_paths[index]
34 | A_img = Image.open(A_path).convert('RGB')
35 | A = self.transform(A_img)
36 | return {'A': A, 'A_paths': A_path}
37 |
38 | def __len__(self):
39 | """Return the total number of images in the dataset."""
40 | return len(self.A_paths)
41 |
--------------------------------------------------------------------------------
/data/template_dataset.py:
--------------------------------------------------------------------------------
1 | """Dataset class template
2 |
3 | This module provides a template for users to implement custom datasets.
4 | You can specify '--dataset_mode template' to use this dataset.
5 | The class name should be consistent with both the filename and its dataset_mode option.
6 | The filename should be _dataset.py
7 | The class name should be Dataset.py
8 | You need to implement the following functions:
9 | -- : Add dataset-specific options and rewrite default values for existing options.
10 | -- <__init__>: Initialize this dataset class.
11 | -- <__getitem__>: Return a data point and its metadata information.
12 | -- <__len__>: Return the number of images.
13 | """
14 | from data.base_dataset import BaseDataset, get_transform
15 | # from data.image_folder import make_dataset
16 | # from PIL import Image
17 |
18 |
19 | class TemplateDataset(BaseDataset):
20 | """A template dataset class for you to implement custom datasets."""
21 | @staticmethod
22 | def modify_commandline_options(parser, is_train):
23 | """Add new dataset-specific options, and rewrite default values for existing options.
24 |
25 | Parameters:
26 | parser -- original option parser
27 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
28 |
29 | Returns:
30 | the modified parser.
31 | """
32 | parser.add_argument('--new_dataset_option', type=float, default=1.0, help='new dataset option')
33 | parser.set_defaults(max_dataset_size=10, new_dataset_option=2.0) # specify dataset-specific default values
34 | return parser
35 |
36 | def __init__(self, opt):
37 | """Initialize this dataset class.
38 |
39 | Parameters:
40 | opt (Option class) -- stores all the experiment flags; needs to be a subclass of BaseOptions
41 |
42 | A few things can be done here.
43 | - save the options (have been done in BaseDataset)
44 | - get image paths and meta information of the dataset.
45 | - define the image transformation.
46 | """
47 | # save the option and dataset root
48 | BaseDataset.__init__(self, opt)
49 | # get the image paths of your dataset;
50 | self.image_paths = [] # You can call sorted(make_dataset(self.root, opt.max_dataset_size)) to get all the image paths under the directory self.root
51 | # define the default transform function. You can use ; You can also define your custom transform function
52 | self.transform = get_transform(opt)
53 |
54 | def __getitem__(self, index):
55 | """Return a data point and its metadata information.
56 |
57 | Parameters:
58 | index -- a random integer for data indexing
59 |
60 | Returns:
61 | a dictionary of data with their names. It usually contains the data itself and its metadata information.
62 |
63 | Step 1: get a random image path: e.g., path = self.image_paths[index]
64 | Step 2: load your data from the disk: e.g., image = Image.open(path).convert('RGB').
65 | Step 3: convert your data to a PyTorch tensor. You can use helpder functions such as self.transform. e.g., data = self.transform(image)
66 | Step 4: return a data point as a dictionary.
67 | """
68 | path = 'temp' # needs to be a string
69 | data_A = None # needs to be a tensor
70 | data_B = None # needs to be a tensor
71 | return {'data_A': data_A, 'data_B': data_B, 'path': path}
72 |
73 | def __len__(self):
74 | """Return the total number of images."""
75 | return len(self.image_paths)
76 |
--------------------------------------------------------------------------------
/data/unaligned_dataset.py:
--------------------------------------------------------------------------------
1 | import os.path
2 | from data.base_dataset import BaseDataset, get_transform
3 | from data.image_folder import make_dataset
4 | from PIL import Image
5 | import random
6 |
7 |
8 | class UnalignedDataset(BaseDataset):
9 | """
10 | This dataset class can load unaligned/unpaired datasets.
11 |
12 | It requires two directories to host training images from domain A '/path/to/data/trainA'
13 | and from domain B '/path/to/data/trainB' respectively.
14 | You can train the model with the dataset flag '--dataroot /path/to/data'.
15 | Similarly, you need to prepare two directories:
16 | '/path/to/data/testA' and '/path/to/data/testB' during test time.
17 | """
18 |
19 | def __init__(self, opt):
20 | """Initialize this dataset class.
21 |
22 | Parameters:
23 | opt (Option class) -- stores all the experiment flags; needs to be a subclass of BaseOptions
24 | """
25 | BaseDataset.__init__(self, opt)
26 | self.dir_A = os.path.join(opt.dataroot, opt.phase + 'A') # create a path '/path/to/data/trainA'
27 | self.dir_B = os.path.join(opt.dataroot, opt.phase + 'B') # create a path '/path/to/data/trainB'
28 |
29 | self.A_paths = sorted(make_dataset(self.dir_A, opt.max_dataset_size)) # load images from '/path/to/data/trainA'
30 | self.B_paths = sorted(make_dataset(self.dir_B, opt.max_dataset_size)) # load images from '/path/to/data/trainB'
31 | self.A_size = len(self.A_paths) # get the size of dataset A
32 | self.B_size = len(self.B_paths) # get the size of dataset B
33 | btoA = self.opt.direction == 'BtoA'
34 | input_nc = self.opt.output_nc if btoA else self.opt.input_nc # get the number of channels of input image
35 | output_nc = self.opt.input_nc if btoA else self.opt.output_nc # get the number of channels of output image
36 | self.transform_A = get_transform(self.opt, grayscale=(input_nc == 1))
37 | self.transform_B = get_transform(self.opt, grayscale=(output_nc == 1))
38 |
39 | def __getitem__(self, index):
40 | """Return a data point and its metadata information.
41 |
42 | Parameters:
43 | index (int) -- a random integer for data indexing
44 |
45 | Returns a dictionary that contains A, B, A_paths and B_paths
46 | A (tensor) -- an image in the input domain
47 | B (tensor) -- its corresponding image in the target domain
48 | A_paths (str) -- image paths
49 | B_paths (str) -- image paths
50 | """
51 | A_path = self.A_paths[index % self.A_size] # make sure index is within then range
52 | if self.opt.serial_batches: # make sure index is within then range
53 | index_B = index % self.B_size
54 | else: # randomize the index for domain B to avoid fixed pairs.
55 | index_B = random.randint(0, self.B_size - 1)
56 | B_path = self.B_paths[index_B]
57 | A_img = Image.open(A_path).convert('RGB')
58 | B_img = Image.open(B_path).convert('RGB')
59 | # apply image transformation
60 | A = self.transform_A(A_img)
61 | B = self.transform_B(B_img)
62 |
63 | return {'A': A, 'B': B, 'A_paths': A_path, 'B_paths': B_path}
64 |
65 | def __len__(self):
66 | """Return the total number of images in the dataset.
67 |
68 | As we have two datasets with potentially different number of images,
69 | we take a maximum of
70 | """
71 | return max(self.A_size, self.B_size)
72 |
--------------------------------------------------------------------------------
/docs/Dockerfile:
--------------------------------------------------------------------------------
1 | FROM nvidia/cuda:9.0-base
2 |
3 | RUN apt update && apt install -y wget unzip curl bzip2 git
4 | RUN curl -LO http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh
5 | RUN bash Miniconda-latest-Linux-x86_64.sh -p /miniconda -b
6 | RUN rm Miniconda-latest-Linux-x86_64.sh
7 | ENV PATH=/miniconda/bin:${PATH}
8 | RUN conda update -y conda
9 |
10 | RUN conda install -y pytorch torchvision -c pytorch
11 | RUN mkdir /workspace/ && cd /workspace/ && git clone https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix.git && cd pytorch-CycleGAN-and-pix2pix && pip install -r requirements.txt
12 |
13 | WORKDIR /workspace
--------------------------------------------------------------------------------
/docs/datasets.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | ### CycleGAN Datasets
4 | Download the CycleGAN datasets using the following script. Some of the datasets are collected by other researchers. Please cite their papers if you use the data.
5 | ```bash
6 | bash ./datasets/download_cyclegan_dataset.sh dataset_name
7 | ```
8 | - `facades`: 400 images from the [CMP Facades dataset](http://cmp.felk.cvut.cz/~tylecr1/facade). [[Citation](datasets/bibtex/facades.tex)]
9 | - `cityscapes`: 2975 images from the [Cityscapes training set](https://www.cityscapes-dataset.com). [[Citation](datasets/bibtex/cityscapes.tex)]
10 | - `maps`: 1096 training images scraped from Google Maps.
11 | - `horse2zebra`: 939 horse images and 1177 zebra images downloaded from [ImageNet](http://www.image-net.org) using keywords `wild horse` and `zebra`
12 | - `apple2orange`: 996 apple images and 1020 orange images downloaded from [ImageNet](http://www.image-net.org) using keywords `apple` and `navel orange`.
13 | - `summer2winter_yosemite`: 1273 summer Yosemite images and 854 winter Yosemite images were downloaded using Flickr API. See more details in our paper.
14 | - `monet2photo`, `vangogh2photo`, `ukiyoe2photo`, `cezanne2photo`: The art images were downloaded from [Wikiart](https://www.wikiart.org/). The real photos are downloaded from Flickr using the combination of the tags *landscape* and *landscapephotography*. The training set size of each class is Monet:1074, Cezanne:584, Van Gogh:401, Ukiyo-e:1433, Photographs:6853.
15 | - `iphone2dslr_flower`: both classes of images were downlaoded from Flickr. The training set size of each class is iPhone:1813, DSLR:3316. See more details in our paper.
16 |
17 | To train a model on your own datasets, you need to create a data folder with two subdirectories `trainA` and `trainB` that contain images from domain A and B. You can test your model on your training set by setting `--phase train` in `test.py`. You can also create subdirectories `testA` and `testB` if you have test data.
18 |
19 | You should **not** expect our method to work on just any random combination of input and output datasets (e.g. `cats<->keyboards`). From our experiments, we find it works better if two datasets share similar visual content. For example, `landscape painting<->landscape photographs` works much better than `portrait painting <-> landscape photographs`. `zebras<->horses` achieves compelling results while `cats<->dogs` completely fails.
20 |
21 | ### pix2pix datasets
22 | Download the pix2pix datasets using the following script. Some of the datasets are collected by other researchers. Please cite their papers if you use the data.
23 | ```bash
24 | bash ./datasets/download_pix2pix_dataset.sh dataset_name
25 | ```
26 | - `facades`: 400 images from [CMP Facades dataset](http://cmp.felk.cvut.cz/~tylecr1/facade). [[Citation](datasets/bibtex/facades.tex)]
27 | - `cityscapes`: 2975 images from the [Cityscapes training set](https://www.cityscapes-dataset.com). [[Citation](datasets/bibtex/cityscapes.tex)]
28 | - `maps`: 1096 training images scraped from Google Maps
29 | - `edges2shoes`: 50k training images from [UT Zappos50K dataset](http://vision.cs.utexas.edu/projects/finegrained/utzap50k). Edges are computed by [HED](https://github.com/s9xie/hed) edge detector + post-processing. [[Citation](datasets/bibtex/shoes.tex)]
30 | - `edges2handbags`: 137K Amazon Handbag images from [iGAN project](https://github.com/junyanz/iGAN). Edges are computed by [HED](https://github.com/s9xie/hed) edge detector + post-processing. [[Citation](datasets/bibtex/handbags.tex)]
31 | - `night2day`: around 20K natural scene images from [Transient Attributes dataset](http://transattr.cs.brown.edu/) [[Citation](datasets/bibtex/transattr.tex)]. To train a `day2night` pix2pix model, you need to add `--direction BtoA`.
32 |
33 | We provide a python script to generate pix2pix training data in the form of pairs of images {A,B}, where A and B are two different depictions of the same underlying scene. For example, these might be pairs {label map, photo} or {bw image, color image}. Then we can learn to translate A to B or B to A:
34 |
35 | Create folder `/path/to/data` with subfolders `A` and `B`. `A` and `B` should each have their own subfolders `train`, `val`, `test`, etc. In `/path/to/data/A/train`, put training images in style A. In `/path/to/data/B/train`, put the corresponding images in style B. Repeat same for other data splits (`val`, `test`, etc).
36 |
37 | Corresponding images in a pair {A,B} must be the same size and have the same filename, e.g., `/path/to/data/A/train/1.jpg` is considered to correspond to `/path/to/data/B/train/1.jpg`.
38 |
39 | Once the data is formatted this way, call:
40 | ```bash
41 | python datasets/combine_A_and_B.py --fold_A /path/to/data/A --fold_B /path/to/data/B --fold_AB /path/to/data
42 | ```
43 |
44 | This will combine each pair of images (A,B) into a single image file, ready for training.
45 |
--------------------------------------------------------------------------------
/docs/docker.md:
--------------------------------------------------------------------------------
1 | # Docker image with pytorch-CycleGAN-and-pix2pix
2 |
3 | We provide both Dockerfile and pre-built Docker container that can run this code repo.
4 |
5 | ## Prerequisite
6 |
7 | - Install [docker-ce](https://docs.docker.com/install/linux/docker-ce/ubuntu/)
8 | - Install [nvidia-docker](https://github.com/NVIDIA/nvidia-docker#quickstart)
9 |
10 | ## Running pre-built Dockerfile
11 |
12 | - Pull the pre-built docker file
13 |
14 | ```bash
15 | docker pull taesungp/pytorch-cyclegan-and-pix2pix
16 | ```
17 |
18 | - Start an interactive docker session. `-p 8097:8097` option is needed if you want to run `visdom` server on the Docker container.
19 |
20 | ```bash
21 | nvidia-docker run -it -p 8097:8097 taesungp/pytorch-cyclegan-and-pix2pix
22 | ```
23 |
24 | - Now you are in the Docker environment. Go to our code repo and start running things.
25 | ```bash
26 | cd /workspace/pytorch-CycleGAN-and-pix2pix
27 | bash datasets/download_pix2pix_dataset.sh facades
28 | python -m visdom.server &
29 | bash scripts/train_pix2pix.sh
30 | ```
31 |
32 | ## Running with Dockerfile
33 |
34 | We also posted the [Dockerfile](Dockerfile). To generate the pre-built file, download the Dockerfile in this directory and run
35 | ```bash
36 | docker build -t [target_tag] .
37 | ```
38 | in the directory that contains the Dockerfile.
39 |
--------------------------------------------------------------------------------
/docs/overview.md:
--------------------------------------------------------------------------------
1 | ## Overview of Code Structure
2 | To help users better understand and use our codebase, we briefly overview the functionality and implementation of each package and each module. Please see the documentation in each file for more details. If you have questions, you may find useful information in [training/test tips](tips.md) and [frequently asked questions](qa.md).
3 |
4 | [train.py](../train.py) is a general-purpose training script. It works for various models (with option `--model`: e.g., `pix2pix`, `cyclegan`, `colorization`) and different datasets (with option `--dataset_mode`: e.g., `aligned`, `unaligned`, `single`, `colorization`). See the main [README](.../README.md) and [training/test tips](tips.md) for more details.
5 |
6 | [test.py](../test.py) is a general-purpose test script. Once you have trained your model with `train.py`, you can use this script to test the model. It will load a saved model from `--checkpoints_dir` and save the results to `--results_dir`. See the main [README](.../README.md) and [training/test tips](tips.md) for more details.
7 |
8 |
9 | [data](../data) directory contains all the modules related to data loading and preprocessing. To add a custom dataset class called `dummy`, you need to add a file called `dummy_dataset.py` and define a subclass `DummyDataset` inherited from `BaseDataset`. You need to implement four functions: `__init__` (initialize the class, you need to first call `BaseDataset.__init__(self, opt)`), `__len__` (return the size of dataset), `__getitem__` (get a data point), and optionally `modify_commandline_options` (add dataset-specific options and set default options). Now you can use the dataset class by specifying flag `--dataset_mode dummy`. See our template dataset [class](../data/template_dataset.py) for an example. Below we explain each file in details.
10 |
11 | * [\_\_init\_\_.py](../data/__init__.py) implements the interface between this package and training and test scripts. `train.py` and `test.py` call `from data import create_dataset` and `dataset = create_dataset(opt)` to create a dataset given the option `opt`.
12 | * [base_dataset.py](../data/base_dataset.py) implements an abstract base class ([ABC](https://docs.python.org/3/library/abc.html)) for datasets. It also includes common transformation functions (e.g., `get_transform`, `__scale_width`), which can be later used in subclasses.
13 | * [image_folder.py](../data/image_folder.py) implements an image folder class. We modify the official PyTorch image folder [code](https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py) so that this class can load images from both the current directory and its subdirectories.
14 | * [template_dataset.py](../data/template_dataset.py) provides a dataset template with detailed documentation. Check out this file if you plan to implement your own dataset.
15 | * [aligned_dataset.py](../data/aligned_dataset.py) includes a dataset class that can load image pairs. It assumes a single image directory `/path/to/data/train`, which contains image pairs in the form of {A,B}. See [here](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md#prepare-your-own-datasets-for-pix2pix) on how to prepare aligned datasets. During test time, you need to prepare a directory `/path/to/data/test` as test data.
16 | * [unaligned_dataset.py](../data/unaligned_dataset.py) includes a dataset class that can load unaligned/unpaired datasets. It assumes that two directories to host training images from domain A `/path/to/data/trainA` and from domain B `/path/to/data/trainB` respectively. Then you can train the model with the dataset flag `--dataroot /path/to/data`. Similarly, you need to prepare two directories `/path/to/data/testA` and `/path/to/data/testB` during test time.
17 | * [single_dataset.py](../data/single_dataset.py) includes a dataset class that can load a set of single images specified by the path `--dataroot /path/to/data`. It can be used for generating CycleGAN results only for one side with the model option `-model test`.
18 | * [colorization_dataset.py](../data/colorization_dataset.py) implements a dataset class that can load a set of nature images in RGB, and convert RGB format into (L, ab) pairs in [Lab](https://en.wikipedia.org/wiki/CIELAB_color_space) color space. It is required by pix2pix-based colorization model (`--model colorization`).
19 |
20 |
21 | [models](../models) directory contains modules related to objective functions, optimizations, and network architectures. To add a custom model class called `dummy`, you need to add a file called `dummy_model.py` and define a subclass `DummyModel` inherited from `BaseModel`. You need to implement four functions: `__init__` (initialize the class; you need to first call `BaseModel.__init__(self, opt)`), `set_input` (unpack data from dataset and apply preprocessing), `forward` (generate intermediate results), `optimize_parameters` (calculate loss, gradients, and update network weights), and optionally `modify_commandline_options` (add model-specific options and set default options). Now you can use the model class by specifying flag `--model dummy`. See our template model [class](../models/template_model.py) for an example. Below we explain each file in details.
22 |
23 | * [\_\_init\_\_.py](../models/__init__.py) implements the interface between this package and training and test scripts. `train.py` and `test.py` call `from models import create_model` and `model = create_model(opt)` to create a model given the option `opt`. You also need to call `model.setup(opt)` to properly initialize the model.
24 | * [base_model.py](../models/base_model.py) implements an abstract base class ([ABC](https://docs.python.org/3/library/abc.html)) for models. It also includes commonly used helper functions (e.g., `setup`, `test`, `update_learning_rate`, `save_networks`, `load_networks`), which can be later used in subclasses.
25 | * [template_model.py](../models/template_model.py) provides a model template with detailed documentation. Check out this file if you plan to implement your own model.
26 | * [pix2pix_model.py](../models/pix2pix_model.py) implements the pix2pix [model](https://phillipi.github.io/pix2pix/), for learning a mapping from input images to output images given paired data. The model training requires `--dataset_mode aligned` dataset. By default, it uses a `--netG unet256` [U-Net](https://arxiv.org/pdf/1505.04597.pdf) generator, a `--netD basic` discriminator (PatchGAN), and a `--gan_mode vanilla` GAN loss (standard cross-entropy objective).
27 | * [colorization_model.py](../models/colorization_model.py) implements a subclass of `Pix2PixModel` for image colorization (black & white image to colorful image). The model training requires `-dataset_model colorization` dataset. It trains a pix2pix model, mapping from L channel to ab channels in [Lab](https://en.wikipedia.org/wiki/CIELAB_color_space) color space. By default, the `colorization` dataset will automatically set `--input_nc 1` and `--output_nc 2`.
28 | * [cycle_gan_model.py](../models/cycle_gan_model.py) implements the CycleGAN [model](https://junyanz.github.io/CycleGAN/), for learning image-to-image translation without paired data. The model training requires `--dataset_mode unaligned` dataset. By default, it uses a `--netG resnet_9blocks` ResNet generator, a `--netD basic` discriminator (PatchGAN introduced by pix2pix), and a least-square GANs [objective](https://arxiv.org/abs/1611.04076) (`--gan_mode lsgan`).
29 | * [networks.py](../models/networks.py) module implements network architectures (both generators and discriminators), as well as normalization layers, initialization methods, optimization scheduler (i.e., learning rate policy), and GAN objective function (`vanilla`, `lsgan`, `wgangp`).
30 | * [test_model.py](../models/test_model.py) implements a model that can be used to generate CycleGAN results for only one direction. This model will automatically set `--dataset_mode single`, which only loads the images from one set. See the test [instruction](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix#apply-a-pre-trained-model-cyclegan) for more details.
31 |
32 | [options](../options) directory includes our option modules: training options, test options, and basic options (used in both training and test). `TrainOptions` and `TestOptions` are both subclasses of `BaseOptions`. They will reuse the options defined in `BaseOptions`.
33 | * [\_\_init\_\_.py](../options/__init__.py) is required to make Python treat the directory `options` as containing packages,
34 | * [base_options.py](../options/base_options.py) includes options that are used in both training and test. It also implements a few helper functions such as parsing, printing, and saving the options. It also gathers additional options defined in `modify_commandline_options` functions in both dataset class and model class.
35 | * [train_options.py](../options/train_options.py) includes options that are only used during training time.
36 | * [test_options.py](../options/test_options.py) includes options that are only used during test time.
37 |
38 |
39 | [util](../util) directory includes a miscellaneous collection of useful helper functions.
40 | * [\_\_init\_\_.py](../util/__init__.py) is required to make Python treat the directory `util` as containing packages,
41 | * [get_data.py](../util/get_data.py) provides a Python script for downloading CycleGAN and pix2pix datasets. Alternatively, You can also use bash scripts such as [download_pix2pix_model.sh](../scripts/download_pix2pix_model.sh) and [download_cyclegan_model.sh](../scripts/download_cyclegan_model.sh).
42 | * [html.py](../util/html.py) implements a module that saves images into a single HTML file. It consists of functions such as `add_header` (add a text header to the HTML file), `add_images` (add a row of images to the HTML file), `save` (save the HTML to the disk). It is based on Python library `dominate`, a Python library for creating and manipulating HTML documents using a DOM API.
43 | * [image_pool.py](../util/image_pool.py) implements an image buffer that stores previously generated images. This buffer enables us to update discriminators using a history of generated images rather than the ones produced by the latest generators. The original idea was discussed in this [paper](http://openaccess.thecvf.com/content_cvpr_2017/papers/Shrivastava_Learning_From_Simulated_CVPR_2017_paper.pdf). The size of the buffer is controlled by the flag `--pool_size`.
44 | * [visualizer.py](../util/visualizer.py) includes several functions that can display/save images and print/save logging information. It uses a Python library `visdom` for display and a Python library `dominate` (wrapped in `HTML`) for creating HTML files with images.
45 | * [util.py](../util/util.py) consists of simple helper functions such as `tensor2im` (convert a tensor array to a numpy image array), `diagnose_network` (calculate and print the mean of average absolute value of gradients), and `mkdirs` (create multiple directories).
46 |
--------------------------------------------------------------------------------
/docs/qa.md:
--------------------------------------------------------------------------------
1 | ## Frequently Asked Questions
2 | Before you post a new question, please first look at the following Q & A and existing GitHub issues. You may also want to read [Training/Test tips](docs/tips.md) for more suggestions.
3 |
4 | #### Connection Error:HTTPConnectionPool ([#230](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/230), [#24](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/24), [#38](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/38))
5 | Similar error messages include “Failed to establish a new connection/Connection refused”.
6 |
7 | Please start the visdom server before starting the training:
8 | ```bash
9 | python -m visdom.server
10 | ```
11 | To install the visdom, you can use the following command:
12 | ```bash
13 | pip install visdom
14 | ```
15 | You can also disable the visdom by setting `--display_id 0`.
16 |
17 | #### My PyTorch errors on CUDA related code.
18 | Try to run the following code snippet to make sure that CUDA is working (assuming using PyTorch >= 0.4):
19 | ```python
20 | import torch
21 | torch.cuda.init()
22 | print(torch.randn(1, device='cuda'))
23 | ```
24 |
25 | If you met an error, it is likely that your PyTorch build does not work with CUDA, e.g., it is installl from the official MacOS binary, or you have a GPU that is too old and not supported anymore. You may run the the code with CPU using `--gpu_ids -1`.
26 |
27 | #### TypeError: Object of type 'Tensor' is not JSON serializable ([#258](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/258))
28 | Similar errors: AttributeError: module 'torch' has no attribute 'device' ([#314](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/314))
29 |
30 | The current code only works with PyTorch 0.4+. An earlier PyTorch version can often cause the above errors.
31 |
32 | #### ValueError: empty range for randrange() ([#390](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/390), [#376](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/376), [#194](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/194))
33 | Similar error messages include "ConnectionRefusedError: [Errno 111] Connection refused"
34 |
35 | It is related to data augmentation step. It often happens when you use `--preprocess crop`. The program will crop random `crop_size x crop_size` patches out of the input training images. But if some of your image sizes (e.g., `256x384`) are smaller than the `crop_size` (e.g., 512), you will get this error. A simple fix will be to use other data augmentation methods such as `resize_and_crop` or `scale_width_and_crop`. Our program will automatically resize the images according to `load_size` before apply `crop_size x crop_size` cropping. Make sure that `load_size >= crop_size`.
36 |
37 |
38 | #### Can I continue/resume my training? ([#350](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/350), [#275](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/275), [#234](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/234), [#87](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/87))
39 | You can use the option `--continue_train`. Also set `--epoch_count` to specify a different starting epoch count. See more discussion in [training/test tips](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md#trainingtest-tips.
40 |
41 | #### Why does my training loss not converge? ([#335](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/335), [#164](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/164), [#30](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/30))
42 | Many GAN losses do not converge (exception: WGAN, WGAN-GP, etc. ) due to the nature of minimax optimization. For DCGAN and LSGAN objective, it is quite normal for the G and D losses to go up and down. It should be fine as long as they do not blow up.
43 |
44 | #### How can I make it work for my own data (e.g., 16-bit png, tiff, hyperspectral images)? ([#309](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/309), [#320](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/), [#202](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/202))
45 | The current code only supports RGB and grayscale images. If you would like to train the model on other data types, please follow the following steps:
46 |
47 | - change the parameters `--input_nc` and `--output_nc` to the number of channels in your input/output images.
48 | - Write your own custom data loader (It is easy as long as you know how to load your data with python). If you write a new data loader class, you need to change the flag `--dataset_mode` accordingly. Alternatively, you can modify the existing data loader. For aligned datasets, change this [line](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/data/aligned_dataset.py#L41); For unaligned datasets, change these two [lines](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/data/unaligned_dataset.py#L57).
49 |
50 | - If you use visdom and HTML to visualize the results, you may also need to change the visualization code.
51 |
52 | #### Multi-GPU Training ([#327](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/327), [#292](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/292), [#137](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/137), [#35](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/35))
53 | You can use Multi-GPU training by setting `--gpu_ids` (e.g., `--gpu_ids 0,1,2,3` for the first four GPUs on your machine.) To fully utilize all the GPUs, you need to increase your batch size. Try `--batch_size 4`, `--batch_size 16`, or even a larger batch_size. Each GPU will process batch_size/#GPUs images. The optimal batch size depends on the number of GPUs you have, GPU memory per GPU, and the resolution of your training images.
54 |
55 | We also recommend that you use the instance normalization for multi-GPU training by setting `--norm instance`. The current batch normalization might not work for multi-GPUs as the batchnorm parameters are not shared across different GPUs. Advanced users can try [synchronized batchnorm](https://github.com/vacancy/Synchronized-BatchNorm-PyTorch).
56 |
57 |
58 | #### Can I run the model on CPU? ([#310](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/310))
59 | Yes, you can set `--gpu_ids -1`. See [training/test tips](tips.md) for more details.
60 |
61 |
62 | #### Are pre-trained models available? ([#10](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/10))
63 | Yes, you can download pretrained models with the bash script `./scripts/download_cyclegan_model.sh`. See [here](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix#apply-a-pre-trained-model-cyclegan) for more details. We are slowly adding more models to the repo.
64 |
65 | #### Out of memory ([#174](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/174))
66 | CycleGAN is more memory-intensive than pix2pix as it requires two generators and two discriminators. If you would like to produce high-resolution images, you can do the following.
67 |
68 | - During training, train CycleGAN on cropped images of the training set. Please be careful not to change the aspect ratio or the scale of the original image, as this can lead to the training/test gap. You can usually do this by using `--preprocess crop` option, or `--preprocess scale_width_and_crop`.
69 |
70 | - Then at test time, you can load only one generator to produce the results in a single direction. This greatly saves GPU memory as you are not loading the discriminators and the other generator in the opposite direction. You can probably take the whole image as input. You can do this using `--model test --dataroot [path to the directory that contains your test images (e.g., ./datasets/horse2zebra/trainA)] --model_suffix _A --preprocess none`. You can use either `--preprocess none` or `--preprocess scale_width --crop_size [your_desired_image_width]`. Please see the [model_suffix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/test_model.py#L16) and [preprocess](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/data/base_dataset.py#L24) for more details.
71 |
72 | #### What is the identity loss? ([#322](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/322), [#373](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/373), [#362](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/pull/362))
73 | We use the identity loss for our photo to painting application. The identity loss can regularize the generator to be close to an identity mapping when fed with real samples from the *target* domain. If something already looks like from the target domain, you should preserve the image without making additional changes. The generator trained with this loss will often be more conservative for unknown content. Please see more details in Sec 5.2 ''Photo generation from paintings'' and Figure 12 in the CycleGAN [paper](https://arxiv.org/pdf/1703.10593.pdf). The loss was first proposed in the Equation 6 of the prior work [[Taigman et al., 2017]](https://arxiv.org/pdf/1611.02200.pdf).
74 |
75 | #### The color gets inverted from the beginning of training ([#249](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/249))
76 | The authors also observe that the generator unnecessarily inverts the color of the input image early in training, and then never learns to undo the inversion. In this case, you can try two things.
77 |
78 | - First, try using identity loss `--lambda_identity 1.0` or `--lambda_identity 0.1`. We observe that the identity loss makes the generator to be more conservative and make fewer unnecessary changes. However, because of this, the change may not be as dramatic.
79 |
80 | - Second, try smaller variance when initializing weights by changing `--init_gain`. We observe that smaller variance in weight initialization results in less color inversion.
81 |
82 | #### For labels2photo Cityscapes evaluation, why does the pretrained FCN-8s model not work well on the original Cityscapes input images? ([#150](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/150))
83 | The model was trained on 256x256 images that are resized/upsampled to 1024x2048, so expected input images to the network are very blurry. The purpose of the resizing was to 1) keep the label maps in the original high resolution untouched and 2) avoid the need of changing the standard FCN training code for Cityscapes.
84 |
85 | #### How do I get the `ground-truth` numbers on the labels2photo Cityscapes evaluation? ([#150](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/150))
86 | You need to resize the original Cityscapes images to 256x256 before running the evaluation code.
87 |
88 |
89 | #### Using resize-conv to reduce checkerboard artifacts ([#190](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/190), [#64](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/64))
90 | This Distill [blog](https://distill.pub/2016/deconv-checkerboard/) discussed one of the potential causes of the checkerboard artifacts. You can fix that issue by switching from "deconvolution" to nearest-neighbor upsampling followed by regular convolution. Here is one implementation provided by [@SsnL](https://github.com/SsnL). You can replace the ConvTranspose2d with the following layers.
91 | ```python
92 | nn.Upsample(scale_factor = 2, mode='bilinear'),
93 | nn.ReflectionPad2d(1),
94 | nn.Conv2d(ngf * mult, int(ngf * mult / 2), kernel_size=3, stride=1, padding=0),
95 | ```
96 | We have also noticed that sometimes the checkboard artifacts will go away if you train long enough. Maybe you can try training your model a bit longer.
97 |
98 | #### pix2pix/CycleGAN has no random noise z ([#152](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/152))
99 | The current pix2pix/CycleGAN model does not take z as input. In both pix2pix and CycleGAN, we tried to add z to the generator: e.g., adding z to a latent state, concatenating with a latent state, applying dropout, etc., but often found the output did not vary significantly as a function of z. Conditional GANs do not need noise as long as the input is sufficiently complex so that the input can kind of play the role of noise. Without noise, the mapping is deterministic.
100 |
101 | Please check out the following papers that show ways of getting z to actually have a substantial effect: e.g., [BicycleGAN](https://github.com/junyanz/BicycleGAN), [AugmentedCycleGAN](https://arxiv.org/abs/1802.10151), [MUNIT](https://arxiv.org/abs/1804.04732), [DRIT](https://arxiv.org/pdf/1808.00948.pdf), etc.
102 |
103 | #### Experiment details (e.g., BW->color) ([#306](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/306))
104 | You can find more training details and hyperparameter settings in the appendix of [CycleGAN](https://arxiv.org/abs/1703.10593) and [pix2pix](https://arxiv.org/abs/1611.07004) papers.
105 |
106 | #### Results with [Cycada](https://arxiv.org/pdf/1711.03213.pdf)
107 | We generated the [result of translating GTA images to Cityscapes-style images](https://junyanz.github.io/CycleGAN/) using our Torch repo. Our PyTorch and Torch implementation seemed to produce a little bit different results, although we have not measured the FCN score using the pytorch-trained model. To reproduce the result of Cycada, please use the Torch repo for now.
108 |
--------------------------------------------------------------------------------
/docs/tips.md:
--------------------------------------------------------------------------------
1 | ## Training/test Tips
2 | #### Training/test options
3 | Please see `options/train_options.py` and `options/base_options.py` for the training flags; see `options/test_options.py` and `options/base_options.py` for the test flags. There are some model-specific flags as well, which are added in the model files, such as `--lambda_A` option in `model/cycle_gan_model.py`. The default values of these options are also adjusted in the model files.
4 | #### CPU/GPU (default `--gpu_ids 0`)
5 | Please set`--gpu_ids -1` to use CPU mode; set `--gpu_ids 0,1,2` for multi-GPU mode. You need a large batch size (e.g., `--batch_size 32`) to benefit from multiple GPUs.
6 |
7 | #### Visualization
8 | During training, the current results can be viewed using two methods. First, if you set `--display_id` > 0, the results and loss plot will appear on a local graphics web server launched by [visdom](https://github.com/facebookresearch/visdom). To do this, you should have `visdom` installed and a server running by the command `python -m visdom.server`. The default server URL is `http://localhost:8097`. `display_id` corresponds to the window ID that is displayed on the `visdom` server. The `visdom` display functionality is turned on by default. To avoid the extra overhead of communicating with `visdom` set `--display_id -1`. Second, the intermediate results are saved to `[opt.checkpoints_dir]/[opt.name]/web/` as an HTML file. To avoid this, set `--no_html`.
9 |
10 | #### Preprocessing
11 | Images can be resized and cropped in different ways using `--preprocess` option. The default option `'resize_and_crop'` resizes the image to be of size `(opt.load_size, opt.load_size)` and does a random crop of size `(opt.crop_size, opt.crop_size)`. `'crop'` skips the resizing step and only performs random cropping. `'scale_width'` resizes the image to have width `opt.crop_size` while keeping the aspect ratio. `'scale_width_and_crop'` first resizes the image to have width `opt.load_size` and then does random cropping of size `(opt.crop_size, opt.crop_size)`. `'none'` tries to skip all these preprocessing steps. However, if the image size is not a multiple of some number depending on the number of downsamplings of the generator, you will get an error because the size of the output image may be different from the size of the input image. Therefore, `'none'` option still tries to adjust the image size to be a multiple of 4. You might need a bigger adjustment if you change the generator architecture. Please see `data/base_datset.py` do see how all these were implemented.
12 |
13 | #### Fine-tuning/resume training
14 | To fine-tune a pre-trained model, or resume the previous training, use the `--continue_train` flag. The program will then load the model based on `epoch`. By default, the program will initialize the epoch count as 1. Set `--epoch_count ` to specify a different starting epoch count.
15 |
16 |
17 | #### Prepare your own datasets for CycleGAN
18 | You need to create two directories to host images from domain A `/path/to/data/trainA` and from domain B `/path/to/data/trainB`. Then you can train the model with the dataset flag `--dataroot /path/to/data`. Optionally, you can create hold-out test datasets at `/path/to/data/testA` and `/path/to/data/testB` to test your model on unseen images.
19 |
20 | #### Prepare your own datasets for pix2pix
21 | Pix2pix's training requires paired data. We provide a python script to generate training data in the form of pairs of images {A,B}, where A and B are two different depictions of the same underlying scene. For example, these might be pairs {label map, photo} or {bw image, color image}. Then we can learn to translate A to B or B to A:
22 |
23 | Create folder `/path/to/data` with subdirectories `A` and `B`. `A` and `B` should each have their own subdirectories `train`, `val`, `test`, etc. In `/path/to/data/A/train`, put training images in style A. In `/path/to/data/B/train`, put the corresponding images in style B. Repeat same for other data splits (`val`, `test`, etc).
24 |
25 | Corresponding images in a pair {A,B} must be the same size and have the same filename, e.g., `/path/to/data/A/train/1.jpg` is considered to correspond to `/path/to/data/B/train/1.jpg`.
26 |
27 | Once the data is formatted this way, call:
28 | ```bash
29 | python datasets/combine_A_and_B.py --fold_A /path/to/data/A --fold_B /path/to/data/B --fold_AB /path/to/data
30 | ```
31 |
32 | This will combine each pair of images (A,B) into a single image file, ready for training.
33 |
34 |
35 | #### About image size
36 | Since the generator architecture in CycleGAN involves a series of downsampling / upsampling operations, the size of the input and output image may not match if the input image size is not a multiple of 4. As a result, you may get a runtime error because the L1 identity loss cannot be enforced with images of different size. Therefore, we slightly resize the image to become multiples of 4 even with `--preprocess none` option. For the same reason, `--crop_size` needs to be a multiple of 4.
37 |
38 | #### Training/Testing with high res images
39 | CycleGAN is quite memory-intensive as four networks (two generators and two discriminators) need to be loaded on one GPU, so a large image cannot be entirely loaded. In this case, we recommend training with cropped images. For example, to generate 1024px results, you can train with `--preprocess scale_width_and_crop --load_size 1024 --crop_size 360`, and test with `--preprocess scale_width --load_size 1024`. This way makes sure the training and test will be at the same scale. At test time, you can afford higher resolution because you don’t need to load all networks.
40 |
41 | #### About loss curve
42 | Unfortunately, the loss curve does not reveal much information in training GANs, and CycleGAN is no exception. To check whether the training has converged or not, we recommend periodically generating a few samples and looking at them.
43 |
44 | #### About batch size
45 | For all experiments in the paper, we set the batch size to be 1. If there is room for memory, you can use higher batch size with batch norm or instance norm. (Note that the default batchnorm does not work well with multi-GPU training. You may consider using [synchronized batchnorm](https://github.com/vacancy/Synchronized-BatchNorm-PyTorch) instead). But please be aware that it can impact the training. In particular, even with Instance Normalization, different batch sizes can lead to different results. Moreover, increasing `--crop_size` may be a good alternative to increasing the batch size.
46 |
47 |
48 | #### Notes on Colorization
49 | No need to run `combine_A_and_B.py` for colorization. Instead, you need to prepare natural images and set `--dataset_mode colorization` and `--model colorization` in the script. The program will automatically convert each RGB image into Lab color space, and create `L -> ab` image pair during the training. Also set `--input_nc 1` and `--output_nc 2`. The training and test directory should be organized as `/your/data/train` and `your/data/test`. See example scripts `scripts/train_colorization.sh` and `scripts/test_colorization` for more details.
50 |
51 | #### Notes on Extracting Edges
52 | We provide python and Matlab scripts to extract coarse edges from photos. Run `scripts/edges/batch_hed.py` to compute [HED](https://github.com/s9xie/hed) edges. Run `scripts/edges/PostprocessHED.m` to simplify edges with additional post-processing steps. Check the code documentation for more details.
53 |
54 | #### Evaluating Labels2Photos on Cityscapes
55 | We provide scripts for running the evaluation of the Labels2Photos task on the Cityscapes **validation** set. We assume that you have installed `caffe` (and `pycaffe`) in your system. If not, see the [official website](http://caffe.berkeleyvision.org/installation.html) for installation instructions. Once `caffe` is successfully installed, download the pre-trained FCN-8s semantic segmentation model (512MB) by running
56 | ```bash
57 | bash ./scripts/eval_cityscapes/download_fcn8s.sh
58 | ```
59 | Then make sure `./scripts/eval_cityscapes/` is in your system's python path. If not, run the following command to add it
60 | ```bash
61 | export PYTHONPATH=${PYTHONPATH}:./scripts/eval_cityscapes/
62 | ```
63 | Now you can run the following command to evaluate your predictions:
64 | ```bash
65 | python ./scripts/eval_cityscapes/evaluate.py --cityscapes_dir /path/to/original/cityscapes/dataset/ --result_dir /path/to/your/predictions/ --output_dir /path/to/output/directory/
66 | ```
67 | Images stored under `--result_dir` should contain your model predictions on the Cityscapes **validation** split, and have the original Cityscapes naming convention (e.g., `frankfurt_000001_038418_leftImg8bit.png`). The script will output a text file under `--output_dir` containing the metric.
68 |
69 | **Further notes**: The pre-trained model is **not** supposed to work on Cityscapes in the original resolution (1024x2048) as it was trained on 256x256 images that are upsampled to 1024x2048. The purpose of the resizing was to 1) keep the label maps in the original high resolution untouched and 2) avoid the need of changing the standard FCN training code for Cityscapes. To get the *ground-truth* numbers in the paper, you need to resize the original Cityscapes images to 256x256 before running the evaluation code.
70 |
--------------------------------------------------------------------------------
/environment.yml:
--------------------------------------------------------------------------------
1 | name: pytorch-CycleGAN-and-pix2pix
2 | channels:
3 | - peterjc123
4 | - defaults
5 | dependencies:
6 | - python=3.5.5
7 | - pytorch=0.4.1
8 | - scipy
9 | - pip:
10 | - dominate==2.3.1
11 | - git+https://github.com/pytorch/vision.git
12 | - Pillow==5.0.0
13 | - numpy==1.14.1
14 | - visdom==0.1.7
15 |
--------------------------------------------------------------------------------
/imgs/edges2cats.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/smartadpole/CycleGAN_Pytorch/6e53bb65f94173c33956cc8af933b858804c830a/imgs/edges2cats.jpg
--------------------------------------------------------------------------------
/imgs/horse2zebra.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/smartadpole/CycleGAN_Pytorch/6e53bb65f94173c33956cc8af933b858804c830a/imgs/horse2zebra.gif
--------------------------------------------------------------------------------
/models/__init__.py:
--------------------------------------------------------------------------------
1 | """This package contains modules related to objective functions, optimizations, and network architectures.
2 |
3 | To add a custom model class called 'dummy', you need to add a file called 'dummy_model.py' and define a subclass DummyModel inherited from BaseModel.
4 | You need to implement the following five functions:
5 | -- <__init__>: initialize the class; first call BaseModel.__init__(self, opt).
6 | -- : unpack data from dataset and apply preprocessing.
7 | -- : produce intermediate results.
8 | -- : calculate loss, gradients, and update network weights.
9 | -- : (optionally) add model-specific options and set default options.
10 |
11 | In the function <__init__>, you need to define four lists:
12 | -- self.loss_names (str list): specify the training losses that you want to plot and save.
13 | -- self.model_names (str list): define networks used in our training.
14 | -- self.visual_names (str list): specify the images that you want to display and save.
15 | -- self.optimizers (optimizer list): define and initialize optimizers. You can define one optimizer for each network. If two networks are updated at the same time, you can use itertools.chain to group them. See cycle_gan_model.py for an usage.
16 |
17 | Now you can use the model class by specifying flag '--model dummy'.
18 | See our template model class 'template_model.py' for more details.
19 | """
20 |
21 | import importlib
22 | from models.base_model import BaseModel
23 |
24 |
25 | def find_model_using_name(model_name):
26 | """Import the module "models/[model_name]_model.py".
27 |
28 | In the file, the class called DatasetNameModel() will
29 | be instantiated. It has to be a subclass of BaseModel,
30 | and it is case-insensitive.
31 | """
32 | model_filename = "models." + model_name + "_model"
33 | modellib = importlib.import_module(model_filename)
34 | model = None
35 | target_model_name = model_name.replace('_', '') + 'model'
36 | for name, cls in modellib.__dict__.items():
37 | if name.lower() == target_model_name.lower() \
38 | and issubclass(cls, BaseModel):
39 | model = cls
40 |
41 | if model is None:
42 | print("In %s.py, there should be a subclass of BaseModel with class name that matches %s in lowercase." % (model_filename, target_model_name))
43 | exit(0)
44 |
45 | return model
46 |
47 |
48 | def get_option_setter(model_name):
49 | """Return the static method of the model class."""
50 | model_class = find_model_using_name(model_name)
51 | return model_class.modify_commandline_options
52 |
53 |
54 | def create_model(opt):
55 | """Create a model given the option.
56 |
57 | This function warps the class CustomDatasetDataLoader.
58 | This is the main interface between this package and 'train.py'/'test.py'
59 |
60 | Example:
61 | >>> from models import create_model
62 | >>> model = create_model(opt)
63 | """
64 | model = find_model_using_name(opt.model)
65 | instance = model(opt)
66 | print("model [%s] was created" % type(instance).__name__)
67 | return instance
68 |
--------------------------------------------------------------------------------
/models/base_model.py:
--------------------------------------------------------------------------------
1 | import os
2 | import torch
3 | from collections import OrderedDict
4 | from abc import ABC, abstractmethod
5 | from . import networks
6 |
7 |
8 | class BaseModel(ABC):
9 | """This class is an abstract base class (ABC) for models.
10 | To create a subclass, you need to implement the following five functions:
11 | -- <__init__>: initialize the class; first call BaseModel.__init__(self, opt).
12 | -- : unpack data from dataset and apply preprocessing.
13 | -- : produce intermediate results.
14 | -- : calculate losses, gradients, and update network weights.
15 | -- : (optionally) add model-specific options and set default options.
16 | """
17 |
18 | def __init__(self, opt):
19 | """Initialize the BaseModel class.
20 |
21 | Parameters:
22 | opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
23 |
24 | When creating your custom class, you need to implement your own initialization.
25 | In this fucntion, you should first call
26 | Then, you need to define four lists:
27 | -- self.loss_names (str list): specify the training losses that you want to plot and save.
28 | -- self.model_names (str list): specify the images that you want to display and save.
29 | -- self.visual_names (str list): define networks used in our training.
30 | -- self.optimizers (optimizer list): define and initialize optimizers. You can define one optimizer for each network. If two networks are updated at the same time, you can use itertools.chain to group them. See cycle_gan_model.py for an example.
31 | """
32 | self.opt = opt
33 | self.gpu_ids = opt.gpu_ids
34 | self.isTrain = opt.isTrain
35 | self.device = torch.device('cuda:{}'.format(self.gpu_ids[0])) if self.gpu_ids else torch.device('cpu') # get device name: CPU or GPU
36 | self.save_dir = os.path.join(opt.checkpoints_dir, opt.name) # save all the checkpoints to save_dir
37 | if opt.preprocess != 'scale_width': # with [scale_width], input images might have different sizes, which hurts the performance of cudnn.benchmark.
38 | torch.backends.cudnn.benchmark = True
39 | self.loss_names = []
40 | self.model_names = []
41 | self.visual_names = []
42 | self.optimizers = []
43 | self.image_paths = []
44 | self.metric = 0 # used for learning rate policy 'plateau'
45 |
46 | @staticmethod
47 | def modify_commandline_options(parser, is_train):
48 | """Add new model-specific options, and rewrite default values for existing options.
49 |
50 | Parameters:
51 | parser -- original option parser
52 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
53 |
54 | Returns:
55 | the modified parser.
56 | """
57 | return parser
58 |
59 | @abstractmethod
60 | def set_input(self, input):
61 | """Unpack input data from the dataloader and perform necessary pre-processing steps.
62 |
63 | Parameters:
64 | input (dict): includes the data itself and its metadata information.
65 | """
66 | pass
67 |
68 | @abstractmethod
69 | def forward(self):
70 | """Run forward pass; called by both functions and ."""
71 | pass
72 |
73 | @abstractmethod
74 | def optimize_parameters(self):
75 | """Calculate losses, gradients, and update network weights; called in every training iteration"""
76 | pass
77 |
78 | def setup(self, opt):
79 | """Load and print networks; create schedulers
80 |
81 | Parameters:
82 | opt (Option class) -- stores all the experiment flags; needs to be a subclass of BaseOptions
83 | """
84 | if self.isTrain:
85 | self.schedulers = [networks.get_scheduler(optimizer, opt) for optimizer in self.optimizers]
86 | if not self.isTrain or opt.continue_train:
87 | load_suffix = 'iter_%d' % opt.load_iter if opt.load_iter > 0 else opt.epoch
88 | self.load_networks(load_suffix)
89 | self.print_networks(opt.verbose)
90 |
91 | def eval(self):
92 | """Make models eval mode during test time"""
93 | for name in self.model_names:
94 | if isinstance(name, str):
95 | net = getattr(self, 'net' + name)
96 | net.eval()
97 |
98 | def test(self):
99 | """Forward function used in test time.
100 |
101 | This function wraps function in no_grad() so we don't save intermediate steps for backprop
102 | It also calls to produce additional visualization results
103 | """
104 | with torch.no_grad():
105 | self.forward()
106 | self.compute_visuals()
107 |
108 | def compute_visuals(self):
109 | """Calculate additional output images for visdom and HTML visualization"""
110 | pass
111 |
112 | def get_image_paths(self):
113 | """ Return image paths that are used to load current data"""
114 | return self.image_paths
115 |
116 | def update_learning_rate(self):
117 | """Update learning rates for all the networks; called at the end of every epoch"""
118 | for scheduler in self.schedulers:
119 | if self.opt.lr_policy == 'plateau':
120 | scheduler.step(self.metric)
121 | else:
122 | scheduler.step()
123 |
124 | lr = self.optimizers[0].param_groups[0]['lr']
125 | print('learning rate = %.7f' % lr)
126 |
127 | def get_current_visuals(self):
128 | """Return visualization images. train.py will display these images with visdom, and save the images to a HTML"""
129 | visual_ret = OrderedDict()
130 | for name in self.visual_names:
131 | if isinstance(name, str):
132 | visual_ret[name] = getattr(self, name)
133 | return visual_ret
134 |
135 | def get_current_losses(self):
136 | """Return traning losses / errors. train.py will print out these errors on console, and save them to a file"""
137 | errors_ret = OrderedDict()
138 | for name in self.loss_names:
139 | if isinstance(name, str):
140 | errors_ret[name] = float(getattr(self, 'loss_' + name)) # float(...) works for both scalar tensor and float number
141 | return errors_ret
142 |
143 | def save_networks(self, epoch):
144 | """Save all the networks to the disk.
145 |
146 | Parameters:
147 | epoch (int) -- current epoch; used in the file name '%s_net_%s.pth' % (epoch, name)
148 | """
149 | for name in self.model_names:
150 | if isinstance(name, str):
151 | save_filename = '%s_net_%s.pth' % (epoch, name)
152 | save_path = os.path.join(self.save_dir, save_filename)
153 | net = getattr(self, 'net' + name)
154 |
155 | if len(self.gpu_ids) > 0 and torch.cuda.is_available():
156 | torch.save(net.module.cpu().state_dict(), save_path)
157 | net.cuda(self.gpu_ids[0])
158 | else:
159 | torch.save(net.cpu().state_dict(), save_path)
160 |
161 | def __patch_instance_norm_state_dict(self, state_dict, module, keys, i=0):
162 | """Fix InstanceNorm checkpoints incompatibility (prior to 0.4)"""
163 | key = keys[i]
164 | if i + 1 == len(keys): # at the end, pointing to a parameter/buffer
165 | if module.__class__.__name__.startswith('InstanceNorm') and \
166 | (key == 'running_mean' or key == 'running_var'):
167 | if getattr(module, key) is None:
168 | state_dict.pop('.'.join(keys))
169 | if module.__class__.__name__.startswith('InstanceNorm') and \
170 | (key == 'num_batches_tracked'):
171 | state_dict.pop('.'.join(keys))
172 | else:
173 | self.__patch_instance_norm_state_dict(state_dict, getattr(module, key), keys, i + 1)
174 |
175 | def load_networks(self, epoch):
176 | """Load all the networks from the disk.
177 |
178 | Parameters:
179 | epoch (int) -- current epoch; used in the file name '%s_net_%s.pth' % (epoch, name)
180 | """
181 | for name in self.model_names:
182 | if isinstance(name, str):
183 | load_filename = '%s_net_%s.pth' % (epoch, name)
184 | load_path = os.path.join(self.save_dir, load_filename)
185 | net = getattr(self, 'net' + name)
186 | if isinstance(net, torch.nn.DataParallel):
187 | net = net.module
188 | print('loading the model from %s' % load_path)
189 | # if you are using PyTorch newer than 0.4 (e.g., built from
190 | # GitHub source), you can remove str() on self.device
191 | state_dict = torch.load(load_path, map_location=str(self.device))
192 | if hasattr(state_dict, '_metadata'):
193 | del state_dict._metadata
194 |
195 | # patch InstanceNorm checkpoints prior to 0.4
196 | for key in list(state_dict.keys()): # need to copy keys here because we mutate in loop
197 | self.__patch_instance_norm_state_dict(state_dict, net, key.split('.'))
198 | net.load_state_dict(state_dict)
199 |
200 | def print_networks(self, verbose):
201 | """Print the total number of parameters in the network and (if verbose) network architecture
202 |
203 | Parameters:
204 | verbose (bool) -- if verbose: print the network architecture
205 | """
206 | print('---------- Networks initialized -------------')
207 | for name in self.model_names:
208 | if isinstance(name, str):
209 | net = getattr(self, 'net' + name)
210 | num_params = 0
211 | for param in net.parameters():
212 | num_params += param.numel()
213 | if verbose:
214 | print(net)
215 | print('[Network %s] Total number of parameters : %.3f M' % (name, num_params / 1e6))
216 | print('-----------------------------------------------')
217 |
218 | def set_requires_grad(self, nets, requires_grad=False):
219 | """Set requies_grad=Fasle for all the networks to avoid unnecessary computations
220 | Parameters:
221 | nets (network list) -- a list of networks
222 | requires_grad (bool) -- whether the networks require gradients or not
223 | """
224 | if not isinstance(nets, list):
225 | nets = [nets]
226 | for net in nets:
227 | if net is not None:
228 | for param in net.parameters():
229 | param.requires_grad = requires_grad
230 |
--------------------------------------------------------------------------------
/models/colorization_model.py:
--------------------------------------------------------------------------------
1 | from .pix2pix_model import Pix2PixModel
2 | import torch
3 | from skimage import color # used for lab2rgb
4 | import numpy as np
5 |
6 |
7 | class ColorizationModel(Pix2PixModel):
8 | """This is a subclass of Pix2PixModel for image colorization (black & white image -> colorful images).
9 |
10 | The model training requires '-dataset_model colorization' dataset.
11 | It trains a pix2pix model, mapping from L channel to ab channels in Lab color space.
12 | By default, the colorization dataset will automatically set '--input_nc 1' and '--output_nc 2'.
13 | """
14 | @staticmethod
15 | def modify_commandline_options(parser, is_train=True):
16 | """Add new dataset-specific options, and rewrite default values for existing options.
17 |
18 | Parameters:
19 | parser -- original option parser
20 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
21 |
22 | Returns:
23 | the modified parser.
24 |
25 | By default, we use 'colorization' dataset for this model.
26 | See the original pix2pix paper (https://arxiv.org/pdf/1611.07004.pdf) and colorization results (Figure 9 in the paper)
27 | """
28 | Pix2PixModel.modify_commandline_options(parser, is_train)
29 | parser.set_defaults(dataset_mode='colorization')
30 | return parser
31 |
32 | def __init__(self, opt):
33 | """Initialize the class.
34 |
35 | Parameters:
36 | opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
37 |
38 | For visualization, we set 'visual_names' as 'real_A' (input real image),
39 | 'real_B_rgb' (ground truth RGB image), and 'fake_B_rgb' (predicted RGB image)
40 | We convert the Lab image 'real_B' (inherited from Pix2pixModel) to a RGB image 'real_B_rgb'.
41 | we convert the Lab image 'fake_B' (inherited from Pix2pixModel) to a RGB image 'fake_B_rgb'.
42 | """
43 | # reuse the pix2pix model
44 | Pix2PixModel.__init__(self, opt)
45 | # specify the images to be visualized.
46 | self.visual_names = ['real_A', 'real_B_rgb', 'fake_B_rgb']
47 |
48 | def lab2rgb(self, L, AB):
49 | """Convert an Lab tensor image to a RGB numpy output
50 | Parameters:
51 | L (1-channel tensor array): L channel images (range: [-1, 1], torch tensor array)
52 | AB (2-channel tensor array): ab channel images (range: [-1, 1], torch tensor array)
53 |
54 | Returns:
55 | rgb (RGB numpy image): rgb output images (range: [0, 255], numpy array)
56 | """
57 | AB2 = AB * 110.0
58 | L2 = (L + 1.0) * 50.0
59 | Lab = torch.cat([L2, AB2], dim=1)
60 | Lab = Lab[0].data.cpu().float().numpy()
61 | Lab = np.transpose(Lab.astype(np.float64), (1, 2, 0))
62 | rgb = color.lab2rgb(Lab) * 255
63 | return rgb
64 |
65 | def compute_visuals(self):
66 | """Calculate additional output images for visdom and HTML visualization"""
67 | self.real_B_rgb = self.lab2rgb(self.real_A, self.real_B)
68 | self.fake_B_rgb = self.lab2rgb(self.real_A, self.fake_B)
69 |
--------------------------------------------------------------------------------
/models/cycle_gan_model.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import itertools
3 | from util.image_pool import ImagePool
4 | from .base_model import BaseModel
5 | from . import networks
6 |
7 |
8 | class CycleGANModel(BaseModel):
9 | """
10 | This class implements the CycleGAN model, for learning image-to-image translation without paired data.
11 |
12 | The model training requires '--dataset_mode unaligned' dataset.
13 | By default, it uses a '--netG resnet_9blocks' ResNet generator,
14 | a '--netD basic' discriminator (PatchGAN introduced by pix2pix),
15 | and a least-square GANs objective ('--gan_mode lsgan').
16 |
17 | CycleGAN paper: https://arxiv.org/pdf/1703.10593.pdf
18 | """
19 | @staticmethod
20 | def modify_commandline_options(parser, is_train=True):
21 | """Add new dataset-specific options, and rewrite default values for existing options.
22 |
23 | Parameters:
24 | parser -- original option parser
25 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
26 |
27 | Returns:
28 | the modified parser.
29 |
30 | For CycleGAN, in addition to GAN losses, we introduce lambda_A, lambda_B, and lambda_identity for the following losses.
31 | A (source domain), B (target domain).
32 | Generators: G_A: A -> B; G_B: B -> A.
33 | Discriminators: D_A: G_A(A) vs. B; D_B: G_B(B) vs. A.
34 | Forward cycle loss: lambda_A * ||G_B(G_A(A)) - A|| (Eqn. (2) in the paper)
35 | Backward cycle loss: lambda_B * ||G_A(G_B(B)) - B|| (Eqn. (2) in the paper)
36 | Identity loss (optional): lambda_identity * (||G_A(B) - B|| * lambda_B + ||G_B(A) - A|| * lambda_A) (Sec 5.2 "Photo generation from paintings" in the paper)
37 | Dropout is not used in the original CycleGAN paper.
38 | """
39 | parser.set_defaults(no_dropout=True) # default CycleGAN did not use dropout
40 | if is_train:
41 | parser.add_argument('--lambda_A', type=float, default=10.0, help='weight for cycle loss (A -> B -> A)')
42 | parser.add_argument('--lambda_B', type=float, default=10.0, help='weight for cycle loss (B -> A -> B)')
43 | parser.add_argument('--lambda_identity', type=float, default=0.5, help='use identity mapping. Setting lambda_identity other than 0 has an effect of scaling the weight of the identity mapping loss. For example, if the weight of the identity loss should be 10 times smaller than the weight of the reconstruction loss, please set lambda_identity = 0.1')
44 |
45 | return parser
46 |
47 | def __init__(self, opt):
48 | """Initialize the CycleGAN class.
49 |
50 | Parameters:
51 | opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
52 | """
53 | BaseModel.__init__(self, opt)
54 | # specify the training losses you want to print out. The training/test scripts will call
55 | self.loss_names = ['D_A', 'G_A', 'cycle_A', 'idt_A', 'D_B', 'G_B', 'cycle_B', 'idt_B']
56 | # specify the images you want to save/display. The training/test scripts will call
57 | visual_names_A = ['real_A', 'fake_B', 'rec_A']
58 | visual_names_B = ['real_B', 'fake_A', 'rec_B']
59 | if self.isTrain and self.opt.lambda_identity > 0.0: # if identity loss is used, we also visualize idt_B=G_A(B) ad idt_A=G_A(B)
60 | visual_names_A.append('idt_B')
61 | visual_names_B.append('idt_A')
62 |
63 | self.visual_names = visual_names_A + visual_names_B # combine visualizations for A and B
64 | # specify the models you want to save to the disk. The training/test scripts will call and .
65 | if self.isTrain:
66 | self.model_names = ['G_A', 'G_B', 'D_A', 'D_B']
67 | else: # during test time, only load Gs
68 | self.model_names = ['G_A', 'G_B']
69 |
70 | # define networks (both Generators and discriminators)
71 | # The naming is different from those used in the paper.
72 | # Code (vs. paper): G_A (G), G_B (F), D_A (D_Y), D_B (D_X)
73 | self.netG_A = networks.define_G(opt.input_nc, opt.output_nc, opt.ngf, opt.netG, opt.norm,
74 | not opt.no_dropout, opt.init_type, opt.init_gain, self.gpu_ids)
75 | self.netG_B = networks.define_G(opt.output_nc, opt.input_nc, opt.ngf, opt.netG, opt.norm,
76 | not opt.no_dropout, opt.init_type, opt.init_gain, self.gpu_ids)
77 |
78 | if self.isTrain: # define discriminators
79 | self.netD_A = networks.define_D(opt.output_nc, opt.ndf, opt.netD,
80 | opt.n_layers_D, opt.norm, opt.init_type, opt.init_gain, self.gpu_ids)
81 | self.netD_B = networks.define_D(opt.input_nc, opt.ndf, opt.netD,
82 | opt.n_layers_D, opt.norm, opt.init_type, opt.init_gain, self.gpu_ids)
83 |
84 | if self.isTrain:
85 | if opt.lambda_identity > 0.0: # only works when input and output images have the same number of channels
86 | assert(opt.input_nc == opt.output_nc)
87 | self.fake_A_pool = ImagePool(opt.pool_size) # create image buffer to store previously generated images
88 | self.fake_B_pool = ImagePool(opt.pool_size) # create image buffer to store previously generated images
89 | # define loss functions
90 | self.criterionGAN = networks.GANLoss(opt.gan_mode).to(self.device) # define GAN loss.
91 | self.criterionCycle = torch.nn.L1Loss()
92 | self.criterionIdt = torch.nn.L1Loss()
93 | # initialize optimizers; schedulers will be automatically created by function .
94 | self.optimizer_G = torch.optim.Adam(itertools.chain(self.netG_A.parameters(), self.netG_B.parameters()), lr=opt.lr, betas=(opt.beta1, 0.999))
95 | self.optimizer_D = torch.optim.Adam(itertools.chain(self.netD_A.parameters(), self.netD_B.parameters()), lr=opt.lr, betas=(opt.beta1, 0.999))
96 | self.optimizers.append(self.optimizer_G)
97 | self.optimizers.append(self.optimizer_D)
98 |
99 | def set_input(self, input):
100 | """Unpack input data from the dataloader and perform necessary pre-processing steps.
101 |
102 | Parameters:
103 | input (dict): include the data itself and its metadata information.
104 |
105 | The option 'direction' can be used to swap domain A and domain B.
106 | """
107 | AtoB = self.opt.direction == 'AtoB'
108 | self.real_A = input['A' if AtoB else 'B'].to(self.device)
109 | self.real_B = input['B' if AtoB else 'A'].to(self.device)
110 | self.image_paths = input['A_paths' if AtoB else 'B_paths']
111 |
112 | def forward(self):
113 | """Run forward pass; called by both functions and ."""
114 | self.fake_B = self.netG_A(self.real_A) # G_A(A)
115 | self.rec_A = self.netG_B(self.fake_B) # G_B(G_A(A))
116 | self.fake_A = self.netG_B(self.real_B) # G_B(B)
117 | self.rec_B = self.netG_A(self.fake_A) # G_A(G_B(B))
118 |
119 | def backward_D_basic(self, netD, real, fake):
120 | """Calculate GAN loss for the discriminator
121 |
122 | Parameters:
123 | netD (network) -- the discriminator D
124 | real (tensor array) -- real images
125 | fake (tensor array) -- images generated by a generator
126 |
127 | Return the discriminator loss.
128 | We also call loss_D.backward() to calculate the gradients.
129 | """
130 | # Real
131 | pred_real = netD(real)
132 | loss_D_real = self.criterionGAN(pred_real, True)
133 | # Fake
134 | pred_fake = netD(fake.detach())
135 | loss_D_fake = self.criterionGAN(pred_fake, False)
136 | # Combined loss and calculate gradients
137 | loss_D = (loss_D_real + loss_D_fake) * 0.5
138 | loss_D.backward()
139 | return loss_D
140 |
141 | def backward_D_A(self):
142 | """Calculate GAN loss for discriminator D_A"""
143 | fake_B = self.fake_B_pool.query(self.fake_B)
144 | self.loss_D_A = self.backward_D_basic(self.netD_A, self.real_B, fake_B)
145 |
146 | def backward_D_B(self):
147 | """Calculate GAN loss for discriminator D_B"""
148 | fake_A = self.fake_A_pool.query(self.fake_A)
149 | self.loss_D_B = self.backward_D_basic(self.netD_B, self.real_A, fake_A)
150 |
151 | def backward_G(self):
152 | """Calculate the loss for generators G_A and G_B"""
153 | lambda_idt = self.opt.lambda_identity
154 | lambda_A = self.opt.lambda_A
155 | lambda_B = self.opt.lambda_B
156 | # Identity loss
157 | if lambda_idt > 0:
158 | # G_A should be identity if real_B is fed: ||G_A(B) - B||
159 | self.idt_A = self.netG_A(self.real_B)
160 | self.loss_idt_A = self.criterionIdt(self.idt_A, self.real_B) * lambda_B * lambda_idt
161 | # G_B should be identity if real_A is fed: ||G_B(A) - A||
162 | self.idt_B = self.netG_B(self.real_A)
163 | self.loss_idt_B = self.criterionIdt(self.idt_B, self.real_A) * lambda_A * lambda_idt
164 | else:
165 | self.loss_idt_A = 0
166 | self.loss_idt_B = 0
167 |
168 | # GAN loss D_A(G_A(A))
169 | self.loss_G_A = self.criterionGAN(self.netD_A(self.fake_B), True)
170 | # GAN loss D_B(G_B(B))
171 | self.loss_G_B = self.criterionGAN(self.netD_B(self.fake_A), True)
172 | # Forward cycle loss || G_B(G_A(A)) - A||
173 | self.loss_cycle_A = self.criterionCycle(self.rec_A, self.real_A) * lambda_A
174 | # Backward cycle loss || G_A(G_B(B)) - B||
175 | self.loss_cycle_B = self.criterionCycle(self.rec_B, self.real_B) * lambda_B
176 | # combined loss and calculate gradients
177 | self.loss_G = self.loss_G_A + self.loss_G_B + self.loss_cycle_A + self.loss_cycle_B + self.loss_idt_A + self.loss_idt_B
178 | self.loss_G.backward()
179 |
180 | def optimize_parameters(self):
181 | """Calculate losses, gradients, and update network weights; called in every training iteration"""
182 | # forward
183 | self.forward() # compute fake images and reconstruction images.
184 | # G_A and G_B
185 | self.set_requires_grad([self.netD_A, self.netD_B], False) # Ds require no gradients when optimizing Gs
186 | self.optimizer_G.zero_grad() # set G_A and G_B's gradients to zero
187 | self.backward_G() # calculate gradients for G_A and G_B
188 | self.optimizer_G.step() # update G_A and G_B's weights
189 | # D_A and D_B
190 | self.set_requires_grad([self.netD_A, self.netD_B], True)
191 | self.optimizer_D.zero_grad() # set D_A and D_B's gradients to zero
192 | self.backward_D_A() # calculate gradients for D_A
193 | self.backward_D_B() # calculate graidents for D_B
194 | self.optimizer_D.step() # update D_A and D_B's weights
195 |
--------------------------------------------------------------------------------
/models/pix2pix_model.py:
--------------------------------------------------------------------------------
1 | import torch
2 | from .base_model import BaseModel
3 | from . import networks
4 |
5 |
6 | class Pix2PixModel(BaseModel):
7 | """ This class implements the pix2pix model, for learning a mapping from input images to output images given paired data.
8 |
9 | The model training requires '--dataset_mode aligned' dataset.
10 | By default, it uses a '--netG unet256' U-Net generator,
11 | a '--netD basic' discriminator (PatchGAN),
12 | and a '--gan_mode' vanilla GAN loss (the cross-entropy objective used in the orignal GAN paper).
13 |
14 | pix2pix paper: https://arxiv.org/pdf/1611.07004.pdf
15 | """
16 | @staticmethod
17 | def modify_commandline_options(parser, is_train=True):
18 | """Add new dataset-specific options, and rewrite default values for existing options.
19 |
20 | Parameters:
21 | parser -- original option parser
22 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
23 |
24 | Returns:
25 | the modified parser.
26 |
27 | For pix2pix, we do not use image buffer
28 | The training objective is: GAN Loss + lambda_L1 * ||G(A)-B||_1
29 | By default, we use vanilla GAN loss, UNet with batchnorm, and aligned datasets.
30 | """
31 | # changing the default values to match the pix2pix paper (https://phillipi.github.io/pix2pix/)
32 | parser.set_defaults(norm='batch', netG='unet_256', dataset_mode='aligned')
33 | if is_train:
34 | parser.set_defaults(pool_size=0, gan_mode='vanilla')
35 | parser.add_argument('--lambda_L1', type=float, default=100.0, help='weight for L1 loss')
36 |
37 | return parser
38 |
39 | def __init__(self, opt):
40 | """Initialize the pix2pix class.
41 |
42 | Parameters:
43 | opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
44 | """
45 | BaseModel.__init__(self, opt)
46 | # specify the training losses you want to print out. The training/test scripts will call
47 | self.loss_names = ['G_GAN', 'G_L1', 'D_real', 'D_fake']
48 | # specify the images you want to save/display. The training/test scripts will call
49 | self.visual_names = ['real_A', 'fake_B', 'real_B']
50 | # specify the models you want to save to the disk. The training/test scripts will call and
51 | if self.isTrain:
52 | self.model_names = ['G', 'D']
53 | else: # during test time, only load G
54 | self.model_names = ['G']
55 | # define networks (both generator and discriminator)
56 | self.netG = networks.define_G(opt.input_nc, opt.output_nc, opt.ngf, opt.netG, opt.norm,
57 | not opt.no_dropout, opt.init_type, opt.init_gain, self.gpu_ids)
58 |
59 | if self.isTrain: # define a discriminator; conditional GANs need to take both input and output images; Therefore, #channels for D is input_nc + output_nc
60 | self.netD = networks.define_D(opt.input_nc + opt.output_nc, opt.ndf, opt.netD,
61 | opt.n_layers_D, opt.norm, opt.init_type, opt.init_gain, self.gpu_ids)
62 |
63 | if self.isTrain:
64 | # define loss functions
65 | self.criterionGAN = networks.GANLoss(opt.gan_mode).to(self.device)
66 | self.criterionL1 = torch.nn.L1Loss()
67 | # initialize optimizers; schedulers will be automatically created by function .
68 | self.optimizer_G = torch.optim.Adam(self.netG.parameters(), lr=opt.lr, betas=(opt.beta1, 0.999))
69 | self.optimizer_D = torch.optim.Adam(self.netD.parameters(), lr=opt.lr, betas=(opt.beta1, 0.999))
70 | self.optimizers.append(self.optimizer_G)
71 | self.optimizers.append(self.optimizer_D)
72 |
73 | def set_input(self, input):
74 | """Unpack input data from the dataloader and perform necessary pre-processing steps.
75 |
76 | Parameters:
77 | input (dict): include the data itself and its metadata information.
78 |
79 | The option 'direction' can be used to swap images in domain A and domain B.
80 | """
81 | AtoB = self.opt.direction == 'AtoB'
82 | self.real_A = input['A' if AtoB else 'B'].to(self.device)
83 | self.real_B = input['B' if AtoB else 'A'].to(self.device)
84 | self.image_paths = input['A_paths' if AtoB else 'B_paths']
85 |
86 | def forward(self):
87 | """Run forward pass; called by both functions and ."""
88 | self.fake_B = self.netG(self.real_A) # G(A)
89 |
90 | def backward_D(self):
91 | """Calculate GAN loss for the discriminator"""
92 | # Fake; stop backprop to the generator by detaching fake_B
93 | fake_AB = torch.cat((self.real_A, self.fake_B), 1) # we use conditional GANs; we need to feed both input and output to the discriminator
94 | pred_fake = self.netD(fake_AB.detach())
95 | self.loss_D_fake = self.criterionGAN(pred_fake, False)
96 | # Real
97 | real_AB = torch.cat((self.real_A, self.real_B), 1)
98 | pred_real = self.netD(real_AB)
99 | self.loss_D_real = self.criterionGAN(pred_real, True)
100 | # combine loss and calculate gradients
101 | self.loss_D = (self.loss_D_fake + self.loss_D_real) * 0.5
102 | self.loss_D.backward()
103 |
104 | def backward_G(self):
105 | """Calculate GAN and L1 loss for the generator"""
106 | # First, G(A) should fake the discriminator
107 | fake_AB = torch.cat((self.real_A, self.fake_B), 1)
108 | pred_fake = self.netD(fake_AB)
109 | self.loss_G_GAN = self.criterionGAN(pred_fake, True)
110 | # Second, G(A) = B
111 | self.loss_G_L1 = self.criterionL1(self.fake_B, self.real_B) * self.opt.lambda_L1
112 | # combine loss and calculate gradients
113 | self.loss_G = self.loss_G_GAN + self.loss_G_L1
114 | self.loss_G.backward()
115 |
116 | def optimize_parameters(self):
117 | self.forward() # compute fake images: G(A)
118 | # update D
119 | self.set_requires_grad(self.netD, True) # enable backprop for D
120 | self.optimizer_D.zero_grad() # set D's gradients to zero
121 | self.backward_D() # calculate gradients for D
122 | self.optimizer_D.step() # update D's weights
123 | # update G
124 | self.set_requires_grad(self.netD, False) # D requires no gradients when optimizing G
125 | self.optimizer_G.zero_grad() # set G's gradients to zero
126 | self.backward_G() # calculate graidents for G
127 | self.optimizer_G.step() # udpate G's weights
128 |
--------------------------------------------------------------------------------
/models/template_model.py:
--------------------------------------------------------------------------------
1 | """Model class template
2 |
3 | This module provides a template for users to implement custom models.
4 | You can specify '--model template' to use this model.
5 | The class name should be consistent with both the filename and its model option.
6 | The filename should be _dataset.py
7 | The class name should be Dataset.py
8 | It implements a simple image-to-image translation baseline based on regression loss.
9 | Given input-output pairs (data_A, data_B), it learns a network netG that can minimize the following L1 loss:
10 | min_ ||netG(data_A) - data_B||_1
11 | You need to implement the following functions:
12 | : Add model-specific options and rewrite default values for existing options.
13 | <__init__>: Initialize this model class.
14 | : Unpack input data and perform data pre-processing.
15 | : Run forward pass. This will be called by both and .
16 | : Update network weights; it will be called in every training iteration.
17 | """
18 | import torch
19 | from .base_model import BaseModel
20 | from . import networks
21 |
22 |
23 | class TemplateModel(BaseModel):
24 | @staticmethod
25 | def modify_commandline_options(parser, is_train=True):
26 | """Add new model-specific options and rewrite default values for existing options.
27 |
28 | Parameters:
29 | parser -- the option parser
30 | is_train -- if it is training phase or test phase. You can use this flag to add training-specific or test-specific options.
31 |
32 | Returns:
33 | the modified parser.
34 | """
35 | parser.set_defaults(dataset_mode='aligned') # You can rewrite default values for this model. For example, this model usually uses aligned dataset as its dataset.
36 | if is_train:
37 | parser.add_argument('--lambda_regression', type=float, default=1.0, help='weight for the regression loss') # You can define new arguments for this model.
38 |
39 | return parser
40 |
41 | def __init__(self, opt):
42 | """Initialize this model class.
43 |
44 | Parameters:
45 | opt -- training/test options
46 |
47 | A few things can be done here.
48 | - (required) call the initialization function of BaseModel
49 | - define loss function, visualization images, model names, and optimizers
50 | """
51 | BaseModel.__init__(self, opt) # call the initialization method of BaseModel
52 | # specify the training losses you want to print out. The program will call base_model.get_current_losses to plot the losses to the console and save them to the disk.
53 | self.loss_names = ['loss_G']
54 | # specify the images you want to save and display. The program will call base_model.get_current_visuals to save and display these images.
55 | self.visual_names = ['data_A', 'data_B', 'output']
56 | # specify the models you want to save to the disk. The program will call base_model.save_networks and base_model.load_networks to save and load networks.
57 | # you can use opt.isTrain to specify different behaviors for training and test. For example, some networks will not be used during test, and you don't need to load them.
58 | self.model_names = ['G']
59 | # define networks; you can use opt.isTrain to specify different behaviors for training and test.
60 | self.netG = networks.define_G(opt.input_nc, opt.output_nc, opt.ngf, opt.netG, gpu_ids=self.gpu_ids)
61 | if self.isTrain: # only defined during training time
62 | # define your loss functions. You can use losses provided by torch.nn such as torch.nn.L1Loss.
63 | # We also provide a GANLoss class "networks.GANLoss". self.criterionGAN = networks.GANLoss().to(self.device)
64 | self.criterionLoss = torch.nn.L1Loss()
65 | # define and initialize optimizers. You can define one optimizer for each network.
66 | # If two networks are updated at the same time, you can use itertools.chain to group them. See cycle_gan_model.py for an example.
67 | self.optimizer = torch.optim.Adam(self.netG.parameters(), lr=opt.lr, betas=(opt.beta1, 0.999))
68 | self.optimizers = [self.optimizer]
69 |
70 | # Our program will automatically call to define schedulers, load networks, and print networks
71 |
72 | def set_input(self, input):
73 | """Unpack input data from the dataloader and perform necessary pre-processing steps.
74 |
75 | Parameters:
76 | input: a dictionary that contains the data itself and its metadata information.
77 | """
78 | AtoB = self.opt.direction == 'AtoB' # use to swap data_A and data_B
79 | self.data_A = input['A' if AtoB else 'B'].to(self.device) # get image data A
80 | self.data_B = input['B' if AtoB else 'A'].to(self.device) # get image data B
81 | self.image_paths = input['A_paths' if AtoB else 'B_paths'] # get image paths
82 |
83 | def forward(self):
84 | """Run forward pass. This will be called by both functions and ."""
85 | self.output = self.netG(self.data_A) # generate output image given the input data_A
86 |
87 | def backward(self):
88 | """Calculate losses, gradients, and update network weights; called in every training iteration"""
89 | # caculate the intermediate results if necessary; here self.output has been computed during function
90 | # calculate loss given the input and intermediate results
91 | self.loss_G = self.criterionLoss(self.output, self.data_B) * self.opt.lambda_regression
92 | self.loss_G.backward() # calculate gradients of network G w.r.t. loss_G
93 |
94 | def optimize_parameters(self):
95 | """Update network weights; it will be called in every training iteration."""
96 | self.forward() # first call forward to calculate intermediate results
97 | self.optimizer.zero_grad() # clear network G's existing gradients
98 | self.backward() # calculate gradients for network G
99 | self.optimizer.step() # update gradients for network G
100 |
--------------------------------------------------------------------------------
/models/test_model.py:
--------------------------------------------------------------------------------
1 | from .base_model import BaseModel
2 | from . import networks
3 |
4 |
5 | class TestModel(BaseModel):
6 | """ This TesteModel can be used to generate CycleGAN results for only one direction.
7 | This model will automatically set '--dataset_mode single', which only loads the images from one collection.
8 |
9 | See the test instruction for more details.
10 | """
11 | @staticmethod
12 | def modify_commandline_options(parser, is_train=True):
13 | """Add new dataset-specific options, and rewrite default values for existing options.
14 |
15 | Parameters:
16 | parser -- original option parser
17 | is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
18 |
19 | Returns:
20 | the modified parser.
21 |
22 | The model can only be used during test time. It requires '--dataset_mode single'.
23 | You need to specify the network using the option '--model_suffix'.
24 | """
25 | assert not is_train, 'TestModel cannot be used during training time'
26 | parser.set_defaults(dataset_mode='single')
27 | parser.add_argument('--model_suffix', type=str, default='', help='In checkpoints_dir, [epoch]_net_G[model_suffix].pth will be loaded as the generator.')
28 |
29 | return parser
30 |
31 | def __init__(self, opt):
32 | """Initialize the pix2pix class.
33 |
34 | Parameters:
35 | opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
36 | """
37 | assert(not opt.isTrain)
38 | BaseModel.__init__(self, opt)
39 | # specify the training losses you want to print out. The training/test scripts will call
40 | self.loss_names = []
41 | # specify the images you want to save/display. The training/test scripts will call
42 | self.visual_names = ['real_A', 'fake_B']
43 | # specify the models you want to save to the disk. The training/test scripts will call and
44 | self.model_names = ['G' + opt.model_suffix] # only generator is needed.
45 | self.netG = networks.define_G(opt.input_nc, opt.output_nc, opt.ngf, opt.netG,
46 | opt.norm, not opt.no_dropout, opt.init_type, opt.init_gain, self.gpu_ids)
47 |
48 | # assigns the model to self.netG_[suffix] so that it can be loaded
49 | # please see
50 | setattr(self, 'netG' + opt.model_suffix, self.netG) # store netG in self.
51 |
52 | def set_input(self, input):
53 | """Unpack input data from the dataloader and perform necessary pre-processing steps.
54 |
55 | Parameters:
56 | input: a dictionary that contains the data itself and its metadata information.
57 |
58 | We need to use 'single_dataset' dataset mode. It only load images from one domain.
59 | """
60 | self.real_A = input['A'].to(self.device)
61 | self.image_paths = input['A_paths']
62 |
63 | def forward(self):
64 | """Run forward pass."""
65 | self.fake_B = self.netG(self.real_A) # G(A)
66 |
67 | def optimize_parameters(self):
68 | """No optimization for test model."""
69 | pass
70 |
--------------------------------------------------------------------------------
/options/__init__.py:
--------------------------------------------------------------------------------
1 | """This package options includes option modules: training options, test options, and basic options (used in both training and test)."""
2 |
--------------------------------------------------------------------------------
/options/base_options.py:
--------------------------------------------------------------------------------
1 | import argparse
2 | import os
3 | from util import util
4 | import torch
5 | import models
6 | import data
7 |
8 |
9 | class BaseOptions():
10 | """This class defines options used during both training and test time.
11 |
12 | It also implements several helper functions such as parsing, printing, and saving the options.
13 | It also gathers additional options defined in functions in both dataset class and model class.
14 | """
15 |
16 | def __init__(self):
17 | """Reset the class; indicates the class hasn't been initailized"""
18 | self.initialized = False
19 |
20 | def initialize(self, parser):
21 | """Define the common options that are used in both training and test."""
22 | # basic parameters
23 | parser.add_argument('--dataroot', required=True, help='path to images (should have subfolders trainA, trainB, valA, valB, etc)')
24 | parser.add_argument('--name', type=str, default='experiment_name', help='name of the experiment. It decides where to store samples and models')
25 | parser.add_argument('--gpu_ids', type=str, default='0', help='gpu ids: e.g. 0 0,1,2, 0,2. use -1 for CPU')
26 | parser.add_argument('--checkpoints_dir', type=str, default='./checkpoints', help='models are saved here')
27 | # model parameters
28 | parser.add_argument('--model', type=str, default='cycle_gan', help='chooses which model to use. [cycle_gan | pix2pix | test | colorization]')
29 | parser.add_argument('--input_nc', type=int, default=3, help='# of input image channels: 3 for RGB and 1 for grayscale')
30 | parser.add_argument('--output_nc', type=int, default=3, help='# of output image channels: 3 for RGB and 1 for grayscale')
31 | parser.add_argument('--ngf', type=int, default=64, help='# of gen filters in the last conv layer')
32 | parser.add_argument('--ndf', type=int, default=64, help='# of discrim filters in the first conv layer')
33 | parser.add_argument('--netD', type=str, default='basic', help='specify discriminator architecture [basic | n_layers | pixel]. The basic model is a 70x70 PatchGAN. n_layers allows you to specify the layers in the discriminator')
34 | parser.add_argument('--netG', type=str, default='resnet_9blocks', help='specify generator architecture [resnet_9blocks | resnet_6blocks | unet_256 | unet_128]')
35 | parser.add_argument('--n_layers_D', type=int, default=3, help='only used if netD==n_layers')
36 | parser.add_argument('--norm', type=str, default='instance', help='instance normalization or batch normalization [instance | batch | none]')
37 | parser.add_argument('--init_type', type=str, default='normal', help='network initialization [normal | xavier | kaiming | orthogonal]')
38 | parser.add_argument('--init_gain', type=float, default=0.02, help='scaling factor for normal, xavier and orthogonal.')
39 | parser.add_argument('--no_dropout', action='store_true', help='no dropout for the generator')
40 | # dataset parameters
41 | parser.add_argument('--dataset_mode', type=str, default='unaligned', help='chooses how datasets are loaded. [unaligned | aligned | single | colorization]')
42 | parser.add_argument('--direction', type=str, default='AtoB', help='AtoB or BtoA')
43 | parser.add_argument('--serial_batches', action='store_true', help='if true, takes images in order to make batches, otherwise takes them randomly')
44 | parser.add_argument('--num_threads', default=4, type=int, help='# threads for loading data')
45 | parser.add_argument('--batch_size', type=int, default=1, help='input batch size')
46 | parser.add_argument('--load_size', type=int, default=286, help='scale images to this size')
47 | parser.add_argument('--crop_size', type=int, default=256, help='then crop to this size')
48 | parser.add_argument('--max_dataset_size', type=int, default=float("inf"), help='Maximum number of samples allowed per dataset. If the dataset directory contains more than max_dataset_size, only a subset is loaded.')
49 | parser.add_argument('--preprocess', type=str, default='resize_and_crop', help='scaling and cropping of images at load time [resize_and_crop | crop | scale_width | scale_width_and_crop | none]')
50 | parser.add_argument('--no_flip', action='store_true', help='if specified, do not flip the images for data augmentation')
51 | parser.add_argument('--display_winsize', type=int, default=256, help='display window size for both visdom and HTML')
52 | # additional parameters
53 | parser.add_argument('--epoch', type=str, default='latest', help='which epoch to load? set to latest to use latest cached model')
54 | parser.add_argument('--load_iter', type=int, default='0', help='which iteration to load? if load_iter > 0, the code will load models by iter_[load_iter]; otherwise, the code will load models by [epoch]')
55 | parser.add_argument('--verbose', action='store_true', help='if specified, print more debugging information')
56 | parser.add_argument('--suffix', default='', type=str, help='customized suffix: opt.name = opt.name + suffix: e.g., {model}_{netG}_size{load_size}')
57 | self.initialized = True
58 | return parser
59 |
60 | def gather_options(self):
61 | """Initialize our parser with basic options(only once).
62 | Add additional model-specific and dataset-specific options.
63 | These options are defined in the function
64 | in model and dataset classes.
65 | """
66 | if not self.initialized: # check if it has been initialized
67 | parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
68 | parser = self.initialize(parser)
69 |
70 | # get the basic options
71 | opt, _ = parser.parse_known_args()
72 |
73 | # modify model-related parser options
74 | model_name = opt.model
75 | model_option_setter = models.get_option_setter(model_name)
76 | parser = model_option_setter(parser, self.isTrain)
77 | opt, _ = parser.parse_known_args() # parse again with new defaults
78 |
79 | # modify dataset-related parser options
80 | dataset_name = opt.dataset_mode
81 | dataset_option_setter = data.get_option_setter(dataset_name)
82 | parser = dataset_option_setter(parser, self.isTrain)
83 |
84 | # save and return the parser
85 | self.parser = parser
86 | return parser.parse_args()
87 |
88 | def print_options(self, opt):
89 | """Print and save options
90 |
91 | It will print both current options and default values(if different).
92 | It will save options into a text file / [checkpoints_dir] / opt.txt
93 | """
94 | message = ''
95 | message += '----------------- Options ---------------\n'
96 | for k, v in sorted(vars(opt).items()):
97 | comment = ''
98 | default = self.parser.get_default(k)
99 | if v != default:
100 | comment = '\t[default: %s]' % str(default)
101 | message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
102 | message += '----------------- End -------------------'
103 | print(message)
104 |
105 | # save to the disk
106 | expr_dir = os.path.join(opt.checkpoints_dir, opt.name)
107 | util.mkdirs(expr_dir)
108 | file_name = os.path.join(expr_dir, '{}_opt.txt'.format(opt.phase))
109 | with open(file_name, 'wt') as opt_file:
110 | opt_file.write(message)
111 | opt_file.write('\n')
112 |
113 | def parse(self):
114 | """Parse our options, create checkpoints directory suffix, and set up gpu device."""
115 | opt = self.gather_options()
116 | opt.isTrain = self.isTrain # train or test
117 |
118 | # process opt.suffix
119 | if opt.suffix:
120 | suffix = ('_' + opt.suffix.format(**vars(opt))) if opt.suffix != '' else ''
121 | opt.name = opt.name + suffix
122 |
123 | self.print_options(opt)
124 |
125 | # set gpu ids
126 | str_ids = opt.gpu_ids.split(',')
127 | opt.gpu_ids = []
128 | for str_id in str_ids:
129 | id = int(str_id)
130 | if id >= 0:
131 | opt.gpu_ids.append(id)
132 | if len(opt.gpu_ids) > 0:
133 | torch.cuda.set_device(opt.gpu_ids[0])
134 |
135 | self.opt = opt
136 | return self.opt
137 |
--------------------------------------------------------------------------------
/options/test_options.py:
--------------------------------------------------------------------------------
1 | from .base_options import BaseOptions
2 |
3 |
4 | class TestOptions(BaseOptions):
5 | """This class includes test options.
6 |
7 | It also includes shared options defined in BaseOptions.
8 | """
9 |
10 | def initialize(self, parser):
11 | parser = BaseOptions.initialize(self, parser) # define shared options
12 | parser.add_argument('--ntest', type=int, default=float("inf"), help='# of test examples.')
13 | parser.add_argument('--results_dir', type=str, default='./results/', help='saves results here.')
14 | parser.add_argument('--aspect_ratio', type=float, default=1.0, help='aspect ratio of result images')
15 | parser.add_argument('--phase', type=str, default='test', help='train, val, test, etc')
16 | # Dropout and Batchnorm has different behavioir during training and test.
17 | parser.add_argument('--eval', action='store_true', help='use eval mode during test time.')
18 | parser.add_argument('--num_test', type=int, default=50, help='how many test images to run')
19 | # rewrite devalue values
20 | parser.set_defaults(model='test')
21 | # To avoid cropping, the load_size should be the same as crop_size
22 | parser.set_defaults(load_size=parser.get_default('crop_size'))
23 | self.isTrain = False
24 | return parser
25 |
--------------------------------------------------------------------------------
/options/train_options.py:
--------------------------------------------------------------------------------
1 | from .base_options import BaseOptions
2 |
3 |
4 | class TrainOptions(BaseOptions):
5 | """This class includes training options.
6 |
7 | It also includes shared options defined in BaseOptions.
8 | """
9 |
10 | def initialize(self, parser):
11 | parser = BaseOptions.initialize(self, parser)
12 | # visdom and HTML visualization parameters
13 | parser.add_argument('--display_freq', type=int, default=400, help='frequency of showing training results on screen')
14 | parser.add_argument('--display_ncols', type=int, default=4, help='if positive, display all images in a single visdom web panel with certain number of images per row.')
15 | parser.add_argument('--display_id', type=int, default=1, help='window id of the web display')
16 | parser.add_argument('--display_server', type=str, default="http://localhost", help='visdom server of the web display')
17 | parser.add_argument('--display_env', type=str, default='main', help='visdom display environment name (default is "main")')
18 | parser.add_argument('--display_port', type=int, default=8097, help='visdom port of the web display')
19 | parser.add_argument('--update_html_freq', type=int, default=1000, help='frequency of saving training results to html')
20 | parser.add_argument('--print_freq', type=int, default=100, help='frequency of showing training results on console')
21 | parser.add_argument('--no_html', action='store_true', help='do not save intermediate training results to [opt.checkpoints_dir]/[opt.name]/web/')
22 | # network saving and loading parameters
23 | parser.add_argument('--save_latest_freq', type=int, default=5000, help='frequency of saving the latest results')
24 | parser.add_argument('--save_epoch_freq', type=int, default=5, help='frequency of saving checkpoints at the end of epochs')
25 | parser.add_argument('--save_by_iter', action='store_true', help='whether saves model by iteration')
26 | parser.add_argument('--continue_train', action='store_true', help='continue training: load the latest model')
27 | parser.add_argument('--epoch_count', type=int, default=1, help='the starting epoch count, we save the model by , +, ...')
28 | parser.add_argument('--phase', type=str, default='train', help='train, val, test, etc')
29 | # training parameters
30 | parser.add_argument('--niter', type=int, default=100, help='# of iter at starting learning rate')
31 | parser.add_argument('--niter_decay', type=int, default=100, help='# of iter to linearly decay learning rate to zero')
32 | parser.add_argument('--beta1', type=float, default=0.5, help='momentum term of adam')
33 | parser.add_argument('--lr', type=float, default=0.0002, help='initial learning rate for adam')
34 | parser.add_argument('--gan_mode', type=str, default='lsgan', help='the type of GAN objective. [vanilla| lsgan | wgangp]. vanilla GAN loss is the cross-entropy objective used in the original GAN paper.')
35 | parser.add_argument('--pool_size', type=int, default=50, help='the size of image buffer that stores previously generated images')
36 | parser.add_argument('--lr_policy', type=str, default='linear', help='learning rate policy. [linear | step | plateau | cosine]')
37 | parser.add_argument('--lr_decay_iters', type=int, default=50, help='multiply by a gamma every lr_decay_iters iterations')
38 |
39 | self.isTrain = True
40 | return parser
41 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | torch>=0.4.1
2 | torchvision>=0.2.1
3 | dominate>=2.3.1
4 | visdom>=0.1.8.3
5 |
--------------------------------------------------------------------------------
/scripts/conda_deps.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | conda install numpy pyyaml mkl mkl-include setuptools cmake cffi typing
3 | conda install pytorch torchvision -c pytorch # add cuda90 if CUDA 9
4 | conda install visdom dominate -c conda-forge # install visdom and dominate
5 |
--------------------------------------------------------------------------------
/scripts/download_cyclegan_model.sh:
--------------------------------------------------------------------------------
1 | FILE=$1
2 |
3 | echo "Note: available models are apple2orange, orange2apple, summer2winter_yosemite, winter2summer_yosemite, horse2zebra, zebra2horse, monet2photo, style_monet, style_cezanne, style_ukiyoe, style_vangogh, sat2map, map2sat, cityscapes_photo2label, cityscapes_label2photo, facades_photo2label, facades_label2photo, iphone2dslr_flower"
4 |
5 | echo "Specified [$FILE]"
6 |
7 | mkdir -p ./checkpoints/${FILE}_pretrained
8 | MODEL_FILE=./checkpoints/${FILE}_pretrained/latest_net_G.pth
9 | URL=http://efrosgans.eecs.berkeley.edu/cyclegan/pretrained_models/$FILE.pth
10 |
11 | wget -N $URL -O $MODEL_FILE
12 |
--------------------------------------------------------------------------------
/scripts/download_pix2pix_model.sh:
--------------------------------------------------------------------------------
1 | FILE=$1
2 |
3 | echo "Note: available models are edges2shoes, sat2map, map2sat, facades_label2photo, and day2night"
4 | echo "Specified [$FILE]"
5 |
6 | mkdir -p ./checkpoints/${FILE}_pretrained
7 | MODEL_FILE=./checkpoints/${FILE}_pretrained/latest_net_G.pth
8 | URL=http://efrosgans.eecs.berkeley.edu/pix2pix/models-pytorch/$FILE.pth
9 |
10 | wget -N $URL -O $MODEL_FILE
11 |
--------------------------------------------------------------------------------
/scripts/edges/PostprocessHED.m:
--------------------------------------------------------------------------------
1 | %%% Prerequisites
2 | % You need to get the cpp file edgesNmsMex.cpp from https://raw.githubusercontent.com/pdollar/edges/master/private/edgesNmsMex.cpp
3 | % and compile it in Matlab: mex edgesNmsMex.cpp
4 | % You also need to download and install Piotr's Computer Vision Matlab Toolbox: https://pdollar.github.io/toolbox/
5 |
6 | %%% parameters
7 | % hed_mat_dir: the hed mat file directory (the output of 'batch_hed.py')
8 | % edge_dir: the output HED edges directory
9 | % image_width: resize the edge map to [image_width, image_width]
10 | % threshold: threshold for image binarization (default 25.0/255.0)
11 | % small_edge: remove small edges (default 5)
12 |
13 | function [] = PostprocessHED(hed_mat_dir, edge_dir, image_width, threshold, small_edge)
14 |
15 | if ~exist(edge_dir, 'dir')
16 | mkdir(edge_dir);
17 | end
18 | fileList = dir(fullfile(hed_mat_dir, '*.mat'));
19 | nFiles = numel(fileList);
20 | fprintf('find %d mat files\n', nFiles);
21 |
22 | for n = 1 : nFiles
23 | if mod(n, 1000) == 0
24 | fprintf('process %d/%d images\n', n, nFiles);
25 | end
26 | fileName = fileList(n).name;
27 | filePath = fullfile(hed_mat_dir, fileName);
28 | jpgName = strrep(fileName, '.mat', '.jpg');
29 | edge_path = fullfile(edge_dir, jpgName);
30 |
31 | if ~exist(edge_path, 'file')
32 | E = GetEdge(filePath);
33 | E = imresize(E,[image_width,image_width]);
34 | E_simple = SimpleEdge(E, threshold, small_edge);
35 | E_simple = uint8(E_simple*255);
36 | imwrite(E_simple, edge_path, 'Quality',100);
37 | end
38 | end
39 | end
40 |
41 |
42 |
43 |
44 | function [E] = GetEdge(filePath)
45 | load(filePath);
46 | E = 1-predict;
47 | end
48 |
49 | function [E4] = SimpleEdge(E, threshold, small_edge)
50 | if nargin <= 1
51 | threshold = 25.0/255.0;
52 | end
53 |
54 | if nargin <= 2
55 | small_edge = 5;
56 | end
57 |
58 | if ndims(E) == 3
59 | E = E(:,:,1);
60 | end
61 |
62 | E1 = 1 - E;
63 | E2 = EdgeNMS(E1);
64 | E3 = double(E2>=max(eps,threshold));
65 | E3 = bwmorph(E3,'thin',inf);
66 | E4 = bwareaopen(E3, small_edge);
67 | E4=1-E4;
68 | end
69 |
70 | function [E_nms] = EdgeNMS( E )
71 | E=single(E);
72 | [Ox,Oy] = gradient2(convTri(E,4));
73 | [Oxx,~] = gradient2(Ox);
74 | [Oxy,Oyy] = gradient2(Oy);
75 | O = mod(atan(Oyy.*sign(-Oxy)./(Oxx+1e-5)),pi);
76 | E_nms = edgesNmsMex(E,O,1,5,1.01,1);
77 | end
78 |
--------------------------------------------------------------------------------
/scripts/edges/batch_hed.py:
--------------------------------------------------------------------------------
1 | # HED batch processing script; modified from https://github.com/s9xie/hed/blob/master/examples/hed/HED-tutorial.ipynb
2 | # Step 1: download the hed repo: https://github.com/s9xie/hed
3 | # Step 2: download the models and protoxt, and put them under {caffe_root}/examples/hed/
4 | # Step 3: put this script under {caffe_root}/examples/hed/
5 | # Step 4: run the following script:
6 | # python batch_hed.py --images_dir=/data/to/path/photos/ --hed_mat_dir=/data/to/path/hed_mat_files/
7 | # The code sometimes crashes after computation is done. Error looks like "Check failed: ... driver shutting down". You can just kill the job.
8 | # For large images, it will produce gpu memory issue. Therefore, you better resize the images before running this script.
9 | # Step 5: run the MATLAB post-processing script "PostprocessHED.m"
10 |
11 |
12 | import numpy as np
13 | from PIL import Image
14 | import os
15 | import argparse
16 | import sys
17 | import scipy.io as sio
18 |
19 |
20 | def parse_args():
21 | parser = argparse.ArgumentParser(description='batch proccesing: photos->edges')
22 | parser.add_argument('--caffe_root', dest='caffe_root', help='caffe root', default='../../', type=str)
23 | parser.add_argument('--caffemodel', dest='caffemodel', help='caffemodel', default='./hed_pretrained_bsds.caffemodel', type=str)
24 | parser.add_argument('--prototxt', dest='prototxt', help='caffe prototxt file', default='./deploy.prototxt', type=str)
25 | parser.add_argument('--images_dir', dest='images_dir', help='directory to store input photos', type=str)
26 | parser.add_argument('--hed_mat_dir', dest='hed_mat_dir', help='directory to store output hed edges in mat file', type=str)
27 | parser.add_argument('--border', dest='border', help='padding border', type=int, default=128)
28 | parser.add_argument('--gpu_id', dest='gpu_id', help='gpu id', type=int, default=1)
29 | args = parser.parse_args()
30 | return args
31 |
32 |
33 | args = parse_args()
34 | for arg in vars(args):
35 | print('[%s] =' % arg, getattr(args, arg))
36 | # Make sure that caffe is on the python path:
37 | caffe_root = args.caffe_root # this file is expected to be in {caffe_root}/examples/hed/
38 | sys.path.insert(0, caffe_root + 'python')
39 | import caffe
40 |
41 |
42 | if not os.path.exists(args.hed_mat_dir):
43 | print('create output directory %s' % args.hed_mat_dir)
44 | os.makedirs(args.hed_mat_dir)
45 |
46 | imgList = os.listdir(args.images_dir)
47 | nImgs = len(imgList)
48 | print('#images = %d' % nImgs)
49 |
50 | caffe.set_mode_gpu()
51 | caffe.set_device(args.gpu_id)
52 | # load net
53 | net = caffe.Net(args.prototxt, args.caffemodel, caffe.TEST)
54 | # pad border
55 | border = args.border
56 |
57 | for i in range(nImgs):
58 | if i % 500 == 0:
59 | print('processing image %d/%d' % (i, nImgs))
60 | im = Image.open(os.path.join(args.images_dir, imgList[i]))
61 |
62 | in_ = np.array(im, dtype=np.float32)
63 | in_ = np.pad(in_, ((border, border), (border, border), (0, 0)), 'reflect')
64 |
65 | in_ = in_[:, :, 0:3]
66 | in_ = in_[:, :, ::-1]
67 | in_ -= np.array((104.00698793, 116.66876762, 122.67891434))
68 | in_ = in_.transpose((2, 0, 1))
69 | # remove the following two lines if testing with cpu
70 |
71 | # shape for input (data blob is N x C x H x W), set data
72 | net.blobs['data'].reshape(1, *in_.shape)
73 | net.blobs['data'].data[...] = in_
74 | # run net and take argmax for prediction
75 | net.forward()
76 | fuse = net.blobs['sigmoid-fuse'].data[0][0, :, :]
77 | # get rid of the border
78 | fuse = fuse[border:-border, border:-border]
79 | # save hed file to the disk
80 | name, ext = os.path.splitext(imgList[i])
81 | sio.savemat(os.path.join(args.hed_mat_dir, name + '.mat'), {'predict': fuse})
82 |
--------------------------------------------------------------------------------
/scripts/eval_cityscapes/caffemodel/deploy.prototxt:
--------------------------------------------------------------------------------
1 | layer {
2 | name: "data"
3 | type: "Input"
4 | top: "data"
5 | input_param {
6 | shape {
7 | dim: 1
8 | dim: 3
9 | dim: 500
10 | dim: 500
11 | }
12 | }
13 | }
14 | layer {
15 | name: "conv1_1"
16 | type: "Convolution"
17 | bottom: "data"
18 | top: "conv1_1"
19 | param {
20 | lr_mult: 1
21 | decay_mult: 1
22 | }
23 | param {
24 | lr_mult: 2
25 | decay_mult: 0
26 | }
27 | convolution_param {
28 | num_output: 64
29 | pad: 100
30 | kernel_size: 3
31 | stride: 1
32 | weight_filler {
33 | type: "gaussian"
34 | std: 0.01
35 | }
36 | bias_filler {
37 | type: "constant"
38 | value: 0
39 | }
40 | }
41 | }
42 | layer {
43 | name: "relu1_1"
44 | type: "ReLU"
45 | bottom: "conv1_1"
46 | top: "conv1_1"
47 | }
48 | layer {
49 | name: "conv1_2"
50 | type: "Convolution"
51 | bottom: "conv1_1"
52 | top: "conv1_2"
53 | param {
54 | lr_mult: 1
55 | decay_mult: 1
56 | }
57 | param {
58 | lr_mult: 2
59 | decay_mult: 0
60 | }
61 | convolution_param {
62 | num_output: 64
63 | pad: 1
64 | kernel_size: 3
65 | stride: 1
66 | weight_filler {
67 | type: "gaussian"
68 | std: 0.01
69 | }
70 | bias_filler {
71 | type: "constant"
72 | value: 0
73 | }
74 | }
75 | }
76 | layer {
77 | name: "relu1_2"
78 | type: "ReLU"
79 | bottom: "conv1_2"
80 | top: "conv1_2"
81 | }
82 | layer {
83 | name: "pool1"
84 | type: "Pooling"
85 | bottom: "conv1_2"
86 | top: "pool1"
87 | pooling_param {
88 | pool: MAX
89 | kernel_size: 2
90 | stride: 2
91 | }
92 | }
93 | layer {
94 | name: "conv2_1"
95 | type: "Convolution"
96 | bottom: "pool1"
97 | top: "conv2_1"
98 | param {
99 | lr_mult: 1
100 | decay_mult: 1
101 | }
102 | param {
103 | lr_mult: 2
104 | decay_mult: 0
105 | }
106 | convolution_param {
107 | num_output: 128
108 | pad: 1
109 | kernel_size: 3
110 | stride: 1
111 | weight_filler {
112 | type: "gaussian"
113 | std: 0.01
114 | }
115 | bias_filler {
116 | type: "constant"
117 | value: 0
118 | }
119 | }
120 | }
121 | layer {
122 | name: "relu2_1"
123 | type: "ReLU"
124 | bottom: "conv2_1"
125 | top: "conv2_1"
126 | }
127 | layer {
128 | name: "conv2_2"
129 | type: "Convolution"
130 | bottom: "conv2_1"
131 | top: "conv2_2"
132 | param {
133 | lr_mult: 1
134 | decay_mult: 1
135 | }
136 | param {
137 | lr_mult: 2
138 | decay_mult: 0
139 | }
140 | convolution_param {
141 | num_output: 128
142 | pad: 1
143 | kernel_size: 3
144 | stride: 1
145 | weight_filler {
146 | type: "gaussian"
147 | std: 0.01
148 | }
149 | bias_filler {
150 | type: "constant"
151 | value: 0
152 | }
153 | }
154 | }
155 | layer {
156 | name: "relu2_2"
157 | type: "ReLU"
158 | bottom: "conv2_2"
159 | top: "conv2_2"
160 | }
161 | layer {
162 | name: "pool2"
163 | type: "Pooling"
164 | bottom: "conv2_2"
165 | top: "pool2"
166 | pooling_param {
167 | pool: MAX
168 | kernel_size: 2
169 | stride: 2
170 | }
171 | }
172 | layer {
173 | name: "conv3_1"
174 | type: "Convolution"
175 | bottom: "pool2"
176 | top: "conv3_1"
177 | param {
178 | lr_mult: 1
179 | decay_mult: 1
180 | }
181 | param {
182 | lr_mult: 2
183 | decay_mult: 0
184 | }
185 | convolution_param {
186 | num_output: 256
187 | pad: 1
188 | kernel_size: 3
189 | stride: 1
190 | weight_filler {
191 | type: "gaussian"
192 | std: 0.01
193 | }
194 | bias_filler {
195 | type: "constant"
196 | value: 0
197 | }
198 | }
199 | }
200 | layer {
201 | name: "relu3_1"
202 | type: "ReLU"
203 | bottom: "conv3_1"
204 | top: "conv3_1"
205 | }
206 | layer {
207 | name: "conv3_2"
208 | type: "Convolution"
209 | bottom: "conv3_1"
210 | top: "conv3_2"
211 | param {
212 | lr_mult: 1
213 | decay_mult: 1
214 | }
215 | param {
216 | lr_mult: 2
217 | decay_mult: 0
218 | }
219 | convolution_param {
220 | num_output: 256
221 | pad: 1
222 | kernel_size: 3
223 | stride: 1
224 | weight_filler {
225 | type: "gaussian"
226 | std: 0.01
227 | }
228 | bias_filler {
229 | type: "constant"
230 | value: 0
231 | }
232 | }
233 | }
234 | layer {
235 | name: "relu3_2"
236 | type: "ReLU"
237 | bottom: "conv3_2"
238 | top: "conv3_2"
239 | }
240 | layer {
241 | name: "conv3_3"
242 | type: "Convolution"
243 | bottom: "conv3_2"
244 | top: "conv3_3"
245 | param {
246 | lr_mult: 1
247 | decay_mult: 1
248 | }
249 | param {
250 | lr_mult: 2
251 | decay_mult: 0
252 | }
253 | convolution_param {
254 | num_output: 256
255 | pad: 1
256 | kernel_size: 3
257 | stride: 1
258 | weight_filler {
259 | type: "gaussian"
260 | std: 0.01
261 | }
262 | bias_filler {
263 | type: "constant"
264 | value: 0
265 | }
266 | }
267 | }
268 | layer {
269 | name: "relu3_3"
270 | type: "ReLU"
271 | bottom: "conv3_3"
272 | top: "conv3_3"
273 | }
274 | layer {
275 | name: "pool3"
276 | type: "Pooling"
277 | bottom: "conv3_3"
278 | top: "pool3"
279 | pooling_param {
280 | pool: MAX
281 | kernel_size: 2
282 | stride: 2
283 | }
284 | }
285 | layer {
286 | name: "conv4_1"
287 | type: "Convolution"
288 | bottom: "pool3"
289 | top: "conv4_1"
290 | param {
291 | lr_mult: 1
292 | decay_mult: 1
293 | }
294 | param {
295 | lr_mult: 2
296 | decay_mult: 0
297 | }
298 | convolution_param {
299 | num_output: 512
300 | pad: 1
301 | kernel_size: 3
302 | stride: 1
303 | weight_filler {
304 | type: "gaussian"
305 | std: 0.01
306 | }
307 | bias_filler {
308 | type: "constant"
309 | value: 0
310 | }
311 | }
312 | }
313 | layer {
314 | name: "relu4_1"
315 | type: "ReLU"
316 | bottom: "conv4_1"
317 | top: "conv4_1"
318 | }
319 | layer {
320 | name: "conv4_2"
321 | type: "Convolution"
322 | bottom: "conv4_1"
323 | top: "conv4_2"
324 | param {
325 | lr_mult: 1
326 | decay_mult: 1
327 | }
328 | param {
329 | lr_mult: 2
330 | decay_mult: 0
331 | }
332 | convolution_param {
333 | num_output: 512
334 | pad: 1
335 | kernel_size: 3
336 | stride: 1
337 | weight_filler {
338 | type: "gaussian"
339 | std: 0.01
340 | }
341 | bias_filler {
342 | type: "constant"
343 | value: 0
344 | }
345 | }
346 | }
347 | layer {
348 | name: "relu4_2"
349 | type: "ReLU"
350 | bottom: "conv4_2"
351 | top: "conv4_2"
352 | }
353 | layer {
354 | name: "conv4_3"
355 | type: "Convolution"
356 | bottom: "conv4_2"
357 | top: "conv4_3"
358 | param {
359 | lr_mult: 1
360 | decay_mult: 1
361 | }
362 | param {
363 | lr_mult: 2
364 | decay_mult: 0
365 | }
366 | convolution_param {
367 | num_output: 512
368 | pad: 1
369 | kernel_size: 3
370 | stride: 1
371 | weight_filler {
372 | type: "gaussian"
373 | std: 0.01
374 | }
375 | bias_filler {
376 | type: "constant"
377 | value: 0
378 | }
379 | }
380 | }
381 | layer {
382 | name: "relu4_3"
383 | type: "ReLU"
384 | bottom: "conv4_3"
385 | top: "conv4_3"
386 | }
387 | layer {
388 | name: "pool4"
389 | type: "Pooling"
390 | bottom: "conv4_3"
391 | top: "pool4"
392 | pooling_param {
393 | pool: MAX
394 | kernel_size: 2
395 | stride: 2
396 | }
397 | }
398 | layer {
399 | name: "conv5_1"
400 | type: "Convolution"
401 | bottom: "pool4"
402 | top: "conv5_1"
403 | param {
404 | lr_mult: 1
405 | decay_mult: 1
406 | }
407 | param {
408 | lr_mult: 2
409 | decay_mult: 0
410 | }
411 | convolution_param {
412 | num_output: 512
413 | pad: 1
414 | kernel_size: 3
415 | stride: 1
416 | weight_filler {
417 | type: "gaussian"
418 | std: 0.01
419 | }
420 | bias_filler {
421 | type: "constant"
422 | value: 0
423 | }
424 | }
425 | }
426 | layer {
427 | name: "relu5_1"
428 | type: "ReLU"
429 | bottom: "conv5_1"
430 | top: "conv5_1"
431 | }
432 | layer {
433 | name: "conv5_2"
434 | type: "Convolution"
435 | bottom: "conv5_1"
436 | top: "conv5_2"
437 | param {
438 | lr_mult: 1
439 | decay_mult: 1
440 | }
441 | param {
442 | lr_mult: 2
443 | decay_mult: 0
444 | }
445 | convolution_param {
446 | num_output: 512
447 | pad: 1
448 | kernel_size: 3
449 | stride: 1
450 | weight_filler {
451 | type: "gaussian"
452 | std: 0.01
453 | }
454 | bias_filler {
455 | type: "constant"
456 | value: 0
457 | }
458 | }
459 | }
460 | layer {
461 | name: "relu5_2"
462 | type: "ReLU"
463 | bottom: "conv5_2"
464 | top: "conv5_2"
465 | }
466 | layer {
467 | name: "conv5_3"
468 | type: "Convolution"
469 | bottom: "conv5_2"
470 | top: "conv5_3"
471 | param {
472 | lr_mult: 1
473 | decay_mult: 1
474 | }
475 | param {
476 | lr_mult: 2
477 | decay_mult: 0
478 | }
479 | convolution_param {
480 | num_output: 512
481 | pad: 1
482 | kernel_size: 3
483 | stride: 1
484 | weight_filler {
485 | type: "gaussian"
486 | std: 0.01
487 | }
488 | bias_filler {
489 | type: "constant"
490 | value: 0
491 | }
492 | }
493 | }
494 | layer {
495 | name: "relu5_3"
496 | type: "ReLU"
497 | bottom: "conv5_3"
498 | top: "conv5_3"
499 | }
500 | layer {
501 | name: "pool5"
502 | type: "Pooling"
503 | bottom: "conv5_3"
504 | top: "pool5"
505 | pooling_param {
506 | pool: MAX
507 | kernel_size: 2
508 | stride: 2
509 | }
510 | }
511 | layer {
512 | name: "fc6_cs"
513 | type: "Convolution"
514 | bottom: "pool5"
515 | top: "fc6_cs"
516 | param {
517 | lr_mult: 1
518 | decay_mult: 1
519 | }
520 | param {
521 | lr_mult: 2
522 | decay_mult: 0
523 | }
524 | convolution_param {
525 | num_output: 4096
526 | pad: 0
527 | kernel_size: 7
528 | stride: 1
529 | weight_filler {
530 | type: "gaussian"
531 | std: 0.01
532 | }
533 | bias_filler {
534 | type: "constant"
535 | value: 0
536 | }
537 | }
538 | }
539 | layer {
540 | name: "relu6_cs"
541 | type: "ReLU"
542 | bottom: "fc6_cs"
543 | top: "fc6_cs"
544 | }
545 | layer {
546 | name: "fc7_cs"
547 | type: "Convolution"
548 | bottom: "fc6_cs"
549 | top: "fc7_cs"
550 | param {
551 | lr_mult: 1
552 | decay_mult: 1
553 | }
554 | param {
555 | lr_mult: 2
556 | decay_mult: 0
557 | }
558 | convolution_param {
559 | num_output: 4096
560 | pad: 0
561 | kernel_size: 1
562 | stride: 1
563 | weight_filler {
564 | type: "gaussian"
565 | std: 0.01
566 | }
567 | bias_filler {
568 | type: "constant"
569 | value: 0
570 | }
571 | }
572 | }
573 | layer {
574 | name: "relu7_cs"
575 | type: "ReLU"
576 | bottom: "fc7_cs"
577 | top: "fc7_cs"
578 | }
579 | layer {
580 | name: "score_fr"
581 | type: "Convolution"
582 | bottom: "fc7_cs"
583 | top: "score_fr"
584 | param {
585 | lr_mult: 1
586 | decay_mult: 1
587 | }
588 | param {
589 | lr_mult: 2
590 | decay_mult: 0
591 | }
592 | convolution_param {
593 | num_output: 20
594 | pad: 0
595 | kernel_size: 1
596 | weight_filler {
597 | type: "xavier"
598 | }
599 | bias_filler {
600 | type: "constant"
601 | }
602 | }
603 | }
604 | layer {
605 | name: "upscore2"
606 | type: "Deconvolution"
607 | bottom: "score_fr"
608 | top: "upscore2"
609 | param {
610 | lr_mult: 1
611 | }
612 | convolution_param {
613 | num_output: 20
614 | bias_term: false
615 | kernel_size: 4
616 | stride: 2
617 | weight_filler {
618 | type: "xavier"
619 | }
620 | bias_filler {
621 | type: "constant"
622 | }
623 | }
624 | }
625 | layer {
626 | name: "score_pool4"
627 | type: "Convolution"
628 | bottom: "pool4"
629 | top: "score_pool4"
630 | param {
631 | lr_mult: 1
632 | decay_mult: 1
633 | }
634 | param {
635 | lr_mult: 2
636 | decay_mult: 0
637 | }
638 | convolution_param {
639 | num_output: 20
640 | pad: 0
641 | kernel_size: 1
642 | weight_filler {
643 | type: "xavier"
644 | }
645 | bias_filler {
646 | type: "constant"
647 | }
648 | }
649 | }
650 | layer {
651 | name: "score_pool4c"
652 | type: "Crop"
653 | bottom: "score_pool4"
654 | bottom: "upscore2"
655 | top: "score_pool4c"
656 | crop_param {
657 | axis: 2
658 | offset: 5
659 | }
660 | }
661 | layer {
662 | name: "fuse_pool4"
663 | type: "Eltwise"
664 | bottom: "upscore2"
665 | bottom: "score_pool4c"
666 | top: "fuse_pool4"
667 | eltwise_param {
668 | operation: SUM
669 | }
670 | }
671 | layer {
672 | name: "upscore_pool4"
673 | type: "Deconvolution"
674 | bottom: "fuse_pool4"
675 | top: "upscore_pool4"
676 | param {
677 | lr_mult: 1
678 | }
679 | convolution_param {
680 | num_output: 20
681 | bias_term: false
682 | kernel_size: 4
683 | stride: 2
684 | weight_filler {
685 | type: "xavier"
686 | }
687 | bias_filler {
688 | type: "constant"
689 | }
690 | }
691 | }
692 | layer {
693 | name: "score_pool3"
694 | type: "Convolution"
695 | bottom: "pool3"
696 | top: "score_pool3"
697 | param {
698 | lr_mult: 1
699 | decay_mult: 1
700 | }
701 | param {
702 | lr_mult: 2
703 | decay_mult: 0
704 | }
705 | convolution_param {
706 | num_output: 20
707 | pad: 0
708 | kernel_size: 1
709 | weight_filler {
710 | type: "xavier"
711 | }
712 | bias_filler {
713 | type: "constant"
714 | }
715 | }
716 | }
717 | layer {
718 | name: "score_pool3c"
719 | type: "Crop"
720 | bottom: "score_pool3"
721 | bottom: "upscore_pool4"
722 | top: "score_pool3c"
723 | crop_param {
724 | axis: 2
725 | offset: 9
726 | }
727 | }
728 | layer {
729 | name: "fuse_pool3"
730 | type: "Eltwise"
731 | bottom: "upscore_pool4"
732 | bottom: "score_pool3c"
733 | top: "fuse_pool3"
734 | eltwise_param {
735 | operation: SUM
736 | }
737 | }
738 | layer {
739 | name: "upscore8"
740 | type: "Deconvolution"
741 | bottom: "fuse_pool3"
742 | top: "upscore8"
743 | param {
744 | lr_mult: 1
745 | }
746 | convolution_param {
747 | num_output: 20
748 | bias_term: false
749 | kernel_size: 16
750 | stride: 8
751 | weight_filler {
752 | type: "xavier"
753 | }
754 | bias_filler {
755 | type: "constant"
756 | }
757 | }
758 | }
759 | layer {
760 | name: "score"
761 | type: "Crop"
762 | bottom: "upscore8"
763 | bottom: "data"
764 | top: "score"
765 | crop_param {
766 | axis: 2
767 | offset: 31
768 | }
769 | }
770 |
--------------------------------------------------------------------------------
/scripts/eval_cityscapes/cityscapes.py:
--------------------------------------------------------------------------------
1 | # The following code is modified from https://github.com/shelhamer/clockwork-fcn
2 | import sys
3 | import os
4 | import glob
5 | import numpy as np
6 | from PIL import Image
7 |
8 |
9 | class cityscapes:
10 | def __init__(self, data_path):
11 | # data_path something like /data2/cityscapes
12 | self.dir = data_path
13 | self.classes = ['road', 'sidewalk', 'building', 'wall', 'fence',
14 | 'pole', 'traffic light', 'traffic sign', 'vegetation', 'terrain',
15 | 'sky', 'person', 'rider', 'car', 'truck',
16 | 'bus', 'train', 'motorcycle', 'bicycle']
17 | self.mean = np.array((72.78044, 83.21195, 73.45286), dtype=np.float32)
18 | # import cityscapes label helper and set up label mappings
19 | sys.path.insert(0, '{}/scripts/helpers/'.format(self.dir))
20 | labels = __import__('labels')
21 | self.id2trainId = {label.id: label.trainId for label in labels.labels} # dictionary mapping from raw IDs to train IDs
22 | self.trainId2color = {label.trainId: label.color for label in labels.labels} # dictionary mapping train IDs to colors as 3-tuples
23 |
24 | def get_dset(self, split):
25 | '''
26 | List images as (city, id) for the specified split
27 |
28 | TODO(shelhamer) generate splits from cityscapes itself, instead of
29 | relying on these separately made text files.
30 | '''
31 | if split == 'train':
32 | dataset = open('{}/ImageSets/segFine/train.txt'.format(self.dir)).read().splitlines()
33 | else:
34 | dataset = open('{}/ImageSets/segFine/val.txt'.format(self.dir)).read().splitlines()
35 | return [(item.split('/')[0], item.split('/')[1]) for item in dataset]
36 |
37 | def load_image(self, split, city, idx):
38 | im = Image.open('{}/leftImg8bit_sequence/{}/{}/{}_leftImg8bit.png'.format(self.dir, split, city, idx))
39 | return im
40 |
41 | def assign_trainIds(self, label):
42 | """
43 | Map the given label IDs to the train IDs appropriate for training
44 | Use the label mapping provided in labels.py from the cityscapes scripts
45 | """
46 | label = np.array(label, dtype=np.float32)
47 | if sys.version_info[0] < 3:
48 | for k, v in self.id2trainId.iteritems():
49 | label[label == k] = v
50 | else:
51 | for k, v in self.id2trainId.items():
52 | label[label == k] = v
53 | return label
54 |
55 | def load_label(self, split, city, idx):
56 | """
57 | Load label image as 1 x height x width integer array of label indices.
58 | The leading singleton dimension is required by the loss.
59 | """
60 | label = Image.open('{}/gtFine/{}/{}/{}_gtFine_labelIds.png'.format(self.dir, split, city, idx))
61 | label = self.assign_trainIds(label) # get proper labels for eval
62 | label = np.array(label, dtype=np.uint8)
63 | label = label[np.newaxis, ...]
64 | return label
65 |
66 | def preprocess(self, im):
67 | """
68 | Preprocess loaded image (by load_image) for Caffe:
69 | - cast to float
70 | - switch channels RGB -> BGR
71 | - subtract mean
72 | - transpose to channel x height x width order
73 | """
74 | in_ = np.array(im, dtype=np.float32)
75 | in_ = in_[:, :, ::-1]
76 | in_ -= self.mean
77 | in_ = in_.transpose((2, 0, 1))
78 | return in_
79 |
80 | def palette(self, label):
81 | '''
82 | Map trainIds to colors as specified in labels.py
83 | '''
84 | if label.ndim == 3:
85 | label = label[0]
86 | color = np.empty((label.shape[0], label.shape[1], 3))
87 | if sys.version_info[0] < 3:
88 | for k, v in self.trainId2color.iteritems():
89 | color[label == k, :] = v
90 | else:
91 | for k, v in self.trainId2color.items():
92 | color[label == k, :] = v
93 | return color
94 |
95 | def make_boundaries(label, thickness=None):
96 | """
97 | Input is an image label, output is a numpy array mask encoding the boundaries of the objects
98 | Extract pixels at the true boundary by dilation - erosion of label.
99 | Don't just pick the void label as it is not exclusive to the boundaries.
100 | """
101 | assert(thickness is not None)
102 | import skimage.morphology as skm
103 | void = 255
104 | mask = np.logical_and(label > 0, label != void)[0]
105 | selem = skm.disk(thickness)
106 | boundaries = np.logical_xor(skm.dilation(mask, selem),
107 | skm.erosion(mask, selem))
108 | return boundaries
109 |
110 | def list_label_frames(self, split):
111 | """
112 | Select labeled frames from a split for evaluation
113 | collected as (city, shot, idx) tuples
114 | """
115 | def file2idx(f):
116 | """Helper to convert file path into frame ID"""
117 | city, shot, frame = (os.path.basename(f).split('_')[:3])
118 | return "_".join([city, shot, frame])
119 | frames = []
120 | cities = [os.path.basename(f) for f in glob.glob('{}/gtFine/{}/*'.format(self.dir, split))]
121 | for c in cities:
122 | files = sorted(glob.glob('{}/gtFine/{}/{}/*labelIds.png'.format(self.dir, split, c)))
123 | frames.extend([file2idx(f) for f in files])
124 | return frames
125 |
126 | def collect_frame_sequence(self, split, idx, length):
127 | """
128 | Collect sequence of frames preceding (and including) a labeled frame
129 | as a list of Images.
130 |
131 | Note: 19 preceding frames are provided for each labeled frame.
132 | """
133 | SEQ_LEN = length
134 | city, shot, frame = idx.split('_')
135 | frame = int(frame)
136 | frame_seq = []
137 | for i in range(frame - SEQ_LEN, frame + 1):
138 | frame_path = '{0}/leftImg8bit_sequence/val/{1}/{1}_{2}_{3:0>6d}_leftImg8bit.png'.format(
139 | self.dir, city, shot, i)
140 | frame_seq.append(Image.open(frame_path))
141 | return frame_seq
142 |
--------------------------------------------------------------------------------
/scripts/eval_cityscapes/download_fcn8s.sh:
--------------------------------------------------------------------------------
1 | URL=http://people.eecs.berkeley.edu/~tinghuiz/projects/pix2pix/fcn-8s-cityscapes/fcn-8s-cityscapes.caffemodel
2 | OUTPUT_FILE=./scripts/eval_cityscapes/caffemodel/fcn-8s-cityscapes.caffemodel
3 | wget -N $URL -O $OUTPUT_FILE
4 |
--------------------------------------------------------------------------------
/scripts/eval_cityscapes/evaluate.py:
--------------------------------------------------------------------------------
1 | import os
2 | import caffe
3 | import argparse
4 | import numpy as np
5 | import scipy.misc
6 | from PIL import Image
7 | from util import segrun, fast_hist, get_scores
8 | from cityscapes import cityscapes
9 |
10 | parser = argparse.ArgumentParser()
11 | parser.add_argument("--cityscapes_dir", type=str, required=True, help="Path to the original cityscapes dataset")
12 | parser.add_argument("--result_dir", type=str, required=True, help="Path to the generated images to be evaluated")
13 | parser.add_argument("--output_dir", type=str, required=True, help="Where to save the evaluation results")
14 | parser.add_argument("--caffemodel_dir", type=str, default='./scripts/eval_cityscapes/caffemodel/', help="Where the FCN-8s caffemodel stored")
15 | parser.add_argument("--gpu_id", type=int, default=0, help="Which gpu id to use")
16 | parser.add_argument("--split", type=str, default='val', help="Data split to be evaluated")
17 | parser.add_argument("--save_output_images", type=int, default=0, help="Whether to save the FCN output images")
18 | args = parser.parse_args()
19 |
20 |
21 | def main():
22 | if not os.path.isdir(args.output_dir):
23 | os.makedirs(args.output_dir)
24 | if args.save_output_images > 0:
25 | output_image_dir = args.output_dir + 'image_outputs/'
26 | if not os.path.isdir(output_image_dir):
27 | os.makedirs(output_image_dir)
28 | CS = cityscapes(args.cityscapes_dir)
29 | n_cl = len(CS.classes)
30 | label_frames = CS.list_label_frames(args.split)
31 | caffe.set_device(args.gpu_id)
32 | caffe.set_mode_gpu()
33 | net = caffe.Net(args.caffemodel_dir + '/deploy.prototxt',
34 | args.caffemodel_dir + 'fcn-8s-cityscapes.caffemodel',
35 | caffe.TEST)
36 |
37 | hist_perframe = np.zeros((n_cl, n_cl))
38 | for i, idx in enumerate(label_frames):
39 | if i % 10 == 0:
40 | print('Evaluating: %d/%d' % (i, len(label_frames)))
41 | city = idx.split('_')[0]
42 | # idx is city_shot_frame
43 | label = CS.load_label(args.split, city, idx)
44 | im_file = args.result_dir + '/' + idx + '_leftImg8bit.png'
45 | im = np.array(Image.open(im_file))
46 | # im = scipy.misc.imresize(im, (256, 256))
47 | im = scipy.misc.imresize(im, (label.shape[1], label.shape[2]))
48 | out = segrun(net, CS.preprocess(im))
49 | hist_perframe += fast_hist(label.flatten(), out.flatten(), n_cl)
50 | if args.save_output_images > 0:
51 | label_im = CS.palette(label)
52 | pred_im = CS.palette(out)
53 | scipy.misc.imsave(output_image_dir + '/' + str(i) + '_pred.jpg', pred_im)
54 | scipy.misc.imsave(output_image_dir + '/' + str(i) + '_gt.jpg', label_im)
55 | scipy.misc.imsave(output_image_dir + '/' + str(i) + '_input.jpg', im)
56 |
57 | mean_pixel_acc, mean_class_acc, mean_class_iou, per_class_acc, per_class_iou = get_scores(hist_perframe)
58 | with open(args.output_dir + '/evaluation_results.txt', 'w') as f:
59 | f.write('Mean pixel accuracy: %f\n' % mean_pixel_acc)
60 | f.write('Mean class accuracy: %f\n' % mean_class_acc)
61 | f.write('Mean class IoU: %f\n' % mean_class_iou)
62 | f.write('************ Per class numbers below ************\n')
63 | for i, cl in enumerate(CS.classes):
64 | while len(cl) < 15:
65 | cl = cl + ' '
66 | f.write('%s: acc = %f, iou = %f\n' % (cl, per_class_acc[i], per_class_iou[i]))
67 |
68 |
69 | main()
70 |
--------------------------------------------------------------------------------
/scripts/eval_cityscapes/util.py:
--------------------------------------------------------------------------------
1 | # The following code is modified from https://github.com/shelhamer/clockwork-fcn
2 | import numpy as np
3 |
4 |
5 | def get_out_scoremap(net):
6 | return net.blobs['score'].data[0].argmax(axis=0).astype(np.uint8)
7 |
8 |
9 | def feed_net(net, in_):
10 | """
11 | Load prepared input into net.
12 | """
13 | net.blobs['data'].reshape(1, *in_.shape)
14 | net.blobs['data'].data[...] = in_
15 |
16 |
17 | def segrun(net, in_):
18 | feed_net(net, in_)
19 | net.forward()
20 | return get_out_scoremap(net)
21 |
22 |
23 | def fast_hist(a, b, n):
24 | k = np.where((a >= 0) & (a < n))[0]
25 | bc = np.bincount(n * a[k].astype(int) + b[k], minlength=n**2)
26 | if len(bc) != n**2:
27 | # ignore this example if dimension mismatch
28 | return 0
29 | return bc.reshape(n, n)
30 |
31 |
32 | def get_scores(hist):
33 | # Mean pixel accuracy
34 | acc = np.diag(hist).sum() / (hist.sum() + 1e-12)
35 |
36 | # Per class accuracy
37 | cl_acc = np.diag(hist) / (hist.sum(1) + 1e-12)
38 |
39 | # Per class IoU
40 | iu = np.diag(hist) / (hist.sum(1) + hist.sum(0) - np.diag(hist) + 1e-12)
41 |
42 | return acc, np.nanmean(cl_acc), np.nanmean(iu), cl_acc, iu
43 |
--------------------------------------------------------------------------------
/scripts/install_deps.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | pip install visdom
3 | pip install dominate
4 |
--------------------------------------------------------------------------------
/scripts/test_before_push.py:
--------------------------------------------------------------------------------
1 | # Simple script to make sure basic usage
2 | # such as training, testing, saving and loading
3 | # runs without errors.
4 | import os
5 |
6 |
7 | def run(command):
8 | print(command)
9 | exit_status = os.system(command)
10 | if exit_status > 0:
11 | exit(1)
12 |
13 |
14 | if __name__ == '__main__':
15 | # download mini datasets
16 | if not os.path.exists('./datasets/mini'):
17 | run('bash ./datasets/download_cyclegan_dataset.sh mini')
18 |
19 | if not os.path.exists('./datasets/mini_pix2pix'):
20 | run('bash ./datasets/download_cyclegan_dataset.sh mini_pix2pix')
21 |
22 | # pretrained cyclegan model
23 | if not os.path.exists('./checkpoints/horse2zebra_pretrained/latest_net_G.pth'):
24 | run('bash ./scripts/download_cyclegan_model.sh horse2zebra')
25 | run('python test.py --model test --dataroot ./datasets/mini --name horse2zebra_pretrained --no_dropout --num_test 1 --no_dropout')
26 |
27 | # pretrained pix2pix model
28 | if not os.path.exists('./checkpoints/facades_label2photo_pretrained/latest_net_G.pth'):
29 | run('bash ./scripts/download_pix2pix_model.sh facades_label2photo')
30 | if not os.path.exists('./datasets/facades'):
31 | run('bash ./datasets/download_pix2pix_dataset.sh facades')
32 | run('python test.py --dataroot ./datasets/facades/ --direction BtoA --model pix2pix --name facades_label2photo_pretrained --num_test 1')
33 |
34 | # cyclegan train/test
35 | run('python train.py --model cycle_gan --name temp_cyclegan --dataroot ./datasets/mini --niter 1 --niter_decay 0 --save_latest_freq 10 --print_freq 1 --display_id -1')
36 | run('python test.py --model test --name temp_cyclegan --dataroot ./datasets/mini --num_test 1 --model_suffix "_A" --no_dropout')
37 |
38 | # pix2pix train/test
39 | run('python train.py --model pix2pix --name temp_pix2pix --dataroot ./datasets/mini_pix2pix --niter 1 --niter_decay 5 --save_latest_freq 10 --display_id -1')
40 | run('python test.py --model pix2pix --name temp_pix2pix --dataroot ./datasets/mini_pix2pix --num_test 1')
41 |
42 | # template train/test
43 | run('python train.py --model template --name temp2 --dataroot ./datasets/mini_pix2pix --niter 1 --niter_decay 0 --save_latest_freq 10 --display_id -1')
44 | run('python test.py --model template --name temp2 --dataroot ./datasets/mini_pix2pix --num_test 1')
45 |
46 | # colorization train/test (optional)
47 | if not os.path.exists('./datasets/mini_colorization'):
48 | run('bash ./datasets/download_cyclegan_dataset.sh mini_colorization')
49 |
50 | run('python train.py --model colorization --name temp_color --dataroot ./datasets/mini_colorization --niter 1 --niter_decay 0 --save_latest_freq 5 --display_id -1')
51 | run('python test.py --model colorization --name temp_color --dataroot ./datasets/mini_colorization --num_test 1')
52 |
--------------------------------------------------------------------------------
/scripts/test_colorization.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | python test.py --dataroot ./datasets/colorization --name color_pix2pix --model colorization
3 |
--------------------------------------------------------------------------------
/scripts/test_cyclegan.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | python test.py --dataroot ./datasets/horse2zebra/testA --name horse2zebra_pretrained --model cycle_gan --phase test --no_dropout
3 |
--------------------------------------------------------------------------------
/scripts/test_pix2pix.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | python test.py --dataroot ./datasets/facades --name facades_pix2pix --model pix2pix --netG unet_256 --direction BtoA --dataset_mode aligned --norm batch
3 |
--------------------------------------------------------------------------------
/scripts/test_single.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | python test.py --dataroot ./datasets/facades/testB/ --name facades_pix2pix --model test --netG unet_256 --direction BtoA --dataset_mode single --norm batch
3 |
--------------------------------------------------------------------------------
/scripts/train_colorization.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | python train.py --dataroot ./datasets/colorization --name color_pix2pix --model colorization
3 |
--------------------------------------------------------------------------------
/scripts/train_cyclegan.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | python train.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan --pool_size 50 --no_dropout
3 |
--------------------------------------------------------------------------------
/scripts/train_pix2pix.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 | python train.py --dataroot ./datasets/facades --name facades_pix2pix --model pix2pix --netG unet_256 --direction BtoA --lambda_L1 100 --dataset_mode aligned --norm batch --pool_size 0
3 |
--------------------------------------------------------------------------------
/test.py:
--------------------------------------------------------------------------------
1 | """General-purpose test script for image-to-image translation.
2 |
3 | Once you have trained your model with train.py, you can use this script to test the model.
4 | It will load a saved model from --checkpoints_dir and save the results to --results_dir.
5 |
6 | It first creates model and dataset given the option. It will hard-code some parameters.
7 | It then runs inference for --num_test images and save results to an HTML file.
8 |
9 | Example (You need to train models first or download pre-trained models from our website):
10 | Test a CycleGAN model (both sides):
11 | python test.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan
12 |
13 | Test a CycleGAN model (one side only):
14 | python test.py --dataroot datasets/horse2zebra/testA --name horse2zebra_pretrained --model test --no_dropout
15 |
16 | The option '--model test' is used for generating CycleGAN results only for one side.
17 | This option will automatically set '--dataset_mode single', which only loads the images from one set.
18 | On the contrary, using '--model cycle_gan' requires loading and generating results in both directions,
19 | which is sometimes unnecessary. The results will be saved at ./results/.
20 | Use '--results_dir ' to specify the results directory.
21 |
22 | Test a pix2pix model:
23 | python test.py --dataroot ./datasets/facades --name facades_pix2pix --model pix2pix --direction BtoA
24 |
25 | See options/base_options.py and options/test_options.py for more test options.
26 | See training and test tips at: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md
27 | See frequently asked questions at: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/qa.md
28 | """
29 | import os
30 | from options.test_options import TestOptions
31 | from data import create_dataset
32 | from models import create_model
33 | from util.visualizer import save_images
34 | from util import html
35 |
36 |
37 | if __name__ == '__main__':
38 | opt = TestOptions().parse() # get test options
39 | # hard-code some parameters for test
40 | opt.num_threads = 0 # test code only supports num_threads = 1
41 | opt.batch_size = 1 # test code only supports batch_size = 1
42 | opt.serial_batches = True # disable data shuffling; comment this line if results on randomly chosen images are needed.
43 | opt.no_flip = True # no flip; comment this line if results on flipped images are needed.
44 | opt.display_id = -1 # no visdom display; the test code saves the results to a HTML file.
45 | dataset = create_dataset(opt) # create a dataset given opt.dataset_mode and other options
46 | model = create_model(opt) # create a model given opt.model and other options
47 | model.setup(opt) # regular setup: load and print networks; create schedulers
48 | # create a website
49 | web_dir = os.path.join(opt.results_dir, opt.name, '%s_%s_%s' % (opt.phase, opt.epoch, os.path.basename(opt.dataroot))) # define the website directory
50 | webpage = html.HTML(web_dir, 'Experiment = %s, Phase = %s, Epoch = %s' % (opt.name, opt.phase, opt.epoch))
51 | # test with eval mode. This only affects layers like batchnorm and dropout.
52 | # For [pix2pix]: we use batchnorm and dropout in the original pix2pix. You can experiment it with and without eval() mode.
53 | # For [CycleGAN]: It should not affect CycleGAN as CycleGAN uses instancenorm without dropout.
54 | if opt.eval:
55 | model.eval()
56 | for i, data in enumerate(dataset):
57 | if i >= opt.num_test: # only apply our model to opt.num_test images.
58 | break
59 | model.set_input(data) # unpack data from data loader
60 | model.test() # run inference
61 | visuals = model.get_current_visuals() # get image results
62 | img_path = model.get_image_paths() # get image paths
63 | if i % 5 == 0: # save images to an HTML file
64 | print('processing (%04d)-th image... %s' % (i, img_path))
65 | save_images(webpage, visuals, img_path, aspect_ratio=opt.aspect_ratio, width=opt.display_winsize)
66 | webpage.save() # save the HTML
67 |
--------------------------------------------------------------------------------
/train.py:
--------------------------------------------------------------------------------
1 | """General-purpose training script for image-to-image translation.
2 |
3 | This script works for various models (with option '--model': e.g., pix2pix, cyclegan, colorization) and
4 | different datasets (with option '--dataset_mode': e.g., aligned, unaligned, single, colorization).
5 | You need to specify the dataset ('--dataroot'), experiment name ('--name'), and model ('--model').
6 |
7 | It first creates model, dataset, and visualizer given the option.
8 | It then does standard network training. During the training, it also visualize/save the images, print/save the loss plot, and save models.
9 | The script supports continue/resume training. Use '--continue_train' to resume your previous training.
10 |
11 | Example:
12 | Train a CycleGAN model:
13 | python train.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan
14 | Train a pix2pix model:
15 | python train.py --dataroot ./datasets/facades --name facades_pix2pix --model pix2pix --direction BtoA
16 |
17 | See options/base_options.py and options/train_options.py for more training options.
18 | See training and test tips at: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md
19 | See frequently asked questions at: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/qa.md
20 | """
21 | import time
22 | from options.train_options import TrainOptions
23 | from data import create_dataset
24 | from models import create_model
25 | from util.visualizer import Visualizer
26 |
27 | if __name__ == '__main__':
28 | opt = TrainOptions().parse() # get training options
29 | dataset = create_dataset(opt) # create a dataset given opt.dataset_mode and other options
30 | dataset_size = len(dataset) # get the number of images in the dataset.
31 | print('The number of training images = %d' % dataset_size)
32 |
33 | model = create_model(opt) # create a model given opt.model and other options
34 | model.setup(opt) # regular setup: load and print networks; create schedulers
35 | visualizer = Visualizer(opt) # create a visualizer that display/save images and plots
36 | total_iters = 0 # the total number of training iterations
37 |
38 | for epoch in range(opt.epoch_count, opt.niter + opt.niter_decay + 1): # outer loop for different epochs; we save the model by , +
39 | epoch_start_time = time.time() # timer for entire epoch
40 | iter_data_time = time.time() # timer for data loading per iteration
41 | epoch_iter = 0 # the number of training iterations in current epoch, reset to 0 every epoch
42 |
43 | for i, data in enumerate(dataset): # inner loop within one epoch
44 | iter_start_time = time.time() # timer for computation per iteration
45 | if total_iters % opt.print_freq == 0:
46 | t_data = iter_start_time - iter_data_time
47 | visualizer.reset()
48 | total_iters += opt.batch_size
49 | epoch_iter += opt.batch_size
50 | model.set_input(data) # unpack data from dataset and apply preprocessing
51 | model.optimize_parameters() # calculate loss functions, get gradients, update network weights
52 |
53 | if total_iters % opt.display_freq == 0: # display images on visdom and save images to a HTML file
54 | save_result = total_iters % opt.update_html_freq == 0
55 | model.compute_visuals()
56 | visualizer.display_current_results(model.get_current_visuals(), epoch, save_result)
57 |
58 | if total_iters % opt.print_freq == 0: # print training losses and save logging information to the disk
59 | losses = model.get_current_losses()
60 | t_comp = (time.time() - iter_start_time) / opt.batch_size
61 | visualizer.print_current_losses(epoch, epoch_iter, losses, t_comp, t_data)
62 | if opt.display_id > 0:
63 | visualizer.plot_current_losses(epoch, float(epoch_iter) / dataset_size, losses)
64 |
65 | if total_iters % opt.save_latest_freq == 0: # cache our latest model every iterations
66 | print('saving the latest model (epoch %d, total_iters %d)' % (epoch, total_iters))
67 | save_suffix = 'iter_%d' % total_iters if opt.save_by_iter else 'latest'
68 | model.save_networks(save_suffix)
69 |
70 | iter_data_time = time.time()
71 | if epoch % opt.save_epoch_freq == 0: # cache our model every epochs
72 | print('saving the model at the end of epoch %d, iters %d' % (epoch, total_iters))
73 | model.save_networks('latest')
74 | model.save_networks(epoch)
75 |
76 | print('End of epoch %d / %d \t Time Taken: %d sec' % (epoch, opt.niter + opt.niter_decay, time.time() - epoch_start_time))
77 | model.update_learning_rate() # update learning rates at the end of every epoch.
78 |
--------------------------------------------------------------------------------
/util/__init__.py:
--------------------------------------------------------------------------------
1 | """This package includes a miscellaneous collection of useful helper functions."""
2 |
--------------------------------------------------------------------------------
/util/get_data.py:
--------------------------------------------------------------------------------
1 | from __future__ import print_function
2 | import os
3 | import tarfile
4 | import requests
5 | from warnings import warn
6 | from zipfile import ZipFile
7 | from bs4 import BeautifulSoup
8 | from os.path import abspath, isdir, join, basename
9 |
10 |
11 | class GetData(object):
12 | """A Python script for downloading CycleGAN or pix2pix datasets.
13 |
14 | Parameters:
15 | technique (str) -- One of: 'cyclegan' or 'pix2pix'.
16 | verbose (bool) -- If True, print additional information.
17 |
18 | Examples:
19 | >>> from util.get_data import GetData
20 | >>> gd = GetData(technique='cyclegan')
21 | >>> new_data_path = gd.get(save_path='./datasets') # options will be displayed.
22 |
23 | Alternatively, You can use bash scripts: 'scripts/download_pix2pix_model.sh'
24 | and 'scripts/download_cyclegan_model.sh'.
25 | """
26 |
27 | def __init__(self, technique='cyclegan', verbose=True):
28 | url_dict = {
29 | 'pix2pix': 'http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/',
30 | 'cyclegan': 'https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets'
31 | }
32 | self.url = url_dict.get(technique.lower())
33 | self._verbose = verbose
34 |
35 | def _print(self, text):
36 | if self._verbose:
37 | print(text)
38 |
39 | @staticmethod
40 | def _get_options(r):
41 | soup = BeautifulSoup(r.text, 'lxml')
42 | options = [h.text for h in soup.find_all('a', href=True)
43 | if h.text.endswith(('.zip', 'tar.gz'))]
44 | return options
45 |
46 | def _present_options(self):
47 | r = requests.get(self.url)
48 | options = self._get_options(r)
49 | print('Options:\n')
50 | for i, o in enumerate(options):
51 | print("{0}: {1}".format(i, o))
52 | choice = input("\nPlease enter the number of the "
53 | "dataset above you wish to download:")
54 | return options[int(choice)]
55 |
56 | def _download_data(self, dataset_url, save_path):
57 | if not isdir(save_path):
58 | os.makedirs(save_path)
59 |
60 | base = basename(dataset_url)
61 | temp_save_path = join(save_path, base)
62 |
63 | with open(temp_save_path, "wb") as f:
64 | r = requests.get(dataset_url)
65 | f.write(r.content)
66 |
67 | if base.endswith('.tar.gz'):
68 | obj = tarfile.open(temp_save_path)
69 | elif base.endswith('.zip'):
70 | obj = ZipFile(temp_save_path, 'r')
71 | else:
72 | raise ValueError("Unknown File Type: {0}.".format(base))
73 |
74 | self._print("Unpacking Data...")
75 | obj.extractall(save_path)
76 | obj.close()
77 | os.remove(temp_save_path)
78 |
79 | def get(self, save_path, dataset=None):
80 | """
81 |
82 | Download a dataset.
83 |
84 | Parameters:
85 | save_path (str) -- A directory to save the data to.
86 | dataset (str) -- (optional). A specific dataset to download.
87 | Note: this must include the file extension.
88 | If None, options will be presented for you
89 | to choose from.
90 |
91 | Returns:
92 | save_path_full (str) -- the absolute path to the downloaded data.
93 |
94 | """
95 | if dataset is None:
96 | selected_dataset = self._present_options()
97 | else:
98 | selected_dataset = dataset
99 |
100 | save_path_full = join(save_path, selected_dataset.split('.')[0])
101 |
102 | if isdir(save_path_full):
103 | warn("\n'{0}' already exists. Voiding Download.".format(
104 | save_path_full))
105 | else:
106 | self._print('Downloading Data...')
107 | url = "{0}/{1}".format(self.url, selected_dataset)
108 | self._download_data(url, save_path=save_path)
109 |
110 | return abspath(save_path_full)
111 |
--------------------------------------------------------------------------------
/util/html.py:
--------------------------------------------------------------------------------
1 | import dominate
2 | from dominate.tags import meta, h3, table, tr, td, p, a, img, br
3 | import os
4 |
5 |
6 | class HTML:
7 | """This HTML class allows us to save images and write texts into a single HTML file.
8 |
9 | It consists of functions such as (add a text header to the HTML file),
10 | (add a row of images to the HTML file), and (save the HTML to the disk).
11 | It is based on Python library 'dominate', a Python library for creating and manipulating HTML documents using a DOM API.
12 | """
13 |
14 | def __init__(self, web_dir, title, refresh=0):
15 | """Initialize the HTML classes
16 |
17 | Parameters:
18 | web_dir (str) -- a directory that stores the webpage. HTML file will be created at /index.html; images will be saved at 0:
32 | with self.doc.head:
33 | meta(http_equiv="refresh", content=str(refresh))
34 |
35 | def get_image_dir(self):
36 | """Return the directory that stores images"""
37 | return self.img_dir
38 |
39 | def add_header(self, text):
40 | """Insert a header to the HTML file
41 |
42 | Parameters:
43 | text (str) -- the header text
44 | """
45 | with self.doc:
46 | h3(text)
47 |
48 | def add_images(self, ims, txts, links, width=400):
49 | """add images to the HTML file
50 |
51 | Parameters:
52 | ims (str list) -- a list of image paths
53 | txts (str list) -- a list of image names shown on the website
54 | links (str list) -- a list of hyperref links; when you click an image, it will redirect you to a new page
55 | """
56 | self.t = table(border=1, style="table-layout: fixed;") # Insert a table
57 | self.doc.add(self.t)
58 | with self.t:
59 | with tr():
60 | for im, txt, link in zip(ims, txts, links):
61 | with td(style="word-wrap: break-word;", halign="center", valign="top"):
62 | with p():
63 | with a(href=os.path.join('images', link)):
64 | img(style="width:%dpx" % width, src=os.path.join('images', im))
65 | br()
66 | p(txt)
67 |
68 | def save(self):
69 | """save the current content to the HMTL file"""
70 | html_file = '%s/index.html' % self.web_dir
71 | f = open(html_file, 'wt')
72 | f.write(self.doc.render())
73 | f.close()
74 |
75 |
76 | if __name__ == '__main__': # we show an example usage here.
77 | html = HTML('web/', 'test_html')
78 | html.add_header('hello world')
79 |
80 | ims, txts, links = [], [], []
81 | for n in range(4):
82 | ims.append('image_%d.png' % n)
83 | txts.append('text_%d' % n)
84 | links.append('image_%d.png' % n)
85 | html.add_images(ims, txts, links)
86 | html.save()
87 |
--------------------------------------------------------------------------------
/util/image_pool.py:
--------------------------------------------------------------------------------
1 | import random
2 | import torch
3 |
4 |
5 | class ImagePool():
6 | """This class implements an image buffer that stores previously generated images.
7 |
8 | This buffer enables us to update discriminators using a history of generated images
9 | rather than the ones produced by the latest generators.
10 | """
11 |
12 | def __init__(self, pool_size):
13 | """Initialize the ImagePool class
14 |
15 | Parameters:
16 | pool_size (int) -- the size of image buffer, if pool_size=0, no buffer will be created
17 | """
18 | self.pool_size = pool_size
19 | if self.pool_size > 0: # create an empty pool
20 | self.num_imgs = 0
21 | self.images = []
22 |
23 | def query(self, images):
24 | """Return an image from the pool.
25 |
26 | Parameters:
27 | images: the latest generated images from the generator
28 |
29 | Returns images from the buffer.
30 |
31 | By 50/100, the buffer will return input images.
32 | By 50/100, the buffer will return images previously stored in the buffer,
33 | and insert the current images to the buffer.
34 | """
35 | if self.pool_size == 0: # if the buffer size is 0, do nothing
36 | return images
37 | return_images = []
38 | for image in images:
39 | image = torch.unsqueeze(image.data, 0)
40 | if self.num_imgs < self.pool_size: # if the buffer is not full; keep inserting current images to the buffer
41 | self.num_imgs = self.num_imgs + 1
42 | self.images.append(image)
43 | return_images.append(image)
44 | else:
45 | p = random.uniform(0, 1)
46 | if p > 0.5: # by 50% chance, the buffer will return a previously stored image, and insert the current image into the buffer
47 | random_id = random.randint(0, self.pool_size - 1) # randint is inclusive
48 | tmp = self.images[random_id].clone()
49 | self.images[random_id] = image
50 | return_images.append(tmp)
51 | else: # by another 50% chance, the buffer will return the current image
52 | return_images.append(image)
53 | return_images = torch.cat(return_images, 0) # collect all the images and return
54 | return return_images
55 |
--------------------------------------------------------------------------------
/util/util.py:
--------------------------------------------------------------------------------
1 | """This module contains simple helper functions """
2 | from __future__ import print_function
3 | import torch
4 | import numpy as np
5 | from PIL import Image
6 | import os
7 |
8 |
9 | def tensor2im(input_image, imtype=np.uint8):
10 | """"Converts a Tensor array into a numpy image array.
11 |
12 | Parameters:
13 | input_image (tensor) -- the input image tensor array
14 | imtype (type) -- the desired type of the converted numpy array
15 | """
16 | if not isinstance(input_image, np.ndarray):
17 | if isinstance(input_image, torch.Tensor): # get the data from a variable
18 | image_tensor = input_image.data
19 | else:
20 | return input_image
21 | image_numpy = image_tensor[0].cpu().float().numpy() # convert it into a numpy array
22 | if image_numpy.shape[0] == 1: # grayscale to RGB
23 | image_numpy = np.tile(image_numpy, (3, 1, 1))
24 | image_numpy = (np.transpose(image_numpy, (1, 2, 0)) + 1) / 2.0 * 255.0 # post-processing: tranpose and scaling
25 | else: # if it is a numpy array, do nothing
26 | image_numpy = input_image
27 | return image_numpy.astype(imtype)
28 |
29 |
30 | def diagnose_network(net, name='network'):
31 | """Calculate and print the mean of average absolute(gradients)
32 |
33 | Parameters:
34 | net (torch network) -- Torch network
35 | name (str) -- the name of the network
36 | """
37 | mean = 0.0
38 | count = 0
39 | for param in net.parameters():
40 | if param.grad is not None:
41 | mean += torch.mean(torch.abs(param.grad.data))
42 | count += 1
43 | if count > 0:
44 | mean = mean / count
45 | print(name)
46 | print(mean)
47 |
48 |
49 | def save_image(image_numpy, image_path):
50 | """Save a numpy image to the disk
51 |
52 | Parameters:
53 | image_numpy (numpy array) -- input numpy array
54 | image_path (str) -- the path of the image
55 | """
56 | image_pil = Image.fromarray(image_numpy)
57 | image_pil.save(image_path)
58 |
59 |
60 | def print_numpy(x, val=True, shp=False):
61 | """Print the mean, min, max, median, std, and size of a numpy array
62 |
63 | Parameters:
64 | val (bool) -- if print the values of the numpy array
65 | shp (bool) -- if print the shape of the numpy array
66 | """
67 | x = x.astype(np.float64)
68 | if shp:
69 | print('shape,', x.shape)
70 | if val:
71 | x = x.flatten()
72 | print('mean = %3.3f, min = %3.3f, max = %3.3f, median = %3.3f, std=%3.3f' % (
73 | np.mean(x), np.min(x), np.max(x), np.median(x), np.std(x)))
74 |
75 |
76 | def mkdirs(paths):
77 | """create empty directories if they don't exist
78 |
79 | Parameters:
80 | paths (str list) -- a list of directory paths
81 | """
82 | if isinstance(paths, list) and not isinstance(paths, str):
83 | for path in paths:
84 | mkdir(path)
85 | else:
86 | mkdir(paths)
87 |
88 |
89 | def mkdir(path):
90 | """create a single empty directory if it didn't exist
91 |
92 | Parameters:
93 | path (str) -- a single directory path
94 | """
95 | if not os.path.exists(path):
96 | os.makedirs(path)
97 |
--------------------------------------------------------------------------------
/util/visualizer.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | import os
3 | import sys
4 | import ntpath
5 | import time
6 | from . import util, html
7 | from subprocess import Popen, PIPE
8 | from scipy.misc import imresize
9 |
10 | if sys.version_info[0] == 2:
11 | VisdomExceptionBase = Exception
12 | else:
13 | VisdomExceptionBase = ConnectionError
14 |
15 |
16 | def save_images(webpage, visuals, image_path, aspect_ratio=1.0, width=256):
17 | """Save images to the disk.
18 |
19 | Parameters:
20 | webpage (the HTML class) -- the HTML webpage class that stores these imaegs (see html.py for more details)
21 | visuals (OrderedDict) -- an ordered dictionary that stores (name, images (either tensor or numpy) ) pairs
22 | image_path (str) -- the string is used to create image paths
23 | aspect_ratio (float) -- the aspect ratio of saved images
24 | width (int) -- the images will be resized to width x width
25 |
26 | This function will save images stored in 'visuals' to the HTML file specified by 'webpage'.
27 | """
28 | image_dir = webpage.get_image_dir()
29 | short_path = ntpath.basename(image_path[0])
30 | name = os.path.splitext(short_path)[0]
31 |
32 | webpage.add_header(name)
33 | ims, txts, links = [], [], []
34 |
35 | for label, im_data in visuals.items():
36 | im = util.tensor2im(im_data)
37 | image_name = '%s_%s.png' % (name, label)
38 | save_path = os.path.join(image_dir, image_name)
39 | h, w, _ = im.shape
40 | if aspect_ratio > 1.0:
41 | im = imresize(im, (h, int(w * aspect_ratio)), interp='bicubic')
42 | if aspect_ratio < 1.0:
43 | im = imresize(im, (int(h / aspect_ratio), w), interp='bicubic')
44 | util.save_image(im, save_path)
45 |
46 | ims.append(image_name)
47 | txts.append(label)
48 | links.append(image_name)
49 | webpage.add_images(ims, txts, links, width=width)
50 |
51 |
52 | class Visualizer():
53 | """This class includes several functions that can display/save images and print/save logging information.
54 |
55 | It uses a Python library 'visdom' for display, and a Python library 'dominate' (wrapped in 'HTML') for creating HTML files with images.
56 | """
57 |
58 | def __init__(self, opt):
59 | """Initialize the Visualizer class
60 |
61 | Parameters:
62 | opt -- stores all the experiment flags; needs to be a subclass of BaseOptions
63 | Step 1: Cache the training/test options
64 | Step 2: connect to a visdom server
65 | Step 3: create an HTML object for saveing HTML filters
66 | Step 4: create a logging file to store training losses
67 | """
68 | self.opt = opt # cache the option
69 | self.display_id = opt.display_id
70 | self.use_html = opt.isTrain and not opt.no_html
71 | self.win_size = opt.display_winsize
72 | self.name = opt.name
73 | self.port = opt.display_port
74 | self.saved = False
75 | if self.display_id > 0: # connect to a visdom server given and
76 | import visdom
77 | self.ncols = opt.display_ncols
78 | self.vis = visdom.Visdom(server=opt.display_server, port=opt.display_port, env=opt.display_env)
79 | if not self.vis.check_connection():
80 | self.create_visdom_connections()
81 |
82 | if self.use_html: # create an HTML object at /web/; images will be saved under /web/images/
83 | self.web_dir = os.path.join(opt.checkpoints_dir, opt.name, 'web')
84 | self.img_dir = os.path.join(self.web_dir, 'images')
85 | print('create web directory %s...' % self.web_dir)
86 | util.mkdirs([self.web_dir, self.img_dir])
87 | # create a logging file to store training losses
88 | self.log_name = os.path.join(opt.checkpoints_dir, opt.name, 'loss_log.txt')
89 | with open(self.log_name, "a") as log_file:
90 | now = time.strftime("%c")
91 | log_file.write('================ Training Loss (%s) ================\n' % now)
92 |
93 | def reset(self):
94 | """Reset the self.saved status"""
95 | self.saved = False
96 |
97 | def create_visdom_connections(self):
98 | """If the program could not connect to Visdom server, this function will start a new server at port < self.port > """
99 | cmd = sys.executable + ' -m visdom.server -p %d &>/dev/null &' % self.port
100 | print('\n\nCould not connect to Visdom server. \n Trying to start a server....')
101 | print('Command: %s' % cmd)
102 | Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
103 |
104 | def display_current_results(self, visuals, epoch, save_result):
105 | """Display current results on visdom; save current results to an HTML file.
106 |
107 | Parameters:
108 | visuals (OrderedDict) - - dictionary of images to display or save
109 | epoch (int) - - the current epoch
110 | save_result (bool) - - if save the current results to an HTML file
111 | """
112 | if self.display_id > 0: # show images in the browser using visdom
113 | ncols = self.ncols
114 | if ncols > 0: # show all the images in one visdom panel
115 | ncols = min(ncols, len(visuals))
116 | h, w = next(iter(visuals.values())).shape[:2]
117 | table_css = """""" % (w, h) # create a table css
121 | # create a table of images.
122 | title = self.name
123 | label_html = ''
124 | label_html_row = ''
125 | images = []
126 | idx = 0
127 | for label, image in visuals.items():
128 | image_numpy = util.tensor2im(image)
129 | label_html_row += '%s | ' % label
130 | images.append(image_numpy.transpose([2, 0, 1]))
131 | idx += 1
132 | if idx % ncols == 0:
133 | label_html += '%s
' % label_html_row
134 | label_html_row = ''
135 | white_image = np.ones_like(image_numpy.transpose([2, 0, 1])) * 255
136 | while idx % ncols != 0:
137 | images.append(white_image)
138 | label_html_row += ' | '
139 | idx += 1
140 | if label_html_row != '':
141 | label_html += '%s
' % label_html_row
142 | try:
143 | self.vis.images(images, nrow=ncols, win=self.display_id + 1,
144 | padding=2, opts=dict(title=title + ' images'))
145 | label_html = '' % label_html
146 | self.vis.text(table_css + label_html, win=self.display_id + 2,
147 | opts=dict(title=title + ' labels'))
148 | except VisdomExceptionBase:
149 | self.create_visdom_connections()
150 |
151 | else: # show each image in a separate visdom panel;
152 | idx = 1
153 | try:
154 | for label, image in visuals.items():
155 | image_numpy = util.tensor2im(image)
156 | self.vis.image(image_numpy.transpose([2, 0, 1]), opts=dict(title=label),
157 | win=self.display_id + idx)
158 | idx += 1
159 | except VisdomExceptionBase:
160 | self.create_visdom_connections()
161 |
162 | if self.use_html and (save_result or not self.saved): # save images to an HTML file if they haven't been saved.
163 | self.saved = True
164 | # save images to the disk
165 | for label, image in visuals.items():
166 | image_numpy = util.tensor2im(image)
167 | img_path = os.path.join(self.img_dir, 'epoch%.3d_%s.png' % (epoch, label))
168 | util.save_image(image_numpy, img_path)
169 |
170 | # update website
171 | webpage = html.HTML(self.web_dir, 'Experiment name = %s' % self.name, refresh=1)
172 | for n in range(epoch, 0, -1):
173 | webpage.add_header('epoch [%d]' % n)
174 | ims, txts, links = [], [], []
175 |
176 | for label, image_numpy in visuals.items():
177 | image_numpy = util.tensor2im(image)
178 | img_path = 'epoch%.3d_%s.png' % (n, label)
179 | ims.append(img_path)
180 | txts.append(label)
181 | links.append(img_path)
182 | webpage.add_images(ims, txts, links, width=self.win_size)
183 | webpage.save()
184 |
185 | def plot_current_losses(self, epoch, counter_ratio, losses):
186 | """display the current losses on visdom display: dictionary of error labels and values
187 |
188 | Parameters:
189 | epoch (int) -- current epoch
190 | counter_ratio (float) -- progress (percentage) in the current epoch, between 0 to 1
191 | losses (OrderedDict) -- training losses stored in the format of (name, float) pairs
192 | """
193 | if not hasattr(self, 'plot_data'):
194 | self.plot_data = {'X': [], 'Y': [], 'legend': list(losses.keys())}
195 | self.plot_data['X'].append(epoch + counter_ratio)
196 | self.plot_data['Y'].append([losses[k] for k in self.plot_data['legend']])
197 | try:
198 | self.vis.line(
199 | X=np.stack([np.array(self.plot_data['X'])] * len(self.plot_data['legend']), 1),
200 | Y=np.array(self.plot_data['Y']),
201 | opts={
202 | 'title': self.name + ' loss over time',
203 | 'legend': self.plot_data['legend'],
204 | 'xlabel': 'epoch',
205 | 'ylabel': 'loss'},
206 | win=self.display_id)
207 | except VisdomExceptionBase:
208 | self.create_visdom_connections()
209 |
210 | # losses: same format as |losses| of plot_current_losses
211 | def print_current_losses(self, epoch, iters, losses, t_comp, t_data):
212 | """print current losses on console; also save the losses to the disk
213 |
214 | Parameters:
215 | epoch (int) -- current epoch
216 | iters (int) -- current training iteration during this epoch (reset to 0 at the end of every epoch)
217 | losses (OrderedDict) -- training losses stored in the format of (name, float) pairs
218 | t_comp (float) -- computational time per data point (normalized by batch_size)
219 | t_data (float) -- data loading time per data point (normalized by batch_size)
220 | """
221 | message = '(epoch: %d, iters: %d, time: %.3f, data: %.3f) ' % (epoch, iters, t_comp, t_data)
222 | for k, v in losses.items():
223 | message += '%s: %.3f ' % (k, v)
224 |
225 | print(message) # print the message
226 | with open(self.log_name, "a") as log_file:
227 | log_file.write('%s\n' % message) # save the message
228 |
--------------------------------------------------------------------------------
2 |
3 |