├── .gitattributes ├── .gitignore ├── README.md ├── datasets ├── README.md └── lsun_bedroom.py ├── evaluations ├── README.md ├── evaluator.py └── requirements.txt ├── patch_diffusion ├── __init__.py ├── dist_util.py ├── fp16_util.py ├── gaussian_diffusion.py ├── image_datasets.py ├── logger.py ├── losses.py ├── nn.py ├── resample.py ├── respace.py ├── script_util.py ├── th_configs.py ├── train_util.py └── unet.py ├── scripts ├── image_nll.py ├── image_sample.py ├── image_train.py ├── super_res_sample.py └── super_res_train.py └── setup.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | __pycache__/ 3 | classify_image_graph_def.pb 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Patch Diffusion 2 | 3 | **UPDATE (Mar 2024): Unfortunately, the model checkpoints were lost.** They were accidentally deleted when I was clearing my personal google drive storage. Hopefully this doesnt cause too much of a detriment. (At this point the patching technique we propose here has become pretty commonplace among diffusion transformers. For those interested in ImageNet-scale models with this technique the [DiT repo](https://github.com/facebookresearch/DiT) might be a good starting point.) 4 | 5 | Code for the paper "Improving Diffusion Model Efficiency Through Patching". The core idea of the paper is to insert a ViT-style patching operation at the beginning of the U-Net, letting it operate on data with smaller height and width. We show in our paper that the optimal prediction for **x** is quite blurry for most timesteps, and therefore convolutions at the original resolution are usually not necessary. This causes a considerable reduction in compute cost: For example, when using a patch size of 4 (P = 4), generating 256x256 images costs only as much as generating 64x64 images normally (with P = 1). 6 | 7 | # Pretrained Models 8 | 9 | **UPDATE: per the above message, the links in this section are broken** 10 | 11 | We include our models for ImageNet 256x256 and FFHQ 1024x1024, as well as 3 LSUN models with P=2, P=4, and P=8. 12 | 13 | You can download them from Google Drive: 14 | 15 | * ImageNet 256x256, Split #0: [imagenet_weights_0.pt](https://drive.google.com/file/d/1--FE31CNDsCqa_ihGaJIwVSoELdwIAfC/view?usp=sharing) 16 | * ImageNet 256x256, Split #1: [imagenet_weights_1.pt](https://drive.google.com/file/d/1-9kmLKUR1fDVckHY0i_83xzV3QHzuDDC/view?usp=sharing) 17 | * FFHQ 1024x1024: [ffhq_weights.pt](https://drive.google.com/file/d/1-4Len8DL1ZzBv---oNurw5UQQrS0tVuQ/view?usp=sharing) 18 | * LSUN 256x256, P=2: [lsun_weights_p2.pt](https://drive.google.com/file/d/1pjQzsyiNWSlyp2HcxSUBf9Hh0EQXr2ES/view?usp=sharing) 19 | * LSUN 256x256, P=4: [lsun_weights_p4.pt](https://drive.google.com/file/d/1-4-e9M2xzmGd2tCTDwcZd6B0m36AqKvz/view?usp=sharing) 20 | * LSUN 256x256, P=8: [lsun_weights_p8.pt](https://drive.google.com/file/d/1-7wvb5coEdoKEmtixBPg_kZbcNXpnApN/view?usp=sharing) 21 | 22 | # Sampling Instructions 23 | 24 | First, clone our repository and change directory into it. 25 | 26 | Then do 27 | 28 | ``` 29 | pip install -e . 30 | ``` 31 | 32 | Assuming you have downloaded the relevant models in ./models, run the following code to sample from our models. It will save the images (in a PNG file) and npz arrays to ./results 33 | 34 | FFHQ: 35 | ``` 36 | SAMPLE_FLAGS="--batch_size 1 --num_samples 1 --timestep_respacing 250" 37 | MODEL_FLAGS="--channel_mult '1,2,2,4,4,4' --class_cond False --patch_size 4 --image_size 1024 --learn_sigma True --noise_schedule linear0.025 --num_channels 128 --num_head_channels 64 --num_res_blocks 2 --use_fp16 True --use_scale_shift_norm True --use_new_attention_order True" 38 | python scripts/image_sample.py $MODEL_FLAGS --save_dir './results' --model_path ./models/ffhq_weights.pt $SAMPLE_FLAGS 39 | ``` 40 | 41 | ImageNet256: 42 | 43 | For ImageNet, we used two techniques to reduce computation cost and boost sample quality: [classifier-free guidance](https://openreview.net/forum?id=qw8AKxfYbI) and splitting. When increasing the guidance scale, classifier-free guidance improves the visual quality of samples, at the expense of sample diversity. 1.5 is a good default value, although larger values such as 2.25 work as well. Unguided sampling (guidance_scale 1.0) is faster, but generally doesn't lead to as good results. 44 | 45 | Splitting uses 2 (or more) different diffusion models during the generation process, where each model learns to denoise data for part of the diffusion process. In our case, one diffusion model denoises data where the signal-to-noise (SNR) ratio of the data is above 0.25, while the other denoises data below 0.25 SNR. Therefore, we set snr_splits to '0.25', and pass in the checkpoint paths to *two* different models in the ```model_path``` argument. Both models need to be downloaded to run ImageNet sampling. 46 | 47 | ``` 48 | SAMPLE_FLAGS="--batch_size 4 --num_samples 4 --guidance_scale 1.5 --timestep_respacing 250" 49 | MODEL_FLAGS="--snr_splits '0.25' --channel_mult '1,2,2,2' --class_cond True --patch_size 4 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 3 --use_fp16 True --use_scale_shift_norm True --use_new_attention_order True" 50 | !python scripts/image_sample.py $MODEL_FLAGS --save_dir './results' --model_path './models/imagenet_weights_0.pt,./models/imagenet_weights_1.pt' $SAMPLE_FLAGS 51 | ``` 52 | 53 | LSUN, with P=2: 54 | ``` 55 | SAMPLE_FLAGS="--batch_size 4 --num_samples 4 --timestep_respacing 250" 56 | MODEL_FLAGS="--channel_mult '1,2,2,4,4' --class_cond False --patch_size 2 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 128 --num_head_channels 64 --num_res_blocks 2 --use_fp16 True --use_scale_shift_norm True --use_new_attention_order True" 57 | !python scripts/image_sample.py $MODEL_FLAGS --save_dir './results' --model_path ./models/lsun_weights_p2.pt $SAMPLE_FLAGS 58 | ``` 59 | 60 | LSUN, with P=4: 61 | ``` 62 | SAMPLE_FLAGS="--batch_size 4 --num_samples 4 --timestep_respacing 250" 63 | MODEL_FLAGS="--channel_mult '1,1,2,2' --class_cond False --patch_size 4 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 2 --use_fp16 True --use_scale_shift_norm True --use_new_attention_order True" 64 | !python scripts/image_sample.py $MODEL_FLAGS --save_dir './results' --model_path ./models/lsun_weights_p4.pt $SAMPLE_FLAGS 65 | ``` 66 | 67 | LSUN, with P=8: 68 | ``` 69 | SAMPLE_FLAGS="--batch_size 4 --num_samples 4 --timestep_respacing 250" 70 | MODEL_FLAGS="--channel_mult '1,1.5,2' --class_cond False --patch_size 8 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 3 --use_fp16 True --use_scale_shift_norm True --use_new_attention_order True" 71 | !python scripts/image_sample.py $MODEL_FLAGS --save_dir './results' --model_path ./models/lsun_weights_p8.pt $SAMPLE_FLAGS 72 | ``` 73 | 74 | # Training: 75 | Make sure all your training images are in the directory data_dir in png/jpg format (they can be in subdirectories). Then, define MODEL_FLAGS, DIFFUSION_FLAGS, and TRAIN_FLAGS. 76 | 77 | For example: 78 | ``` 79 | MODEL_FLAGS="--image_size 256 --num_channels 128 --num_res_blocks 2 --channel_mult '1,1,2,2' --patch_size 4 --learn_sigma True" 80 | DIFFUSION_FLAGS="--diffusion_steps 1000 --noise_schedule linear" 81 | TRAIN_FLAGS="--lr 1e-4 --batch_size 128" 82 | ``` 83 | 84 | Through the ```--weight_schedule``` argument, we also support different weight schedules of $\lambda_t$, where the objective is $\lambda_t\lVert\textbf{x} - \textbf{x}_\theta(\textbf{z}_t, t) \rVert$. The default is "sqrt_snr", where $\lambda_t$ = $\sqrt{\frac{\alpha}{1-\alpha}}$. However we also include support for the [P2 loss weighting](https://arxiv.org/abs/2204.00227) ("p2") , as well as "snr", "snr+1" and "truncated_snr" schedules from the [progressive distillation](https://arxiv.org/abs/2202.00512) paper. 85 | 86 | To train a model with splitting, add ``` "--snr_splits '{snr_split_values}'" ``` to MODEL_FLAGS and add ```--schedule_sampler uniform_split_{num}``` where num is the split index starting from 0. Model splitting is described in Section 4.1 of the paper. Note: the SNR is defined as alpha/(1-alpha). 87 | 88 | 89 | 90 | Then use: 91 | ``` 92 | python scripts/image_train.py --data_dir path/to/images $MODEL_FLAGS $DIFFUSION_FLAGS $TRAIN_FLAGS 93 | ``` 94 | 95 | 96 | We trained our models for a relatively short duration: our ImageNet models trained for a combined 32 V-100 days (approximately), while our FFHQ model trained for roughly 14 V-100 days. Our LSUN models trained for about 5 V-100 days each. In general, longer training is recommended if you have the budget for it - it improves results. 97 | 98 | # Acknowledgements: 99 | 100 | Our repository builds on top of ADM's [guided diffusion](https://github.com/openai/guided-diffusion) repository - Thanks for sharing! 101 | 102 | # Citation: 103 | 104 | If you find this work helpful to your research, please cite us: 105 | 106 | @misc{https://doi.org/10.48550/arxiv.2207.04316, 107 | doi = {10.48550/ARXIV.2207.04316}, 108 | 109 | url = {https://arxiv.org/abs/2207.04316}, 110 | 111 | author = {Luhman, Troy and Luhman, Eric}, 112 | 113 | keywords = {Machine Learning (cs.LG), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences}, 114 | 115 | title = {Improving Diffusion Model Efficiency Through Patching}, 116 | 117 | publisher = {arXiv}, 118 | 119 | year = {2022}, 120 | 121 | copyright = {Creative Commons Attribution 4.0 International} 122 | } 123 | -------------------------------------------------------------------------------- /datasets/README.md: -------------------------------------------------------------------------------- 1 | # Downloading datasets 2 | 3 | This directory includes instructions and scripts for downloading ImageNet and LSUN bedrooms for use in this codebase. 4 | 5 | ## Class-conditional ImageNet 6 | 7 | For our class-conditional models, we use the official ILSVRC2012 dataset with manual center cropping and downsampling. To obtain this dataset, navigate to [this page on image-net.org](http://www.image-net.org/challenges/LSVRC/2012/downloads) and sign in (or create an account if you do not already have one). Then click on the link reading "Training images (Task 1 & 2)". This is a 138GB tar file containing 1000 sub-tar files, one per class. 8 | 9 | Once the file is downloaded, extract it and look inside. You should see 1000 `.tar` files. You need to extract each of these, which may be impractical to do by hand on your operating system. To automate the process on a Unix-based system, you can `cd` into the directory and run this short shell script: 10 | 11 | ``` 12 | for file in *.tar; do tar xf "$file"; rm "$file"; done 13 | ``` 14 | 15 | This will extract and remove each tar file in turn. 16 | 17 | Once all of the images have been extracted, the resulting directory should be usable as a data directory (the `--data_dir` argument for the training script). The filenames should all start with WNID (class ids) followed by underscores, like `n01440764_2708.JPEG`. Conveniently (but not by accident) this is how the automated data-loader expects to discover class labels. 18 | 19 | ## LSUN bedroom 20 | 21 | To download and pre-process LSUN bedroom, clone [fyu/lsun](https://github.com/fyu/lsun) on GitHub and run their download script `python3 download.py bedroom`. The result will be an "lmdb" database named like `bedroom_train_lmdb`. You can pass this to our [lsun_bedroom.py](lsun_bedroom.py) script like so: 22 | 23 | ``` 24 | python lsun_bedroom.py bedroom_train_lmdb lsun_train_output_dir 25 | ``` 26 | 27 | This creates a directory called `lsun_train_output_dir`. This directory can be passed to the training scripts via the `--data_dir` argument. 28 | -------------------------------------------------------------------------------- /datasets/lsun_bedroom.py: -------------------------------------------------------------------------------- 1 | """ 2 | Convert an LSUN lmdb database into a directory of images. 3 | """ 4 | 5 | import argparse 6 | import io 7 | import os 8 | 9 | from PIL import Image 10 | import lmdb 11 | import numpy as np 12 | 13 | 14 | def read_images(lmdb_path, image_size): 15 | env = lmdb.open(lmdb_path, map_size=1099511627776, max_readers=100, readonly=True) 16 | with env.begin(write=False) as transaction: 17 | cursor = transaction.cursor() 18 | for _, webp_data in cursor: 19 | img = Image.open(io.BytesIO(webp_data)) 20 | width, height = img.size 21 | scale = image_size / min(width, height) 22 | img = img.resize( 23 | (int(round(scale * width)), int(round(scale * height))), 24 | resample=Image.BOX, 25 | ) 26 | arr = np.array(img) 27 | h, w, _ = arr.shape 28 | h_off = (h - image_size) // 2 29 | w_off = (w - image_size) // 2 30 | arr = arr[h_off : h_off + image_size, w_off : w_off + image_size] 31 | yield arr 32 | 33 | 34 | def dump_images(out_dir, images, prefix): 35 | if not os.path.exists(out_dir): 36 | os.mkdir(out_dir) 37 | for i, img in enumerate(images): 38 | Image.fromarray(img).save(os.path.join(out_dir, f"{prefix}_{i:07d}.png")) 39 | 40 | 41 | def main(): 42 | parser = argparse.ArgumentParser() 43 | parser.add_argument("--image-size", help="new image size", type=int, default=256) 44 | parser.add_argument("--prefix", help="class name", type=str, default="bedroom") 45 | parser.add_argument("lmdb_path", help="path to an LSUN lmdb database") 46 | parser.add_argument("out_dir", help="path to output directory") 47 | args = parser.parse_args() 48 | 49 | images = read_images(args.lmdb_path, args.image_size) 50 | dump_images(args.out_dir, images, args.prefix) 51 | 52 | 53 | if __name__ == "__main__": 54 | main() 55 | -------------------------------------------------------------------------------- /evaluations/README.md: -------------------------------------------------------------------------------- 1 | # Evaluations 2 | 3 | To compare different generative models, we use FID, sFID, Precision, Recall, and Inception Score. These metrics can all be calculated using batches of samples, which we store in `.npz` (numpy) files. 4 | 5 | # Download batches 6 | 7 | We provide pre-computed sample batches for the reference datasets, our diffusion models, and several baselines we compare against. These are all stored in `.npz` format. 8 | 9 | Reference dataset batches contain pre-computed statistics over the whole dataset, as well as 10,000 images for computing Precision and Recall. All other batches contain 50,000 images which can be used to compute statistics and Precision/Recall. 10 | 11 | Here are links to download all of the sample and reference batches: 12 | 13 | * LSUN 14 | * LSUN bedroom: [reference batch](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/bedroom/VIRTUAL_lsun_bedroom256.npz) 15 | * [ADM (dropout)](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/bedroom/admnet_dropout_lsun_bedroom.npz) 16 | * [DDPM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/bedroom/ddpm_lsun_bedroom.npz) 17 | * [IDDPM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/bedroom/iddpm_lsun_bedroom.npz) 18 | * [StyleGAN](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/bedroom/stylegan_lsun_bedroom.npz) 19 | * LSUN cat: [reference batch](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/cat/VIRTUAL_lsun_cat256.npz) 20 | * [ADM (dropout)](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/cat/admnet_dropout_lsun_cat.npz) 21 | * [StyleGAN2](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/cat/stylegan2_lsun_cat.npz) 22 | * LSUN horse: [reference batch](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/horse/VIRTUAL_lsun_horse256.npz) 23 | * [ADM (dropout)](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/horse/admnet_dropout_lsun_horse.npz) 24 | * [ADM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/lsun/horse/admnet_lsun_horse.npz) 25 | * ImageNet 64x64: [reference batch](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/64/VIRTUAL_imagenet64_labeled.npz) 26 | * [ADM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/64/admnet_imagenet64.npz) 27 | * [IDDPM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/64/iddpm_imagenet64.npz) 28 | * [BigGAN](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/64/biggan_deep_imagenet64.npz) 29 | * ImageNet 128x128: [reference batch](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/128/VIRTUAL_imagenet128_labeled.npz) 30 | * [ADM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/128/admnet_imagenet128.npz) 31 | * [ADM-G](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/128/admnet_guided_imagenet128.npz) 32 | * [ADM-G, 25 steps](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/128/admnet_guided_25step_imagenet128.npz) 33 | * [BigGAN-deep (trunc=1.0)](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/128/biggan_deep_trunc1_imagenet128.npz) 34 | * ImageNet 256x256: [reference batch](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/VIRTUAL_imagenet256_labeled.npz) 35 | * [ADM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/admnet_imagenet256.npz) 36 | * [ADM-G](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/admnet_guided_imagenet256.npz) 37 | * [ADM-G, 25 step](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/admnet_guided_25step_imagenet256.npz) 38 | * [ADM-G + ADM-U](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/admnet_guided_upsampled_imagenet256.npz) 39 | * [ADM-U](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/admnet_upsampled_imagenet256.npz) 40 | * [BigGAN-deep (trunc=1.0)](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/biggan_deep_trunc1_imagenet256.npz) 41 | * ImageNet 512x512: [reference batch](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/512/VIRTUAL_imagenet512.npz) 42 | * [ADM](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/512/admnet_imagenet512.npz) 43 | * [ADM-G](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/512/admnet_guided_imagenet512.npz) 44 | * [ADM-G, 25 step](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/512/admnet_guided_25step_imagenet512.npz) 45 | * [ADM-G + ADM-U](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/512/admnet_guided_upsampled_imagenet512.npz) 46 | * [ADM-U](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/512/admnet_upsampled_imagenet512.npz) 47 | * [BigGAN-deep (trunc=1.0)](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/512/biggan_deep_trunc1_imagenet512.npz) 48 | 49 | # Run evaluations 50 | 51 | First, generate or download a batch of samples and download the corresponding reference batch for the given dataset. For this example, we'll use ImageNet 256x256, so the refernce batch is `VIRTUAL_imagenet256_labeled.npz` and we can use the sample batch `admnet_guided_upsampled_imagenet256.npz`. 52 | 53 | Next, run the `evaluator.py` script. The requirements of this script can be found in [requirements.txt](requirements.txt). Pass two arguments to the script: the reference batch and the sample batch. The script will download the InceptionV3 model used for evaluations into the current working directory (if it is not already present). This file is roughly 100MB. 54 | 55 | The output of the script will look something like this, where the first `...` is a bunch of verbose TensorFlow logging: 56 | 57 | ``` 58 | $ python evaluator.py VIRTUAL_imagenet256_labeled.npz admnet_guided_upsampled_imagenet256.npz 59 | ... 60 | computing reference batch activations... 61 | computing/reading reference batch statistics... 62 | computing sample batch activations... 63 | computing/reading sample batch statistics... 64 | Computing evaluations... 65 | Inception Score: 215.8370361328125 66 | FID: 3.9425574129223264 67 | sFID: 6.140433703346162 68 | Precision: 0.8265 69 | Recall: 0.5309 70 | ``` 71 | -------------------------------------------------------------------------------- /evaluations/evaluator.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import io 3 | import os 4 | import random 5 | import warnings 6 | import zipfile 7 | from abc import ABC, abstractmethod 8 | from contextlib import contextmanager 9 | from functools import partial 10 | from multiprocessing import cpu_count 11 | from multiprocessing.pool import ThreadPool 12 | from typing import Iterable, Optional, Tuple 13 | 14 | import numpy as np 15 | import requests 16 | import tensorflow.compat.v1 as tf 17 | from scipy import linalg 18 | from tqdm.auto import tqdm 19 | 20 | INCEPTION_V3_URL = "https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/classify_image_graph_def.pb" 21 | INCEPTION_V3_PATH = "classify_image_graph_def.pb" 22 | 23 | FID_POOL_NAME = "pool_3:0" 24 | FID_SPATIAL_NAME = "mixed_6/conv:0" 25 | 26 | 27 | def main(): 28 | parser = argparse.ArgumentParser() 29 | parser.add_argument("ref_batch", help="path to reference batch npz file") 30 | parser.add_argument("sample_batch", help="path to sample batch npz file") 31 | args = parser.parse_args() 32 | 33 | config = tf.ConfigProto( 34 | allow_soft_placement=True # allows DecodeJpeg to run on CPU in Inception graph 35 | ) 36 | config.gpu_options.allow_growth = True 37 | evaluator = Evaluator(tf.Session(config=config)) 38 | 39 | print("warming up TensorFlow...") 40 | # This will cause TF to print a bunch of verbose stuff now rather 41 | # than after the next print(), to help prevent confusion. 42 | evaluator.warmup() 43 | 44 | print("computing reference batch activations...") 45 | ref_acts = evaluator.read_activations(args.ref_batch) 46 | print("computing/reading reference batch statistics...") 47 | ref_stats, ref_stats_spatial = evaluator.read_statistics(args.ref_batch, ref_acts) 48 | 49 | print("computing sample batch activations...") 50 | sample_acts = evaluator.read_activations(args.sample_batch) 51 | print("computing/reading sample batch statistics...") 52 | sample_stats, sample_stats_spatial = evaluator.read_statistics(args.sample_batch, sample_acts) 53 | 54 | print("Computing evaluations...") 55 | print("Inception Score:", evaluator.compute_inception_score(sample_acts[0])) 56 | print("FID:", sample_stats.frechet_distance(ref_stats)) 57 | print("sFID:", sample_stats_spatial.frechet_distance(ref_stats_spatial)) 58 | prec, recall = evaluator.compute_prec_recall(ref_acts[0], sample_acts[0]) 59 | print("Precision:", prec) 60 | print("Recall:", recall) 61 | 62 | 63 | class InvalidFIDException(Exception): 64 | pass 65 | 66 | 67 | class FIDStatistics: 68 | def __init__(self, mu: np.ndarray, sigma: np.ndarray): 69 | self.mu = mu 70 | self.sigma = sigma 71 | 72 | def frechet_distance(self, other, eps=1e-6): 73 | """ 74 | Compute the Frechet distance between two sets of statistics. 75 | """ 76 | # https://github.com/bioinf-jku/TTUR/blob/73ab375cdf952a12686d9aa7978567771084da42/fid.py#L132 77 | mu1, sigma1 = self.mu, self.sigma 78 | mu2, sigma2 = other.mu, other.sigma 79 | 80 | mu1 = np.atleast_1d(mu1) 81 | mu2 = np.atleast_1d(mu2) 82 | 83 | sigma1 = np.atleast_2d(sigma1) 84 | sigma2 = np.atleast_2d(sigma2) 85 | 86 | assert ( 87 | mu1.shape == mu2.shape 88 | ), f"Training and test mean vectors have different lengths: {mu1.shape}, {mu2.shape}" 89 | assert ( 90 | sigma1.shape == sigma2.shape 91 | ), f"Training and test covariances have different dimensions: {sigma1.shape}, {sigma2.shape}" 92 | 93 | diff = mu1 - mu2 94 | 95 | # product might be almost singular 96 | covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False) 97 | if not np.isfinite(covmean).all(): 98 | msg = ( 99 | "fid calculation produces singular product; adding %s to diagonal of cov estimates" 100 | % eps 101 | ) 102 | warnings.warn(msg) 103 | offset = np.eye(sigma1.shape[0]) * eps 104 | covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset)) 105 | 106 | # numerical error might give slight imaginary component 107 | if np.iscomplexobj(covmean): 108 | if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3): 109 | m = np.max(np.abs(covmean.imag)) 110 | raise ValueError("Imaginary component {}".format(m)) 111 | covmean = covmean.real 112 | 113 | tr_covmean = np.trace(covmean) 114 | 115 | return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean 116 | 117 | 118 | class Evaluator: 119 | def __init__( 120 | self, 121 | session, 122 | batch_size=64, 123 | softmax_batch_size=512, 124 | ): 125 | self.sess = session 126 | self.batch_size = batch_size 127 | self.softmax_batch_size = softmax_batch_size 128 | self.manifold_estimator = ManifoldEstimator(session) 129 | with self.sess.graph.as_default(): 130 | self.image_input = tf.placeholder(tf.float32, shape=[None, None, None, 3]) 131 | self.softmax_input = tf.placeholder(tf.float32, shape=[None, 2048]) 132 | self.pool_features, self.spatial_features = _create_feature_graph(self.image_input) 133 | self.softmax = _create_softmax_graph(self.softmax_input) 134 | 135 | def warmup(self): 136 | self.compute_activations(np.zeros([1, 8, 64, 64, 3])) 137 | 138 | def read_activations(self, npz_path: str) -> Tuple[np.ndarray, np.ndarray]: 139 | with open_npz_array(npz_path, "arr_0") as reader: 140 | return self.compute_activations(reader.read_batches(self.batch_size)) 141 | 142 | def compute_activations(self, batches: Iterable[np.ndarray]) -> Tuple[np.ndarray, np.ndarray]: 143 | """ 144 | Compute image features for downstream evals. 145 | 146 | :param batches: a iterator over NHWC numpy arrays in [0, 255]. 147 | :return: a tuple of numpy arrays of shape [N x X], where X is a feature 148 | dimension. The tuple is (pool_3, spatial). 149 | """ 150 | preds = [] 151 | spatial_preds = [] 152 | for batch in tqdm(batches): 153 | batch = batch.astype(np.float32) 154 | pred, spatial_pred = self.sess.run( 155 | [self.pool_features, self.spatial_features], {self.image_input: batch} 156 | ) 157 | preds.append(pred.reshape([pred.shape[0], -1])) 158 | spatial_preds.append(spatial_pred.reshape([spatial_pred.shape[0], -1])) 159 | return ( 160 | np.concatenate(preds, axis=0), 161 | np.concatenate(spatial_preds, axis=0), 162 | ) 163 | 164 | def read_statistics( 165 | self, npz_path: str, activations: Tuple[np.ndarray, np.ndarray] 166 | ) -> Tuple[FIDStatistics, FIDStatistics]: 167 | obj = np.load(npz_path) 168 | if "mu" in list(obj.keys()): 169 | return FIDStatistics(obj["mu"], obj["sigma"]), FIDStatistics( 170 | obj["mu_s"], obj["sigma_s"] 171 | ) 172 | return tuple(self.compute_statistics(x) for x in activations) 173 | 174 | def compute_statistics(self, activations: np.ndarray) -> FIDStatistics: 175 | mu = np.mean(activations, axis=0) 176 | sigma = np.cov(activations, rowvar=False) 177 | return FIDStatistics(mu, sigma) 178 | 179 | def compute_inception_score(self, activations: np.ndarray, split_size: int = 5000) -> float: 180 | softmax_out = [] 181 | for i in range(0, len(activations), self.softmax_batch_size): 182 | acts = activations[i : i + self.softmax_batch_size] 183 | softmax_out.append(self.sess.run(self.softmax, feed_dict={self.softmax_input: acts})) 184 | preds = np.concatenate(softmax_out, axis=0) 185 | # https://github.com/openai/improved-gan/blob/4f5d1ec5c16a7eceb206f42bfc652693601e1d5c/inception_score/model.py#L46 186 | scores = [] 187 | for i in range(0, len(preds), split_size): 188 | part = preds[i : i + split_size] 189 | kl = part * (np.log(part) - np.log(np.expand_dims(np.mean(part, 0), 0))) 190 | kl = np.mean(np.sum(kl, 1)) 191 | scores.append(np.exp(kl)) 192 | return float(np.mean(scores)) 193 | 194 | def compute_prec_recall( 195 | self, activations_ref: np.ndarray, activations_sample: np.ndarray 196 | ) -> Tuple[float, float]: 197 | radii_1 = self.manifold_estimator.manifold_radii(activations_ref) 198 | radii_2 = self.manifold_estimator.manifold_radii(activations_sample) 199 | pr = self.manifold_estimator.evaluate_pr( 200 | activations_ref, radii_1, activations_sample, radii_2 201 | ) 202 | return (float(pr[0][0]), float(pr[1][0])) 203 | 204 | 205 | class ManifoldEstimator: 206 | """ 207 | A helper for comparing manifolds of feature vectors. 208 | 209 | Adapted from https://github.com/kynkaat/improved-precision-and-recall-metric/blob/f60f25e5ad933a79135c783fcda53de30f42c9b9/precision_recall.py#L57 210 | """ 211 | 212 | def __init__( 213 | self, 214 | session, 215 | row_batch_size=10000, 216 | col_batch_size=10000, 217 | nhood_sizes=(3,), 218 | clamp_to_percentile=None, 219 | eps=1e-5, 220 | ): 221 | """ 222 | Estimate the manifold of given feature vectors. 223 | 224 | :param session: the TensorFlow session. 225 | :param row_batch_size: row batch size to compute pairwise distances 226 | (parameter to trade-off between memory usage and performance). 227 | :param col_batch_size: column batch size to compute pairwise distances. 228 | :param nhood_sizes: number of neighbors used to estimate the manifold. 229 | :param clamp_to_percentile: prune hyperspheres that have radius larger than 230 | the given percentile. 231 | :param eps: small number for numerical stability. 232 | """ 233 | self.distance_block = DistanceBlock(session) 234 | self.row_batch_size = row_batch_size 235 | self.col_batch_size = col_batch_size 236 | self.nhood_sizes = nhood_sizes 237 | self.num_nhoods = len(nhood_sizes) 238 | self.clamp_to_percentile = clamp_to_percentile 239 | self.eps = eps 240 | 241 | def warmup(self): 242 | feats, radii = ( 243 | np.zeros([1, 2048], dtype=np.float32), 244 | np.zeros([1, 1], dtype=np.float32), 245 | ) 246 | self.evaluate_pr(feats, radii, feats, radii) 247 | 248 | def manifold_radii(self, features: np.ndarray) -> np.ndarray: 249 | num_images = len(features) 250 | 251 | # Estimate manifold of features by calculating distances to k-NN of each sample. 252 | radii = np.zeros([num_images, self.num_nhoods], dtype=np.float32) 253 | distance_batch = np.zeros([self.row_batch_size, num_images], dtype=np.float32) 254 | seq = np.arange(max(self.nhood_sizes) + 1, dtype=np.int32) 255 | 256 | for begin1 in range(0, num_images, self.row_batch_size): 257 | end1 = min(begin1 + self.row_batch_size, num_images) 258 | row_batch = features[begin1:end1] 259 | 260 | for begin2 in range(0, num_images, self.col_batch_size): 261 | end2 = min(begin2 + self.col_batch_size, num_images) 262 | col_batch = features[begin2:end2] 263 | 264 | # Compute distances between batches. 265 | distance_batch[ 266 | 0 : end1 - begin1, begin2:end2 267 | ] = self.distance_block.pairwise_distances(row_batch, col_batch) 268 | 269 | # Find the k-nearest neighbor from the current batch. 270 | radii[begin1:end1, :] = np.concatenate( 271 | [ 272 | x[:, self.nhood_sizes] 273 | for x in _numpy_partition(distance_batch[0 : end1 - begin1, :], seq, axis=1) 274 | ], 275 | axis=0, 276 | ) 277 | 278 | if self.clamp_to_percentile is not None: 279 | max_distances = np.percentile(radii, self.clamp_to_percentile, axis=0) 280 | radii[radii > max_distances] = 0 281 | return radii 282 | 283 | def evaluate(self, features: np.ndarray, radii: np.ndarray, eval_features: np.ndarray): 284 | """ 285 | Evaluate if new feature vectors are at the manifold. 286 | """ 287 | num_eval_images = eval_features.shape[0] 288 | num_ref_images = radii.shape[0] 289 | distance_batch = np.zeros([self.row_batch_size, num_ref_images], dtype=np.float32) 290 | batch_predictions = np.zeros([num_eval_images, self.num_nhoods], dtype=np.int32) 291 | max_realism_score = np.zeros([num_eval_images], dtype=np.float32) 292 | nearest_indices = np.zeros([num_eval_images], dtype=np.int32) 293 | 294 | for begin1 in range(0, num_eval_images, self.row_batch_size): 295 | end1 = min(begin1 + self.row_batch_size, num_eval_images) 296 | feature_batch = eval_features[begin1:end1] 297 | 298 | for begin2 in range(0, num_ref_images, self.col_batch_size): 299 | end2 = min(begin2 + self.col_batch_size, num_ref_images) 300 | ref_batch = features[begin2:end2] 301 | 302 | distance_batch[ 303 | 0 : end1 - begin1, begin2:end2 304 | ] = self.distance_block.pairwise_distances(feature_batch, ref_batch) 305 | 306 | # From the minibatch of new feature vectors, determine if they are in the estimated manifold. 307 | # If a feature vector is inside a hypersphere of some reference sample, then 308 | # the new sample lies at the estimated manifold. 309 | # The radii of the hyperspheres are determined from distances of neighborhood size k. 310 | samples_in_manifold = distance_batch[0 : end1 - begin1, :, None] <= radii 311 | batch_predictions[begin1:end1] = np.any(samples_in_manifold, axis=1).astype(np.int32) 312 | 313 | max_realism_score[begin1:end1] = np.max( 314 | radii[:, 0] / (distance_batch[0 : end1 - begin1, :] + self.eps), axis=1 315 | ) 316 | nearest_indices[begin1:end1] = np.argmin(distance_batch[0 : end1 - begin1, :], axis=1) 317 | 318 | return { 319 | "fraction": float(np.mean(batch_predictions)), 320 | "batch_predictions": batch_predictions, 321 | "max_realisim_score": max_realism_score, 322 | "nearest_indices": nearest_indices, 323 | } 324 | 325 | def evaluate_pr( 326 | self, 327 | features_1: np.ndarray, 328 | radii_1: np.ndarray, 329 | features_2: np.ndarray, 330 | radii_2: np.ndarray, 331 | ) -> Tuple[np.ndarray, np.ndarray]: 332 | """ 333 | Evaluate precision and recall efficiently. 334 | 335 | :param features_1: [N1 x D] feature vectors for reference batch. 336 | :param radii_1: [N1 x K1] radii for reference vectors. 337 | :param features_2: [N2 x D] feature vectors for the other batch. 338 | :param radii_2: [N x K2] radii for other vectors. 339 | :return: a tuple of arrays for (precision, recall): 340 | - precision: an np.ndarray of length K1 341 | - recall: an np.ndarray of length K2 342 | """ 343 | features_1_status = np.zeros([len(features_1), radii_2.shape[1]], dtype=np.bool) 344 | features_2_status = np.zeros([len(features_2), radii_1.shape[1]], dtype=np.bool) 345 | for begin_1 in range(0, len(features_1), self.row_batch_size): 346 | end_1 = begin_1 + self.row_batch_size 347 | batch_1 = features_1[begin_1:end_1] 348 | for begin_2 in range(0, len(features_2), self.col_batch_size): 349 | end_2 = begin_2 + self.col_batch_size 350 | batch_2 = features_2[begin_2:end_2] 351 | batch_1_in, batch_2_in = self.distance_block.less_thans( 352 | batch_1, radii_1[begin_1:end_1], batch_2, radii_2[begin_2:end_2] 353 | ) 354 | features_1_status[begin_1:end_1] |= batch_1_in 355 | features_2_status[begin_2:end_2] |= batch_2_in 356 | return ( 357 | np.mean(features_2_status.astype(np.float64), axis=0), 358 | np.mean(features_1_status.astype(np.float64), axis=0), 359 | ) 360 | 361 | 362 | class DistanceBlock: 363 | """ 364 | Calculate pairwise distances between vectors. 365 | 366 | Adapted from https://github.com/kynkaat/improved-precision-and-recall-metric/blob/f60f25e5ad933a79135c783fcda53de30f42c9b9/precision_recall.py#L34 367 | """ 368 | 369 | def __init__(self, session): 370 | self.session = session 371 | 372 | # Initialize TF graph to calculate pairwise distances. 373 | with session.graph.as_default(): 374 | self._features_batch1 = tf.placeholder(tf.float32, shape=[None, None]) 375 | self._features_batch2 = tf.placeholder(tf.float32, shape=[None, None]) 376 | distance_block_16 = _batch_pairwise_distances( 377 | tf.cast(self._features_batch1, tf.float16), 378 | tf.cast(self._features_batch2, tf.float16), 379 | ) 380 | self.distance_block = tf.cond( 381 | tf.reduce_all(tf.math.is_finite(distance_block_16)), 382 | lambda: tf.cast(distance_block_16, tf.float32), 383 | lambda: _batch_pairwise_distances(self._features_batch1, self._features_batch2), 384 | ) 385 | 386 | # Extra logic for less thans. 387 | self._radii1 = tf.placeholder(tf.float32, shape=[None, None]) 388 | self._radii2 = tf.placeholder(tf.float32, shape=[None, None]) 389 | dist32 = tf.cast(self.distance_block, tf.float32)[..., None] 390 | self._batch_1_in = tf.math.reduce_any(dist32 <= self._radii2, axis=1) 391 | self._batch_2_in = tf.math.reduce_any(dist32 <= self._radii1[:, None], axis=0) 392 | 393 | def pairwise_distances(self, U, V): 394 | """ 395 | Evaluate pairwise distances between two batches of feature vectors. 396 | """ 397 | return self.session.run( 398 | self.distance_block, 399 | feed_dict={self._features_batch1: U, self._features_batch2: V}, 400 | ) 401 | 402 | def less_thans(self, batch_1, radii_1, batch_2, radii_2): 403 | return self.session.run( 404 | [self._batch_1_in, self._batch_2_in], 405 | feed_dict={ 406 | self._features_batch1: batch_1, 407 | self._features_batch2: batch_2, 408 | self._radii1: radii_1, 409 | self._radii2: radii_2, 410 | }, 411 | ) 412 | 413 | 414 | def _batch_pairwise_distances(U, V): 415 | """ 416 | Compute pairwise distances between two batches of feature vectors. 417 | """ 418 | with tf.variable_scope("pairwise_dist_block"): 419 | # Squared norms of each row in U and V. 420 | norm_u = tf.reduce_sum(tf.square(U), 1) 421 | norm_v = tf.reduce_sum(tf.square(V), 1) 422 | 423 | # norm_u as a column and norm_v as a row vectors. 424 | norm_u = tf.reshape(norm_u, [-1, 1]) 425 | norm_v = tf.reshape(norm_v, [1, -1]) 426 | 427 | # Pairwise squared Euclidean distances. 428 | D = tf.maximum(norm_u - 2 * tf.matmul(U, V, False, True) + norm_v, 0.0) 429 | 430 | return D 431 | 432 | 433 | class NpzArrayReader(ABC): 434 | @abstractmethod 435 | def read_batch(self, batch_size: int) -> Optional[np.ndarray]: 436 | pass 437 | 438 | @abstractmethod 439 | def remaining(self) -> int: 440 | pass 441 | 442 | def read_batches(self, batch_size: int) -> Iterable[np.ndarray]: 443 | def gen_fn(): 444 | while True: 445 | batch = self.read_batch(batch_size) 446 | if batch is None: 447 | break 448 | yield batch 449 | 450 | rem = self.remaining() 451 | num_batches = rem // batch_size + int(rem % batch_size != 0) 452 | return BatchIterator(gen_fn, num_batches) 453 | 454 | 455 | class BatchIterator: 456 | def __init__(self, gen_fn, length): 457 | self.gen_fn = gen_fn 458 | self.length = length 459 | 460 | def __len__(self): 461 | return self.length 462 | 463 | def __iter__(self): 464 | return self.gen_fn() 465 | 466 | 467 | class StreamingNpzArrayReader(NpzArrayReader): 468 | def __init__(self, arr_f, shape, dtype): 469 | self.arr_f = arr_f 470 | self.shape = shape 471 | self.dtype = dtype 472 | self.idx = 0 473 | 474 | def read_batch(self, batch_size: int) -> Optional[np.ndarray]: 475 | if self.idx >= self.shape[0]: 476 | return None 477 | 478 | bs = min(batch_size, self.shape[0] - self.idx) 479 | self.idx += bs 480 | 481 | if self.dtype.itemsize == 0: 482 | return np.ndarray([bs, *self.shape[1:]], dtype=self.dtype) 483 | 484 | read_count = bs * np.prod(self.shape[1:]) 485 | read_size = int(read_count * self.dtype.itemsize) 486 | data = _read_bytes(self.arr_f, read_size, "array data") 487 | return np.frombuffer(data, dtype=self.dtype).reshape([bs, *self.shape[1:]]) 488 | 489 | def remaining(self) -> int: 490 | return max(0, self.shape[0] - self.idx) 491 | 492 | 493 | class MemoryNpzArrayReader(NpzArrayReader): 494 | def __init__(self, arr): 495 | self.arr = arr 496 | self.idx = 0 497 | 498 | @classmethod 499 | def load(cls, path: str, arr_name: str): 500 | with open(path, "rb") as f: 501 | arr = np.load(f)[arr_name] 502 | return cls(arr) 503 | 504 | def read_batch(self, batch_size: int) -> Optional[np.ndarray]: 505 | if self.idx >= self.arr.shape[0]: 506 | return None 507 | 508 | res = self.arr[self.idx : self.idx + batch_size] 509 | self.idx += batch_size 510 | return res 511 | 512 | def remaining(self) -> int: 513 | return max(0, self.arr.shape[0] - self.idx) 514 | 515 | 516 | @contextmanager 517 | def open_npz_array(path: str, arr_name: str) -> NpzArrayReader: 518 | with _open_npy_file(path, arr_name) as arr_f: 519 | version = np.lib.format.read_magic(arr_f) 520 | if version == (1, 0): 521 | header = np.lib.format.read_array_header_1_0(arr_f) 522 | elif version == (2, 0): 523 | header = np.lib.format.read_array_header_2_0(arr_f) 524 | else: 525 | yield MemoryNpzArrayReader.load(path, arr_name) 526 | return 527 | shape, fortran, dtype = header 528 | if fortran or dtype.hasobject: 529 | yield MemoryNpzArrayReader.load(path, arr_name) 530 | else: 531 | yield StreamingNpzArrayReader(arr_f, shape, dtype) 532 | 533 | 534 | def _read_bytes(fp, size, error_template="ran out of data"): 535 | """ 536 | Copied from: https://github.com/numpy/numpy/blob/fb215c76967739268de71aa4bda55dd1b062bc2e/numpy/lib/format.py#L788-L886 537 | 538 | Read from file-like object until size bytes are read. 539 | Raises ValueError if not EOF is encountered before size bytes are read. 540 | Non-blocking objects only supported if they derive from io objects. 541 | Required as e.g. ZipExtFile in python 2.6 can return less data than 542 | requested. 543 | """ 544 | data = bytes() 545 | while True: 546 | # io files (default in python3) return None or raise on 547 | # would-block, python2 file will truncate, probably nothing can be 548 | # done about that. note that regular files can't be non-blocking 549 | try: 550 | r = fp.read(size - len(data)) 551 | data += r 552 | if len(r) == 0 or len(data) == size: 553 | break 554 | except io.BlockingIOError: 555 | pass 556 | if len(data) != size: 557 | msg = "EOF: reading %s, expected %d bytes got %d" 558 | raise ValueError(msg % (error_template, size, len(data))) 559 | else: 560 | return data 561 | 562 | 563 | @contextmanager 564 | def _open_npy_file(path: str, arr_name: str): 565 | with open(path, "rb") as f: 566 | with zipfile.ZipFile(f, "r") as zip_f: 567 | if f"{arr_name}.npy" not in zip_f.namelist(): 568 | raise ValueError(f"missing {arr_name} in npz file") 569 | with zip_f.open(f"{arr_name}.npy", "r") as arr_f: 570 | yield arr_f 571 | 572 | 573 | def _download_inception_model(): 574 | if os.path.exists(INCEPTION_V3_PATH): 575 | return 576 | print("downloading InceptionV3 model...") 577 | with requests.get(INCEPTION_V3_URL, stream=True) as r: 578 | r.raise_for_status() 579 | tmp_path = INCEPTION_V3_PATH + ".tmp" 580 | with open(tmp_path, "wb") as f: 581 | for chunk in tqdm(r.iter_content(chunk_size=8192)): 582 | f.write(chunk) 583 | os.rename(tmp_path, INCEPTION_V3_PATH) 584 | 585 | 586 | def _create_feature_graph(input_batch): 587 | _download_inception_model() 588 | prefix = f"{random.randrange(2**32)}_{random.randrange(2**32)}" 589 | with open(INCEPTION_V3_PATH, "rb") as f: 590 | graph_def = tf.GraphDef() 591 | graph_def.ParseFromString(f.read()) 592 | pool3, spatial = tf.import_graph_def( 593 | graph_def, 594 | input_map={f"ExpandDims:0": input_batch}, 595 | return_elements=[FID_POOL_NAME, FID_SPATIAL_NAME], 596 | name=prefix, 597 | ) 598 | _update_shapes(pool3) 599 | spatial = spatial[..., :7] 600 | return pool3, spatial 601 | 602 | 603 | def _create_softmax_graph(input_batch): 604 | _download_inception_model() 605 | prefix = f"{random.randrange(2**32)}_{random.randrange(2**32)}" 606 | with open(INCEPTION_V3_PATH, "rb") as f: 607 | graph_def = tf.GraphDef() 608 | graph_def.ParseFromString(f.read()) 609 | (matmul,) = tf.import_graph_def( 610 | graph_def, return_elements=[f"softmax/logits/MatMul"], name=prefix 611 | ) 612 | w = matmul.inputs[1] 613 | logits = tf.matmul(input_batch, w) 614 | return tf.nn.softmax(logits) 615 | 616 | 617 | def _update_shapes(pool3): 618 | # https://github.com/bioinf-jku/TTUR/blob/73ab375cdf952a12686d9aa7978567771084da42/fid.py#L50-L63 619 | ops = pool3.graph.get_operations() 620 | for op in ops: 621 | for o in op.outputs: 622 | shape = o.get_shape() 623 | if shape._dims is not None: # pylint: disable=protected-access 624 | # shape = [s.value for s in shape] TF 1.x 625 | shape = [s for s in shape] # TF 2.x 626 | new_shape = [] 627 | for j, s in enumerate(shape): 628 | if s == 1 and j == 0: 629 | new_shape.append(None) 630 | else: 631 | new_shape.append(s) 632 | o.__dict__["_shape_val"] = tf.TensorShape(new_shape) 633 | return pool3 634 | 635 | 636 | def _numpy_partition(arr, kth, **kwargs): 637 | num_workers = min(cpu_count(), len(arr)) 638 | chunk_size = len(arr) // num_workers 639 | extra = len(arr) % num_workers 640 | 641 | start_idx = 0 642 | batches = [] 643 | for i in range(num_workers): 644 | size = chunk_size + (1 if i < extra else 0) 645 | batches.append(arr[start_idx : start_idx + size]) 646 | start_idx += size 647 | 648 | with ThreadPool(num_workers) as pool: 649 | return list(pool.map(partial(np.partition, kth=kth, **kwargs), batches)) 650 | 651 | 652 | if __name__ == "__main__": 653 | main() 654 | -------------------------------------------------------------------------------- /evaluations/requirements.txt: -------------------------------------------------------------------------------- 1 | tensorflow-gpu>=2.0 2 | scipy 3 | requests 4 | tqdm -------------------------------------------------------------------------------- /patch_diffusion/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Codebase for "Improved Denoising Diffusion Probabilistic Models". 3 | """ 4 | -------------------------------------------------------------------------------- /patch_diffusion/dist_util.py: -------------------------------------------------------------------------------- 1 | """ 2 | Helpers for distributed training. 3 | """ 4 | 5 | import io 6 | import os 7 | import socket 8 | 9 | import blobfile as bf 10 | from mpi4py import MPI 11 | import torch as th 12 | import torch.distributed as dist 13 | 14 | # Change this to reflect your cluster layout. 15 | # The GPU for a given rank is (rank % GPUS_PER_NODE). 16 | GPUS_PER_NODE = 8 17 | 18 | SETUP_RETRY_COUNT = 3 19 | 20 | 21 | def setup_dist(): 22 | """ 23 | Setup a distributed process group. 24 | """ 25 | if dist.is_initialized(): 26 | return 27 | os.environ["CUDA_VISIBLE_DEVICES"] = f"{MPI.COMM_WORLD.Get_rank() % GPUS_PER_NODE}" 28 | 29 | comm = MPI.COMM_WORLD 30 | backend = "gloo" if not th.cuda.is_available() else "nccl" 31 | 32 | if backend == "gloo": 33 | hostname = "localhost" 34 | else: 35 | hostname = socket.gethostbyname(socket.getfqdn()) 36 | os.environ["MASTER_ADDR"] = comm.bcast(hostname, root=0) 37 | os.environ["RANK"] = str(comm.rank) 38 | os.environ["WORLD_SIZE"] = str(comm.size) 39 | 40 | port = comm.bcast(_find_free_port(), root=0) 41 | os.environ["MASTER_PORT"] = str(port) 42 | dist.init_process_group(backend=backend, init_method="env://") 43 | 44 | 45 | def dev(): 46 | """ 47 | Get the device to use for torch.distributed. 48 | """ 49 | if th.cuda.is_available(): 50 | return th.device(f"cuda") 51 | return th.device("cpu") 52 | 53 | 54 | def load_state_dict(path, **kwargs): 55 | """ 56 | Load a PyTorch file without redundant fetches across MPI ranks. 57 | """ 58 | chunk_size = 2 ** 30 # MPI has a relatively small size limit 59 | if MPI.COMM_WORLD.Get_rank() == 0: 60 | with bf.BlobFile(path, "rb") as f: 61 | data = f.read() 62 | num_chunks = len(data) // chunk_size 63 | if len(data) % chunk_size: 64 | num_chunks += 1 65 | MPI.COMM_WORLD.bcast(num_chunks) 66 | for i in range(0, len(data), chunk_size): 67 | MPI.COMM_WORLD.bcast(data[i : i + chunk_size]) 68 | else: 69 | num_chunks = MPI.COMM_WORLD.bcast(None) 70 | data = bytes() 71 | for _ in range(num_chunks): 72 | data += MPI.COMM_WORLD.bcast(None) 73 | 74 | return th.load(io.BytesIO(data), **kwargs) 75 | 76 | 77 | def sync_params(params): 78 | """ 79 | Synchronize a sequence of Tensors across ranks from rank 0. 80 | """ 81 | for p in params: 82 | with th.no_grad(): 83 | dist.broadcast(p, 0) 84 | 85 | 86 | def _find_free_port(): 87 | try: 88 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 89 | s.bind(("", 0)) 90 | s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 91 | return s.getsockname()[1] 92 | finally: 93 | s.close() 94 | -------------------------------------------------------------------------------- /patch_diffusion/fp16_util.py: -------------------------------------------------------------------------------- 1 | """ 2 | Helpers to train with 16-bit precision. 3 | """ 4 | 5 | import numpy as np 6 | import torch as th 7 | import torch.nn as nn 8 | from torch._utils import _flatten_dense_tensors, _unflatten_dense_tensors 9 | 10 | from . import logger 11 | 12 | INITIAL_LOG_LOSS_SCALE = 20.0 13 | 14 | 15 | def convert_module_to_f16(l): 16 | """ 17 | Convert primitive modules to float16. 18 | """ 19 | if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Conv3d)): 20 | l.weight.data = l.weight.data.half() 21 | if l.bias is not None: 22 | l.bias.data = l.bias.data.half() 23 | 24 | 25 | def convert_module_to_f32(l): 26 | """ 27 | Convert primitive modules to float32, undoing convert_module_to_f16(). 28 | """ 29 | if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Conv3d)): 30 | l.weight.data = l.weight.data.float() 31 | if l.bias is not None: 32 | l.bias.data = l.bias.data.float() 33 | 34 | 35 | def make_master_params(param_groups_and_shapes): 36 | """ 37 | Copy model parameters into a (differently-shaped) list of full-precision 38 | parameters. 39 | """ 40 | master_params = [] 41 | for param_group, shape in param_groups_and_shapes: 42 | master_param = nn.Parameter( 43 | _flatten_dense_tensors( 44 | [param.detach().float() for (_, param) in param_group] 45 | ).view(shape) 46 | ) 47 | master_param.requires_grad = True 48 | master_params.append(master_param) 49 | return master_params 50 | 51 | 52 | def model_grads_to_master_grads(param_groups_and_shapes, master_params): 53 | """ 54 | Copy the gradients from the model parameters into the master parameters 55 | from make_master_params(). 56 | """ 57 | for master_param, (param_group, shape) in zip( 58 | master_params, param_groups_and_shapes 59 | ): 60 | master_param.grad = _flatten_dense_tensors( 61 | [param_grad_or_zeros(param) for (_, param) in param_group] 62 | ).view(shape) 63 | 64 | 65 | def master_params_to_model_params(param_groups_and_shapes, master_params): 66 | """ 67 | Copy the master parameter data back into the model parameters. 68 | """ 69 | # Without copying to a list, if a generator is passed, this will 70 | # silently not copy any parameters. 71 | for master_param, (param_group, _) in zip(master_params, param_groups_and_shapes): 72 | for (_, param), unflat_master_param in zip( 73 | param_group, unflatten_master_params(param_group, master_param.view(-1)) 74 | ): 75 | param.detach().copy_(unflat_master_param) 76 | 77 | 78 | def unflatten_master_params(param_group, master_param): 79 | return _unflatten_dense_tensors(master_param, [param for (_, param) in param_group]) 80 | 81 | 82 | def get_param_groups_and_shapes(named_model_params): 83 | named_model_params = list(named_model_params) 84 | scalar_vector_named_params = ( 85 | [(n, p) for (n, p) in named_model_params if p.ndim <= 1], 86 | (-1), 87 | ) 88 | matrix_named_params = ( 89 | [(n, p) for (n, p) in named_model_params if p.ndim > 1], 90 | (1, -1), 91 | ) 92 | return [scalar_vector_named_params, matrix_named_params] 93 | 94 | 95 | def master_params_to_state_dict( 96 | model, param_groups_and_shapes, master_params, use_fp16 97 | ): 98 | if use_fp16: 99 | state_dict = model.state_dict() 100 | for master_param, (param_group, _) in zip( 101 | master_params, param_groups_and_shapes 102 | ): 103 | for (name, _), unflat_master_param in zip( 104 | param_group, unflatten_master_params(param_group, master_param.view(-1)) 105 | ): 106 | assert name in state_dict 107 | state_dict[name] = unflat_master_param 108 | else: 109 | state_dict = model.state_dict() 110 | for i, (name, _value) in enumerate(model.named_parameters()): 111 | assert name in state_dict 112 | state_dict[name] = master_params[i] 113 | return state_dict 114 | 115 | 116 | def state_dict_to_master_params(model, state_dict, use_fp16): 117 | if use_fp16: 118 | named_model_params = [ 119 | (name, state_dict[name]) for name, _ in model.named_parameters() 120 | ] 121 | param_groups_and_shapes = get_param_groups_and_shapes(named_model_params) 122 | master_params = make_master_params(param_groups_and_shapes) 123 | else: 124 | master_params = [state_dict[name] for name, _ in model.named_parameters()] 125 | return master_params 126 | 127 | 128 | def zero_master_grads(master_params): 129 | for param in master_params: 130 | param.grad = None 131 | 132 | 133 | def zero_grad(model_params): 134 | for param in model_params: 135 | # Taken from https://pytorch.org/docs/stable/_modules/torch/optim/optimizer.html#Optimizer.add_param_group 136 | if param.grad is not None: 137 | param.grad.detach_() 138 | param.grad.zero_() 139 | 140 | 141 | def param_grad_or_zeros(param): 142 | if param.grad is not None: 143 | return param.grad.data.detach() 144 | else: 145 | return th.zeros_like(param) 146 | 147 | 148 | class MixedPrecisionTrainer: 149 | def __init__( 150 | self, 151 | *, 152 | model, 153 | use_fp16=False, 154 | fp16_scale_growth=1e-3, 155 | initial_lg_loss_scale=INITIAL_LOG_LOSS_SCALE, 156 | ): 157 | self.model = model 158 | self.use_fp16 = use_fp16 159 | self.fp16_scale_growth = fp16_scale_growth 160 | 161 | self.model_params = list(self.model.parameters()) 162 | self.master_params = self.model_params 163 | self.param_groups_and_shapes = None 164 | self.lg_loss_scale = initial_lg_loss_scale 165 | 166 | if self.use_fp16: 167 | self.param_groups_and_shapes = get_param_groups_and_shapes( 168 | self.model.named_parameters() 169 | ) 170 | self.master_params = make_master_params(self.param_groups_and_shapes) 171 | self.model.convert_to_fp16() 172 | 173 | def zero_grad(self): 174 | zero_grad(self.model_params) 175 | 176 | def backward(self, loss: th.Tensor): 177 | if self.use_fp16: 178 | loss_scale = 2 ** self.lg_loss_scale 179 | (loss * loss_scale).backward() 180 | else: 181 | loss.backward() 182 | 183 | def optimize(self, opt: th.optim.Optimizer): 184 | if self.use_fp16: 185 | return self._optimize_fp16(opt) 186 | else: 187 | return self._optimize_normal(opt) 188 | 189 | def _optimize_fp16(self, opt: th.optim.Optimizer): 190 | logger.logkv_mean("lg_loss_scale", self.lg_loss_scale) 191 | model_grads_to_master_grads(self.param_groups_and_shapes, self.master_params) 192 | grad_norm, param_norm = self._compute_norms(grad_scale=2 ** self.lg_loss_scale) 193 | if check_overflow(grad_norm): 194 | self.lg_loss_scale -= 1 195 | logger.log(f"Found NaN, decreased lg_loss_scale to {self.lg_loss_scale}") 196 | zero_master_grads(self.master_params) 197 | return False 198 | 199 | logger.logkv_mean("grad_norm", grad_norm) 200 | logger.logkv_mean("param_norm", param_norm) 201 | 202 | self.master_params[0].grad.mul_(1.0 / (2 ** self.lg_loss_scale)) 203 | opt.step() 204 | zero_master_grads(self.master_params) 205 | master_params_to_model_params(self.param_groups_and_shapes, self.master_params) 206 | self.lg_loss_scale += self.fp16_scale_growth 207 | return True 208 | 209 | def _optimize_normal(self, opt: th.optim.Optimizer): 210 | grad_norm, param_norm = self._compute_norms() 211 | logger.logkv_mean("grad_norm", grad_norm) 212 | logger.logkv_mean("param_norm", param_norm) 213 | opt.step() 214 | return True 215 | 216 | def _compute_norms(self, grad_scale=1.0): 217 | grad_norm = 0.0 218 | param_norm = 0.0 219 | for p in self.master_params: 220 | with th.no_grad(): 221 | param_norm += th.norm(p, p=2, dtype=th.float32).item() ** 2 222 | if p.grad is not None: 223 | grad_norm += th.norm(p.grad, p=2, dtype=th.float32).item() ** 2 224 | return np.sqrt(grad_norm) / grad_scale, np.sqrt(param_norm) 225 | 226 | def master_params_to_state_dict(self, master_params): 227 | return master_params_to_state_dict( 228 | self.model, self.param_groups_and_shapes, master_params, self.use_fp16 229 | ) 230 | 231 | def state_dict_to_master_params(self, state_dict): 232 | return state_dict_to_master_params(self.model, state_dict, self.use_fp16) 233 | 234 | 235 | def check_overflow(value): 236 | return (value == float("inf")) or (value == -float("inf")) or (value != value) 237 | -------------------------------------------------------------------------------- /patch_diffusion/image_datasets.py: -------------------------------------------------------------------------------- 1 | import math 2 | import random 3 | 4 | from PIL import Image 5 | import blobfile as bf 6 | from mpi4py import MPI 7 | import numpy as np 8 | from torch.utils.data import DataLoader, Dataset 9 | 10 | 11 | def load_data( 12 | *, 13 | data_dir, 14 | batch_size, 15 | image_size, 16 | class_cond=False, 17 | deterministic=False, 18 | random_crop=False, 19 | random_flip=True, 20 | ): 21 | """ 22 | For a dataset, create a generator over (images, kwargs) pairs. 23 | 24 | Each images is an NCHW float tensor, and the kwargs dict contains zero or 25 | more keys, each of which map to a batched Tensor of their own. 26 | The kwargs dict can be used for class labels, in which case the key is "y" 27 | and the values are integer tensors of class labels. 28 | 29 | :param data_dir: a dataset directory. 30 | :param batch_size: the batch size of each returned pair. 31 | :param image_size: the size to which images are resized. 32 | :param class_cond: if True, include a "y" key in returned dicts for class 33 | label. If classes are not available and this is true, an 34 | exception will be raised. 35 | :param deterministic: if True, yield results in a deterministic order. 36 | :param random_crop: if True, randomly crop the images for augmentation. 37 | :param random_flip: if True, randomly flip the images for augmentation. 38 | """ 39 | if not data_dir: 40 | raise ValueError("unspecified data directory") 41 | all_files = _list_image_files_recursively(data_dir) 42 | classes = None 43 | if class_cond: 44 | # Assume classes are the first part of the filename, 45 | # before an underscore. 46 | classes = [bf.basename(path).split("_")[0] for path in all_files] 47 | dataset = ImageDataset( 48 | image_size, 49 | all_files, 50 | classes=classes, 51 | shard=MPI.COMM_WORLD.Get_rank(), 52 | num_shards=MPI.COMM_WORLD.Get_size(), 53 | random_crop=random_crop, 54 | random_flip=random_flip, 55 | ) 56 | if deterministic: 57 | loader = DataLoader( 58 | dataset, batch_size=batch_size, shuffle=False, num_workers=1, drop_last=True 59 | ) 60 | else: 61 | loader = DataLoader( 62 | dataset, batch_size=batch_size, shuffle=True, num_workers=1, drop_last=True 63 | ) 64 | while True: 65 | yield from loader 66 | 67 | 68 | def _list_image_files_recursively(data_dir): 69 | results = [] 70 | for entry in sorted(bf.listdir(data_dir)): 71 | full_path = bf.join(data_dir, entry) 72 | ext = entry.split(".")[-1] 73 | if "." in entry and ext.lower() in ["jpg", "jpeg", "png", "gif"]: 74 | results.append(full_path) 75 | elif bf.isdir(full_path): 76 | results.extend(_list_image_files_recursively(full_path)) 77 | return results 78 | 79 | 80 | class ImageDataset(Dataset): 81 | def __init__( 82 | self, 83 | resolution, 84 | image_paths, 85 | classes=None, 86 | shard=0, 87 | num_shards=1, 88 | random_crop=False, 89 | random_flip=True, 90 | ): 91 | super().__init__() 92 | self.resolution = resolution 93 | self.local_images = image_paths[shard:][::num_shards] 94 | self.local_classes = None if classes is None else classes[shard:][::num_shards] 95 | self.random_crop = random_crop 96 | self.random_flip = random_flip 97 | 98 | def __len__(self): 99 | return len(self.local_images) 100 | 101 | def __getitem__(self, idx): 102 | path = self.local_images[idx] 103 | with bf.BlobFile(path, "rb") as f: 104 | pil_image = Image.open(f) 105 | pil_image.load() 106 | pil_image = pil_image.convert("RGB") 107 | 108 | if self.random_crop: 109 | arr = random_crop_arr(pil_image, self.resolution) 110 | else: 111 | arr = center_crop_arr(pil_image, self.resolution) 112 | 113 | if self.random_flip and random.random() < 0.5: 114 | arr = arr[:, ::-1] 115 | 116 | arr = arr.astype(np.float32) / 127.5 - 1 117 | 118 | out_dict = {} 119 | if self.local_classes is not None: 120 | out_dict["y"] = np.array(self.local_classes[idx], dtype=np.int64) 121 | return np.transpose(arr, [2, 0, 1]), out_dict 122 | 123 | 124 | def center_crop_arr(pil_image, image_size): 125 | # We are not on a new enough PIL to support the `reducing_gap` 126 | # argument, which uses BOX downsampling at powers of two first. 127 | # Thus, we do it by hand to improve downsample quality. 128 | while min(*pil_image.size) >= 2 * image_size: 129 | pil_image = pil_image.resize( 130 | tuple(x // 2 for x in pil_image.size), resample=Image.BOX 131 | ) 132 | 133 | scale = image_size / min(*pil_image.size) 134 | pil_image = pil_image.resize( 135 | tuple(round(x * scale) for x in pil_image.size), resample=Image.BICUBIC 136 | ) 137 | 138 | arr = np.array(pil_image) 139 | crop_y = (arr.shape[0] - image_size) // 2 140 | crop_x = (arr.shape[1] - image_size) // 2 141 | return arr[crop_y : crop_y + image_size, crop_x : crop_x + image_size] 142 | 143 | 144 | def random_crop_arr(pil_image, image_size, min_crop_frac=0.8, max_crop_frac=1.0): 145 | min_smaller_dim_size = math.ceil(image_size / max_crop_frac) 146 | max_smaller_dim_size = math.ceil(image_size / min_crop_frac) 147 | smaller_dim_size = random.randrange(min_smaller_dim_size, max_smaller_dim_size + 1) 148 | 149 | # We are not on a new enough PIL to support the `reducing_gap` 150 | # argument, which uses BOX downsampling at powers of two first. 151 | # Thus, we do it by hand to improve downsample quality. 152 | while min(*pil_image.size) >= 2 * smaller_dim_size: 153 | pil_image = pil_image.resize( 154 | tuple(x // 2 for x in pil_image.size), resample=Image.BOX 155 | ) 156 | 157 | scale = smaller_dim_size / min(*pil_image.size) 158 | pil_image = pil_image.resize( 159 | tuple(round(x * scale) for x in pil_image.size), resample=Image.BICUBIC 160 | ) 161 | 162 | arr = np.array(pil_image) 163 | crop_y = random.randrange(arr.shape[0] - image_size + 1) 164 | crop_x = random.randrange(arr.shape[1] - image_size + 1) 165 | return arr[crop_y : crop_y + image_size, crop_x : crop_x + image_size] 166 | -------------------------------------------------------------------------------- /patch_diffusion/logger.py: -------------------------------------------------------------------------------- 1 | """ 2 | Logger copied from OpenAI baselines to avoid extra RL-based dependencies: 3 | https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/logger.py 4 | """ 5 | 6 | import os 7 | import sys 8 | import shutil 9 | import os.path as osp 10 | import json 11 | import time 12 | import datetime 13 | import tempfile 14 | import warnings 15 | from collections import defaultdict 16 | from contextlib import contextmanager 17 | 18 | DEBUG = 10 19 | INFO = 20 20 | WARN = 30 21 | ERROR = 40 22 | 23 | DISABLED = 50 24 | 25 | 26 | class KVWriter(object): 27 | def writekvs(self, kvs): 28 | raise NotImplementedError 29 | 30 | 31 | class SeqWriter(object): 32 | def writeseq(self, seq): 33 | raise NotImplementedError 34 | 35 | 36 | class HumanOutputFormat(KVWriter, SeqWriter): 37 | def __init__(self, filename_or_file): 38 | if isinstance(filename_or_file, str): 39 | self.file = open(filename_or_file, "wt") 40 | self.own_file = True 41 | else: 42 | assert hasattr(filename_or_file, "read"), ( 43 | "expected file or str, got %s" % filename_or_file 44 | ) 45 | self.file = filename_or_file 46 | self.own_file = False 47 | 48 | def writekvs(self, kvs): 49 | # Create strings for printing 50 | key2str = {} 51 | for (key, val) in sorted(kvs.items()): 52 | if hasattr(val, "__float__"): 53 | valstr = "%-8.3g" % val 54 | else: 55 | valstr = str(val) 56 | key2str[self._truncate(key)] = self._truncate(valstr) 57 | 58 | # Find max widths 59 | if len(key2str) == 0: 60 | print("WARNING: tried to write empty key-value dict") 61 | return 62 | else: 63 | keywidth = max(map(len, key2str.keys())) 64 | valwidth = max(map(len, key2str.values())) 65 | 66 | # Write out the data 67 | dashes = "-" * (keywidth + valwidth + 7) 68 | lines = [dashes] 69 | for (key, val) in sorted(key2str.items(), key=lambda kv: kv[0].lower()): 70 | lines.append( 71 | "| %s%s | %s%s |" 72 | % (key, " " * (keywidth - len(key)), val, " " * (valwidth - len(val))) 73 | ) 74 | lines.append(dashes) 75 | self.file.write("\n".join(lines) + "\n") 76 | 77 | # Flush the output to the file 78 | self.file.flush() 79 | 80 | def _truncate(self, s): 81 | maxlen = 30 82 | return s[: maxlen - 3] + "..." if len(s) > maxlen else s 83 | 84 | def writeseq(self, seq): 85 | seq = list(seq) 86 | for (i, elem) in enumerate(seq): 87 | self.file.write(elem) 88 | if i < len(seq) - 1: # add space unless this is the last one 89 | self.file.write(" ") 90 | self.file.write("\n") 91 | self.file.flush() 92 | 93 | def close(self): 94 | if self.own_file: 95 | self.file.close() 96 | 97 | 98 | class JSONOutputFormat(KVWriter): 99 | def __init__(self, filename): 100 | self.file = open(filename, "wt") 101 | 102 | def writekvs(self, kvs): 103 | for k, v in sorted(kvs.items()): 104 | if hasattr(v, "dtype"): 105 | kvs[k] = float(v) 106 | self.file.write(json.dumps(kvs) + "\n") 107 | self.file.flush() 108 | 109 | def close(self): 110 | self.file.close() 111 | 112 | 113 | class CSVOutputFormat(KVWriter): 114 | def __init__(self, filename): 115 | self.file = open(filename, "w+t") 116 | self.keys = [] 117 | self.sep = "," 118 | 119 | def writekvs(self, kvs): 120 | # Add our current row to the history 121 | extra_keys = list(kvs.keys() - self.keys) 122 | extra_keys.sort() 123 | if extra_keys: 124 | self.keys.extend(extra_keys) 125 | self.file.seek(0) 126 | lines = self.file.readlines() 127 | self.file.seek(0) 128 | for (i, k) in enumerate(self.keys): 129 | if i > 0: 130 | self.file.write(",") 131 | self.file.write(k) 132 | self.file.write("\n") 133 | for line in lines[1:]: 134 | self.file.write(line[:-1]) 135 | self.file.write(self.sep * len(extra_keys)) 136 | self.file.write("\n") 137 | for (i, k) in enumerate(self.keys): 138 | if i > 0: 139 | self.file.write(",") 140 | v = kvs.get(k) 141 | if v is not None: 142 | self.file.write(str(v)) 143 | self.file.write("\n") 144 | self.file.flush() 145 | 146 | def close(self): 147 | self.file.close() 148 | 149 | 150 | class TensorBoardOutputFormat(KVWriter): 151 | """ 152 | Dumps key/value pairs into TensorBoard's numeric format. 153 | """ 154 | 155 | def __init__(self, dir): 156 | os.makedirs(dir, exist_ok=True) 157 | self.dir = dir 158 | self.step = 1 159 | prefix = "events" 160 | path = osp.join(osp.abspath(dir), prefix) 161 | import tensorflow as tf 162 | from tensorflow.python import pywrap_tensorflow 163 | from tensorflow.core.util import event_pb2 164 | from tensorflow.python.util import compat 165 | 166 | self.tf = tf 167 | self.event_pb2 = event_pb2 168 | self.pywrap_tensorflow = pywrap_tensorflow 169 | self.writer = pywrap_tensorflow.EventsWriter(compat.as_bytes(path)) 170 | 171 | def writekvs(self, kvs): 172 | def summary_val(k, v): 173 | kwargs = {"tag": k, "simple_value": float(v)} 174 | return self.tf.Summary.Value(**kwargs) 175 | 176 | summary = self.tf.Summary(value=[summary_val(k, v) for k, v in kvs.items()]) 177 | event = self.event_pb2.Event(wall_time=time.time(), summary=summary) 178 | event.step = ( 179 | self.step 180 | ) # is there any reason why you'd want to specify the step? 181 | self.writer.WriteEvent(event) 182 | self.writer.Flush() 183 | self.step += 1 184 | 185 | def close(self): 186 | if self.writer: 187 | self.writer.Close() 188 | self.writer = None 189 | 190 | 191 | def make_output_format(format, ev_dir, log_suffix=""): 192 | os.makedirs(ev_dir, exist_ok=True) 193 | if format == "stdout": 194 | return HumanOutputFormat(sys.stdout) 195 | elif format == "log": 196 | return HumanOutputFormat(osp.join(ev_dir, "log%s.txt" % log_suffix)) 197 | elif format == "json": 198 | return JSONOutputFormat(osp.join(ev_dir, "progress%s.json" % log_suffix)) 199 | elif format == "csv": 200 | return CSVOutputFormat(osp.join(ev_dir, "progress%s.csv" % log_suffix)) 201 | elif format == "tensorboard": 202 | return TensorBoardOutputFormat(osp.join(ev_dir, "tb%s" % log_suffix)) 203 | else: 204 | raise ValueError("Unknown format specified: %s" % (format,)) 205 | 206 | 207 | # ================================================================ 208 | # API 209 | # ================================================================ 210 | 211 | 212 | def logkv(key, val): 213 | """ 214 | Log a value of some diagnostic 215 | Call this once for each diagnostic quantity, each iteration 216 | If called many times, last value will be used. 217 | """ 218 | get_current().logkv(key, val) 219 | 220 | 221 | def logkv_mean(key, val): 222 | """ 223 | The same as logkv(), but if called many times, values averaged. 224 | """ 225 | get_current().logkv_mean(key, val) 226 | 227 | 228 | def logkvs(d): 229 | """ 230 | Log a dictionary of key-value pairs 231 | """ 232 | for (k, v) in d.items(): 233 | logkv(k, v) 234 | 235 | 236 | def dumpkvs(): 237 | """ 238 | Write all of the diagnostics from the current iteration 239 | """ 240 | return get_current().dumpkvs() 241 | 242 | 243 | def getkvs(): 244 | return get_current().name2val 245 | 246 | 247 | def log(*args, level=INFO): 248 | """ 249 | Write the sequence of args, with no separators, to the console and output files (if you've configured an output file). 250 | """ 251 | get_current().log(*args, level=level) 252 | 253 | 254 | def debug(*args): 255 | log(*args, level=DEBUG) 256 | 257 | 258 | def info(*args): 259 | log(*args, level=INFO) 260 | 261 | 262 | def warn(*args): 263 | log(*args, level=WARN) 264 | 265 | 266 | def error(*args): 267 | log(*args, level=ERROR) 268 | 269 | 270 | def set_level(level): 271 | """ 272 | Set logging threshold on current logger. 273 | """ 274 | get_current().set_level(level) 275 | 276 | 277 | def set_comm(comm): 278 | get_current().set_comm(comm) 279 | 280 | 281 | def get_dir(): 282 | """ 283 | Get directory that log files are being written to. 284 | will be None if there is no output directory (i.e., if you didn't call start) 285 | """ 286 | return get_current().get_dir() 287 | 288 | 289 | record_tabular = logkv 290 | dump_tabular = dumpkvs 291 | 292 | 293 | @contextmanager 294 | def profile_kv(scopename): 295 | logkey = "wait_" + scopename 296 | tstart = time.time() 297 | try: 298 | yield 299 | finally: 300 | get_current().name2val[logkey] += time.time() - tstart 301 | 302 | 303 | def profile(n): 304 | """ 305 | Usage: 306 | @profile("my_func") 307 | def my_func(): code 308 | """ 309 | 310 | def decorator_with_name(func): 311 | def func_wrapper(*args, **kwargs): 312 | with profile_kv(n): 313 | return func(*args, **kwargs) 314 | 315 | return func_wrapper 316 | 317 | return decorator_with_name 318 | 319 | 320 | # ================================================================ 321 | # Backend 322 | # ================================================================ 323 | 324 | 325 | def get_current(): 326 | if Logger.CURRENT is None: 327 | _configure_default_logger() 328 | 329 | return Logger.CURRENT 330 | 331 | 332 | class Logger(object): 333 | DEFAULT = None # A logger with no output files. (See right below class definition) 334 | # So that you can still log to the terminal without setting up any output files 335 | CURRENT = None # Current logger being used by the free functions above 336 | 337 | def __init__(self, dir, output_formats, comm=None): 338 | self.name2val = defaultdict(float) # values this iteration 339 | self.name2cnt = defaultdict(int) 340 | self.level = INFO 341 | self.dir = dir 342 | self.output_formats = output_formats 343 | self.comm = comm 344 | 345 | # Logging API, forwarded 346 | # ---------------------------------------- 347 | def logkv(self, key, val): 348 | self.name2val[key] = val 349 | 350 | def logkv_mean(self, key, val): 351 | oldval, cnt = self.name2val[key], self.name2cnt[key] 352 | self.name2val[key] = oldval * cnt / (cnt + 1) + val / (cnt + 1) 353 | self.name2cnt[key] = cnt + 1 354 | 355 | def dumpkvs(self): 356 | if self.comm is None: 357 | d = self.name2val 358 | else: 359 | d = mpi_weighted_mean( 360 | self.comm, 361 | { 362 | name: (val, self.name2cnt.get(name, 1)) 363 | for (name, val) in self.name2val.items() 364 | }, 365 | ) 366 | if self.comm.rank != 0: 367 | d["dummy"] = 1 # so we don't get a warning about empty dict 368 | out = d.copy() # Return the dict for unit testing purposes 369 | for fmt in self.output_formats: 370 | if isinstance(fmt, KVWriter): 371 | fmt.writekvs(d) 372 | self.name2val.clear() 373 | self.name2cnt.clear() 374 | return out 375 | 376 | def log(self, *args, level=INFO): 377 | if self.level <= level: 378 | self._do_log(args) 379 | 380 | # Configuration 381 | # ---------------------------------------- 382 | def set_level(self, level): 383 | self.level = level 384 | 385 | def set_comm(self, comm): 386 | self.comm = comm 387 | 388 | def get_dir(self): 389 | return self.dir 390 | 391 | def close(self): 392 | for fmt in self.output_formats: 393 | fmt.close() 394 | 395 | # Misc 396 | # ---------------------------------------- 397 | def _do_log(self, args): 398 | for fmt in self.output_formats: 399 | if isinstance(fmt, SeqWriter): 400 | fmt.writeseq(map(str, args)) 401 | 402 | 403 | def get_rank_without_mpi_import(): 404 | # check environment variables here instead of importing mpi4py 405 | # to avoid calling MPI_Init() when this module is imported 406 | for varname in ["PMI_RANK", "OMPI_COMM_WORLD_RANK"]: 407 | if varname in os.environ: 408 | return int(os.environ[varname]) 409 | return 0 410 | 411 | 412 | def mpi_weighted_mean(comm, local_name2valcount): 413 | """ 414 | Copied from: https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/mpi_util.py#L110 415 | Perform a weighted average over dicts that are each on a different node 416 | Input: local_name2valcount: dict mapping key -> (value, count) 417 | Returns: key -> mean 418 | """ 419 | all_name2valcount = comm.gather(local_name2valcount) 420 | if comm.rank == 0: 421 | name2sum = defaultdict(float) 422 | name2count = defaultdict(float) 423 | for n2vc in all_name2valcount: 424 | for (name, (val, count)) in n2vc.items(): 425 | try: 426 | val = float(val) 427 | except ValueError: 428 | if comm.rank == 0: 429 | warnings.warn( 430 | "WARNING: tried to compute mean on non-float {}={}".format( 431 | name, val 432 | ) 433 | ) 434 | else: 435 | name2sum[name] += val * count 436 | name2count[name] += count 437 | return {name: name2sum[name] / name2count[name] for name in name2sum} 438 | else: 439 | return {} 440 | 441 | 442 | def configure(dir=None, format_strs=None, comm=None, log_suffix=""): 443 | """ 444 | If comm is provided, average all numerical stats across that comm 445 | """ 446 | if dir is None: 447 | dir = os.getenv("OPENAI_LOGDIR") 448 | if dir is None: 449 | dir = osp.join( 450 | tempfile.gettempdir(), 451 | datetime.datetime.now().strftime("openai-%Y-%m-%d-%H-%M-%S-%f"), 452 | ) 453 | assert isinstance(dir, str) 454 | dir = os.path.expanduser(dir) 455 | os.makedirs(os.path.expanduser(dir), exist_ok=True) 456 | 457 | rank = get_rank_without_mpi_import() 458 | if rank > 0: 459 | log_suffix = log_suffix + "-rank%03i" % rank 460 | 461 | if format_strs is None: 462 | if rank == 0: 463 | format_strs = os.getenv("OPENAI_LOG_FORMAT", "stdout,log,csv").split(",") 464 | else: 465 | format_strs = os.getenv("OPENAI_LOG_FORMAT_MPI", "log").split(",") 466 | format_strs = filter(None, format_strs) 467 | output_formats = [make_output_format(f, dir, log_suffix) for f in format_strs] 468 | 469 | Logger.CURRENT = Logger(dir=dir, output_formats=output_formats, comm=comm) 470 | if output_formats: 471 | log("Logging to %s" % dir) 472 | 473 | 474 | def _configure_default_logger(): 475 | configure() 476 | Logger.DEFAULT = Logger.CURRENT 477 | 478 | 479 | def reset(): 480 | if Logger.CURRENT is not Logger.DEFAULT: 481 | Logger.CURRENT.close() 482 | Logger.CURRENT = Logger.DEFAULT 483 | log("Reset logger") 484 | 485 | 486 | @contextmanager 487 | def scoped_configure(dir=None, format_strs=None, comm=None): 488 | prevlogger = Logger.CURRENT 489 | configure(dir=dir, format_strs=format_strs, comm=comm) 490 | try: 491 | yield 492 | finally: 493 | Logger.CURRENT.close() 494 | Logger.CURRENT = prevlogger 495 | 496 | -------------------------------------------------------------------------------- /patch_diffusion/losses.py: -------------------------------------------------------------------------------- 1 | """ 2 | Helpers for various likelihood-based losses. These are ported from the original 3 | Ho et al. diffusion models codebase: 4 | https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/utils.py 5 | """ 6 | 7 | import numpy as np 8 | 9 | import torch as th 10 | 11 | 12 | def normal_kl(mean1, logvar1, mean2, logvar2): 13 | """ 14 | Compute the KL divergence between two gaussians. 15 | 16 | Shapes are automatically broadcasted, so batches can be compared to 17 | scalars, among other use cases. 18 | """ 19 | tensor = None 20 | for obj in (mean1, logvar1, mean2, logvar2): 21 | if isinstance(obj, th.Tensor): 22 | tensor = obj 23 | break 24 | assert tensor is not None, "at least one argument must be a Tensor" 25 | 26 | # Force variances to be Tensors. Broadcasting helps convert scalars to 27 | # Tensors, but it does not work for th.exp(). 28 | logvar1, logvar2 = [ 29 | x if isinstance(x, th.Tensor) else th.tensor(x).to(tensor) 30 | for x in (logvar1, logvar2) 31 | ] 32 | 33 | return 0.5 * ( 34 | -1.0 35 | + logvar2 36 | - logvar1 37 | + th.exp(logvar1 - logvar2) 38 | + ((mean1 - mean2) ** 2) * th.exp(-logvar2) 39 | ) 40 | 41 | 42 | def approx_standard_normal_cdf(x): 43 | """ 44 | A fast approximation of the cumulative distribution function of the 45 | standard normal. 46 | """ 47 | return 0.5 * (1.0 + th.tanh(np.sqrt(2.0 / np.pi) * (x + 0.044715 * th.pow(x, 3)))) 48 | 49 | 50 | def discretized_gaussian_log_likelihood(x, *, means, log_scales): 51 | """ 52 | Compute the log-likelihood of a Gaussian distribution discretizing to a 53 | given image. 54 | 55 | :param x: the target images. It is assumed that this was uint8 values, 56 | rescaled to the range [-1, 1]. 57 | :param means: the Gaussian mean Tensor. 58 | :param log_scales: the Gaussian log stddev Tensor. 59 | :return: a tensor like x of log probabilities (in nats). 60 | """ 61 | assert x.shape == means.shape == log_scales.shape 62 | centered_x = x - means 63 | inv_stdv = th.exp(-log_scales) 64 | plus_in = inv_stdv * (centered_x + 1.0 / 255.0) 65 | cdf_plus = approx_standard_normal_cdf(plus_in) 66 | min_in = inv_stdv * (centered_x - 1.0 / 255.0) 67 | cdf_min = approx_standard_normal_cdf(min_in) 68 | log_cdf_plus = th.log(cdf_plus.clamp(min=1e-12)) 69 | log_one_minus_cdf_min = th.log((1.0 - cdf_min).clamp(min=1e-12)) 70 | cdf_delta = cdf_plus - cdf_min 71 | log_probs = th.where( 72 | x < -0.999, 73 | log_cdf_plus, 74 | th.where(x > 0.999, log_one_minus_cdf_min, th.log(cdf_delta.clamp(min=1e-12))), 75 | ) 76 | assert log_probs.shape == x.shape 77 | return log_probs 78 | -------------------------------------------------------------------------------- /patch_diffusion/nn.py: -------------------------------------------------------------------------------- 1 | """ 2 | Various utilities for neural networks. 3 | """ 4 | 5 | import math 6 | 7 | import torch as th 8 | import torch.nn as nn 9 | 10 | 11 | # PyTorch 1.7 has SiLU, but we support PyTorch 1.5. 12 | class SiLU(nn.Module): 13 | def forward(self, x): 14 | return x * th.sigmoid(x) 15 | 16 | 17 | class GroupNorm32(nn.GroupNorm): 18 | def forward(self, x): 19 | return super().forward(x.float()).type(x.dtype) 20 | 21 | def conv_nd(dims, *args, **kwargs): 22 | """ 23 | Create a 1D, 2D, or 3D convolution module. 24 | """ 25 | if dims == 1: 26 | return nn.Conv1d(*args, **kwargs) 27 | elif dims == 2: 28 | return nn.Conv2d(*args, **kwargs) 29 | elif dims == 3: 30 | return nn.Conv3d(*args, **kwargs) 31 | raise ValueError(f"unsupported dimensions: {dims}") 32 | 33 | 34 | def linear(*args, **kwargs): 35 | """ 36 | Create a linear module. 37 | """ 38 | return nn.Linear(*args, **kwargs) 39 | 40 | 41 | def avg_pool_nd(dims, *args, **kwargs): 42 | """ 43 | Create a 1D, 2D, or 3D average pooling module. 44 | """ 45 | if dims == 1: 46 | return nn.AvgPool1d(*args, **kwargs) 47 | elif dims == 2: 48 | return nn.AvgPool2d(*args, **kwargs) 49 | elif dims == 3: 50 | return nn.AvgPool3d(*args, **kwargs) 51 | raise ValueError(f"unsupported dimensions: {dims}") 52 | 53 | 54 | def update_ema(target_params, source_params, rate=0.99): 55 | """ 56 | Update target parameters to be closer to those of source parameters using 57 | an exponential moving average. 58 | 59 | :param target_params: the target parameter sequence. 60 | :param source_params: the source parameter sequence. 61 | :param rate: the EMA rate (closer to 1 means slower). 62 | """ 63 | for targ, src in zip(target_params, source_params): 64 | targ.detach().mul_(rate).add_(src, alpha=1 - rate) 65 | 66 | 67 | def zero_module(module): 68 | """ 69 | Zero out the parameters of a module and return it. 70 | """ 71 | for p in module.parameters(): 72 | p.detach().zero_() 73 | return module 74 | 75 | 76 | def scale_module(module, scale): 77 | """ 78 | Scale the parameters of a module and return it. 79 | """ 80 | for p in module.parameters(): 81 | p.detach().mul_(scale) 82 | return module 83 | 84 | 85 | def mean_flat(tensor): 86 | """ 87 | Take the mean over all non-batch dimensions. 88 | """ 89 | return tensor.mean(dim=list(range(1, len(tensor.shape)))) 90 | 91 | 92 | def normalization(channels): 93 | """ 94 | Make a standard normalization layer. 95 | 96 | :param channels: number of input channels. 97 | :return: an nn.Module for normalization. 98 | """ 99 | return GroupNorm32(32, channels) 100 | 101 | 102 | def timestep_embedding(timesteps, dim, max_period=10000): 103 | """ 104 | Create sinusoidal timestep embeddings. 105 | 106 | :param timesteps: a 1-D Tensor of N indices, one per batch element. 107 | These may be fractional. 108 | :param dim: the dimension of the output. 109 | :param max_period: controls the minimum frequency of the embeddings. 110 | 111 | :return: an [N x dim] Tensor of positional embeddings. 112 | 113 | We changed the timestep_embedding function a little bit from ADM's to match the implementation at https://github.com/hojonathanho/diffusion/blob/master/diffusion_tf/nn.py 114 | """ 115 | half = dim // 2 116 | freqs = th.exp( 117 | -math.log(max_period) * th.arange(start=0, end=half, dtype=th.float32) /(half - 1) 118 | ).to(device=timesteps.device) 119 | args = timesteps[:, None].float() * freqs[None] 120 | embedding = th.cat([th.sin(args), th.cos(args)], dim=-1) 121 | 122 | if dim % 2: 123 | embedding = th.cat([embedding, th.zeros_like(embedding[:, :1])], dim=-1) 124 | return embedding 125 | 126 | def timestep_embedding_2d(tens, dim, resolution): 127 | """ 128 | Creates 2-dimensional sinusoidal timestep embeddings. 129 | 130 | :param tens: any tensor that's placed on the correct device. we used the timesteps tensor. 131 | :param dim: the dimension of the output. 132 | :param resolution: the resolution of the data we want to add positional encoding to. 133 | :return: an [dim x resolution x resolution] Tensor of 2-d positional embeddings. 134 | 135 | """ 136 | omega = 64 / resolution #higher resolutions need longer wavelengths 137 | half_dim = dim // 2 138 | arange = th.arange(start=0, end=resolution, dtype=th.float32) 139 | 140 | emb = (th.log(th.tensor(10000)) / (half_dim - 1)).float() 141 | emb = th.exp(th.arange(start=0, end=half_dim, dtype=th.float32) * -emb) 142 | 143 | emb = arange[:, None] * emb[None, :] 144 | emb = th.sin(emb * omega) 145 | 146 | emb_x = th.repeat_interleave(emb[None, ...], resolution, dim=0) 147 | emb_y = th.repeat_interleave(emb[:, None, :], resolution, dim=1) 148 | emb = th.cat([emb_x, emb_y], dim=-1) 149 | return emb[None, ...].to(tens.device).float().permute(0, 3, 1, 2) 150 | 151 | def checkpoint(func, inputs, params, flag): 152 | """ 153 | Evaluate a function without caching intermediate activations, allowing for 154 | reduced memory at the expense of extra compute in the backward pass. 155 | 156 | :param func: the function to evaluate. 157 | :param inputs: the argument sequence to pass to `func`. 158 | :param params: a sequence of parameters `func` depends on but does not 159 | explicitly take as arguments. 160 | :param flag: if False, disable gradient checkpointing. 161 | """ 162 | if flag: 163 | args = tuple(inputs) + tuple(params) 164 | return CheckpointFunction.apply(func, len(inputs), *args) 165 | else: 166 | return func(*inputs) 167 | 168 | 169 | class CheckpointFunction(th.autograd.Function): 170 | @staticmethod 171 | def forward(ctx, run_function, length, *args): 172 | ctx.run_function = run_function 173 | ctx.input_tensors = list(args[:length]) 174 | ctx.input_params = list(args[length:]) 175 | with th.no_grad(): 176 | output_tensors = ctx.run_function(*ctx.input_tensors) 177 | return output_tensors 178 | 179 | @staticmethod 180 | def backward(ctx, *output_grads): 181 | ctx.input_tensors = [x.detach().requires_grad_(True) for x in ctx.input_tensors] 182 | with th.enable_grad(): 183 | # Fixes a bug where the first op in run_function modifies the 184 | # Tensor storage in place, which is not allowed for detach()'d 185 | # Tensors. 186 | shallow_copies = [x.view_as(x) for x in ctx.input_tensors] 187 | output_tensors = ctx.run_function(*shallow_copies) 188 | input_grads = th.autograd.grad( 189 | output_tensors, 190 | ctx.input_tensors + ctx.input_params, 191 | output_grads, 192 | allow_unused=True, 193 | ) 194 | del ctx.input_tensors 195 | del ctx.input_params 196 | del output_tensors 197 | return (None, None) + input_grads 198 | 199 | -------------------------------------------------------------------------------- /patch_diffusion/resample.py: -------------------------------------------------------------------------------- 1 | from abc import ABC, abstractmethod 2 | from asyncio import start_server 3 | 4 | import numpy as np 5 | import torch as th 6 | import torch.distributed as dist 7 | 8 | 9 | def create_named_schedule_sampler(name, diffusion): 10 | """ 11 | Create a ScheduleSampler from a library of pre-defined samplers. 12 | 13 | :param name: the name of the sampler. 14 | :param diffusion: the diffusion object to sample for. 15 | """ 16 | if name == "uniform": 17 | return UniformSampler(diffusion) 18 | elif name.startswith("uniform_split_"): 19 | return UniformSplitSampler(diffusion, split_num=int(name[-1])) 20 | elif name == "loss-second-moment": 21 | return LossSecondMomentResampler(diffusion) 22 | else: 23 | raise NotImplementedError(f"unknown schedule sampler: {name}") 24 | 25 | 26 | class ScheduleSampler(ABC): 27 | """ 28 | A distribution over timesteps in the diffusion process, intended to reduce 29 | variance of the objective. 30 | 31 | By default, samplers perform unbiased importance sampling, in which the 32 | objective's mean is unchanged. 33 | However, subclasses may override sample() to change how the resampled 34 | terms are reweighted, allowing for actual changes in the objective. 35 | """ 36 | 37 | @abstractmethod 38 | def weights(self): 39 | """ 40 | Get a numpy array of weights, one per diffusion step. 41 | 42 | The weights needn't be normalized, but must be positive. 43 | """ 44 | 45 | def sample(self, batch_size, device): 46 | """ 47 | Importance-sample timesteps for a batch. 48 | 49 | :param batch_size: the number of timesteps. 50 | :param device: the torch device to save to. 51 | :return: a tuple (timesteps, weights): 52 | - timesteps: a tensor of timestep indices. 53 | - weights: a tensor of weights to scale the resulting losses. 54 | """ 55 | w = self.weights() 56 | p = w / np.sum(w) 57 | indices_np = np.random.choice(len(p), size=(batch_size,), p=p) 58 | indices = th.from_numpy(indices_np).long().to(device) 59 | weights_np = 1 / (len(p) * p[indices_np]) 60 | weights = th.from_numpy(weights_np).float().to(device) 61 | return indices, weights 62 | 63 | 64 | class UniformSampler(ScheduleSampler): 65 | def __init__(self, diffusion): 66 | self.diffusion = diffusion 67 | self._weights = np.ones([diffusion.num_timesteps]) 68 | 69 | def weights(self): 70 | return self._weights 71 | 72 | class UniformSplitSampler(ScheduleSampler): 73 | def __init__(self, diffusion, split_num): 74 | self.diffusion = diffusion 75 | chosen_timesteps = [ 76 | diffusion.snr_splits[split_num] > diffusion.snrs[i] >= diffusion.snr_splits[split_num+1] for i in range(diffusion.num_timesteps) 77 | ] 78 | self._weights = np.array(chosen_timesteps).astype('float32') 79 | 80 | 81 | def weights(self): 82 | return self._weights 83 | 84 | class LossAwareSampler(ScheduleSampler): 85 | def update_with_local_losses(self, local_ts, local_losses): 86 | """ 87 | Update the reweighting using losses from a model. 88 | 89 | Call this method from each rank with a batch of timesteps and the 90 | corresponding losses for each of those timesteps. 91 | This method will perform synchronization to make sure all of the ranks 92 | maintain the exact same reweighting. 93 | 94 | :param local_ts: an integer Tensor of timesteps. 95 | :param local_losses: a 1D Tensor of losses. 96 | """ 97 | batch_sizes = [ 98 | th.tensor([0], dtype=th.int32, device=local_ts.device) 99 | for _ in range(dist.get_world_size()) 100 | ] 101 | dist.all_gather( 102 | batch_sizes, 103 | th.tensor([len(local_ts)], dtype=th.int32, device=local_ts.device), 104 | ) 105 | 106 | # Pad all_gather batches to be the maximum batch size. 107 | batch_sizes = [x.item() for x in batch_sizes] 108 | max_bs = max(batch_sizes) 109 | 110 | timestep_batches = [th.zeros(max_bs).to(local_ts) for bs in batch_sizes] 111 | loss_batches = [th.zeros(max_bs).to(local_losses) for bs in batch_sizes] 112 | dist.all_gather(timestep_batches, local_ts) 113 | dist.all_gather(loss_batches, local_losses) 114 | timesteps = [ 115 | x.item() for y, bs in zip(timestep_batches, batch_sizes) for x in y[:bs] 116 | ] 117 | losses = [x.item() for y, bs in zip(loss_batches, batch_sizes) for x in y[:bs]] 118 | self.update_with_all_losses(timesteps, losses) 119 | 120 | @abstractmethod 121 | def update_with_all_losses(self, ts, losses): 122 | """ 123 | Update the reweighting using losses from a model. 124 | 125 | Sub-classes should override this method to update the reweighting 126 | using losses from the model. 127 | 128 | This method directly updates the reweighting without synchronizing 129 | between workers. It is called by update_with_local_losses from all 130 | ranks with identical arguments. Thus, it should have deterministic 131 | behavior to maintain state across workers. 132 | 133 | :param ts: a list of int timesteps. 134 | :param losses: a list of float losses, one per timestep. 135 | """ 136 | 137 | 138 | class LossSecondMomentResampler(LossAwareSampler): 139 | def __init__(self, diffusion, history_per_term=10, uniform_prob=0.001): 140 | self.diffusion = diffusion 141 | self.history_per_term = history_per_term 142 | self.uniform_prob = uniform_prob 143 | self._loss_history = np.zeros( 144 | [diffusion.num_timesteps, history_per_term], dtype=np.float64 145 | ) 146 | self._loss_counts = np.zeros([diffusion.num_timesteps], dtype=np.int) 147 | 148 | def weights(self): 149 | if not self._warmed_up(): 150 | return np.ones([self.diffusion.num_timesteps], dtype=np.float64) 151 | weights = np.sqrt(np.mean(self._loss_history ** 2, axis=-1)) 152 | weights /= np.sum(weights) 153 | weights *= 1 - self.uniform_prob 154 | weights += self.uniform_prob / len(weights) 155 | return weights 156 | 157 | def update_with_all_losses(self, ts, losses): 158 | for t, loss in zip(ts, losses): 159 | if self._loss_counts[t] == self.history_per_term: 160 | # Shift out the oldest loss term. 161 | self._loss_history[t, :-1] = self._loss_history[t, 1:] 162 | self._loss_history[t, -1] = loss 163 | else: 164 | self._loss_history[t, self._loss_counts[t]] = loss 165 | self._loss_counts[t] += 1 166 | 167 | def _warmed_up(self): 168 | return (self._loss_counts == self.history_per_term).all() 169 | -------------------------------------------------------------------------------- /patch_diffusion/respace.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch as th 3 | 4 | from .gaussian_diffusion import GaussianDiffusion 5 | 6 | 7 | def space_timesteps(num_timesteps, section_counts): 8 | """ 9 | Create a list of timesteps to use from an original diffusion process, 10 | given the number of timesteps we want to take from equally-sized portions 11 | of the original process. 12 | 13 | For example, if there's 300 timesteps and the section counts are [10,15,20] 14 | then the first 100 timesteps are strided to be 10 timesteps, the second 100 15 | are strided to be 15 timesteps, and the final 100 are strided to be 20. 16 | 17 | If the stride is a string starting with "ddim", then the fixed striding 18 | from the DDIM paper is used, and only one section is allowed. 19 | 20 | :param num_timesteps: the number of diffusion steps in the original 21 | process to divide up. 22 | :param section_counts: either a list of numbers, or a string containing 23 | comma-separated numbers, indicating the step count 24 | per section. As a special case, use "ddimN" where N 25 | is a number of steps to use the striding from the 26 | DDIM paper. 27 | :return: a set of diffusion steps from the original process to use. 28 | """ 29 | if isinstance(section_counts, str): 30 | if section_counts.startswith("ddim"): 31 | desired_count = int(section_counts[len("ddim") :]) 32 | for i in range(1, num_timesteps): 33 | if len(range(0, num_timesteps, i)) == desired_count: 34 | return set(range(0, num_timesteps, i)) 35 | raise ValueError( 36 | f"cannot create exactly {num_timesteps} steps with an integer stride" 37 | ) 38 | section_counts = [int(x) for x in section_counts.split(",")] 39 | size_per = num_timesteps // len(section_counts) 40 | extra = num_timesteps % len(section_counts) 41 | start_idx = 0 42 | all_steps = [] 43 | for i, section_count in enumerate(section_counts): 44 | size = size_per + (1 if i < extra else 0) 45 | if size < section_count: 46 | raise ValueError( 47 | f"cannot divide section of {size} steps into {section_count}" 48 | ) 49 | if section_count <= 1: 50 | frac_stride = 1 51 | else: 52 | frac_stride = (size - 1) / (section_count - 1) 53 | cur_idx = 0.0 54 | taken_steps = [] 55 | for _ in range(section_count): 56 | taken_steps.append(start_idx + round(cur_idx)) 57 | cur_idx += frac_stride 58 | all_steps += taken_steps 59 | start_idx += size 60 | return set(all_steps) 61 | 62 | 63 | class SpacedDiffusion(GaussianDiffusion): 64 | """ 65 | A diffusion process which can skip steps in a base diffusion process. 66 | 67 | :param use_timesteps: a collection (sequence or set) of timesteps from the 68 | original diffusion process to retain. 69 | :param kwargs: the kwargs to create the base diffusion process. 70 | """ 71 | 72 | def __init__(self, use_timesteps, **kwargs): 73 | self.use_timesteps = set(use_timesteps) 74 | self.timestep_map = [] 75 | self.original_num_steps = len(kwargs["betas"]) 76 | 77 | base_diffusion = GaussianDiffusion(**kwargs) # pylint: disable=missing-kwoa 78 | last_alpha_cumprod = 1.0 79 | new_betas = [] 80 | for i, alpha_cumprod in enumerate(base_diffusion.alphas_cumprod): 81 | if i in self.use_timesteps: 82 | new_betas.append(1 - alpha_cumprod / last_alpha_cumprod) 83 | last_alpha_cumprod = alpha_cumprod 84 | self.timestep_map.append(i) 85 | kwargs["betas"] = np.array(new_betas) 86 | super().__init__(**kwargs) 87 | 88 | self.base_diffusion = base_diffusion 89 | 90 | def p_mean_variance( 91 | self, model, *args, **kwargs 92 | ): # pylint: disable=signature-differs 93 | return super().p_mean_variance(self._wrap_model(model), *args, **kwargs) 94 | 95 | def training_losses( 96 | self, model, *args, **kwargs 97 | ): # pylint: disable=signature-differs 98 | return super().training_losses(self._wrap_model(model), *args, **kwargs) 99 | 100 | def condition_mean(self, cond_fn, *args, **kwargs): 101 | return super().condition_mean(self._wrap_model(cond_fn), *args, **kwargs) 102 | 103 | def condition_score(self, cond_fn, *args, **kwargs): 104 | return super().condition_score(self._wrap_model(cond_fn), *args, **kwargs) 105 | 106 | def _wrap_model(self, model): 107 | if isinstance(model, _WrappedModel): 108 | return model 109 | return _WrappedModel( 110 | model, self.timestep_map, self.rescale_timesteps, self.original_num_steps 111 | ) 112 | 113 | def _scale_timesteps(self, t): 114 | # Scaling is done by the wrapped model. 115 | return t 116 | 117 | 118 | class _WrappedModel: 119 | def __init__(self, model, timestep_map, rescale_timesteps, original_num_steps): 120 | self.model = model 121 | self.timestep_map = timestep_map 122 | self.rescale_timesteps = rescale_timesteps 123 | self.original_num_steps = original_num_steps 124 | 125 | def __call__(self, x, ts, **kwargs): 126 | map_tensor = th.tensor(self.timestep_map, device=ts.device, dtype=ts.dtype) 127 | new_ts = map_tensor[ts] 128 | if self.rescale_timesteps: 129 | new_ts = new_ts.float() * (1000.0 / self.original_num_steps) 130 | return self.model(x, new_ts, **kwargs) 131 | -------------------------------------------------------------------------------- /patch_diffusion/script_util.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import inspect 3 | 4 | from . import gaussian_diffusion as gd 5 | from .respace import SpacedDiffusion, space_timesteps 6 | from .unet import SuperResModel, UNetModel, EncoderUNetModel 7 | 8 | NUM_CLASSES = 1000 9 | 10 | def diffusion_defaults(): 11 | """ 12 | Defaults for image and classifier training. 13 | We set the default output to xstart -> eps is difficult to predict at very low SNR levels 14 | """ 15 | return dict( 16 | learn_sigma=False, 17 | diffusion_steps=1000, 18 | noise_schedule="linear", 19 | timestep_respacing="", 20 | use_kl=False, 21 | model_mean_type="xstart", 22 | rescale_timesteps=False, 23 | rescale_learned_sigmas=False, 24 | snr_splits="", 25 | weight_schedule="sqrt_snr" 26 | ) 27 | 28 | 29 | def classifier_defaults(): 30 | """ 31 | Defaults for classifier models. 32 | """ 33 | return dict( 34 | image_size=64, 35 | classifier_use_fp16=False, 36 | classifier_width=128, 37 | classifier_depth=2, 38 | classifier_attention_resolutions="32,16,8", # 16 39 | classifier_use_scale_shift_norm=True, # False 40 | classifier_resblock_updown=True, # False 41 | classifier_pool="attention", 42 | ) 43 | 44 | 45 | def model_and_diffusion_defaults(): 46 | """ 47 | Defaults for image training - which includes using patching (p=4) and classifier-free guidance. 48 | """ 49 | res = dict( 50 | image_size=64, 51 | num_channels=128, 52 | num_res_blocks=2, 53 | num_heads=4, 54 | num_heads_upsample=-1, 55 | num_head_channels=-1, 56 | attention_resolutions="16,8", 57 | channel_mult="", 58 | dropout=0.0, 59 | class_cond=False, 60 | use_checkpoint=False, 61 | use_scale_shift_norm=True, 62 | resblock_updown=False, 63 | use_fp16=False, 64 | use_new_attention_order=False, 65 | 66 | patch_size=4, 67 | classifier_free=True, 68 | snr_splits="", 69 | weight_schedule="sqrt_snr" 70 | 71 | ) 72 | res.update(diffusion_defaults()) 73 | return res 74 | 75 | 76 | def classifier_and_diffusion_defaults(): 77 | res = classifier_defaults() 78 | res.update(diffusion_defaults()) 79 | return res 80 | 81 | 82 | def create_model_and_diffusion( 83 | image_size, 84 | class_cond, 85 | learn_sigma, 86 | num_channels, 87 | num_res_blocks, 88 | 89 | patch_size, 90 | classifier_free, 91 | 92 | channel_mult, 93 | num_heads, 94 | num_head_channels, 95 | num_heads_upsample, 96 | attention_resolutions, 97 | dropout, 98 | diffusion_steps, 99 | noise_schedule, 100 | timestep_respacing, 101 | use_kl, 102 | model_mean_type, 103 | rescale_timesteps, 104 | rescale_learned_sigmas, 105 | use_checkpoint, 106 | use_scale_shift_norm, 107 | resblock_updown, 108 | use_fp16, 109 | use_new_attention_order, 110 | snr_splits, 111 | weight_schedule 112 | ): 113 | model = create_model( 114 | image_size, 115 | num_channels, 116 | num_res_blocks, 117 | patch_size, 118 | classifier_free, 119 | channel_mult=channel_mult, 120 | learn_sigma=learn_sigma, 121 | class_cond=class_cond, 122 | use_checkpoint=use_checkpoint, 123 | attention_resolutions=attention_resolutions, 124 | num_heads=num_heads, 125 | num_head_channels=num_head_channels, 126 | num_heads_upsample=num_heads_upsample, 127 | use_scale_shift_norm=use_scale_shift_norm, 128 | dropout=dropout, 129 | resblock_updown=resblock_updown, 130 | use_fp16=use_fp16, 131 | use_new_attention_order=use_new_attention_order, 132 | ) 133 | diffusion = create_gaussian_diffusion( 134 | steps=diffusion_steps, 135 | learn_sigma=learn_sigma, 136 | noise_schedule=noise_schedule, 137 | use_kl=use_kl, 138 | model_mean_type=model_mean_type, 139 | rescale_timesteps=rescale_timesteps, 140 | rescale_learned_sigmas=rescale_learned_sigmas, 141 | timestep_respacing=timestep_respacing, 142 | snr_splits=snr_splits, 143 | weight_schedule=weight_schedule 144 | ) 145 | return model, diffusion 146 | 147 | 148 | def create_model( 149 | image_size, #real image size, not the dimensions of the image after patches have been extracted. 150 | num_channels, 151 | num_res_blocks, 152 | 153 | patch_size=4, 154 | classifier_free=True, 155 | 156 | channel_mult="", 157 | learn_sigma=False, 158 | class_cond=False, 159 | use_checkpoint=False, 160 | attention_resolutions="16", 161 | num_heads=1, 162 | num_head_channels=-1, 163 | num_heads_upsample=-1, 164 | use_scale_shift_norm=False, 165 | dropout=0, 166 | resblock_updown=False, 167 | use_fp16=False, 168 | use_new_attention_order=False, 169 | ): 170 | assert image_size%patch_size == 0, "patch size must evenly divide image size." 171 | 172 | input_res = image_size//patch_size 173 | 174 | if channel_mult == "": 175 | if input_res == 256: 176 | channel_mult = (1, 1, 2, 2, 4, 4) 177 | elif input_res == 128: 178 | channel_mult = (1, 1, 2, 3, 4) 179 | elif input_res == 64: 180 | channel_mult = (1, 2, 3, 4) 181 | elif input_res == 32: 182 | channel_mult = (1, 2, 3) 183 | elif input_res == 16: 184 | channel_mult == (1, 2) 185 | else: 186 | raise ValueError(f"unsupported image size: {image_size}") 187 | else: 188 | channel_mult = tuple(float(ch_mult) for ch_mult in channel_mult.split(",")) 189 | 190 | attention_ds = [] 191 | for res in attention_resolutions.split(","): 192 | attention_ds.append(input_res // int(res)) 193 | 194 | return UNetModel( 195 | image_size=image_size, 196 | in_channels=3, 197 | model_channels=num_channels, 198 | out_channels=(3 if not learn_sigma else 6), 199 | num_res_blocks=num_res_blocks, 200 | attention_resolutions=tuple(attention_ds), 201 | dropout=dropout, 202 | channel_mult=channel_mult, 203 | patch_size=patch_size, 204 | classifier_free=classifier_free, 205 | num_classes=(NUM_CLASSES if class_cond else None), 206 | use_checkpoint=use_checkpoint, 207 | use_fp16=use_fp16, 208 | num_heads=num_heads, 209 | num_head_channels=num_head_channels, 210 | num_heads_upsample=num_heads_upsample, 211 | use_scale_shift_norm=use_scale_shift_norm, 212 | resblock_updown=resblock_updown, 213 | use_new_attention_order=use_new_attention_order, 214 | ) 215 | 216 | 217 | def create_classifier_and_diffusion( 218 | image_size, 219 | classifier_use_fp16, 220 | classifier_width, 221 | classifier_depth, 222 | classifier_attention_resolutions, 223 | classifier_use_scale_shift_norm, 224 | classifier_resblock_updown, 225 | classifier_pool, 226 | learn_sigma, 227 | diffusion_steps, 228 | noise_schedule, 229 | timestep_respacing, 230 | use_kl, 231 | model_mean_type, 232 | rescale_timesteps, 233 | rescale_learned_sigmas, 234 | ): 235 | classifier = create_classifier( 236 | image_size, 237 | classifier_use_fp16, 238 | classifier_width, 239 | classifier_depth, 240 | classifier_attention_resolutions, 241 | classifier_use_scale_shift_norm, 242 | classifier_resblock_updown, 243 | classifier_pool, 244 | ) 245 | diffusion = create_gaussian_diffusion( 246 | steps=diffusion_steps, 247 | learn_sigma=learn_sigma, 248 | noise_schedule=noise_schedule, 249 | use_kl=use_kl, 250 | model_mean_type=model_mean_type, 251 | rescale_timesteps=rescale_timesteps, 252 | rescale_learned_sigmas=rescale_learned_sigmas, 253 | timestep_respacing=timestep_respacing, 254 | ) 255 | return classifier, diffusion 256 | 257 | 258 | def create_classifier( 259 | image_size, 260 | classifier_use_fp16, 261 | classifier_width, 262 | classifier_depth, 263 | classifier_attention_resolutions, 264 | classifier_use_scale_shift_norm, 265 | classifier_resblock_updown, 266 | classifier_pool, 267 | ): 268 | if image_size == 512: 269 | channel_mult = (0.5, 1, 1, 2, 2, 4, 4) 270 | elif image_size == 256: 271 | channel_mult = (1, 1, 2, 2, 4, 4) 272 | elif image_size == 128: 273 | channel_mult = (1, 1, 2, 3, 4) 274 | elif image_size == 64: 275 | channel_mult = (1, 2, 3, 4) 276 | else: 277 | raise ValueError(f"unsupported image size: {image_size}") 278 | 279 | attention_ds = [] 280 | for res in classifier_attention_resolutions.split(","): 281 | attention_ds.append(image_size // int(res)) 282 | 283 | return EncoderUNetModel( 284 | image_size=image_size, 285 | in_channels=3, 286 | model_channels=classifier_width, 287 | out_channels=1000, 288 | num_res_blocks=classifier_depth, 289 | attention_resolutions=tuple(attention_ds), 290 | channel_mult=channel_mult, 291 | use_fp16=classifier_use_fp16, 292 | num_head_channels=64, 293 | use_scale_shift_norm=classifier_use_scale_shift_norm, 294 | resblock_updown=classifier_resblock_updown, 295 | pool=classifier_pool, 296 | ) 297 | 298 | 299 | def sr_model_and_diffusion_defaults(): 300 | res = model_and_diffusion_defaults() 301 | res["large_size"] = 256 302 | res["small_size"] = 64 303 | arg_names = inspect.getfullargspec(sr_create_model_and_diffusion)[0] 304 | for k in res.copy().keys(): 305 | if k not in arg_names: 306 | del res[k] 307 | return res 308 | 309 | 310 | def sr_create_model_and_diffusion( 311 | large_size, 312 | small_size, 313 | class_cond, 314 | learn_sigma, 315 | num_channels, 316 | num_res_blocks, 317 | num_heads, 318 | num_head_channels, 319 | num_heads_upsample, 320 | attention_resolutions, 321 | patch_size, 322 | classifier_free, 323 | dropout, 324 | diffusion_steps, 325 | noise_schedule, 326 | timestep_respacing, 327 | use_kl, 328 | model_mean_type, 329 | rescale_timesteps, 330 | rescale_learned_sigmas, 331 | use_checkpoint, 332 | use_scale_shift_norm, 333 | resblock_updown, 334 | use_fp16, 335 | ): 336 | model = sr_create_model( 337 | large_size, 338 | small_size, 339 | num_channels, 340 | num_res_blocks, 341 | learn_sigma=learn_sigma, 342 | class_cond=class_cond, 343 | use_checkpoint=use_checkpoint, 344 | attention_resolutions=attention_resolutions, 345 | patch_size=patch_size, 346 | classifier_free=classifier_free, 347 | num_heads=num_heads, 348 | num_head_channels=num_head_channels, 349 | num_heads_upsample=num_heads_upsample, 350 | use_scale_shift_norm=use_scale_shift_norm, 351 | dropout=dropout, 352 | resblock_updown=resblock_updown, 353 | use_fp16=use_fp16, 354 | ) 355 | diffusion = create_gaussian_diffusion( 356 | steps=diffusion_steps, 357 | learn_sigma=learn_sigma, 358 | noise_schedule=noise_schedule, 359 | use_kl=use_kl, 360 | model_mean_type=model_mean_type, 361 | rescale_timesteps=rescale_timesteps, 362 | rescale_learned_sigmas=rescale_learned_sigmas, 363 | timestep_respacing=timestep_respacing, 364 | ) 365 | return model, diffusion 366 | 367 | 368 | def sr_create_model( 369 | large_size, 370 | small_size, 371 | num_channels, 372 | num_res_blocks, 373 | learn_sigma, 374 | class_cond, 375 | use_checkpoint, 376 | attention_resolutions, 377 | patch_size, 378 | classifier_free, 379 | num_heads, 380 | num_head_channels, 381 | num_heads_upsample, 382 | use_scale_shift_norm, 383 | dropout, 384 | resblock_updown, 385 | use_fp16, 386 | ): 387 | _ = small_size # hack to prevent unused variable 388 | 389 | if large_size == 512: 390 | channel_mult = (1, 1, 2, 2, 4, 4) 391 | elif large_size == 256: 392 | channel_mult = (1, 1, 2, 2, 4, 4) 393 | elif large_size == 64: 394 | channel_mult = (1, 2, 3, 4) 395 | else: 396 | raise ValueError(f"unsupported large size: {large_size}") 397 | 398 | attention_ds = [] 399 | for res in attention_resolutions.split(","): 400 | attention_ds.append(large_size // int(res)) 401 | 402 | return SuperResModel( 403 | image_size=large_size, 404 | in_channels=3, 405 | model_channels=num_channels, 406 | out_channels=(3 if not learn_sigma else 6), 407 | num_res_blocks=num_res_blocks, 408 | attention_resolutions=tuple(attention_ds), 409 | patch_size=patch_size, 410 | classifier_free=classifier_free, 411 | dropout=dropout, 412 | channel_mult=channel_mult, 413 | num_classes=(NUM_CLASSES if class_cond else None), 414 | use_checkpoint=use_checkpoint, 415 | num_heads=num_heads, 416 | num_head_channels=num_head_channels, 417 | num_heads_upsample=num_heads_upsample, 418 | use_scale_shift_norm=use_scale_shift_norm, 419 | resblock_updown=resblock_updown, 420 | use_fp16=use_fp16, 421 | ) 422 | 423 | 424 | def create_gaussian_diffusion( 425 | *, 426 | steps=1000, 427 | learn_sigma=False, 428 | sigma_small=False, 429 | noise_schedule="linear", 430 | use_kl=False, 431 | model_mean_type='xstart', 432 | rescale_timesteps=False, 433 | rescale_learned_sigmas=False, 434 | timestep_respacing="", 435 | snr_splits="", 436 | weight_schedule="sqrt_snr" 437 | ): 438 | betas = gd.get_named_beta_schedule(noise_schedule, steps) 439 | if use_kl: 440 | loss_type = gd.LossType.RESCALED_KL 441 | elif rescale_learned_sigmas: 442 | loss_type = gd.LossType.RESCALED_MSE 443 | else: 444 | loss_type = gd.LossType.MSE 445 | if not timestep_respacing: 446 | timestep_respacing = [steps] 447 | 448 | if model_mean_type.lower() in ['eps', 'epsilon']: 449 | model_mean_type = gd.ModelMeanType.EPSILON 450 | elif model_mean_type.lower() in ['xstart', 'x0', 'xo']: 451 | model_mean_type = gd.ModelMeanType.START_X 452 | elif model_mean_type == 'v': 453 | model_mean_type = gd.ModelMeanType.V 454 | else: 455 | raise NotImplementedError() 456 | 457 | return SpacedDiffusion( 458 | use_timesteps=space_timesteps(steps, timestep_respacing), 459 | betas=betas, 460 | model_mean_type=model_mean_type, 461 | model_var_type=( 462 | ( 463 | gd.ModelVarType.FIXED_LARGE 464 | if not sigma_small 465 | else gd.ModelVarType.FIXED_SMALL 466 | ) 467 | if not learn_sigma 468 | else gd.ModelVarType.LEARNED_RANGE 469 | ), 470 | loss_type=loss_type, 471 | rescale_timesteps=rescale_timesteps, 472 | snr_splits=snr_splits, 473 | weight_schedule=weight_schedule 474 | ) 475 | 476 | 477 | def add_dict_to_argparser(parser, default_dict): 478 | for k, v in default_dict.items(): 479 | v_type = type(v) 480 | if v is None: 481 | v_type = str 482 | elif isinstance(v, bool): 483 | v_type = str2bool 484 | parser.add_argument(f"--{k}", default=v, type=v_type) 485 | 486 | 487 | def args_to_dict(args, keys): 488 | return {k: getattr(args, k) for k in keys} 489 | 490 | 491 | def str2bool(v): 492 | """ 493 | https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse 494 | """ 495 | if isinstance(v, bool): 496 | return v 497 | if v.lower() in ("yes", "true", "t", "y", "1"): 498 | return True 499 | elif v.lower() in ("no", "false", "f", "n", "0"): 500 | return False 501 | else: 502 | raise argparse.ArgumentTypeError("boolean value expected") 503 | -------------------------------------------------------------------------------- /patch_diffusion/th_configs.py: -------------------------------------------------------------------------------- 1 | def make_th_model_ffhq(): 2 | defaults = dict( 3 | clip_denoised=True, 4 | batch_size=8, 5 | use_ddim=False, 6 | model_path=None, 7 | timestep_respacing=1000, 8 | num_samples=1000 9 | ) 10 | defaults.update(model_and_diffusion_defaults()) 11 | 12 | args = ConfigDict(defaults) 13 | 14 | args.attention_resolutions = '16,8' 15 | args.channel_mult = '1,2,2,4,4,4' 16 | args.use_conv=False 17 | args.class_cond=False 18 | args.image_size=1024 19 | 20 | args.patch_size=4 21 | args.classifier_free=True #doesnt matter for unconditional models. 22 | args.output_type='x0' #doesnt matter for unet, move arg to diffusion object 23 | args.modified_arch=True 24 | 25 | args.learn_sigma=True 26 | args.num_channels=128 27 | args.resblock_updown=False 28 | args.use_scale_shift_norm=True 29 | args.use_fp16 = False 30 | args.use_new_attention_order = True 31 | args.num_head_channels = 64 32 | 33 | print(args_to_dict(args, model_and_diffusion_defaults().keys())) 34 | model, diffusion = create_model_and_diffusion( 35 | **args_to_dict(args, model_and_diffusion_defaults().keys()) 36 | ) 37 | model.eval() 38 | return model 39 | 40 | def make_th_model_net(): 41 | defaults = dict( 42 | clip_denoised=True, 43 | batch_size=8, 44 | use_ddim=False, 45 | model_path=None, 46 | timestep_respacing=1000, 47 | num_samples=1000 48 | ) 49 | defaults.update(model_and_diffusion_defaults()) 50 | 51 | args = ConfigDict(defaults) 52 | 53 | args.attention_resolutions = '16,8' 54 | args.channel_mult = '1,2,2,2' 55 | args.use_conv=False 56 | args.class_cond=True 57 | args.image_size=256 58 | 59 | args.patch_size=4 60 | args.classifier_free=True #doesnt matter for unconditional models. 61 | args.output_type='x0' #doesnt matter for unet, move arg to diffusion object 62 | args.modified_arch=True 63 | 64 | args.learn_sigma=True 65 | args.num_channels=256 66 | args.num_res_blocks=3 67 | args.resblock_updown=False 68 | args.use_scale_shift_norm=True 69 | args.use_fp16 = False 70 | args.use_new_attention_order = True 71 | args.num_head_channels = 64 72 | 73 | print(args_to_dict(args, model_and_diffusion_defaults().keys())) 74 | model, diffusion = create_model_and_diffusion( 75 | **args_to_dict(args, model_and_diffusion_defaults().keys()) 76 | ) 77 | model.eval() 78 | return model -------------------------------------------------------------------------------- /patch_diffusion/train_util.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import functools 3 | import os 4 | 5 | import blobfile as bf 6 | import torch as th 7 | import torch.distributed as dist 8 | from torch.nn.parallel.distributed import DistributedDataParallel as DDP 9 | from torch.optim import AdamW 10 | 11 | from . import dist_util, logger 12 | from .fp16_util import MixedPrecisionTrainer 13 | from .nn import update_ema 14 | from .resample import LossAwareSampler, UniformSampler 15 | 16 | # For ImageNet experiments, this was a good default value. 17 | # We found that the lg_loss_scale quickly climbed to 18 | # 20-21 within the first ~1K steps of training. 19 | INITIAL_LOG_LOSS_SCALE = 20.0 20 | 21 | 22 | class TrainLoop: 23 | def __init__( 24 | self, 25 | *, 26 | model, 27 | diffusion, 28 | data, 29 | batch_size, 30 | microbatch, 31 | lr, 32 | ema_rate, 33 | log_interval, 34 | save_interval, 35 | resume_checkpoint, 36 | use_fp16=False, 37 | fp16_scale_growth=1e-3, 38 | schedule_sampler=None, 39 | weight_decay=0.0, 40 | lr_anneal_steps=0, 41 | weight_schedule="sqrt_snr" 42 | ): 43 | self.model = model 44 | self.diffusion = diffusion 45 | self.data = data 46 | self.batch_size = batch_size 47 | self.microbatch = microbatch if microbatch > 0 else batch_size 48 | self.lr = lr 49 | self.ema_rate = ( 50 | [ema_rate] 51 | if isinstance(ema_rate, float) 52 | else [float(x) for x in ema_rate.split(",")] 53 | ) 54 | self.log_interval = log_interval 55 | self.save_interval = save_interval 56 | self.resume_checkpoint = resume_checkpoint 57 | self.use_fp16 = use_fp16 58 | self.fp16_scale_growth = fp16_scale_growth 59 | self.schedule_sampler = schedule_sampler or UniformSampler(diffusion) 60 | self.weight_decay = weight_decay 61 | self.lr_anneal_steps = lr_anneal_steps 62 | self.weight_schedule = weight_schedule 63 | 64 | self.step = 0 65 | self.resume_step = 0 66 | self.global_batch = self.batch_size * dist.get_world_size() 67 | 68 | self.sync_cuda = th.cuda.is_available() 69 | 70 | self._load_and_sync_parameters() 71 | self.mp_trainer = MixedPrecisionTrainer( 72 | model=self.model, 73 | use_fp16=self.use_fp16, 74 | fp16_scale_growth=fp16_scale_growth, 75 | ) 76 | 77 | self.opt = AdamW( 78 | self.mp_trainer.master_params, lr=self.lr, weight_decay=self.weight_decay 79 | ) 80 | if self.resume_step: 81 | self._load_optimizer_state() 82 | # Model was resumed, either due to a restart or a checkpoint 83 | # being specified at the command line. 84 | self.ema_params = [ 85 | self._load_ema_parameters(rate) for rate in self.ema_rate 86 | ] 87 | else: 88 | self.ema_params = [ 89 | copy.deepcopy(self.mp_trainer.master_params) 90 | for _ in range(len(self.ema_rate)) 91 | ] 92 | 93 | if th.cuda.is_available(): 94 | self.use_ddp = True 95 | self.ddp_model = DDP( 96 | self.model, 97 | device_ids=[dist_util.dev()], 98 | output_device=dist_util.dev(), 99 | broadcast_buffers=False, 100 | bucket_cap_mb=128, 101 | find_unused_parameters=False, 102 | ) 103 | else: 104 | if dist.get_world_size() > 1: 105 | logger.warn( 106 | "Distributed training requires CUDA. " 107 | "Gradients will not be synchronized properly!" 108 | ) 109 | self.use_ddp = False 110 | self.ddp_model = self.model 111 | 112 | def _load_and_sync_parameters(self): 113 | resume_checkpoint = find_resume_checkpoint() or self.resume_checkpoint 114 | 115 | if resume_checkpoint: 116 | self.resume_step = parse_resume_step_from_filename(resume_checkpoint) 117 | if dist.get_rank() == 0: 118 | logger.log(f"loading model from checkpoint: {resume_checkpoint}...") 119 | self.model.load_state_dict( 120 | dist_util.load_state_dict( 121 | resume_checkpoint, map_location=dist_util.dev() 122 | ) 123 | ) 124 | 125 | dist_util.sync_params(self.model.parameters()) 126 | 127 | def _load_ema_parameters(self, rate): 128 | ema_params = copy.deepcopy(self.mp_trainer.master_params) 129 | 130 | main_checkpoint = find_resume_checkpoint() or self.resume_checkpoint 131 | ema_checkpoint = find_ema_checkpoint(main_checkpoint, self.resume_step, rate) 132 | if ema_checkpoint: 133 | if dist.get_rank() == 0: 134 | logger.log(f"loading EMA from checkpoint: {ema_checkpoint}...") 135 | state_dict = dist_util.load_state_dict( 136 | ema_checkpoint, map_location=dist_util.dev() 137 | ) 138 | ema_params = self.mp_trainer.state_dict_to_master_params(state_dict) 139 | 140 | dist_util.sync_params(ema_params) 141 | return ema_params 142 | 143 | def _load_optimizer_state(self): 144 | main_checkpoint = find_resume_checkpoint() or self.resume_checkpoint 145 | opt_checkpoint = bf.join( 146 | bf.dirname(main_checkpoint), f"opt{self.resume_step:06}.pt" 147 | ) 148 | if bf.exists(opt_checkpoint): 149 | logger.log(f"loading optimizer state from checkpoint: {opt_checkpoint}") 150 | state_dict = dist_util.load_state_dict( 151 | opt_checkpoint, map_location=dist_util.dev() 152 | ) 153 | self.opt.load_state_dict(state_dict) 154 | 155 | def run_loop(self): 156 | while ( 157 | not self.lr_anneal_steps 158 | or self.step + self.resume_step < self.lr_anneal_steps 159 | ): 160 | batch, cond = next(self.data) 161 | self.run_step(batch, cond) 162 | if self.step % self.log_interval == 0: 163 | logger.dumpkvs() 164 | if self.step % self.save_interval == 0: 165 | self.save() 166 | # Run for a finite amount of time in integration tests. 167 | if os.environ.get("DIFFUSION_TRAINING_TEST", "") and self.step > 0: 168 | return 169 | self.step += 1 170 | # Save the last checkpoint if it wasn't already saved. 171 | if (self.step - 1) % self.save_interval != 0: 172 | self.save() 173 | 174 | def run_step(self, batch, cond): 175 | self.forward_backward(batch, cond) 176 | took_step = self.mp_trainer.optimize(self.opt) 177 | if took_step: 178 | self._update_ema() 179 | self._anneal_lr() 180 | self.log_step() 181 | 182 | def forward_backward(self, batch, cond): 183 | self.mp_trainer.zero_grad() 184 | for i in range(0, batch.shape[0], self.microbatch): 185 | micro = batch[i : i + self.microbatch].to(dist_util.dev()) 186 | micro_cond = { 187 | k: v[i : i + self.microbatch].to(dist_util.dev()) 188 | for k, v in cond.items() 189 | } 190 | last_batch = (i + self.microbatch) >= batch.shape[0] 191 | t, weights = self.schedule_sampler.sample(micro.shape[0], dist_util.dev()) 192 | 193 | compute_losses = functools.partial( 194 | self.diffusion.training_losses, 195 | self.ddp_model, 196 | micro, 197 | t, 198 | model_kwargs=micro_cond, 199 | #weight_schedule=self.weight_schedule 200 | ) 201 | 202 | if last_batch or not self.use_ddp: 203 | losses = compute_losses() 204 | else: 205 | with self.ddp_model.no_sync(): 206 | losses = compute_losses() 207 | 208 | if isinstance(self.schedule_sampler, LossAwareSampler): 209 | self.schedule_sampler.update_with_local_losses( 210 | t, losses["loss"].detach() 211 | ) 212 | 213 | loss = (losses["loss"] * weights).mean() 214 | log_loss_dict( 215 | self.diffusion, t, {k: v * weights for k, v in losses.items()} 216 | ) 217 | self.mp_trainer.backward(loss) 218 | 219 | def _update_ema(self): 220 | for rate, params in zip(self.ema_rate, self.ema_params): 221 | update_ema(params, self.mp_trainer.master_params, rate=rate) 222 | 223 | def _anneal_lr(self): 224 | if not self.lr_anneal_steps: 225 | return 226 | frac_done = (self.step + self.resume_step) / self.lr_anneal_steps 227 | lr = self.lr * (1 - frac_done) 228 | for param_group in self.opt.param_groups: 229 | param_group["lr"] = lr 230 | 231 | def log_step(self): 232 | logger.logkv("step", self.step + self.resume_step) 233 | logger.logkv("samples", (self.step + self.resume_step + 1) * self.global_batch) 234 | 235 | def save(self): 236 | def save_checkpoint(rate, params): 237 | state_dict = self.mp_trainer.master_params_to_state_dict(params) 238 | if dist.get_rank() == 0: 239 | logger.log(f"saving model {rate}...") 240 | if not rate: 241 | filename = f"model{(self.step+self.resume_step):06d}.pt" 242 | else: 243 | filename = f"ema_{rate}_{(self.step+self.resume_step):06d}.pt" 244 | with bf.BlobFile(bf.join(get_blob_logdir(), filename), "wb") as f: 245 | th.save(state_dict, f) 246 | 247 | save_checkpoint(0, self.mp_trainer.master_params) 248 | for rate, params in zip(self.ema_rate, self.ema_params): 249 | save_checkpoint(rate, params) 250 | 251 | if dist.get_rank() == 0: 252 | with bf.BlobFile( 253 | bf.join(get_blob_logdir(), f"opt{(self.step+self.resume_step):06d}.pt"), 254 | "wb", 255 | ) as f: 256 | th.save(self.opt.state_dict(), f) 257 | 258 | dist.barrier() 259 | 260 | 261 | def parse_resume_step_from_filename(filename): 262 | """ 263 | Parse filenames of the form path/to/modelNNNNNN.pt, where NNNNNN is the 264 | checkpoint's number of steps. 265 | """ 266 | split = filename.split("model") 267 | if len(split) < 2: 268 | return 0 269 | split1 = split[-1].split(".")[0] 270 | try: 271 | return int(split1) 272 | except ValueError: 273 | return 0 274 | 275 | 276 | def get_blob_logdir(): 277 | # You can change this to be a separate path to save checkpoints to 278 | # a blobstore or some external drive. 279 | return logger.get_dir() 280 | 281 | 282 | def find_resume_checkpoint(): 283 | # On your infrastructure, you may want to override this to automatically 284 | # discover the latest checkpoint on your blob storage, etc. 285 | return None 286 | 287 | 288 | def find_ema_checkpoint(main_checkpoint, step, rate): 289 | if main_checkpoint is None: 290 | return None 291 | filename = f"ema_{rate}_{(step):06d}.pt" 292 | path = bf.join(bf.dirname(main_checkpoint), filename) 293 | if bf.exists(path): 294 | return path 295 | return None 296 | 297 | 298 | def log_loss_dict(diffusion, ts, losses): 299 | for key, values in losses.items(): 300 | logger.logkv_mean(key, values.mean().item()) 301 | # Log the quantiles (four quartiles, in particular). 302 | for sub_t, sub_loss in zip(ts.cpu().numpy(), values.detach().cpu().numpy()): 303 | quartile = int(4 * sub_t / diffusion.num_timesteps) 304 | logger.logkv_mean(f"{key}_q{quartile}", sub_loss) 305 | -------------------------------------------------------------------------------- /patch_diffusion/unet.py: -------------------------------------------------------------------------------- 1 | from abc import abstractmethod 2 | 3 | import math 4 | 5 | import numpy as np 6 | import torch as th 7 | import torch.nn as nn 8 | import torch.nn.functional as F 9 | 10 | from .fp16_util import convert_module_to_f16, convert_module_to_f32 11 | from .nn import ( 12 | checkpoint, 13 | conv_nd, 14 | linear, 15 | avg_pool_nd, 16 | zero_module, 17 | normalization, 18 | timestep_embedding, 19 | timestep_embedding_2d 20 | ) 21 | 22 | 23 | class AttentionPool2d(nn.Module): 24 | """ 25 | Adapted from CLIP: https://github.com/openai/CLIP/blob/main/clip/model.py 26 | """ 27 | 28 | def __init__( 29 | self, 30 | spacial_dim: int, 31 | embed_dim: int, 32 | num_heads_channels: int, 33 | output_dim: int = None, 34 | ): 35 | super().__init__() 36 | self.positional_embedding = nn.Parameter( 37 | th.randn(embed_dim, spacial_dim ** 2 + 1) / embed_dim ** 0.5 38 | ) 39 | self.qkv_proj = conv_nd(1, embed_dim, 3 * embed_dim, 1) 40 | self.c_proj = conv_nd(1, embed_dim, output_dim or embed_dim, 1) 41 | self.num_heads = embed_dim // num_heads_channels 42 | self.attention = QKVAttention(self.num_heads) 43 | 44 | def forward(self, x): 45 | b, c, *_spatial = x.shape 46 | x = x.reshape(b, c, -1) # NC(HW) 47 | x = th.cat([x.mean(dim=-1, keepdim=True), x], dim=-1) # NC(HW+1) 48 | x = x + self.positional_embedding[None, :, :].to(x.dtype) # NC(HW+1) 49 | x = self.qkv_proj(x) 50 | x = self.attention(x) 51 | x = self.c_proj(x) 52 | return x[:, :, 0] 53 | 54 | 55 | class TimestepBlock(nn.Module): 56 | """ 57 | Any module where forward() takes timestep embeddings as a second argument. 58 | """ 59 | 60 | @abstractmethod 61 | def forward(self, x, emb): 62 | """ 63 | Apply the module to `x` given `emb` timestep embeddings. 64 | """ 65 | 66 | 67 | class TimestepEmbedSequential(nn.Sequential, TimestepBlock): 68 | """ 69 | A sequential module that passes timestep embeddings to the children that 70 | support it as an extra input. 71 | """ 72 | 73 | def forward(self, x, emb): 74 | for layer in self: 75 | if isinstance(layer, TimestepBlock): 76 | x = layer(x, emb) 77 | else: 78 | x = layer(x) 79 | return x 80 | 81 | 82 | class Upsample(nn.Module): 83 | """ 84 | An upsampling layer with an optional convolution. 85 | 86 | :param channels: channels in the inputs and outputs. 87 | :param use_conv: a bool determining if a convolution is applied. 88 | :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then 89 | upsampling occurs in the inner-two dimensions. 90 | """ 91 | 92 | def __init__(self, channels, use_conv, dims=2, out_channels=None): 93 | super().__init__() 94 | self.channels = channels 95 | self.out_channels = out_channels or channels 96 | self.use_conv = use_conv 97 | self.dims = dims 98 | if use_conv: 99 | self.conv = conv_nd(dims, self.channels, self.out_channels, 3, padding=1) 100 | 101 | def forward(self, x): 102 | assert x.shape[1] == self.channels 103 | if self.dims == 3: 104 | x = F.interpolate( 105 | x, (x.shape[2], x.shape[3] * 2, x.shape[4] * 2), mode="nearest" 106 | ) 107 | else: 108 | x = F.interpolate(x, scale_factor=2, mode="nearest") 109 | if self.use_conv: 110 | x = self.conv(x) 111 | return x 112 | 113 | # a class for strided conv2d in pytorch that matches tensorflow's strided conv behavior with 'same' padding. 114 | # code segment taken from https://github.com/pytorch/pytorch/issues/67551, 115 | # for more info about torch vs tf padding see https://gist.github.com/Yangqing/47772de7eb3d5dbbff50ffb0d7a98964 116 | class Conv2dSame(th.nn.Conv2d): 117 | def calc_same_pad(self, i: int, k: int, s: int, d: int) -> int: 118 | return max((math.ceil(i / s) - 1) * s + (k - 1) * d + 1 - i, 0) 119 | 120 | def forward(self, x: th.Tensor) -> th.Tensor: 121 | ih, iw = x.size()[-2:] 122 | 123 | pad_h = self.calc_same_pad(i=ih, k=self.kernel_size[0], s=self.stride[0], d=self.dilation[0]) 124 | pad_w = self.calc_same_pad(i=iw, k=self.kernel_size[1], s=self.stride[1], d=self.dilation[1]) 125 | 126 | if pad_h > 0 or pad_w > 0: 127 | x = F.pad( 128 | x, [pad_w // 2, pad_w - pad_w // 2, pad_h // 2, pad_h - pad_h // 2] 129 | ) 130 | return F.conv2d( 131 | x, 132 | self.weight, 133 | self.bias, 134 | self.stride, 135 | self.padding, 136 | self.dilation, 137 | self.groups, 138 | ) 139 | 140 | class Downsample(nn.Module): 141 | """ 142 | A downsampling layer with an optional convolution. 143 | 144 | :param channels: channels in the inputs and outputs. 145 | :param use_conv: a bool determining if a convolution is applied. 146 | :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then 147 | downsampling occurs in the inner-two dimensions. 148 | """ 149 | 150 | def __init__(self, channels, use_conv, dims=2, out_channels=None): 151 | super().__init__() 152 | self.channels = channels 153 | self.out_channels = out_channels or channels 154 | self.use_conv = use_conv 155 | self.dims = dims 156 | stride = 2 if dims != 3 else (1, 2, 2) 157 | if use_conv: 158 | if dims==2: 159 | self.op = Conv2dSame(self.channels, self.out_channels, 3, stride=stride) 160 | else: 161 | self.op = conv_nd(dims, self.channels, self.out_channels, 3, stride=stride, padding=1) 162 | else: 163 | assert self.channels == self.out_channels 164 | self.op = avg_pool_nd(dims, kernel_size=stride, stride=stride) 165 | 166 | def forward(self, x): 167 | assert x.shape[1] == self.channels 168 | return self.op(x) 169 | 170 | 171 | class ResBlock(TimestepBlock): 172 | """ 173 | 174 | A residual block that can optionally change the number of channels. 175 | 176 | :param channels: the number of input channels. 177 | :param emb_channels: the number of timestep embedding channels. 178 | :param dropout: the rate of dropout. 179 | :param out_channels: if specified, the number of out channels. 180 | :param use_conv: if True and out_channels is specified, use a spatial 181 | convolution instead of a smaller 1x1 convolution to change the 182 | channels in the skip connection. 183 | :param class_emb_channels: specifies the input dimension of the class embedding projection 184 | :param dims: determines if the signal is 1D, 2D, or 3D. 185 | :param use_checkpoint: if True, use gradient checkpointing on this module. 186 | :param up: if True, use this block for upsampling. 187 | :param down: if True, use this block for downsampling. 188 | """ 189 | 190 | def __init__( 191 | self, 192 | channels, 193 | emb_channels, 194 | dropout, 195 | out_channels=None, 196 | use_conv=False, 197 | class_emb_channels=None, 198 | dims=2, 199 | use_checkpoint=False, 200 | use_scale_shift_norm=True, 201 | up=False, down=False 202 | ): 203 | super().__init__() 204 | 205 | self.channels = channels 206 | self.emb_channels = emb_channels 207 | 208 | self.use_scale_shift_norm = use_scale_shift_norm 209 | self.class_emb_channels = class_emb_channels 210 | 211 | self.dropout = dropout 212 | self.out_channels = out_channels or channels 213 | self.use_checkpoint = use_checkpoint 214 | self.use_scale_shift_norm = use_scale_shift_norm 215 | 216 | self.in_layers = nn.Sequential( 217 | normalization(channels), 218 | nn.SiLU(), 219 | conv_nd(dims, channels, self.out_channels, 3, padding=1), 220 | ) 221 | 222 | self.updown = up or down 223 | 224 | if up: 225 | self.h_upd = Upsample(channels, False, dims) 226 | self.x_upd = Upsample(channels, False, dims) 227 | elif down: 228 | self.h_upd = Downsample(channels, False, dims) 229 | self.x_upd = Downsample(channels, False, dims) 230 | else: 231 | self.h_upd = self.x_upd = nn.Identity() 232 | 233 | if self.class_emb_channels: 234 | self.classemb_proj = linear(class_emb_channels, self.out_channels) 235 | 236 | self.emb_layers = zero_module( 237 | linear(emb_channels, 2 * self.out_channels) 238 | ) 239 | 240 | self.out_norm = lambda x: F.group_norm(x.float(), 32, eps=1e-3).type(x.dtype) #our arch didn't use an affine transform. 241 | 242 | self.out_layers = nn.Sequential( 243 | nn.SiLU(), 244 | nn.Dropout(p=dropout), 245 | zero_module( 246 | conv_nd(dims, self.out_channels, self.out_channels, 3, padding=1) 247 | ), 248 | ) 249 | 250 | if self.out_channels == channels: 251 | self.skip_connection = nn.Identity() 252 | elif use_conv: 253 | self.skip_connection = conv_nd( 254 | dims, channels, self.out_channels, 3, padding=1 255 | ) 256 | else: 257 | self.skip_connection = conv_nd(dims, channels, self.out_channels, 1) 258 | 259 | def forward(self, x, emb): 260 | """ 261 | Apply the block to a Tensor, conditioned on a timestep embedding. 262 | 263 | :param x: an [N x C x ...] Tensor of features. 264 | :param emb: an [N x emb_channels] Tensor of timestep embeddings. 265 | :return: an [N x C x ...] Tensor of outputs. 266 | """ 267 | return checkpoint( 268 | self._forward, (x, emb), self.parameters(), self.use_checkpoint 269 | ) 270 | 271 | def _forward(self, x, conditioning): 272 | if self.class_emb_channels: c_emb, emb = conditioning 273 | else: c_emb, emb = None, conditioning 274 | 275 | if self.updown: 276 | in_rest, in_conv = self.in_layers[:-1], self.in_layers[-1] 277 | h = in_rest(x) 278 | h = self.h_upd(h) 279 | x = self.x_upd(x) 280 | h = in_conv(h) 281 | else: 282 | h = self.in_layers(x) 283 | 284 | emb_out = self.emb_layers(emb).type(h.dtype) 285 | 286 | while len(emb_out.shape) < len(h.shape): 287 | emb_out = emb_out[..., None] 288 | 289 | if self.use_scale_shift_norm: 290 | scale, shift = th.chunk(emb_out, 2, dim=1) 291 | 292 | h = self.out_norm(h) 293 | h = h * (1 + scale) + shift 294 | h = self.out_layers(h) 295 | else: 296 | h = h + emb_out 297 | h = self.out_layers(h) 298 | 299 | if self.class_emb_channels: 300 | cemb_out = self.classemb_proj(c_emb) + 2. 301 | while len(cemb_out.shape) < len(h.shape): 302 | cemb_out = cemb_out[..., None] 303 | h = h * F.sigmoid(cemb_out).type(h.dtype) 304 | 305 | out = self.skip_connection(x) + h 306 | return out 307 | 308 | 309 | class AttentionBlock(nn.Module): 310 | """ 311 | An attention block that allows spatial positions to attend to each other. 312 | 313 | Originally ported from here, but adapted to the N-d case. 314 | https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/models/unet.py#L66. 315 | """ 316 | 317 | def __init__( 318 | self, 319 | channels, 320 | num_heads=1, 321 | num_head_channels=-1, 322 | use_checkpoint=False, 323 | use_new_attention_order=False, 324 | ): 325 | super().__init__() 326 | self.channels = channels 327 | 328 | if num_head_channels == -1: 329 | self.num_heads = num_heads 330 | else: 331 | assert ( 332 | channels % num_head_channels == 0 333 | ), f"q,k,v channels {channels} is not divisible by num_head_channels {num_head_channels}" 334 | self.num_heads = channels // num_head_channels 335 | 336 | self.use_checkpoint = use_checkpoint 337 | self.norm = normalization(channels) 338 | self.qkv = conv_nd(1, channels, channels * 3, 1) 339 | 340 | if use_new_attention_order: 341 | # split qkv before split heads 342 | self.attention = QKVAttention(self.num_heads) 343 | else: 344 | # split heads before split qkv 345 | self.attention = QKVAttentionLegacy(self.num_heads) 346 | 347 | self.proj_out = zero_module(conv_nd(1, channels, channels, 1)) 348 | 349 | def forward(self, x): 350 | return checkpoint(self._forward, (x,), self.parameters(), True) 351 | 352 | def _forward(self, x): 353 | b, c, *spatial = x.shape 354 | x = x.reshape(b, c, -1) 355 | qkv = self.qkv(self.norm(x)) 356 | h = self.attention(qkv) 357 | h = self.proj_out(h) 358 | return (x + h).reshape(b, c, *spatial) 359 | 360 | 361 | def count_flops_attn(model, _x, y): 362 | """ 363 | A counter for the `thop` package to count the operations in an 364 | attention operation. 365 | Meant to be used like: 366 | macs, params = thop.profile( 367 | model, 368 | inputs=(inputs, timestamps), 369 | custom_ops={QKVAttention: QKVAttention.count_flops}, 370 | ) 371 | """ 372 | b, c, *spatial = y[0].shape 373 | num_spatial = int(np.prod(spatial)) 374 | # We perform two matmuls with the same number of ops. 375 | # The first computes the weight matrix, the second computes 376 | # the combination of the value vectors. 377 | matmul_ops = 2 * b * (num_spatial ** 2) * c 378 | model.total_ops += th.DoubleTensor([matmul_ops]) 379 | 380 | 381 | class QKVAttentionLegacy(nn.Module): 382 | """ 383 | A module which performs QKV attention. Matches legacy QKVAttention + input/ouput heads shaping 384 | """ 385 | 386 | def __init__(self, n_heads): 387 | super().__init__() 388 | self.n_heads = n_heads 389 | 390 | def forward(self, qkv): 391 | """ 392 | Apply QKV attention. 393 | 394 | :param qkv: an [N x (H * 3 * C) x T] tensor of Qs, Ks, and Vs. 395 | :return: an [N x (H * C) x T] tensor after attention. 396 | """ 397 | bs, width, length = qkv.shape 398 | assert width % (3 * self.n_heads) == 0 399 | ch = width // (3 * self.n_heads) 400 | q, k, v = qkv.reshape(bs * self.n_heads, ch * 3, length).split(ch, dim=1) 401 | scale = 1 / math.sqrt(math.sqrt(ch)) 402 | weight = th.einsum( 403 | "bct,bcs->bts", q * scale, k * scale 404 | ) # More stable with f16 than dividing afterwards 405 | weight = th.softmax(weight.float(), dim=-1).type(weight.dtype) 406 | a = th.einsum("bts,bcs->bct", weight, v) 407 | return a.reshape(bs, -1, length) 408 | 409 | @staticmethod 410 | def count_flops(model, _x, y): 411 | return count_flops_attn(model, _x, y) 412 | 413 | 414 | class QKVAttention(nn.Module): 415 | """ 416 | A module which performs QKV attention and splits in a different order. 417 | """ 418 | 419 | def __init__(self, n_heads): 420 | super().__init__() 421 | self.n_heads = n_heads 422 | 423 | def forward(self, qkv): 424 | """ 425 | Apply QKV attention. 426 | 427 | :param qkv: an [N x (3 * H * C) x T] tensor of Qs, Ks, and Vs. 428 | :return: an [N x (H * C) x T] tensor after attention. 429 | """ 430 | bs, width, length = qkv.shape 431 | assert width % (3 * self.n_heads) == 0 432 | ch = width // (3 * self.n_heads) 433 | q, k, v = qkv.chunk(3, dim=1) 434 | 435 | scale = 1 / math.sqrt(math.sqrt(ch)) 436 | weight = th.einsum( 437 | "bct,bcs->bts", 438 | (q * scale).view(bs * self.n_heads, ch, length), 439 | (k * scale).view(bs * self.n_heads, ch, length), 440 | ) # More stable with f16 than dividing afterwards 441 | weight = th.softmax(weight.float(), dim=-1).type(weight.dtype) 442 | 443 | a = th.einsum("bts,bcs->bct", weight, v.reshape(bs * self.n_heads, ch, length)) 444 | return a.reshape(bs, -1, length) 445 | 446 | @staticmethod 447 | def count_flops(model, _x, y): 448 | return count_flops_attn(model, _x, y) 449 | 450 | class UNetModel(nn.Module): 451 | """ 452 | The full UNet model with attention and timestep embedding. 453 | :param in_channels: channels in the input Tensor. 454 | :param model_channels: base channel count for the model. 455 | :param out_channels: channels in the output Tensor. 456 | :param num_res_blocks: number of residual blocks per downsample. 457 | :param attention_resolutions: a collection of downsample rates at which 458 | attention will take place. May be a set, list, or tuple. 459 | For example, if this contains 4, then at 4x downsampling, attention 460 | will be used. 461 | 462 | :param patch_size: how large each patch extracted should be. For image data, 463 | the height and width of the input are each reduced by a factor of patch_size, 464 | while the # of channels is increased by a factor of patch_size^2. 465 | :param classifier_free: whether to use classifier-free diffusion guidance for class-conditional models. 466 | We encourage setting this to True. 467 | 468 | :param dropout: the dropout probability. 469 | :param channel_mult: channel multiplier for each level of the UNet. 470 | :param conv_resample: if True, use learned convolutions for upsampling and 471 | downsampling. 472 | :param dims: determines if the signal is 1D, 2D, or 3D. 473 | :param num_classes: if specified (as an int), then this model will be 474 | class-conditional with `num_classes` classes. 475 | :param use_checkpoint: use gradient checkpointing to reduce memory usage. 476 | :param num_heads: the number of attention heads in each attention layer. 477 | :param num_heads_channels: if specified, ignore num_heads and instead use 478 | a fixed channel width per attention head. 479 | :param num_heads_upsample: works with num_heads to set a different number 480 | of heads for upsampling. Deprecated. 481 | :param use_scale_shift_norm: use a FiLM-like conditioning mechanism. 482 | :param resblock_updown: use residual blocks for up/downsampling. 483 | :param use_new_attention_order: use a different attention pattern for potentially 484 | increased efficiency. 485 | """ 486 | 487 | def __init__( 488 | self, 489 | image_size, 490 | in_channels, 491 | model_channels, 492 | out_channels, 493 | num_res_blocks, 494 | attention_resolutions, 495 | 496 | patch_size=4, 497 | classifier_free=True, 498 | 499 | dropout=0, 500 | channel_mult=(1, 2, 4, 8), 501 | conv_resample=True, 502 | dims=2, 503 | num_classes=None, 504 | use_checkpoint=False, 505 | use_fp16=False, 506 | num_heads=1, 507 | num_head_channels=-1, 508 | num_heads_upsample=-1, 509 | use_scale_shift_norm=False, 510 | resblock_updown=False, 511 | use_new_attention_order=False, 512 | ): 513 | super().__init__() 514 | 515 | if num_heads_upsample == -1: 516 | num_heads_upsample = num_heads 517 | 518 | in_channels = in_channels * patch_size * patch_size 519 | out_channels = out_channels * patch_size * patch_size 520 | 521 | self.image_size = image_size 522 | self.in_channels = in_channels 523 | self.model_channels = model_channels 524 | self.out_channels = out_channels 525 | self.num_res_blocks = num_res_blocks 526 | self.attention_resolutions = attention_resolutions 527 | 528 | self.patch_size = patch_size 529 | self.classifier_free = classifier_free 530 | 531 | self.dropout = dropout 532 | self.channel_mult = channel_mult 533 | self.conv_resample = conv_resample 534 | self.num_classes = num_classes 535 | self.use_checkpoint = use_checkpoint 536 | self.dtype = th.float16 if use_fp16 else th.float32 537 | self.num_heads = num_heads 538 | self.num_head_channels = num_head_channels 539 | self.num_heads_upsample = num_heads_upsample 540 | 541 | self.learn_sigma = (out_channels == in_channels * 2) 542 | 543 | if self.num_classes: 544 | class_emb_channels = int(channel_mult[0] * model_channels) 545 | nc = self.num_classes + 1 if self.classifier_free else self.num_classes 546 | self.label_embs = nn.Embedding(nc, class_emb_channels) 547 | else: 548 | class_emb_channels = None 549 | 550 | time_embed_dim = int(model_channels * channel_mult[-1]) #our imagenet model used a smaller time embedding dim because we used model splitting - we thought that smaller timestep ranges didn't need as many dims. 551 | 552 | self.time_embed = nn.Sequential( 553 | linear(model_channels, model_channels * 4), 554 | nn.SiLU(), 555 | linear(model_channels * 4, time_embed_dim), 556 | nn.SiLU() 557 | ) 558 | 559 | ch = input_ch = int(channel_mult[0] * model_channels) 560 | self.input_blocks = nn.ModuleList( 561 | [TimestepEmbedSequential(conv_nd(dims, in_channels, ch, 3, padding=1))] 562 | ) 563 | self._feature_size = ch 564 | input_block_chans = [ch] 565 | ds = 1 566 | for level, mult in enumerate(channel_mult): 567 | for _ in range(num_res_blocks): 568 | layers = [ 569 | ResBlock( 570 | ch, 571 | time_embed_dim, 572 | dropout, 573 | out_channels=int(mult * model_channels), 574 | dims=dims, 575 | use_checkpoint=use_checkpoint, 576 | use_scale_shift_norm=use_scale_shift_norm, 577 | class_emb_channels=class_emb_channels, 578 | ) 579 | ] 580 | ch = int(mult * model_channels) 581 | if ds in attention_resolutions: 582 | layers.append( 583 | AttentionBlock( 584 | ch, 585 | use_checkpoint=use_checkpoint, 586 | num_heads=num_heads, 587 | num_head_channels=num_head_channels, 588 | use_new_attention_order=use_new_attention_order, 589 | ) 590 | ) 591 | self.input_blocks.append(TimestepEmbedSequential(*layers)) 592 | self._feature_size += ch 593 | input_block_chans.append(ch) 594 | if level != len(channel_mult) - 1: 595 | out_ch = int(model_channels * channel_mult[level+1]) 596 | self.input_blocks.append( 597 | TimestepEmbedSequential( 598 | ResBlock( 599 | ch, 600 | time_embed_dim, 601 | dropout, 602 | out_channels=out_ch, 603 | dims=dims, 604 | use_checkpoint=use_checkpoint, 605 | use_scale_shift_norm=use_scale_shift_norm, 606 | down=True, 607 | class_emb_channels=class_emb_channels, 608 | ) 609 | if resblock_updown 610 | else Downsample( 611 | ch, conv_resample, dims=dims, out_channels=out_ch 612 | ) 613 | ) 614 | ) 615 | ch = out_ch 616 | input_block_chans.append(ch) 617 | ds *= 2 618 | self._feature_size += ch 619 | 620 | self.middle_block = TimestepEmbedSequential( 621 | ResBlock( 622 | ch, 623 | time_embed_dim, 624 | dropout, 625 | dims=dims, 626 | use_checkpoint=use_checkpoint, 627 | use_scale_shift_norm=use_scale_shift_norm, 628 | class_emb_channels=class_emb_channels, 629 | ), 630 | AttentionBlock( 631 | ch, 632 | use_checkpoint=use_checkpoint, 633 | num_heads=num_heads, 634 | num_head_channels=num_head_channels, 635 | use_new_attention_order=use_new_attention_order, 636 | ), 637 | ResBlock( 638 | ch, 639 | time_embed_dim, 640 | dropout, 641 | dims=dims, 642 | use_checkpoint=use_checkpoint, 643 | use_scale_shift_norm=use_scale_shift_norm, 644 | class_emb_channels=class_emb_channels, 645 | ), 646 | ) 647 | self._feature_size += ch 648 | 649 | self.output_blocks = nn.ModuleList([]) 650 | for level, mult in list(enumerate(channel_mult))[::-1]: 651 | for i in range(num_res_blocks + 1): 652 | ich = input_block_chans.pop() 653 | 654 | if ich != ch: ch = ich%ch 655 | layers = [ 656 | ResBlock( 657 | ch + ich, 658 | time_embed_dim, 659 | dropout, 660 | out_channels=int(model_channels * mult), 661 | dims=dims, 662 | use_checkpoint=use_checkpoint, 663 | use_scale_shift_norm=use_scale_shift_norm, 664 | class_emb_channels=class_emb_channels, 665 | ) 666 | ] 667 | ch = int(model_channels * mult) 668 | if ds in attention_resolutions: 669 | layers.append( 670 | AttentionBlock( 671 | ch, 672 | use_checkpoint=use_checkpoint, 673 | num_heads=num_heads_upsample, 674 | num_head_channels=num_head_channels, 675 | use_new_attention_order=use_new_attention_order, 676 | ) 677 | ) 678 | if level and i == num_res_blocks: 679 | out_ch = int(model_channels*channel_mult[level-1]) 680 | 681 | layers.append( 682 | ResBlock( 683 | ch, 684 | time_embed_dim, 685 | dropout, 686 | out_channels=out_ch, 687 | dims=dims, 688 | use_checkpoint=use_checkpoint, 689 | use_scale_shift_norm=use_scale_shift_norm, 690 | up=True, 691 | class_emb_channels=class_emb_channels 692 | ) 693 | if resblock_updown 694 | else Upsample(ch, conv_resample, dims=dims, out_channels=out_ch) 695 | ) 696 | ds //= 2 697 | self.output_blocks.append(TimestepEmbedSequential(*layers)) 698 | self._feature_size += ch 699 | 700 | self.out = nn.Sequential( 701 | normalization(ch), 702 | nn.SiLU(), 703 | ) 704 | 705 | if self.learn_sigma: 706 | self.variance_output_t_emb = conv_nd(dims, time_embed_dim, model_channels * 2, 1) 707 | self.variance_output_model = nn.Sequential( 708 | conv_nd(dims, model_channels*2, model_channels, 3, padding=1), 709 | nn.SiLU(), 710 | conv_nd(dims, model_channels, in_channels, 3, padding=1), 711 | nn.Sigmoid() 712 | ) 713 | self.out.append( 714 | zero_module(conv_nd(dims, input_ch, in_channels, 3, padding=1)) 715 | ) 716 | else: 717 | self.out.append( 718 | zero_module(conv_nd(dims, input_ch, out_channels, 3, padding=1)) 719 | ) 720 | 721 | def convert_to_fp16(self): 722 | """ 723 | Convert the torso of the model to float16. 724 | """ 725 | self.input_blocks.apply(convert_module_to_f16) 726 | self.middle_block.apply(convert_module_to_f16) 727 | self.output_blocks.apply(convert_module_to_f16) 728 | 729 | def convert_to_fp32(self): 730 | """ 731 | Convert the torso of the model to float32. 732 | """ 733 | self.input_blocks.apply(convert_module_to_f32) 734 | self.middle_block.apply(convert_module_to_f32) 735 | self.output_blocks.apply(convert_module_to_f32) 736 | 737 | def forward(self, x, timesteps, y=None): 738 | """ 739 | Apply the model to an input batch. 740 | 741 | :param x: an [N x C x ...] Tensor of inputs. 742 | :param timesteps: a 1-D batch of timesteps. 743 | :param y: an [N] Tensor of labels, if class-conditional. 744 | :return: an [N x C x ...] Tensor of outputs. 745 | """ 746 | x = self.to_patches(x) 747 | 748 | assert (y is not None) == ( 749 | self.num_classes is not None 750 | ), "must specify y if and only if the model is class-conditional" 751 | 752 | hs = [] 753 | t_emb = self.time_embed(timestep_embedding(timesteps, self.model_channels))#.type(self.dtype) 754 | 755 | if self.num_classes is not None: 756 | assert y.shape == (x.shape[0],) 757 | c_emb = self.label_embs(y) 758 | emb = (c_emb, t_emb) 759 | else: 760 | emb = t_emb 761 | 762 | x_dtype = x.dtype 763 | x = self.input_blocks[0](x.type(self.dtype), emb) 764 | hs.append(x) 765 | 766 | x += timestep_embedding_2d(timesteps, x.shape[1], x.shape[-1]).type(self.dtype) 767 | h = x.clone() 768 | 769 | for module in self.input_blocks[1:]: 770 | h = module(h, emb) 771 | hs.append(h) 772 | 773 | h = self.middle_block(h, emb) 774 | 775 | for module in self.output_blocks: 776 | h = th.cat([h, hs.pop()], dim=1) 777 | h = module(h, emb) 778 | 779 | h = h.type(x_dtype) 780 | 781 | if self.learn_sigma: 782 | concat = th.cat([x, h], axis=1) 783 | while len(t_emb.shape) < len(concat.shape): 784 | t_emb = t_emb[..., None] 785 | concat = concat.detach().requires_grad_(True) + self.variance_output_t_emb(F.silu(t_emb)) 786 | mean_output, var_output = self.out(h), self.variance_output_model(concat) 787 | return th.cat([ 788 | self.from_patches(mean_output), self.from_patches(var_output) 789 | ], dim=1) 790 | 791 | else: 792 | outs = self.out(h) 793 | return self.from_patches(outs) 794 | 795 | def to_patches(self, x): 796 | p = self.patch_size 797 | B, C, H, W = x.shape 798 | 799 | x = x.permute(0, 2, 3, 1).reshape(B, H, W//p, C*p) 800 | x = x.permute(0, 2, 1, 3).reshape(B, W//p, H//p, C*p*p) 801 | return x.permute(0, 3, 2, 1) 802 | 803 | def from_patches(self, x): 804 | p = self.patch_size 805 | B, C, H, W = x.shape 806 | 807 | x = x.permute(0,3,2,1).reshape(B, W, H*p, C//p) 808 | x = x.permute(0,2,1,3).reshape(B, H*p, W*p, C//(p*p)) 809 | return x.permute(0, 3, 1, 2) 810 | 811 | def get_classifier_free_label(self, y): 812 | if not self.classifier_free: 813 | raise RuntimeError("You are trying to get the unconditional label for a model that wasn't trained with classifier-free guidance.") 814 | 815 | return (th.ones_like(y) * self.num_classes).type(y.dtype) 816 | 817 | class SuperResModel(UNetModel): 818 | """ 819 | A UNetModel that performs super-resolution. 820 | 821 | Expects an extra kwarg `low_res` to condition on a low-resolution image. 822 | 823 | In principle, it's possible to use patching in a super-resolution model - it would just be conditioned on an image with fewer channels. 824 | for example, one can do a super-resolution from (64, 64, 3) -> (256, 256, 3) by setting patch_size=4 for the SuperResModel object. 825 | 826 | In the call function, it essentially would do: 827 | x = cat(x, upsampled) #[B, 6, 256, 256] 828 | x = to_patches(x, self.patch_size) #[B, 96, 64, 64] 829 | ... rest of layers execute ... 830 | x = from_patches(x, self.patch_size) #[B, 3, 256, 256] 831 | 832 | """ 833 | 834 | def __init__(self, image_size, in_channels, *args, **kwargs): 835 | super().__init__(image_size, in_channels * 2, *args, **kwargs) 836 | 837 | def forward(self, x, timesteps, low_res=None, **kwargs): 838 | _, _, new_height, new_width = x.shape 839 | upsampled = F.interpolate(low_res, (new_height, new_width), mode="bilinear") 840 | x = th.cat([x, upsampled], dim=1) 841 | return super().forward(x, timesteps, **kwargs) 842 | 843 | 844 | class EncoderUNetModel(nn.Module): 845 | """ 846 | The half UNet model with attention and timestep embedding. 847 | 848 | 849 | For usage, see UNet. 850 | """ 851 | 852 | def __init__( 853 | self, 854 | image_size, 855 | in_channels, 856 | model_channels, 857 | out_channels, 858 | num_res_blocks, 859 | attention_resolutions, 860 | dropout=0, 861 | channel_mult=(1, 2, 4, 8), 862 | conv_resample=True, 863 | dims=2, 864 | use_checkpoint=False, 865 | use_fp16=False, 866 | num_heads=1, 867 | num_head_channels=-1, 868 | num_heads_upsample=-1, 869 | use_scale_shift_norm=False, 870 | resblock_updown=False, 871 | use_new_attention_order=False, 872 | pool="adaptive", 873 | ): 874 | super().__init__() 875 | 876 | if num_heads_upsample == -1: 877 | num_heads_upsample = num_heads 878 | 879 | self.in_channels = in_channels 880 | self.model_channels = model_channels 881 | self.out_channels = out_channels 882 | self.num_res_blocks = num_res_blocks 883 | self.attention_resolutions = attention_resolutions 884 | self.dropout = dropout 885 | self.channel_mult = channel_mult 886 | self.conv_resample = conv_resample 887 | self.use_checkpoint = use_checkpoint 888 | self.dtype = th.float16 if use_fp16 else th.float32 889 | self.num_heads = num_heads 890 | self.num_head_channels = num_head_channels 891 | self.num_heads_upsample = num_heads_upsample 892 | 893 | time_embed_dim = model_channels * 4 894 | self.time_embed = nn.Sequential( 895 | linear(model_channels, time_embed_dim), 896 | nn.SiLU(), 897 | linear(time_embed_dim, time_embed_dim) 898 | ) 899 | 900 | ch = int(channel_mult[0] * model_channels) 901 | self.input_blocks = nn.ModuleList( 902 | [TimestepEmbedSequential(conv_nd(dims, in_channels, ch, 3, padding=1))] 903 | ) 904 | self._feature_size = ch 905 | input_block_chans = [ch] 906 | ds = 1 907 | for level, mult in enumerate(channel_mult): 908 | for _ in range(num_res_blocks): 909 | layers = [ 910 | ResBlock( 911 | ch, 912 | time_embed_dim, 913 | dropout, 914 | out_channels=int(mult * model_channels), 915 | dims=dims, 916 | use_checkpoint=use_checkpoint, 917 | use_scale_shift_norm=use_scale_shift_norm, 918 | ) 919 | ] 920 | ch = int(mult * model_channels) 921 | if ds in attention_resolutions: 922 | layers.append( 923 | AttentionBlock( 924 | ch, 925 | use_checkpoint=use_checkpoint, 926 | num_heads=num_heads, 927 | num_head_channels=num_head_channels, 928 | use_new_attention_order=use_new_attention_order, 929 | ) 930 | ) 931 | self.input_blocks.append(TimestepEmbedSequential(*layers)) 932 | self._feature_size += ch 933 | input_block_chans.append(ch) 934 | if level != len(channel_mult) - 1: 935 | out_ch = ch 936 | self.input_blocks.append( 937 | TimestepEmbedSequential( 938 | ResBlock( 939 | ch, 940 | time_embed_dim, 941 | dropout, 942 | out_channels=out_ch, 943 | dims=dims, 944 | use_checkpoint=use_checkpoint, 945 | use_scale_shift_norm=use_scale_shift_norm, 946 | down=True, 947 | ) 948 | if resblock_updown 949 | else Downsample( 950 | ch, conv_resample, dims=dims, out_channels=out_ch 951 | ) 952 | ) 953 | ) 954 | ch = out_ch 955 | input_block_chans.append(ch) 956 | ds *= 2 957 | self._feature_size += ch 958 | 959 | self.middle_block = TimestepEmbedSequential( 960 | ResBlock( 961 | ch, 962 | time_embed_dim, 963 | dropout, 964 | dims=dims, 965 | use_checkpoint=use_checkpoint, 966 | use_scale_shift_norm=use_scale_shift_norm, 967 | ), 968 | AttentionBlock( 969 | ch, 970 | use_checkpoint=use_checkpoint, 971 | num_heads=num_heads, 972 | num_head_channels=num_head_channels, 973 | use_new_attention_order=use_new_attention_order, 974 | ), 975 | ResBlock( 976 | ch, 977 | time_embed_dim, 978 | dropout, 979 | dims=dims, 980 | use_checkpoint=use_checkpoint, 981 | use_scale_shift_norm=use_scale_shift_norm, 982 | ), 983 | ) 984 | self._feature_size += ch 985 | self.pool = pool 986 | if pool == "adaptive": 987 | self.out = nn.Sequential( 988 | normalization(ch), 989 | nn.SiLU(), 990 | nn.AdaptiveAvgPool2d((1, 1)), 991 | zero_module(conv_nd(dims, ch, out_channels, 1)), 992 | nn.Flatten(), 993 | ) 994 | elif pool == "attention": 995 | assert num_head_channels != -1 996 | self.out = nn.Sequential( 997 | normalization(ch), 998 | nn.SiLU(), 999 | AttentionPool2d( 1000 | (image_size // ds), ch, num_head_channels, out_channels 1001 | ), 1002 | ) 1003 | elif pool == "spatial": 1004 | self.out = nn.Sequential( 1005 | nn.Linear(self._feature_size, 2048), 1006 | nn.ReLU(), 1007 | nn.Linear(2048, self.out_channels), 1008 | ) 1009 | elif pool == "spatial_v2": 1010 | self.out = nn.Sequential( 1011 | nn.Linear(self._feature_size, 2048), 1012 | normalization(2048), 1013 | nn.SiLU(), 1014 | nn.Linear(2048, self.out_channels), 1015 | ) 1016 | else: 1017 | raise NotImplementedError(f"Unexpected {pool} pooling") 1018 | 1019 | def convert_to_fp16(self): 1020 | """ 1021 | Convert the torso of the model to float16. 1022 | """ 1023 | self.input_blocks.apply(convert_module_to_f16) 1024 | self.middle_block.apply(convert_module_to_f16) 1025 | 1026 | def convert_to_fp32(self): 1027 | """ 1028 | Convert the torso of the model to float32. 1029 | """ 1030 | self.input_blocks.apply(convert_module_to_f32) 1031 | self.middle_block.apply(convert_module_to_f32) 1032 | 1033 | def forward(self, x, timesteps): 1034 | """ 1035 | Apply the model to an input batch. 1036 | 1037 | :param x: an [N x C x ...] Tensor of inputs. 1038 | :param timesteps: a 1-D batch of timesteps. 1039 | :return: an [N x K] Tensor of outputs. 1040 | """ 1041 | emb = self.time_embed(timestep_embedding(timesteps, self.model_channels)) 1042 | 1043 | results = [] 1044 | h = x.type(self.dtype) 1045 | for module in self.input_blocks: 1046 | h = module(h, emb) 1047 | if self.pool.startswith("spatial"): 1048 | results.append(h.type(x.dtype).mean(dim=(2, 3))) 1049 | h = self.middle_block(h, emb) 1050 | if self.pool.startswith("spatial"): 1051 | results.append(h.type(x.dtype).mean(dim=(2, 3))) 1052 | h = th.cat(results, axis=-1) 1053 | return self.out(h) 1054 | else: 1055 | h = h.type(x.dtype) 1056 | return self.out(h) 1057 | -------------------------------------------------------------------------------- /scripts/image_nll.py: -------------------------------------------------------------------------------- 1 | """ 2 | Approximate the bits/dimension for an image model. 3 | """ 4 | 5 | import argparse 6 | import os 7 | 8 | import numpy as np 9 | import torch.distributed as dist 10 | 11 | from patch_diffusion import dist_util, logger 12 | from patch_diffusion.image_datasets import load_data 13 | from patch_diffusion.script_util import ( 14 | model_and_diffusion_defaults, 15 | create_model_and_diffusion, 16 | add_dict_to_argparser, 17 | args_to_dict, 18 | ) 19 | 20 | 21 | def main(): 22 | args = create_argparser().parse_args() 23 | 24 | dist_util.setup_dist() 25 | logger.configure() 26 | 27 | logger.log("creating model and diffusion...") 28 | model, diffusion = create_model_and_diffusion( 29 | **args_to_dict(args, model_and_diffusion_defaults().keys()) 30 | ) 31 | model.load_state_dict( 32 | dist_util.load_state_dict(args.model_path, map_location="cpu") 33 | ) 34 | model.to(dist_util.dev()) 35 | model.eval() 36 | 37 | logger.log("creating data loader...") 38 | data = load_data( 39 | data_dir=args.data_dir, 40 | batch_size=args.batch_size, 41 | image_size=args.image_size, 42 | class_cond=args.class_cond, 43 | deterministic=True, 44 | ) 45 | 46 | logger.log("evaluating...") 47 | run_bpd_evaluation(model, diffusion, data, args.num_samples, args.clip_denoised) 48 | 49 | 50 | def run_bpd_evaluation(model, diffusion, data, num_samples, clip_denoised): 51 | all_bpd = [] 52 | all_metrics = {"vb": [], "mse": [], "xstart_mse": []} 53 | num_complete = 0 54 | while num_complete < num_samples: 55 | batch, model_kwargs = next(data) 56 | batch = batch.to(dist_util.dev()) 57 | model_kwargs = {k: v.to(dist_util.dev()) for k, v in model_kwargs.items()} 58 | minibatch_metrics = diffusion.calc_bpd_loop( 59 | model, batch, clip_denoised=clip_denoised, model_kwargs=model_kwargs 60 | ) 61 | 62 | for key, term_list in all_metrics.items(): 63 | terms = minibatch_metrics[key].mean(dim=0) / dist.get_world_size() 64 | dist.all_reduce(terms) 65 | term_list.append(terms.detach().cpu().numpy()) 66 | 67 | total_bpd = minibatch_metrics["total_bpd"] 68 | total_bpd = total_bpd.mean() / dist.get_world_size() 69 | dist.all_reduce(total_bpd) 70 | all_bpd.append(total_bpd.item()) 71 | num_complete += dist.get_world_size() * batch.shape[0] 72 | 73 | logger.log(f"done {num_complete} samples: bpd={np.mean(all_bpd)}") 74 | 75 | if dist.get_rank() == 0: 76 | for name, terms in all_metrics.items(): 77 | out_path = os.path.join(logger.get_dir(), f"{name}_terms.npz") 78 | logger.log(f"saving {name} terms to {out_path}") 79 | np.savez(out_path, np.mean(np.stack(terms), axis=0)) 80 | 81 | dist.barrier() 82 | logger.log("evaluation complete") 83 | 84 | 85 | def create_argparser(): 86 | defaults = dict( 87 | data_dir="", clip_denoised=True, num_samples=1000, batch_size=1, model_path="" 88 | ) 89 | defaults.update(model_and_diffusion_defaults()) 90 | parser = argparse.ArgumentParser() 91 | add_dict_to_argparser(parser, defaults) 92 | return parser 93 | 94 | 95 | if __name__ == "__main__": 96 | main() 97 | -------------------------------------------------------------------------------- /scripts/image_sample.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generate a large batch of image samples from a model and save them as a large 3 | numpy array. This can be used to produce samples for FID evaluation. 4 | """ 5 | 6 | import argparse 7 | import os 8 | 9 | import numpy as np 10 | import torch as th 11 | import torch.distributed as dist 12 | import matplotlib.pyplot as plt 13 | 14 | from patch_diffusion import dist_util, logger 15 | from patch_diffusion.script_util import ( 16 | NUM_CLASSES, 17 | model_and_diffusion_defaults, 18 | create_model_and_diffusion, 19 | add_dict_to_argparser, 20 | args_to_dict, 21 | ) 22 | 23 | def save_images(images, figure_path, figdims='4,4', scale='5'): 24 | figdims = [int(d) for d in figdims.split(',')] 25 | scale = float(scale) 26 | 27 | if figdims is None: 28 | m = len(images)//10 + 1 29 | n = 10 30 | else: 31 | m, n = figdims 32 | 33 | plt.figure(figsize=(scale*n, scale*m)) 34 | 35 | for i in range(len(images[:m*n])): 36 | plt.subplot(m, n, i+1) 37 | plt.imshow(images[i]) 38 | plt.axis('off') 39 | 40 | plt.tight_layout() 41 | plt.savefig(figure_path) 42 | print(f"saved image samples at {figure_path}") 43 | 44 | def main(): 45 | args = create_argparser().parse_args() 46 | 47 | dist_util.setup_dist() 48 | logger.configure() 49 | 50 | logger.log("creating model and diffusion...") 51 | 52 | 53 | model_names = args.model_path.split(",") 54 | models = [] 55 | 56 | for model_name in model_names: 57 | model, diffusion = create_model_and_diffusion( 58 | **args_to_dict(args, model_and_diffusion_defaults().keys()) 59 | ) 60 | model.load_state_dict( 61 | dist_util.load_state_dict(model_name, map_location="cpu") 62 | ) 63 | 64 | model.to(dist_util.dev()) 65 | if args.use_fp16: 66 | model.convert_to_fp16() 67 | model.eval() 68 | 69 | models.append(model) 70 | 71 | 72 | if model.classifier_free and model.num_classes and args.guidance_scale != 1.0: 73 | model_fns = [diffusion.make_classifier_free_fn(model, args.guidance_scale) for model in models] 74 | 75 | def denoised_fn(x0): 76 | s = th.quantile(th.abs(x0).reshape([x0.shape[0], -1]), 0.995, dim=-1, interpolation='nearest') 77 | s = th.maximum(s, th.ones_like(s)) 78 | s = s[:, None, None, None] 79 | x0 = x0.clamp(-s, s) / s 80 | return x0 81 | else: 82 | model_fns = models 83 | denoised_fn = None 84 | 85 | logger.log("sampling...") 86 | all_images = [] 87 | all_labels = [] 88 | while len(all_images) * args.batch_size < args.num_samples: 89 | model_kwargs = {} 90 | if args.class_cond: 91 | classes = th.randint( 92 | low=0, high=NUM_CLASSES, size=(args.batch_size,), device=dist_util.dev() 93 | ) 94 | model_kwargs["y"] = classes 95 | sample_fn = ( 96 | diffusion.p_sample_loop if not args.use_ddim else diffusion.ddim_sample_loop 97 | ) 98 | sample = sample_fn( 99 | model_fns, 100 | (args.batch_size, 3, args.image_size, args.image_size), 101 | clip_denoised=args.clip_denoised, 102 | model_kwargs=model_kwargs, 103 | denoised_fn=denoised_fn, 104 | device=dist_util.dev() 105 | ) 106 | sample = ((sample + 1) * 127.5).clamp(0, 255).to(th.uint8) 107 | sample = sample.permute(0, 2, 3, 1) 108 | sample = sample.contiguous() 109 | 110 | gathered_samples = [th.zeros_like(sample) for _ in range(dist.get_world_size())] 111 | dist.all_gather(gathered_samples, sample) # gather not supported with NCCL 112 | all_images.extend([sample.cpu().numpy() for sample in gathered_samples]) 113 | if args.class_cond: 114 | gathered_labels = [ 115 | th.zeros_like(classes) for _ in range(dist.get_world_size()) 116 | ] 117 | dist.all_gather(gathered_labels, classes) 118 | all_labels.extend([labels.cpu().numpy() for labels in gathered_labels]) 119 | logger.log(f"created {len(all_images) * args.batch_size} samples") 120 | 121 | arr = np.concatenate(all_images, axis=0) 122 | arr = arr[: args.num_samples] 123 | if args.class_cond: 124 | label_arr = np.concatenate(all_labels, axis=0) 125 | label_arr = label_arr[: args.num_samples] 126 | if dist.get_rank() == 0: 127 | shape_str = "x".join([str(x) for x in arr.shape]) 128 | 129 | samples_index = len(os.listdir(args.save_dir))//2 130 | 131 | out_path = os.path.join(args.save_dir, f"samples_{shape_str}_{samples_index}.npz") 132 | if os.path.exists(out_path): 133 | print(f"Warning, there is already an npz file {out_path}, saving to a different file...") 134 | new_rands = np.random.randint(0, high=1e6) 135 | samples_index += new_rands 136 | out_path = os.path.join(args.save_dir, f"samples_{shape_str}_{samples_index}.npz") 137 | 138 | logger.log(f"saving to {out_path}") 139 | if args.class_cond: 140 | np.savez(out_path, arr, label_arr) 141 | else: 142 | np.savez(out_path, arr) 143 | 144 | out_path = os.path.join(args.save_dir, f"samples_{shape_str}_{samples_index}.png") 145 | if os.path.exists(out_path): 146 | print(f"Warning, there is already a png file {out_path}, overwriting this file...") 147 | 148 | save_images(arr, out_path, args.figdims, args.figscale) 149 | 150 | dist.barrier() 151 | logger.log("sampling complete") 152 | 153 | 154 | def create_argparser(): 155 | defaults = dict( 156 | clip_denoised=True, 157 | num_samples=10000, 158 | batch_size=16, 159 | use_ddim=False, 160 | model_path="", 161 | guidance_scale=1.5, 162 | save_dir="", 163 | figdims="4,4", 164 | figscale="5" 165 | ) 166 | defaults.update(model_and_diffusion_defaults()) 167 | parser = argparse.ArgumentParser() 168 | add_dict_to_argparser(parser, defaults) 169 | return parser 170 | 171 | 172 | if __name__ == "__main__": 173 | main() 174 | -------------------------------------------------------------------------------- /scripts/image_train.py: -------------------------------------------------------------------------------- 1 | """ 2 | Train a diffusion model on images. 3 | """ 4 | 5 | import argparse 6 | 7 | from patch_diffusion import dist_util, logger 8 | from patch_diffusion.image_datasets import load_data 9 | from patch_diffusion.resample import create_named_schedule_sampler 10 | from patch_diffusion.script_util import ( 11 | model_and_diffusion_defaults, 12 | create_model_and_diffusion, 13 | args_to_dict, 14 | add_dict_to_argparser, 15 | ) 16 | from patch_diffusion.train_util import TrainLoop 17 | 18 | 19 | def main(): 20 | args = create_argparser().parse_args() 21 | 22 | dist_util.setup_dist() 23 | logger.configure() 24 | 25 | logger.log("creating model and diffusion...") 26 | model, diffusion = create_model_and_diffusion( 27 | **args_to_dict(args, model_and_diffusion_defaults().keys()) 28 | ) 29 | model.to(dist_util.dev()) 30 | schedule_sampler = create_named_schedule_sampler(args.schedule_sampler, diffusion) 31 | 32 | logger.log("creating data loader...") 33 | data = load_data( 34 | data_dir=args.data_dir, 35 | batch_size=args.batch_size, 36 | image_size=args.image_size, 37 | class_cond=args.class_cond, 38 | ) 39 | 40 | logger.log("training...") 41 | TrainLoop( 42 | model=model, 43 | diffusion=diffusion, 44 | data=data, 45 | batch_size=args.batch_size, 46 | microbatch=args.microbatch, 47 | lr=args.lr, 48 | ema_rate=args.ema_rate, 49 | log_interval=args.log_interval, 50 | save_interval=args.save_interval, 51 | resume_checkpoint=args.resume_checkpoint, 52 | use_fp16=args.use_fp16, 53 | fp16_scale_growth=args.fp16_scale_growth, 54 | schedule_sampler=schedule_sampler, 55 | weight_decay=args.weight_decay, 56 | lr_anneal_steps=args.lr_anneal_steps, 57 | weight_schedule=args.weight_schedule 58 | ).run_loop() 59 | 60 | 61 | def create_argparser(): 62 | defaults = dict( 63 | data_dir="", 64 | schedule_sampler="uniform", 65 | lr=1e-4, 66 | weight_decay=0.0, 67 | lr_anneal_steps=0, 68 | batch_size=1, 69 | microbatch=-1, # -1 disables microbatches 70 | ema_rate="0.9999", # comma-separated list of EMA values 71 | log_interval=10, 72 | save_interval=10000, 73 | resume_checkpoint="", 74 | use_fp16=False, 75 | fp16_scale_growth=1e-3, 76 | weight_schedule="sqrt_snr" 77 | ) 78 | defaults.update(model_and_diffusion_defaults()) 79 | parser = argparse.ArgumentParser() 80 | add_dict_to_argparser(parser, defaults) 81 | return parser 82 | 83 | 84 | if __name__ == "__main__": 85 | main() 86 | -------------------------------------------------------------------------------- /scripts/super_res_sample.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generate a large batch of samples from a super resolution model, given a batch 3 | of samples from a regular model from image_sample.py. 4 | """ 5 | 6 | import argparse 7 | import os 8 | 9 | import blobfile as bf 10 | import numpy as np 11 | import torch as th 12 | import torch.distributed as dist 13 | 14 | from patch_diffusion import dist_util, logger 15 | from patch_diffusion.script_util import ( 16 | sr_model_and_diffusion_defaults, 17 | sr_create_model_and_diffusion, 18 | args_to_dict, 19 | add_dict_to_argparser, 20 | ) 21 | 22 | 23 | def main(): 24 | args = create_argparser().parse_args() 25 | 26 | dist_util.setup_dist() 27 | logger.configure() 28 | 29 | logger.log("creating model...") 30 | model, diffusion = sr_create_model_and_diffusion( 31 | **args_to_dict(args, sr_model_and_diffusion_defaults().keys()) 32 | ) 33 | model.load_state_dict( 34 | dist_util.load_state_dict(args.model_path, map_location="cpu") 35 | ) 36 | model.to(dist_util.dev()) 37 | if args.use_fp16: 38 | model.convert_to_fp16() 39 | model.eval() 40 | 41 | logger.log("loading data...") 42 | data = load_data_for_worker(args.base_samples, args.batch_size, args.class_cond) 43 | 44 | logger.log("creating samples...") 45 | all_images = [] 46 | while len(all_images) * args.batch_size < args.num_samples: 47 | model_kwargs = next(data) 48 | model_kwargs = {k: v.to(dist_util.dev()) for k, v in model_kwargs.items()} 49 | sample = diffusion.p_sample_loop( 50 | model, 51 | (args.batch_size, 3, args.large_size, args.large_size), 52 | clip_denoised=args.clip_denoised, 53 | model_kwargs=model_kwargs, 54 | ) 55 | sample = ((sample + 1) * 127.5).clamp(0, 255).to(th.uint8) 56 | sample = sample.permute(0, 2, 3, 1) 57 | sample = sample.contiguous() 58 | 59 | all_samples = [th.zeros_like(sample) for _ in range(dist.get_world_size())] 60 | dist.all_gather(all_samples, sample) # gather not supported with NCCL 61 | for sample in all_samples: 62 | all_images.append(sample.cpu().numpy()) 63 | logger.log(f"created {len(all_images) * args.batch_size} samples") 64 | 65 | arr = np.concatenate(all_images, axis=0) 66 | arr = arr[: args.num_samples] 67 | if dist.get_rank() == 0: 68 | shape_str = "x".join([str(x) for x in arr.shape]) 69 | out_path = os.path.join(logger.get_dir(), f"samples_{shape_str}.npz") 70 | logger.log(f"saving to {out_path}") 71 | np.savez(out_path, arr) 72 | 73 | dist.barrier() 74 | logger.log("sampling complete") 75 | 76 | 77 | def load_data_for_worker(base_samples, batch_size, class_cond): 78 | with bf.BlobFile(base_samples, "rb") as f: 79 | obj = np.load(f) 80 | image_arr = obj["arr_0"] 81 | if class_cond: 82 | label_arr = obj["arr_1"] 83 | rank = dist.get_rank() 84 | num_ranks = dist.get_world_size() 85 | buffer = [] 86 | label_buffer = [] 87 | while True: 88 | for i in range(rank, len(image_arr), num_ranks): 89 | buffer.append(image_arr[i]) 90 | if class_cond: 91 | label_buffer.append(label_arr[i]) 92 | if len(buffer) == batch_size: 93 | batch = th.from_numpy(np.stack(buffer)).float() 94 | batch = batch / 127.5 - 1.0 95 | batch = batch.permute(0, 3, 1, 2) 96 | res = dict(low_res=batch) 97 | if class_cond: 98 | res["y"] = th.from_numpy(np.stack(label_buffer)) 99 | yield res 100 | buffer, label_buffer = [], [] 101 | 102 | 103 | def create_argparser(): 104 | defaults = dict( 105 | clip_denoised=True, 106 | num_samples=10000, 107 | batch_size=16, 108 | use_ddim=False, 109 | base_samples="", 110 | model_path="", 111 | ) 112 | defaults.update(sr_model_and_diffusion_defaults()) 113 | parser = argparse.ArgumentParser() 114 | add_dict_to_argparser(parser, defaults) 115 | return parser 116 | 117 | 118 | if __name__ == "__main__": 119 | main() 120 | -------------------------------------------------------------------------------- /scripts/super_res_train.py: -------------------------------------------------------------------------------- 1 | """ 2 | Train a super-resolution model. 3 | """ 4 | 5 | import argparse 6 | 7 | import torch.nn.functional as F 8 | 9 | from patch_diffusion import dist_util, logger 10 | from patch_diffusion.image_datasets import load_data 11 | from patch_diffusion.resample import create_named_schedule_sampler 12 | from patch_diffusion.script_util import ( 13 | sr_model_and_diffusion_defaults, 14 | sr_create_model_and_diffusion, 15 | args_to_dict, 16 | add_dict_to_argparser, 17 | ) 18 | from patch_diffusion.train_util import TrainLoop 19 | 20 | 21 | def main(): 22 | args = create_argparser().parse_args() 23 | 24 | dist_util.setup_dist() 25 | logger.configure() 26 | 27 | logger.log("creating model...") 28 | model, diffusion = sr_create_model_and_diffusion( 29 | **args_to_dict(args, sr_model_and_diffusion_defaults().keys()) 30 | ) 31 | model.to(dist_util.dev()) 32 | schedule_sampler = create_named_schedule_sampler(args.schedule_sampler, diffusion) 33 | 34 | logger.log("creating data loader...") 35 | data = load_superres_data( 36 | args.data_dir, 37 | args.batch_size, 38 | large_size=args.large_size, 39 | small_size=args.small_size, 40 | class_cond=args.class_cond, 41 | ) 42 | 43 | logger.log("training...") 44 | TrainLoop( 45 | model=model, 46 | diffusion=diffusion, 47 | data=data, 48 | batch_size=args.batch_size, 49 | microbatch=args.microbatch, 50 | lr=args.lr, 51 | ema_rate=args.ema_rate, 52 | log_interval=args.log_interval, 53 | save_interval=args.save_interval, 54 | resume_checkpoint=args.resume_checkpoint, 55 | use_fp16=args.use_fp16, 56 | fp16_scale_growth=args.fp16_scale_growth, 57 | schedule_sampler=schedule_sampler, 58 | weight_decay=args.weight_decay, 59 | lr_anneal_steps=args.lr_anneal_steps, 60 | ).run_loop() 61 | 62 | 63 | def load_superres_data(data_dir, batch_size, large_size, small_size, class_cond=False): 64 | data = load_data( 65 | data_dir=data_dir, 66 | batch_size=batch_size, 67 | image_size=large_size, 68 | class_cond=class_cond, 69 | ) 70 | for large_batch, model_kwargs in data: 71 | model_kwargs["low_res"] = F.interpolate(large_batch, small_size, mode="area") 72 | yield large_batch, model_kwargs 73 | 74 | 75 | def create_argparser(): 76 | defaults = dict( 77 | data_dir="", 78 | schedule_sampler="uniform", 79 | lr=1e-4, 80 | weight_decay=0.0, 81 | lr_anneal_steps=0, 82 | batch_size=1, 83 | microbatch=-1, 84 | ema_rate="0.9999", 85 | log_interval=10, 86 | save_interval=10000, 87 | resume_checkpoint="", 88 | use_fp16=False, 89 | fp16_scale_growth=1e-3, 90 | ) 91 | defaults.update(sr_model_and_diffusion_defaults()) 92 | parser = argparse.ArgumentParser() 93 | add_dict_to_argparser(parser, defaults) 94 | return parser 95 | 96 | 97 | if __name__ == "__main__": 98 | main() 99 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | name="patch-diffusion", 5 | py_modules=["patch_diffusion"], 6 | install_requires=["blobfile>=1.0.5", "torch", "tqdm"], 7 | ) 8 | --------------------------------------------------------------------------------