├── .gitignore ├── README.md ├── config.py ├── download.py ├── dp_pca.py ├── dp_utils.py ├── environments.txt ├── evaluation ├── train-classifier-celebA.py ├── train-classifier-fmnist.py ├── train-classifier-hair.py └── train-classifier-mnist.py ├── fid.py ├── gen_data.py ├── inception_score.py ├── input.py ├── main.py ├── mnist_cnn_icp_eval.py ├── model.py ├── ops.py ├── pate_core.py ├── rdp_utils.py ├── requirements.txt ├── temp.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | .DS_Store 3 | evaluation/.DS_Store -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DataLens 2 | 3 | This is the official code base for our ACM CCS 2021 paper: 4 | 5 | ["DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation".](https://arxiv.org/abs/2103.11109) 6 | 7 | Boxin Wang*, Fan Wu*, Yunhui Long*, Luka Rimanic, Ce Zhang, Bo Li 8 | 9 | ## Citation 10 | ``` 11 | @article{wang2021datalens, 12 | title={DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation}, 13 | author={Wang, Boxin and Wu, Fan and Long, Yunhui and Rimanic, Luka and Zhang, Ce and Li, Bo}, 14 | journal={ACM Conference on Computer and Communications Security (CCS)}, 15 | year={2021} 16 | } 17 | ``` 18 | 19 | ## Usage 20 | 21 | ### Prepare your environment 22 | 23 | The project is tested on Python 3.6, but a higher version of Python should also work. Download required packages 24 | 25 | ```shell script 26 | pip install -r requirements.txt 27 | ``` 28 | 29 | ### Prepare your data 30 | Please store the training data in `$data_dir`. By default, `$data_dir` is set to `../../data`. 31 | 32 | We provide a script to download the MNIST and Fashion Mnist datasets. 33 | 34 | ```shell script 35 | python download.py [dataset_name] 36 | ``` 37 | 38 | For MNIST, you can run 39 | 40 | ```shell script 41 | python download.py mnist 42 | ``` 43 | 44 | For Fashion-MNIST, you can run 45 | 46 | ```shell script 47 | python download.py fashion_mnist 48 | ``` 49 | 50 | For CelebA and Places365 datasets, please refer to their official websites for downloading. 51 | 52 | ### Training 53 | 54 | ```shell script 55 | python main.py --checkpoint_dir [checkpoint_dir] --dataset [dataset_name] --train --stochastic --signsgd --topk [topk] 56 | ``` 57 | 58 | For example, to train the Datalens on Fashion-MNIST given eps=1 and delta=1e-5 59 | ```shell script 60 | python main.py --checkpoint_dir fmnist_z_dim_50_topk_200_teacher_4000_sigma_5000_thresh_0.7_pt_30_d_step_2_stochastic_1e-5/ \ 61 | --topk 200 --signsgd --norandom_proj --shuffle --teachers_batch 80 --batch_teachers 50 \ 62 | --dataset fashion_mnist --train --max_eps 1 --train --thresh 0.7 --sigma 5000 --nopretrain \ 63 | --z_dim 50 --nosave_epoch --epoch 300 --save_vote --d_step 2 --pretrain_teacher 10 --stochastic --max_grad 1e-5 64 | ``` 65 | 66 | By default, after it reaches the max epsilon, it will generate 10 batches of 10,000 DP samples as `eps-1.00.data-{i}.pkl` (i=0,...9) in `checkpoint_dir`. 67 | 68 | #### More example commands (eps=1): 69 | MNIST 70 | ```shell script 71 | python main.py --checkpoint_dir [checkpoint-dir] \ 72 | --topk 200 --signsgd --norandom_proj --shuffle --teachers_batch 80 --batch_teachers 50 \ 73 | --dataset mnist --train --max_eps 1 --train --thresh 0.7 --sigma 5000 --nopretrain \ 74 | --z_dim 50 --save_epoch --epoch 300 --save_vote --d_step 2 --pretrain_teacher 10 --stochastic --max_grad 1e-5 75 | ``` 76 | 77 | Fashion-MNIST 78 | ```shell script 79 | python main.py --checkpoint_dir [checkpoint-dir] \ 80 | --topk 200 --signsgd --norandom_proj --shuffle --teachers_batch 80 --batch_teachers 50 \ 81 | --dataset fashion_mnist --train --max_eps 1 --train --thresh 0.9 --sigma 5000 --nopretrain \ 82 | --z_dim 50 --nosave_epoch --epoch 300 --save_vote --d_step 2 --pretrain_teacher 10 --stochastic --max_grad 1e-5 83 | ``` 84 | 85 | 86 | CelebA 87 | ```shell script 88 | python main.py --checkpoint_dir [checkpoint-dir] \ 89 | --topk 700 --signsgd --norandom_proj --shuffle --teachers_batch 100 --batch_teachers 60 \ 90 | --dataset celebA-gender-train --train --max_eps 1 --train --thresh 0.85 --sigma 9000 --nopretrain \ 91 | --z_dim 100 --nosave_epoch --epoch 300 --save_vote --d_step 2 --pretrain_teacher 30 --stochastic --max_grad 1e-5 92 | ``` 93 | 94 | Hair 95 | ```shell script 96 | python main.py --checkpoint_dir [checkpoint-dir]\ 97 | --topk 700 --signsgd --norandom_proj --shuffle --teachers_batch 100 --batch_teachers 80 \ 98 | --dataset celebA-hair-trn --train --max_eps 1 --train --thresh 0.9 --sigma 9000 --nopretrain \ 99 | --z_dim 100 --save_epoch --epoch 300 --save_vote --d_step 2 --pretrain_teacher 30 --stochastic --max_grad 1e-5 100 | ``` 101 | 102 | #### More example commands (eps=10): 103 | 104 | MNIST: 105 | ```shell script 106 | python main.py --checkpoint_dir [checkpoint-dir]/ \ 107 | --topk 300 --signsgd --norandom_proj --shuffle --teachers_batch 80 --batch_teachers 50 \ 108 | --dataset mnist --train --max_eps 10 --train --thresh 0.2 --sigma 800 --nopretrain \ 109 | --z_dim 50 --nosave_epoch --epoch 300 --save_vote --pretrain_teacher 10 --d_step 2 --stochastic --max_grad 1e-5 110 | ``` 111 | 112 | Fashion-MNIST 113 | ```shell script 114 | python main.py --checkpoint_dir [checkpoint-dir] / \ 115 | --topk 350 --signsgd --norandom_proj --shuffle --teachers_batch 80 --batch_teachers 50 \ 116 | --dataset fashion_mnist --train --max_eps 10 --train --thresh 0.27 --sigma 1000 --nopretrain \ 117 | --z_dim 64 --nosave_epoch --epoch 300 --save_vote --pretrain_teacher 10 --d_step 2 --stochastic --max_grad 1e-5 118 | ``` 119 | 120 | ```shell script 121 | python main.py --checkpoint_dir [checkpoint-dir] / \ 122 | --topk 350 --signsgd --norandom_proj --shuffle --teachers_batch 80 --batch_teachers 50 \ 123 | --dataset fashion_mnist --train --max_eps 10 --train --thresh 0.27 --sigma 1000 --nopretrain \ 124 | --z_dim 64 --nosave_epoch --epoch 300 --save_vote --pretrain_teacher 10 --d_step 2 125 | ``` 126 | 127 | CelebA-Gender 128 | ```shell script 129 | python main.py --checkpoint_dir [checkpoint-dir] / \ 130 | --topk 500 --signsgd --norandom_proj --shuffle --teachers_batch 100 --batch_teachers 60 \ 131 | --dataset celebA-gender-train --train --max_eps 10 --train --thresh 0.12 --sigma 700 --nopretrain \ 132 | --z_dim 100 --nosave_epoch --epoch 300 --save_vote --pretrain_teacher 30 --d_step 2 --stochastic 133 | ``` 134 | 135 | CelebA-Hair 136 | ```shell script 137 | python main.py --checkpoint_dir [checkpoint-dir] / \ 138 | --topk 500 --signsgd --norandom_proj --shuffle --teachers_batch 80 --batch_teachers 50 \ 139 | --dataset celebA-hair-trn --train --max_eps 10 --train --thresh 0.25 --sigma 700 --nopretrain \ 140 | --z_dim 100 --nosave_epoch --epoch 300 --save_vote --pretrain_teacher 30 --d_step 2 --stochastic 141 | ``` 142 | 143 | ### Training Args 144 | 145 | ``` 146 | main.py: 147 | --ae: AE model name 148 | (default: '') 149 | --batch_size: The size of batch images [64] 150 | (default: '30') 151 | (an integer) 152 | --batch_teachers: Number of teacher models in one batch 153 | (default: '1') 154 | (an integer) 155 | --beta1: Momentum term of adam [0.5] 156 | (default: '0.5') 157 | (a number) 158 | --checkpoint_dir: Directory name to save the checkpoints [checkpoint] 159 | (default: 'checkpoint') 160 | --checkpoint_name: checkpoint model name [checkpoint] 161 | (default: 'checkpoint') 162 | --[no]crop: True for cropping 163 | (default: 'false') 164 | --d_step: steps of the discriminator 165 | (default: '1') 166 | (an integer) 167 | --data_dir: Root directory of dataset [data] 168 | (default: '../../data') 169 | --dataset: The name of dataset [cinic, celebA, mnist, lsun, fire-small] 170 | (default: 'slt') 171 | --delta: delta for differential privacy 172 | (default: '1e-05') 173 | (a number) 174 | --epoch: Epoch for training teacher models 175 | (default: '1000') 176 | (an integer) 177 | --[no]finetune_ae: Finetune ae 178 | (default: 'false') 179 | --g_epoch: Epoch for training the student models 180 | (default: '500') 181 | (an integer) 182 | --g_step: steps of the generator 183 | (default: '1') 184 | (an integer) 185 | --generator_dir: Directory name to save the generator 186 | (default: 'generator') 187 | --hid_dim: Dimmension of hidden dim 188 | (default: '512') 189 | (an integer) 190 | --[no]increasing_dim: Increase the projection dimension for each epoch 191 | (default: 'false') 192 | --input_height: The size of image to use (will be center cropped). 193 | (default: '32') 194 | (an integer) 195 | --input_width: The size of image to use (will be center cropped). If None, same value as input_height [None] 196 | (default: '32') 197 | (an integer) 198 | --klevel: Levels of gradient quantization 199 | (default: '4') 200 | (an integer) 201 | --[no]klevelsgd: Apply klevel sgd for gradient agggregation 202 | (default: 'false') 203 | --learning_rate: Learning rate of for adam 204 | (default: '0.001') 205 | (a number) 206 | --[no]load_d: True for loading the pretrained models w/ discriminator, False for not load [True] 207 | (default: 'true') 208 | --loss: AE reconstruction loss 209 | (default: 'l1') 210 | --max_eps: maximum epsilon 211 | (default: '1.0') 212 | (a number) 213 | --max_grad: maximum gradient for signsgd aggregation 214 | (default: '0.0') 215 | (a number) 216 | --[no]mean_kernel: Apply Mean Kernel for gradient agggregation 217 | (default: 'false') 218 | --[no]non_private: Do not apply differential privacy 219 | (default: 'false') 220 | --orders: rdp orders 221 | (default: '200') 222 | (an integer) 223 | --output_height: The size of the output images to produce [64] 224 | (default: '32') 225 | (an integer) 226 | --output_width: The size of the output images to produce. If None, same value as output_height [None] 227 | (default: '32') 228 | (an integer) 229 | --[no]pca: Apply pca for gradient aggregation 230 | (default: 'false') 231 | --pca_dim: principal dimensions for pca 232 | (default:'10') 233 | (a number) 234 | --[no]pretrain: True for loading the pretrained models, False for not load [True] 235 | (default: 'true') 236 | --pretrain_teacher: Pretrain teacher for epochs 237 | (default: '0') 238 | (an integer) 239 | --proj_mat: #/ projection mat 240 | (default: '1') 241 | (an integer) 242 | --[no]random_label: random labels for training data, only used when pretraining some models 243 | (default: 'false') 244 | --[no]random_proj: Apply pca for gradient aggregation 245 | (default: 'true') 246 | --sample_dir: Directory name to save the image samples [samples] 247 | (default: 'samples') 248 | --sample_step: Number of teacher models in one batch 249 | (default: '10') 250 | (an integer) 251 | --[no]save_epoch: Save each epoch per 0.1 eps 252 | (default: 'false') 253 | --[no]save_vote: Save voting results 254 | (default: 'false') 255 | --[no]shuffle: Evenly distribute dataset 256 | (default: 'true') 257 | --sigma: Scale of gaussian noise for gradient aggregation 258 | (default: '2000.0') 259 | (a number) 260 | --sigma_thresh: Scale of gaussian noise for thresh gnmax 261 | (default: '4500.0') 262 | (a number) 263 | --[no]signsgd: Apply sign sgd for gradient agggregation 264 | (default: 'false') 265 | --[no]signsgd_dept: Apply sign sgd for gradient agggregation with data dependent bound 266 | (default: 'false') 267 | --[no]signsgd_nothresh: Apply sign sgd for gradient agggregation 268 | (default: 'false') 269 | --[no]simple_gan: Use fc to build GAN 270 | (default: 'false') 271 | --[no]sketchsgd: Apply sketch sgd for gradient agggregation 272 | (default: 'false') 273 | --[no]small: Use a smaller discriminator 274 | (default: 'false') 275 | --step_size: Step size for gradient aggregation 276 | (default: '0.0001') 277 | (a number) 278 | --[no]stochastic: Apply stochastic sign sgd for gradient agggregation 279 | (default: 'false') 280 | --[no]tanh: Use tanh as activation func 281 | (default: 'false') 282 | --teacher_dir: Directory name to save the teacher [teacher] 283 | (default: 'teacher') 284 | --teachers_batch: Number of batch 285 | (default: '1') 286 | (an integer) 287 | --thresh: threshhold for threshgmax 288 | (default: '0.5') 289 | (a number) 290 | --topk: Number of top k gradients 291 | (default: '50') 292 | (an integer) 293 | --[no]train: True for training, False for testing [False] 294 | (default: 'false') 295 | --[no]train_ae: Train ae 296 | (default: 'false') 297 | --train_size: The size of train images [np.inf] 298 | (default: 'inf') 299 | (a number) 300 | --[no]wgan: Train wgan 301 | (default: 'false') 302 | --y_dim: #/ y dim 303 | (default: '10') 304 | (an integer) 305 | --z_dim: #/ z dim 306 | (default: '100') 307 | (an integer) 308 | ``` 309 | 310 | 311 | ## Generating synthetic samples 312 | 313 | ```shell script 314 | python main.py --checkpoint_dir [checkpoint_dir] --dataset [dataset_name] 315 | ``` 316 | 317 | ## Evaluate the synthetic records 318 | 319 | We train a classifier on synthetic samples and test it on real samples. We put the evaluation script under the `evaluation` folder. 320 | 321 | For MNIST, 322 | ```shell script 323 | python evaluation/train-classifier-mnist.py --data [DP_data_dir] 324 | ``` 325 | 326 | 327 | For Fashion-MNIST, 328 | ```shell script 329 | python evaluation/train-classifier-fmnist.py --data [DP_data_dir] 330 | ``` 331 | 332 | For CelebA-Gender, 333 | ```shell script 334 | python evaluation/train-classifier-celebA.py --data [DP_data_dir] 335 | ``` 336 | 337 | For CelebA-Hair, 338 | ```shell script 339 | python evaluation/train-classifier-hair.py --data [DP_data_dir] 340 | ``` 341 | 342 | The `[DP_data_dir]` is where your generated DP samples are located. In the Fashion-MNIST example above, we have generated 10 bathces of DP samples in `$checkpoint_dir/eps-1.00.data-{i}.pkl` (i=0,...,9). During evaluation, you should run with the prefix of the `data_dir`, where the program will concatenate all of the generated DP samples and use it as the training data. 343 | 344 | ```shell script 345 | python evaluation/train-classifier-fmnist.py --data $checkpoint_dir/eps-1.00.data 346 | ``` 347 | -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import pickle 3 | import os 4 | from utils import mkdir 5 | 6 | RESULT_DIR = './../results' 7 | 8 | 9 | def parse_arguments(): 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('--random_seed', '-s', type=int, default=1, help='random seed') 12 | parser.add_argument('--dataset', '-data', type=str, default='mnist', choices=['mnist', 'fashionmnist', 'hair', 'gender'], 13 | help=' dataset name') 14 | parser.add_argument('--num_discriminators', '-ndis', type=int, default=1000, help='number of discriminators') 15 | parser.add_argument('--noise_multiplier', '-noise', type=float, default=1.07, help='noise multiplier') 16 | parser.add_argument('--z_dim', '-zdim', type=int, default=10, help='latent code dimensionality') 17 | parser.add_argument('--model_dim', '-mdim', type=int, default=64, help='model dimensionality') 18 | parser.add_argument('--batchsize', '-bs', type=int, default=32, help='batch size') 19 | parser.add_argument('--L_gp', '-lgp', type=float, default=10, help='gradient penalty lambda hyperparameter') 20 | parser.add_argument('--L_epsilon', '-lep', type=float, default=0.001, help='epsilon penalty (used in PGGAN)') 21 | parser.add_argument('--critic_iters', '-diters', type=int, default=5, help='number of critic iters per gen iter') 22 | parser.add_argument('--latent_type', '-latent', type=str, default='bernoulli', choices=['normal', 'bernoulli'], 23 | help='latent distribution') 24 | parser.add_argument('--iterations', '-iters', type=int, default=20000, help='iterations for training') 25 | parser.add_argument('--pretrain_iterations', '-piters', type=int, default=100, help='iterations for pre-training') 26 | parser.add_argument('--num_workers', '-nwork', type=int, default=0, help='number of workers') 27 | parser.add_argument('--net_ids', '-ids', type=int, nargs='+', help='the index list for the discriminator') 28 | parser.add_argument('--print_step', '-pstep', type=int, default=100, help='number of steps to print') 29 | parser.add_argument('--vis_step', '-vstep', type=int, default=1000, help='number of steps to vis & eval') 30 | parser.add_argument('--save_step', '-sstep', type=int, default=5000, help='number of steps to save') 31 | parser.add_argument('--load_dir', '-ldir', type=str, help='checkpoint dir (for loading pre-trained models)') 32 | parser.add_argument('--pretrain', action='store_true', default=False, help='if performing pre-training') 33 | parser.add_argument('--num_gpus', '-ngpus', type=int, default=1, help='number of gpus') 34 | parser.add_argument('--gen_arch', '-gen', type=str, default='ResNet', choices=['DCGAN', 'ResNet'], 35 | help='generator architecture') 36 | parser.add_argument('--run', '-run', type=int, default=1, help='index number of run') 37 | parser.add_argument('--exp_name', '-name', type=str, 38 | help='output folder name; will be automatically generated if not specified') 39 | args = parser.parse_args() 40 | return args 41 | 42 | 43 | def save_config(args): 44 | ''' 45 | store the config and set up result dir 46 | :param args: 47 | :return: 48 | ''' 49 | ### set up experiment name 50 | if args.exp_name is None: 51 | exp_name = '{}_Ndis{}_Noise{}_Zdim{}_Mdim{}_BS{}_Lgp{}_Lep{}_Diters{}_{}_Run{}'.format( 52 | args.gen_arch, 53 | args.num_discriminators, 54 | args.noise_multiplier, 55 | args.z_dim, 56 | args.model_dim, 57 | args.batchsize, 58 | args.L_gp, 59 | args.L_epsilon, 60 | args.critic_iters, 61 | args.latent_type, 62 | args.run) 63 | args.exp_name = exp_name 64 | 65 | if args.pretrain: 66 | save_dir = os.path.join(RESULT_DIR, args.dataset, 'pretrain', args.exp_name) 67 | else: 68 | save_dir = os.path.join(RESULT_DIR, args.dataset, 'main', args.exp_name) 69 | args.save_dir = save_dir 70 | 71 | ### save config 72 | mkdir(save_dir) 73 | config = vars(args) 74 | pickle.dump(config, open(os.path.join(save_dir, 'params.pkl'), 'wb'), protocol=2) 75 | with open(os.path.join(save_dir, 'params.txt'), 'w') as f: 76 | for k, v in config.items(): 77 | kv_str = k + ':' + str(v) + '\n' 78 | print(kv_str) 79 | f.writelines(kv_str) 80 | 81 | 82 | def load_config(args): 83 | ''' 84 | load the config 85 | :param args: 86 | :return: 87 | ''' 88 | assert args.exp_name is not None, "Please specify the experiment name" 89 | if args.pretrain: 90 | save_dir = os.path.join(RESULT_DIR, args.dataset, 'pretrain', args.exp_name) 91 | else: 92 | save_dir = os.path.join(RESULT_DIR, args.dataset, 'main', args.exp_name) 93 | assert os.path.exists(save_dir) 94 | 95 | ### load config 96 | config = pickle.load(open(os.path.join(save_dir, 'params.pkl'), 'rb')) 97 | return config 98 | -------------------------------------------------------------------------------- /download.py: -------------------------------------------------------------------------------- 1 | """ 2 | Modification of https://github.com/stanfordnlp/treelstm/blob/master/scripts/download.py 3 | 4 | Downloads the following: 5 | - Celeb-A dataset 6 | - LSUN dataset 7 | - MNIST dataset 8 | """ 9 | 10 | from __future__ import print_function 11 | import os 12 | import sys 13 | import gzip 14 | import json 15 | import shutil 16 | import zipfile 17 | import argparse 18 | import requests 19 | import subprocess 20 | from tqdm import tqdm 21 | from six.moves import urllib 22 | 23 | parser = argparse.ArgumentParser(description='Download dataset for DCGAN.') 24 | parser.add_argument('datasets', metavar='N', type=str, nargs='+', choices=['celebA', 'lsun', 'mnist','fashion'], 25 | help='name of dataset to download [celebA, lsun, mnist]') 26 | 27 | def download(url, dirpath): 28 | filename = url.split('/')[-1] 29 | filepath = os.path.join(dirpath, filename) 30 | u = urllib.request.urlopen(url) 31 | f = open(filepath, 'wb') 32 | filesize = int(u.headers["Content-Length"]) 33 | print("Downloading: %s Bytes: %s" % (filename, filesize)) 34 | 35 | downloaded = 0 36 | block_sz = 8192 37 | status_width = 70 38 | while True: 39 | buf = u.read(block_sz) 40 | if not buf: 41 | print('') 42 | break 43 | else: 44 | print('', end='\r') 45 | downloaded += len(buf) 46 | f.write(buf) 47 | status = (("[%-" + str(status_width + 1) + "s] %3.2f%%") % 48 | ('=' * int(float(downloaded) / filesize * status_width) + '>', downloaded * 100. / filesize)) 49 | print(status, end='') 50 | sys.stdout.flush() 51 | f.close() 52 | return filepath 53 | 54 | def download_file_from_google_drive(id, destination): 55 | URL = "https://docs.google.com/uc?export=download" 56 | session = requests.Session() 57 | 58 | response = session.get(URL, params={ 'id': id }, stream=True) 59 | token = get_confirm_token(response) 60 | 61 | if token: 62 | params = { 'id' : id, 'confirm' : token } 63 | response = session.get(URL, params=params, stream=True) 64 | 65 | save_response_content(response, destination) 66 | 67 | def get_confirm_token(response): 68 | for key, value in response.cookies.items(): 69 | if key.startswith('download_warning'): 70 | return value 71 | return None 72 | 73 | def save_response_content(response, destination, chunk_size=32*1024): 74 | total_size = int(response.headers.get('content-length', 0)) 75 | with open(destination, "wb") as f: 76 | for chunk in tqdm(response.iter_content(chunk_size), total=total_size, 77 | unit='B', unit_scale=True, desc=destination): 78 | if chunk: # filter out keep-alive new chunks 79 | f.write(chunk) 80 | 81 | def unzip(filepath): 82 | print("Extracting: " + filepath) 83 | dirpath = os.path.dirname(filepath) 84 | with zipfile.ZipFile(filepath) as zf: 85 | zf.extractall(dirpath) 86 | os.remove(filepath) 87 | 88 | 89 | def _list_categories(tag): 90 | url = 'http://lsun.cs.princeton.edu/htbin/list.cgi?tag=' + tag 91 | f = urllib.request.urlopen(url) 92 | return json.loads(f.read()) 93 | 94 | def _download_lsun(out_dir, category, set_name, tag): 95 | url = 'http://lsun.cs.princeton.edu/htbin/download.cgi?tag={tag}' \ 96 | '&category={category}&set={set_name}'.format(**locals()) 97 | print(url) 98 | if set_name == 'test': 99 | out_name = 'test_lmdb.zip' 100 | else: 101 | out_name = '{category}_{set_name}_lmdb.zip'.format(**locals()) 102 | out_path = os.path.join(out_dir, out_name) 103 | cmd = ['curl', url, '-o', out_path] 104 | print('Downloading', category, set_name, 'set') 105 | subprocess.call(cmd) 106 | 107 | def download_lsun(dirpath): 108 | data_dir = os.path.join(dirpath, 'lsun') 109 | if os.path.exists(data_dir): 110 | print('Found LSUN - skip') 111 | return 112 | else: 113 | os.mkdir(data_dir) 114 | 115 | tag = 'latest' 116 | #categories = _list_categories(tag) 117 | categories = ['bedroom'] 118 | 119 | for category in categories: 120 | _download_lsun(data_dir, category, 'train', tag) 121 | _download_lsun(data_dir, category, 'val', tag) 122 | _download_lsun(data_dir, '', 'test', tag) 123 | 124 | def download_fashion_mnist(dirpath): 125 | data_dir = os.path.join(dirpath, 'fashion_mnist') 126 | if os.path.exists(data_dir): 127 | print('Found MNIST - skip') 128 | return 129 | else: 130 | os.mkdir(data_dir) 131 | url_base = 'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/' 132 | file_names = ['train-images-idx3-ubyte.gz', 133 | 'train-labels-idx1-ubyte.gz', 134 | 't10k-images-idx3-ubyte.gz', 135 | 't10k-labels-idx1-ubyte.gz'] 136 | for file_name in file_names: 137 | url = (url_base+file_name).format(**locals()) 138 | print(url) 139 | out_path = os.path.join(data_dir,file_name) 140 | cmd = ['curl', url, '-o', out_path] 141 | print('Downloading ', file_name) 142 | subprocess.call(cmd) 143 | cmd = ['gzip', '-d', out_path] 144 | print('Decompressing ', file_name) 145 | subprocess.call(cmd) 146 | 147 | def download_mnist(dirpath): 148 | data_dir = os.path.join(dirpath, 'mnist') 149 | if os.path.exists(data_dir): 150 | print('Found MNIST - skip') 151 | return 152 | else: 153 | os.mkdir(data_dir) 154 | url_base = 'http://yann.lecun.com/exdb/mnist/' 155 | file_names = ['train-images-idx3-ubyte.gz', 156 | 'train-labels-idx1-ubyte.gz', 157 | 't10k-images-idx3-ubyte.gz', 158 | 't10k-labels-idx1-ubyte.gz'] 159 | for file_name in file_names: 160 | url = (url_base+file_name).format(**locals()) 161 | print(url) 162 | out_path = os.path.join(data_dir,file_name) 163 | cmd = ['curl', url, '-o', out_path] 164 | print('Downloading ', file_name) 165 | subprocess.call(cmd) 166 | cmd = ['gzip', '-d', out_path] 167 | print('Decompressing ', file_name) 168 | subprocess.call(cmd) 169 | 170 | def prepare_data_dir(path = './data'): 171 | if not os.path.exists(path): 172 | os.mkdir(path) 173 | 174 | if __name__ == '__main__': 175 | args = parser.parse_args() 176 | prepare_data_dir() 177 | 178 | if 'lsun' in args.datasets: 179 | download_lsun('./data') 180 | if 'mnist' in args.datasets: 181 | download_mnist('./data') 182 | if 'fashion' in args.datasets: 183 | download_fashion_mnist('./data') 184 | -------------------------------------------------------------------------------- /dp_pca.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The TensorFlow Authors. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ============================================================================== 15 | 16 | """ Combined sanitizer.py and dp_pca.py under tensorflow/models/research/differential-privacy """ 17 | 18 | """Differentially private optimizers. 19 | """ 20 | import tensorflow as tf 21 | import collections 22 | from sklearn.preprocessing import normalize 23 | from rdp_utils import gaussian_rdp 24 | import numpy as np 25 | 26 | def ComputeDPPrincipalProjection(data, projection_dims, orders, sigma): 27 | """Compute differentially private projection. 28 | 29 | Args: 30 | data: the input data, each row is a data vector. 31 | projection_dims: the projection dimension. 32 | sigma: sigma for gaussian noise 33 | Returns: 34 | A projection matrix with projection_dims columns. 35 | """ 36 | 37 | # Normalize each row. 38 | normalized_data = normalize(data, norm='l2', axis=1) 39 | covar = np.matmul(np.transpose(normalized_data), normalized_data) 40 | 41 | # Since the data is already normalized, there is no need to clip 42 | # the covariance matrix. 43 | 44 | gaussian_noise, rdp_budget = gaussian_rdp(covar.reshape([1,-1]), 1.0, orders, sigma) 45 | 46 | saned_covar = covar + gaussian_noise.reshape(covar.shape) 47 | 48 | # Symmetrize saned_covar. This also reduces the noise variance. 49 | saned_covar = 0.5 * (saned_covar + np.transpose(saned_covar)) 50 | 51 | # Compute the eigen decomposition of the covariance matrix, and 52 | # return the top projection_dims eigen vectors, represented as columns of 53 | # the projection matrix. 54 | eigvals, eigvecs = np.linalg.eig(saned_covar) 55 | 56 | topk_indices = eigvals.argsort()[::-1][:projection_dims] 57 | topk_indices = np.reshape(topk_indices, [projection_dims]) 58 | 59 | # Gather and return the corresponding eigenvectors. 60 | return np.transpose(np.take(np.transpose(eigvecs), topk_indices, axis=0)), rdp_budget -------------------------------------------------------------------------------- /dp_utils.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The TensorFlow Authors. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ============================================================================== 15 | 16 | """Utils for building and training NN models. 17 | """ 18 | from __future__ import division 19 | 20 | import math 21 | 22 | import numpy 23 | import tensorflow as tf 24 | 25 | 26 | def BatchClipByL2norm(t, upper_bound, name=None): 27 | """Clip an array of tensors by L2 norm. 28 | 29 | Shrink each dimension-0 slice of tensor (for matrix it is each row) such 30 | that the l2 norm is at most upper_bound. Here we clip each row as it 31 | corresponds to each example in the batch. 32 | 33 | Args: 34 | t: the input tensor. 35 | upper_bound: the upperbound of the L2 norm. 36 | name: optional name. 37 | Returns: 38 | the clipped tensor. 39 | """ 40 | 41 | assert upper_bound > 0 42 | with tf.name_scope(values=[t, upper_bound], name=name, 43 | default_name="batch_clip_by_l2norm") as name: 44 | saved_shape = tf.shape(t) 45 | batch_size = tf.slice(saved_shape, [0], [1]) 46 | t2 = tf.reshape(t, tf.concat(axis=0, values=[batch_size, [-1]])) 47 | upper_bound_inv = tf.fill(tf.slice(saved_shape, [0], [1]), 48 | tf.constant(1.0/upper_bound)) 49 | # Add a small number to avoid divide by 0 50 | l2norm_inv = tf.rsqrt(tf.reduce_sum(t2 * t2, [1]) + 0.000001) 51 | scale = tf.minimum(l2norm_inv, upper_bound_inv) * upper_bound 52 | clipped_t = tf.matmul(tf.diag(scale), t2) 53 | clipped_t = tf.reshape(clipped_t, saved_shape, name=name) 54 | return clipped_t 55 | 56 | 57 | def AddGaussianNoise(t, sigma, name=None): 58 | """Add i.i.d. Gaussian noise (0, sigma^2) to every entry of t. 59 | 60 | Args: 61 | t: the input tensor. 62 | sigma: the stddev of the Gaussian noise. 63 | name: optional name. 64 | Returns: 65 | the noisy tensor. 66 | """ 67 | 68 | with tf.name_scope(values=[t, sigma], name=name, 69 | default_name="add_gaussian_noise") as name: 70 | noisy_t = t + tf.random_normal(tf.shape(t), stddev=sigma) 71 | return noisy_t 72 | -------------------------------------------------------------------------------- /environments.txt: -------------------------------------------------------------------------------- 1 | # packages 2 | # 3 | keras-applications 1.0.7 py_0 https://repo.continuum.io/pkgs/main 4 | keras-base 2.2.4 py36_0 anaconda 5 | keras-gpu 2.2.4 0 anaconda 6 | keras-preprocessing 1.0.9 py_0 https://repo.continuum.io/pkgs/main 7 | matplotlib 3.0.3 py36h5429711_0 https://repo.continuum.io/pkgs/main 8 | matplotlib-base 3.0.3 py36h5f35d83_1 conda-forge 9 | numpy 1.16.2 py36h7e9f1db_0 https://repo.continuum.io/pkgs/main 10 | numpy-base 1.16.2 py36hde5b4d6_0 https://repo.continuum.io/pkgs/main 11 | scikit-image 0.15.0 py36he1b5a44_0 conda-forge 12 | scikit-learn 0.20.3 py36hd81dba3_0 https://repo.continuum.io/pkgs/main 13 | scipy 1.2.1 py36h7c811a0_0 https://repo.continuum.io/pkgs/main 14 | tensorboard 1.12.2 py36he6710b0_0 https://repo.continuum.io/pkgs/main 15 | tensorflow 1.12.0 gpu_py36he68c306_0 https://repo.continuum.io/pkgs/main 16 | tensorflow-base 1.12.0 gpu_py36h8e0ae2d_0 https://repo.continuum.io/pkgs/main 17 | tensorflow-gpu 1.12.0 h0d30ee6_0 https://repo.continuum.io/pkgs/main -------------------------------------------------------------------------------- /evaluation/train-classifier-celebA.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import numpy as np 5 | import argparse 6 | 7 | parser = argparse.ArgumentParser(description='Train classifier and evaluate their accuracy') 8 | parser.add_argument('--data', type=str, help='datafile name') 9 | parser.add_argument('--range', type=int, default=10, help="range of pickles") 10 | 11 | args = parser.parse_args() 12 | 13 | 14 | import joblib 15 | from tqdm import tqdm 16 | try: 17 | data = joblib.load(args.data) 18 | except: 19 | data = np.zeros((100000, 12290)) 20 | dim = 0 21 | for i in tqdm(range(args.range)): 22 | x = joblib.load(args.data + f'-{i}.pkl') 23 | data[dim: dim+len(x)] = x 24 | dim += len(x) 25 | print(args.data) 26 | print(data.shape) 27 | 28 | 29 | 30 | import tensorflow as tf 31 | 32 | config = tf.ConfigProto() 33 | config.gpu_options.allow_growth = True 34 | # config.gpu_options.per_process_gpu_memory_fraction = 0.3 35 | tf.keras.backend.set_session(tf.Session(config=config)); 36 | 37 | 38 | 39 | 40 | def load_celeb(): 41 | celebA_directory = '../../data/celebA/' 42 | tst_x = joblib.load(celebA_directory + 'celebA-tst-x.pkl') 43 | tst_y = joblib.load(celebA_directory + 'celebA-tst-gender.pkl') 44 | print(tst_y.sum(), len(tst_y)) 45 | from keras.utils import np_utils 46 | tst_y = np_utils.to_categorical(tst_y, 2) 47 | return tst_x, tst_y 48 | 49 | 50 | 51 | x_test, y_test = load_celeb() 52 | 53 | 54 | 55 | def pipeline(data): 56 | print(data.shape) 57 | x, label = np.hsplit(data, [-2]) 58 | nb_classes = 2 59 | label = label.reshape((label.shape[0], nb_classes),order='F') 60 | x = x.reshape(x.shape[0], 64, 64, 3) 61 | 62 | from keras.models import Sequential 63 | from keras.layers.core import Dense, Dropout, Activation, Flatten 64 | from keras.layers.pooling import MaxPooling2D 65 | from keras.layers.convolutional import Convolution2D, Conv2D 66 | from keras.optimizers import Adam 67 | from keras import optimizers 68 | 69 | 70 | model = Sequential() 71 | model.add(Conv2D(32, kernel_size=3, activation='relu', input_shape=(64, 64, 3), name='Conv2D-1')) 72 | model.add(MaxPooling2D(pool_size=2, name='MaxPool')) 73 | model.add(Dropout(0.2, name='Dropout-1')) 74 | model.add(Conv2D(64, kernel_size=3, activation='relu', name='Conv2D-2')) 75 | model.add(Dropout(0.25, name='Dropout-2')) 76 | model.add(Flatten(name='flatten')) 77 | model.add(Dense(64, activation='relu', name='Dense')) 78 | model.add(Dense(nb_classes, activation='softmax', name='Output')) 79 | sgd = optimizers.sgd(lr=1e-4) #, decay=1e-6, momentum=0.9, nesterov=True) 80 | 81 | 82 | model.compile(loss='categorical_crossentropy', 83 | optimizer=sgd, 84 | metrics=['accuracy']) 85 | 86 | print(x.shape) 87 | print(label.shape) 88 | print(x_test.shape) 89 | print(y_test.shape) 90 | evals = model.fit(x, label, batch_size=256, epochs=250, validation_data=(x_test, y_test), shuffle=True) 91 | return evals.history 92 | 93 | 94 | 95 | 96 | 97 | hist = pipeline(data) 98 | print("Max acc:", max(hist['val_acc'])) 99 | -------------------------------------------------------------------------------- /evaluation/train-classifier-fmnist.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import numpy as np 5 | import argparse 6 | 7 | parser = argparse.ArgumentParser(description='Train classifier and evaluate their accuracy') 8 | parser.add_argument('--data', type=str, help='datafile name') 9 | parser.add_argument('--range', type=int, default=1, help="range of pickles") 10 | 11 | args = parser.parse_args() 12 | 13 | data = np.zeros((100000, 794)) 14 | dim = 0 15 | import joblib 16 | from tqdm import tqdm 17 | for i in tqdm(range(args.range)): 18 | x = joblib.load(args.data + f'-{i}.pkl') 19 | data[dim: dim+len(x)] = x 20 | dim += len(x) 21 | print(args.data) 22 | print(data.shape) 23 | 24 | 25 | 26 | import tensorflow as tf 27 | 28 | config = tf.ConfigProto() 29 | config.gpu_options.allow_growth = True 30 | # config.gpu_options.per_process_gpu_memory_fraction = 0.3 31 | tf.keras.backend.set_session(tf.Session(config=config)); 32 | 33 | 34 | 35 | def pipeline(data): 36 | x, label = np.hsplit(data, [-10]) 37 | nb_classes = 10 38 | label = label.reshape((label.shape[0], nb_classes), order='F') 39 | x = x.reshape(x.shape[0], 28, 28, 1) 40 | from keras.datasets import fashion_mnist 41 | (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data() 42 | from keras.utils import np_utils 43 | y_train = np_utils.to_categorical(y_train, 10) 44 | y_test = np_utils.to_categorical(y_test, 10) 45 | x_train = x_train.reshape(x_train.shape[0], 28, 28, 1) 46 | x_train = x_train.astype('float32') / 255. 47 | x_test = x_test.reshape(x_test.shape[0], 28, 28, 1) 48 | x_test = x_test.astype('float32') / 255. 49 | 50 | from keras.models import Sequential 51 | from keras.layers.core import Dense, Dropout, Activation, Flatten 52 | from keras.layers.pooling import MaxPooling2D 53 | from keras.layers.convolutional import Convolution2D, Conv2D 54 | from keras.optimizers import Adam 55 | from keras import optimizers 56 | 57 | model = Sequential() 58 | model.add(Conv2D(32, kernel_size=3, activation='relu', input_shape=(28, 28, 1), name='Conv2D-1')) 59 | model.add(MaxPooling2D(pool_size=2, name='MaxPool')) 60 | model.add(Dropout(0.2, name='Dropout-1')) 61 | model.add(Conv2D(64, kernel_size=3, activation='relu', name='Conv2D-2')) 62 | model.add(Dropout(0.25, name='Dropout-2')) 63 | model.add(Flatten(name='flatten')) 64 | model.add(Dense(64, activation='relu', name='Dense')) 65 | model.add(Dense(nb_classes, activation='softmax', name='Output')) 66 | sgd = optimizers.sgd(lr=2e-3) # , decay=1e-6, momentum=0.9, nesterov=True) 67 | 68 | model.compile(loss='categorical_crossentropy', 69 | optimizer=sgd, 70 | metrics=['accuracy']) 71 | 72 | print(x.shape) 73 | print(label.shape) 74 | print(x_test.shape) 75 | print(y_test.shape) 76 | train_accs = [] 77 | eval_accs = [] 78 | history = model.fit(x, label, batch_size=512, epochs=600, validation_data=(x_test, y_test), shuffle=True) 79 | if 'acc' in history.history: 80 | train_accs = history.history['acc'] 81 | else: 82 | train_accs = history.history['accuracy'] 83 | if 'val_acc' in history.history: 84 | eval_accs = history.history['val_acc'] 85 | else: 86 | eval_accs = history.history['val_accuracy'] 87 | return train_accs, eval_accs 88 | 89 | 90 | 91 | 92 | train_accs, eval_accs = pipeline(data) 93 | print("Max eval acc:", max(eval_accs)) 94 | print("Max train acc:", max(train_accs)) 95 | -------------------------------------------------------------------------------- /evaluation/train-classifier-hair.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | # In[13]: 5 | import numpy as np 6 | import argparse 7 | 8 | parser = argparse.ArgumentParser(description='Train classifier and evaluate their accuracy') 9 | parser.add_argument('--data', type=str, help='datafile name') 10 | parser.add_argument('--range', type=int, default=10, help="range of pickles") 11 | 12 | args = parser.parse_args() 13 | 14 | data = np.zeros((100000, 12291)) 15 | dim = 0 16 | import joblib 17 | from tqdm import tqdm 18 | for i in tqdm(range(args.range)): 19 | x = joblib.load(args.data + f'-{i}.pkl') 20 | data[dim: dim+len(x)] = x 21 | dim += len(x) 22 | print(args.data) 23 | print(data.shape) 24 | 25 | 26 | import tensorflow as tf 27 | 28 | config = tf.ConfigProto() 29 | config.gpu_options.allow_growth = True 30 | # config.gpu_options.per_process_gpu_memory_fraction = 0.3 31 | tf.keras.backend.set_session(tf.Session(config=config)); 32 | 33 | 34 | # In[5]: 35 | 36 | 37 | def load_celeb(): 38 | celebA_directory = '../../data/celebA/' 39 | tst_x = joblib.load(celebA_directory + 'celeb-tst-ups-hair-x.pkl') 40 | tst_y = joblib.load(celebA_directory + 'celeb-tst-ups-hair-y.pkl') 41 | print(tst_y.sum(), len(tst_y)) 42 | from keras.utils import np_utils 43 | tst_y = np_utils.to_categorical(tst_y, 3) 44 | return tst_x, tst_y 45 | 46 | 47 | x_test, y_test = load_celeb() 48 | 49 | 50 | 51 | def pipeline(data): 52 | print(data.shape) 53 | x, label = np.hsplit(data, [-3]) 54 | nb_classes = 3 55 | label = label.reshape((label.shape[0], nb_classes),order='F') 56 | x = x.reshape(x.shape[0], 64, 64, 3) 57 | 58 | from keras.models import Sequential 59 | from keras.layers.core import Dense, Dropout, Activation, Flatten 60 | from keras.layers.pooling import MaxPooling2D 61 | from keras.layers.convolutional import Convolution2D, Conv2D 62 | from keras.optimizers import Adam 63 | from keras import optimizers 64 | 65 | 66 | model = Sequential() 67 | model.add(Conv2D(32, kernel_size=3, activation='relu', input_shape=(64, 64, 3), name='Conv2D-1')) 68 | model.add(MaxPooling2D(pool_size=2, name='MaxPool')) 69 | model.add(Dropout(0.2, name='Dropout-1')) 70 | model.add(Conv2D(64, kernel_size=3, activation='relu', name='Conv2D-2')) 71 | model.add(Dropout(0.25, name='Dropout-2')) 72 | model.add(Flatten(name='flatten')) 73 | model.add(Dense(64, activation='relu', name='Dense')) 74 | model.add(Dense(nb_classes, activation='softmax', name='Output')) 75 | sgd = optimizers.sgd(lr=1e-4) #, decay=1e-6, momentum=0.9, nesterov=True) 76 | 77 | 78 | model.compile(loss='categorical_crossentropy', 79 | optimizer=sgd, 80 | metrics=['accuracy']) 81 | 82 | print(x.shape) 83 | print(label.shape) 84 | print(x_test.shape) 85 | print(y_test.shape) 86 | evals = model.fit(x, label, batch_size=256, epochs=250, validation_data=(x_test, y_test), shuffle=True) 87 | return evals.history 88 | 89 | 90 | 91 | hist = pipeline(data) 92 | print("Max acc:", max(hist['val_acc'])) 93 | 94 | -------------------------------------------------------------------------------- /evaluation/train-classifier-mnist.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import numpy as np 5 | import argparse 6 | 7 | parser = argparse.ArgumentParser(description='Train classifier and evaluate their accuracy') 8 | parser.add_argument('--data', type=str, help='datafile name') 9 | parser.add_argument('--range', type=int, default=10, help="range of pickles") 10 | 11 | args = parser.parse_args() 12 | 13 | data = np.zeros((100000, 794)) 14 | dim = 0 15 | import joblib 16 | from tqdm import tqdm 17 | for i in tqdm(range(args.range)): 18 | x = joblib.load(args.data + f'-{i}.pkl') 19 | data[dim: dim+len(x)] = x 20 | dim += len(x) 21 | print(args.data) 22 | print(data.shape) 23 | 24 | 25 | import tensorflow as tf 26 | 27 | config = tf.ConfigProto() 28 | config.gpu_options.allow_growth = True 29 | tf.keras.backend.set_session(tf.Session(config=config)); 30 | 31 | 32 | def pipeline(data): 33 | x, label = np.hsplit(data, [-10]) 34 | nb_classes = 10 35 | label = label.reshape((label.shape[0], nb_classes), order='F') 36 | x = x.reshape(x.shape[0], 28, 28, 1) 37 | from keras.datasets import mnist 38 | (x_train, y_train), (x_test, y_test) = mnist.load_data() 39 | from keras.utils import np_utils 40 | y_train = np_utils.to_categorical(y_train, 10) 41 | y_test = np_utils.to_categorical(y_test, 10) 42 | x_train = x_train.reshape(x_train.shape[0], 28, 28, 1) 43 | x_train = x_train.astype('float32') / 255. 44 | x_test = x_test.reshape(x_test.shape[0], 28, 28, 1) 45 | x_test = x_test.astype('float32') / 255. 46 | 47 | from keras.models import Sequential 48 | from keras.layers.core import Dense, Dropout, Activation, Flatten 49 | from keras.layers.pooling import MaxPooling2D 50 | from keras.layers.convolutional import Convolution2D, Conv2D 51 | from keras.optimizers import Adam 52 | from keras import optimizers 53 | 54 | model = Sequential() 55 | model.add(Conv2D(32, kernel_size=3, activation='relu', input_shape=(28, 28, 1), name='Conv2D-1')) 56 | model.add(MaxPooling2D(pool_size=2, name='MaxPool')) 57 | model.add(Dropout(0.2, name='Dropout-1')) 58 | model.add(Conv2D(64, kernel_size=3, activation='relu', name='Conv2D-2')) 59 | model.add(Dropout(0.25, name='Dropout-2')) 60 | model.add(Flatten(name='flatten')) 61 | model.add(Dense(64, activation='relu', name='Dense')) 62 | model.add(Dense(nb_classes, activation='softmax', name='Output')) 63 | sgd = optimizers.sgd(lr=2e-3) 64 | 65 | model.compile(loss='categorical_crossentropy', 66 | optimizer=sgd, 67 | metrics=['accuracy']) 68 | 69 | print(x.shape) 70 | print(label.shape) 71 | print(x_test.shape) 72 | print(y_test.shape) 73 | train_accs = [] 74 | eval_accs = [] 75 | history = model.fit(x, label, batch_size=512, epochs=600, validation_data=(x_test, y_test), shuffle=True) 76 | if 'acc' in history.history: 77 | train_accs = history.history['acc'] 78 | else: 79 | train_accs = history.history['accuracy'] 80 | if 'val_acc' in history.history: 81 | eval_accs = history.history['val_acc'] 82 | else: 83 | eval_accs = history.history['val_accuracy'] 84 | return train_accs, eval_accs 85 | 86 | 87 | 88 | 89 | train_accs, eval_accs = pipeline(data) 90 | print("Max eval acc:", max(eval_accs)) 91 | print("Max train acc:", max(train_accs)) 92 | 93 | -------------------------------------------------------------------------------- /fid.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | ''' Calculates the Frechet Inception Distance (FID) to evalulate GANs. 3 | The FID metric calculates the distance between two distributions of images. 4 | Typically, we have summary statistics (mean & covariance matrix) of one 5 | of these distributions, while the 2nd distribution is given by a GAN. 6 | When run as a stand-alone program, it compares the distribution of 7 | images that are stored as PNG/JPEG at a specified location with a 8 | distribution given by summary statistics (in pickle format). 9 | The FID is calculated by assuming that X_1 and X_2 are the activations of 10 | the pool_3 layer of the inception net for generated samples and real world 11 | samples respectivly. 12 | See --help to see further details. 13 | ''' 14 | 15 | from __future__ import absolute_import, division, print_function 16 | import numpy as np 17 | import os 18 | import gzip, pickle 19 | import tensorflow as tf 20 | from scipy import linalg 21 | import pathlib 22 | import urllib 23 | 24 | 25 | class InvalidFIDException(Exception): 26 | pass 27 | 28 | 29 | def create_inception_graph(pth): 30 | """Creates a graph from saved GraphDef file.""" 31 | # Creates graph from saved graph_def.pb. 32 | with tf.gfile.FastGFile( pth, 'rb') as f: 33 | graph_def = tf.GraphDef() 34 | graph_def.ParseFromString( f.read()) 35 | _ = tf.import_graph_def( graph_def, name='FID_Inception_Net') 36 | #------------------------------------------------------------------------------- 37 | 38 | 39 | # code for handling inception net derived from 40 | # https://github.com/openai/improved-gan/blob/master/inception_score/model.py 41 | def _get_inception_layer(sess): 42 | """Prepares inception net for batched usage and returns pool_3 layer. """ 43 | layername = 'FID_Inception_Net/pool_3:0' 44 | pool3 = sess.graph.get_tensor_by_name(layername) 45 | ops = pool3.graph.get_operations() 46 | for op_idx, op in enumerate(ops): 47 | for o in op.outputs: 48 | shape = o.get_shape() 49 | if shape._dims != []: 50 | shape = [s.value for s in shape] 51 | new_shape = [] 52 | for j, s in enumerate(shape): 53 | if s == 1 and j == 0: 54 | new_shape.append(None) 55 | else: 56 | new_shape.append(s) 57 | o.__dict__['_shape_val'] = tf.TensorShape(new_shape) 58 | return pool3 59 | #------------------------------------------------------------------------------- 60 | 61 | 62 | def get_activations(images, sess, batch_size=50, verbose=False): 63 | """Calculates the activations of the pool_3 layer for all images. 64 | Params: 65 | -- images : Numpy array of dimension (n_images, hi, wi, 3). The values 66 | must lie between 0 and 256. 67 | -- sess : current session 68 | -- batch_size : the images numpy array is split into batches with batch size 69 | batch_size. A reasonable batch size depends on the disposable hardware. 70 | -- verbose : If set to True and parameter out_step is given, the number of calculated 71 | batches is reported. 72 | Returns: 73 | -- A numpy array of dimension (num images, 2048) that contains the 74 | activations of the given tensor when feeding inception with the query tensor. 75 | """ 76 | inception_layer = _get_inception_layer(sess) 77 | d0 = images.shape[0] 78 | if batch_size > d0: 79 | print("warning: batch size is bigger than the data size. setting batch size to data size") 80 | batch_size = d0 81 | n_batches = d0//batch_size 82 | n_used_imgs = n_batches*batch_size 83 | pred_arr = np.empty((n_used_imgs,2048)) 84 | from tqdm import tqdm 85 | for i in tqdm(range(n_batches)): 86 | if verbose: 87 | print("\rPropagating batch %d/%d" % (i+1, n_batches), end="", flush=True) 88 | start = i*batch_size 89 | end = start + batch_size 90 | batch = images[start:end] 91 | pred = sess.run(inception_layer, {'FID_Inception_Net/ExpandDims:0': batch}) 92 | pred_arr[start:end] = pred.reshape(batch_size,-1) 93 | if verbose: 94 | print(" done") 95 | return pred_arr 96 | #------------------------------------------------------------------------------- 97 | 98 | 99 | def calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6): 100 | """Numpy implementation of the Frechet Distance. 101 | The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1) 102 | and X_2 ~ N(mu_2, C_2) is 103 | d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)). 104 | Stable version by Dougal J. Sutherland. 105 | Params: 106 | -- mu1 : Numpy array containing the activations of the pool_3 layer of the 107 | inception net ( like returned by the function 'get_predictions') 108 | for generated samples. 109 | -- mu2 : The sample mean over activations of the pool_3 layer, precalcualted 110 | on an representive data set. 111 | -- sigma1: The covariance matrix over activations of the pool_3 layer for 112 | generated samples. 113 | -- sigma2: The covariance matrix over activations of the pool_3 layer, 114 | precalcualted on an representive data set. 115 | Returns: 116 | -- : The Frechet Distance. 117 | """ 118 | 119 | mu1 = np.atleast_1d(mu1) 120 | mu2 = np.atleast_1d(mu2) 121 | 122 | sigma1 = np.atleast_2d(sigma1) 123 | sigma2 = np.atleast_2d(sigma2) 124 | 125 | assert mu1.shape == mu2.shape, "Training and test mean vectors have different lengths" 126 | assert sigma1.shape == sigma2.shape, "Training and test covariances have different dimensions" 127 | 128 | diff = mu1 - mu2 129 | 130 | # product might be almost singular 131 | covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False) 132 | if not np.isfinite(covmean).all(): 133 | msg = "fid calculation produces singular product; adding %s to diagonal of cov estimates" % eps 134 | warnings.warn(msg) 135 | offset = np.eye(sigma1.shape[0]) * eps 136 | covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset)) 137 | 138 | # numerical error might give slight imaginary component 139 | if np.iscomplexobj(covmean): 140 | if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3): 141 | m = np.max(np.abs(covmean.imag)) 142 | raise ValueError("Imaginary component {}".format(m)) 143 | covmean = covmean.real 144 | 145 | tr_covmean = np.trace(covmean) 146 | 147 | return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean 148 | #------------------------------------------------------------------------------- 149 | 150 | 151 | def calculate_activation_statistics(images, sess, batch_size=512, verbose=False): 152 | """Calculation of the statistics used by the FID. 153 | Params: 154 | -- images : Numpy array of dimension (n_images, hi, wi, 3). The values 155 | must lie between 0 and 255. 156 | -- sess : current session 157 | -- batch_size : the images numpy array is split into batches with batch size 158 | batch_size. A reasonable batch size depends on the available hardware. 159 | -- verbose : If set to True and parameter out_step is given, the number of calculated 160 | batches is reported. 161 | Returns: 162 | -- mu : The mean over samples of the activations of the pool_3 layer of 163 | the incption model. 164 | -- sigma : The covariance matrix of the activations of the pool_3 layer of 165 | the incption model. 166 | """ 167 | act = get_activations(images, sess, batch_size, verbose) 168 | mu = np.mean(act, axis=0) 169 | sigma = np.cov(act, rowvar=False) 170 | return mu, sigma 171 | #------------------------------------------------------------------------------- 172 | 173 | 174 | #------------------------------------------------------------------------------- 175 | # The following functions aren't needed for calculating the FID 176 | # they're just here to make this module work as a stand-alone script 177 | # for calculating FID scores 178 | #------------------------------------------------------------------------------- 179 | def check_or_download_inception(inception_path): 180 | ''' Checks if the path to the inception file is valid, or downloads 181 | the file if it is not present. ''' 182 | INCEPTION_URL = 'http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz' 183 | if inception_path is None: 184 | inception_path = '/tmp' 185 | inception_path = pathlib.Path(inception_path) 186 | model_file = inception_path / 'classify_image_graph_def.pb' 187 | if not model_file.exists(): 188 | print("Downloading Inception model") 189 | from urllib import request 190 | import tarfile 191 | fn, _ = request.urlretrieve(INCEPTION_URL) 192 | with tarfile.open(fn, mode='r') as f: 193 | f.extract('classify_image_graph_def.pb', str(model_file.parent)) 194 | return str(model_file) 195 | 196 | 197 | def _handle_path(path, sess): 198 | if path.endswith('.npz'): 199 | f = np.load(path) 200 | m, s = f['mu'][:], f['sigma'][:] 201 | f.close() 202 | else: 203 | path = pathlib.Path(path) 204 | files = list(path.glob('*.jpg')) + list(path.glob('*.png')) 205 | x = np.array([imread(str(fn)).astype(np.float32) for fn in files]) 206 | m, s = calculate_activation_statistics(x, sess) 207 | return m, s 208 | 209 | 210 | def calculate_fid_given_paths(paths, inception_path): 211 | ''' Calculates the FID of two paths. ''' 212 | inception_path = check_or_download_inception(inception_path) 213 | 214 | for p in paths: 215 | if not os.path.exists(p): 216 | raise RuntimeError("Invalid path: %s" % p) 217 | 218 | create_inception_graph(str(inception_path)) 219 | with tf.Session() as sess: 220 | sess.run(tf.global_variables_initializer()) 221 | m1, s1 = _handle_path(paths[0], sess) 222 | m2, s2 = _handle_path(paths[1], sess) 223 | fid_value = calculate_frechet_distance(m1, s1, m2, s2) 224 | return fid_value 225 | 226 | def preprocess_mnist(data): 227 | from sklearn.utils import shuffle 228 | data = shuffle(data) 229 | data = data[:50000,:784].reshape(-1, 28, 28, 1) * 255 230 | data = np.concatenate((data, data, data), axis=3) 231 | data = np.float32(data) 232 | return data 233 | 234 | def calculate_fid_given_pkls(real_path, synthetic_path, inception_path): 235 | inception_path = check_or_download_inception(inception_path) 236 | import joblib 237 | 238 | real = joblib.load(real_path) 239 | fake = joblib.load(synthetic_path) 240 | 241 | 242 | real = preprocess_mnist(real) 243 | fake = preprocess_mnist(fake) 244 | 245 | create_inception_graph(str(inception_path)) 246 | with tf.Session() as sess: 247 | real_stats = calculate_activation_statistics(real, sess, verbose=True) 248 | fake_stats = calculate_activation_statistics(fake, sess, verbose=True) 249 | 250 | fid_value = calculate_frechet_distance(real_stats[0], real_stats[1], fake_stats[0], fake_stats[1]) 251 | return fid_value 252 | 253 | 254 | if __name__ == "__main__": 255 | from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter 256 | parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter) 257 | parser.add_argument("--path1", type=str, 258 | help='Path to the generated images or to .npz statistic files') 259 | parser.add_argument("--path2", type=str, 260 | help='Path to the generated images or to .npz statistic files') 261 | parser.add_argument("-i", "--inception", type=str, default=None, 262 | help='Path to Inception model (will be downloaded if not provided)') 263 | parser.add_argument("--gpu", default="", type=str, 264 | help='GPU to use (leave blank for CPU only)') 265 | args = parser.parse_args() 266 | # os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu 267 | # fid_value = calculate_fid_given_paths(args.path, args.inception) 268 | fid_value = calculate_fid_given_pkls(args.path1, args.path2, args.inception) 269 | print("FID: ", fid_value) -------------------------------------------------------------------------------- /gen_data.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import argparse 3 | 4 | x = [2, 2, 5, 8, 8, 9, 10, 11, 11, 12, 15, 28, 32, 42, 46, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101] 5 | y = [[-1, 2], [-1, 2], [-1, 5], [-1, 8], [-1, 8], [-1, 9], [-1, 10], [-1, 11], [-1, 11], [-1, 12], [-1, 15], [-1, 28], [-1, 32], [-1, 42], [-1, 46], [1451516400, 1483225200], [1451602843, 1483311589], [1451602962, 1483311743], [1451603006, 1483311791], [1451603122, 1483311883], [1451603285, 1483311990], [1451604148, 1483312891], [1451604963, 1483314738], [-122.513648358854, -122.332574620522], [1, 5], [1451603253, 1483316817], [1, 83], [37.6168823239251, 37.8544643401172], [955490400, 1539122400]] 6 | for i in range(1,29): 7 | x[i] += x[i - 1] 8 | 9 | FEATURES = [ 10 | 'ALS Unit', 11 | 'Final Priority', 12 | 'Call Type Group', 13 | 'Original Priority', 14 | 'Priority', 15 | 'City', 16 | 'Unit Type', 17 | 'Fire Prevention District', 18 | 'Battalion', 19 | 'Supervisor District', 20 | 'Call Final Disposition', 21 | 'Zipcode of Incident', 22 | 'Call Type', 23 | 'Neighborhooods - Analysis Boundaries', 24 | 'Station Area', 25 | 'Watch Date', 26 | 'Received DtTm', 27 | 'Entry DtTm', 28 | 'Dispatch DtTm', 29 | 'Response DtTm', 30 | 'On Scene DtTm', 31 | 'Transport DtTm', 32 | 'Hospital DtTm', 33 | 'Location - Lng', 34 | 'Number of Alarms', 35 | 'Available DtTm', 36 | 'Unit sequence in call dispatch', 37 | 'Location - Lat', 38 | 'Call Date', 39 | 'Unit ID', 40 | 'Box', 41 | 'Address', 42 | ] 43 | 44 | def data2str(ans, n_dim=29): 45 | temp = "" 46 | for i in range(n_dim): 47 | if (i == 0): 48 | tmp = ans[:x[i]] 49 | else: 50 | tmp = ans[x[i - 1]:x[i]] 51 | _ = np.argmax(tmp) 52 | if (i == 0): 53 | temp += str(_) 54 | else: 55 | if (x[i] - x[i - 1] == 101): 56 | if (_ == 100): 57 | temp += "," 58 | else: 59 | step = float(y[i][1] - y[i][0]) / 100 60 | value = y[i][0] + (_ + 0.5) * step 61 | if (i != 23 and i != 27): 62 | temp += "," + str(int(round(value))) 63 | else: 64 | 65 | temp += "," + str(value) 66 | else: 67 | temp += "," + str(_) 68 | return temp 69 | 70 | def batch2str(data, out_file, n_dim=29, n_features=20): 71 | g = open(out_file, "w+") 72 | temp = '' 73 | for i in range(n_features): 74 | if i > 0: 75 | temp += ',' 76 | temp += FEATURES[i] 77 | g.write(temp + "\n") 78 | 79 | for i in range(data.shape[0]): 80 | temp = data2str(data[i,:], n_dim = n_dim) 81 | g.write(temp + "\n") 82 | g.close() 83 | 84 | -------------------------------------------------------------------------------- /inception_score.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch import nn 3 | from torch.autograd import Variable 4 | from torch.nn import functional as F 5 | import torch.utils.data 6 | 7 | from torchvision.models.inception import inception_v3 8 | 9 | import numpy as np 10 | from scipy.stats import entropy 11 | 12 | def inception_score(imgs, cuda=True, batch_size=32, resize=False, splits=1): 13 | """Computes the inception score of the generated images imgs 14 | imgs -- Torch dataset of (3xHxW) numpy images normalized in the range [-1, 1] 15 | cuda -- whether or not to run on GPU 16 | batch_size -- batch size for feeding into Inception v3 17 | splits -- number of splits 18 | """ 19 | N = len(imgs) 20 | 21 | assert batch_size > 0 22 | assert N > batch_size 23 | 24 | # Set up dtype 25 | if cuda: 26 | dtype = torch.cuda.FloatTensor 27 | else: 28 | if torch.cuda.is_available(): 29 | print("WARNING: You have a CUDA device, so you should probably set cuda=True") 30 | dtype = torch.FloatTensor 31 | 32 | # Set up dataloader 33 | dataloader = torch.utils.data.DataLoader(imgs, batch_size=batch_size) 34 | 35 | # Load inception model 36 | inception_model.eval(); 37 | up = nn.Upsample(size=(299, 299), mode='bilinear').type(dtype) 38 | def get_pred(x): 39 | if resize: 40 | x = up(x) 41 | x = inception_model(x) 42 | return F.softmax(x).data.cpu().numpy() 43 | 44 | # Get predictions 45 | preds = np.zeros((N, 1000)) 46 | 47 | from tqdm import tqdm 48 | for i, batch in tqdm(enumerate(dataloader, 0)): 49 | batch = batch.type(dtype) 50 | batchv = Variable(batch) 51 | batch_size_i = batch.size()[0] 52 | 53 | preds[i*batch_size:i*batch_size + batch_size_i] = get_pred(batchv) 54 | 55 | # Now compute the mean kl-div 56 | split_scores = [] 57 | 58 | for k in range(splits): 59 | part = preds[k * (N // splits): (k+1) * (N // splits), :] 60 | py = np.mean(part, axis=0) 61 | scores = [] 62 | for i in range(part.shape[0]): 63 | pyx = part[i, :] 64 | scores.append(entropy(pyx, py)) 65 | split_scores.append(np.exp(np.mean(scores))) 66 | 67 | return np.mean(split_scores), np.std(split_scores) 68 | 69 | class MnistDataset(torch.utils.data.Dataset): 70 | def __init__(self, orig, transform): 71 | self.transform = transform 72 | self.orig = orig 73 | 74 | def __getitem__(self, index): 75 | return self.transform(self.orig[index]) 76 | 77 | def __len__(self): 78 | return len(self.orig) 79 | 80 | dtype = torch.cuda.FloatTensor 81 | inception_model = inception_v3(pretrained=True, transform_input=False).type(dtype) 82 | 83 | def pipeline(datapath): 84 | import joblib 85 | import torchvision.transforms as transforms 86 | data = joblib.load(datapath) 87 | from sklearn.utils import shuffle 88 | data = shuffle(data) 89 | data = data[:10000,:784].reshape(-1, 28, 28, 1) 90 | data = np.concatenate((data, data, data), axis=3) 91 | data = np.float32(data) 92 | transform = transforms.Compose([ 93 | transforms.ToTensor(), 94 | transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) 95 | ]) 96 | data = MnistDataset(data, transform=transform) 97 | print(inception_score(data, cuda=True, batch_size=64, resize=True, splits=10)) 98 | 99 | 100 | 101 | if __name__ == '__main__': 102 | # (2.358112431133864, 0.04182949711390127) 103 | pipeline('/home/ubuntu/disk2/wbxshm/mnist_z_dim_50_topk_200_teacher_4000_sigma_5000_thresh_0.7_pt_30_d_step_2_v2_stochastic_b_1e-5_v2/eps-1.00.data') 104 | -------------------------------------------------------------------------------- /input.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The TensorFlow Authors. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ============================================================================== 15 | 16 | # TODO update load mnist 17 | 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import _pickle as cPickle 23 | import gzip 24 | import math 25 | import numpy as np 26 | import os 27 | from scipy.io import loadmat as loadmat 28 | from six.moves import urllib 29 | from six.moves import xrange 30 | import sys 31 | import tarfile 32 | 33 | import tensorflow as tf 34 | 35 | def create_dir_if_needed(dest_directory): 36 | """ 37 | Create directory if doesn't exist 38 | :param dest_directory: 39 | :return: True if everything went well 40 | """ 41 | if not tf.gfile.IsDirectory(dest_directory): 42 | tf.gfile.MakeDirs(dest_directory) 43 | 44 | return True 45 | 46 | # Test if file already exists 47 | if not tf.gfile.Exists(filepath): 48 | def _progress(count, block_size, total_size): 49 | sys.stdout.write('\r>> Downloading %s %.1f%%' % (filename, 50 | float(count * block_size) / float(total_size) * 100.0)) 51 | sys.stdout.flush() 52 | filepath, _ = urllib.request.urlretrieve(file_url, filepath, _progress) 53 | print() 54 | statinfo = os.stat(filepath) 55 | print('Successfully downloaded', filename, statinfo.st_size, 'bytes.') 56 | 57 | return result 58 | 59 | 60 | def image_whitening(data): 61 | """ 62 | Subtracts mean of image and divides by adjusted standard variance (for 63 | stability). Operations are per image but performed for the entire array. 64 | :param image: 4D array (ID, Height, Weight, Channel) 65 | :return: 4D array (ID, Height, Weight, Channel) 66 | """ 67 | assert len(np.shape(data)) == 4 68 | 69 | # Compute number of pixels in image 70 | nb_pixels = np.shape(data)[1] * np.shape(data)[2] * np.shape(data)[3] 71 | 72 | # Subtract mean 73 | mean = np.mean(data, axis=(1,2,3)) 74 | 75 | ones = np.ones(np.shape(data)[1:4], dtype=np.float32) 76 | for i in xrange(len(data)): 77 | data[i, :, :, :] -= mean[i] * ones 78 | 79 | # Compute adjusted standard variance 80 | adj_std_var = np.maximum(np.ones(len(data), dtype=np.float32) / math.sqrt(nb_pixels), np.std(data, axis=(1,2,3))) #NOLINT(long-line) 81 | 82 | # Divide image 83 | for i in xrange(len(data)): 84 | data[i, :, :, :] = data[i, :, :, :] / adj_std_var[i] 85 | 86 | print(np.shape(data)) 87 | 88 | return data 89 | 90 | def ld_mnist(data_dir, dataset_name): 91 | 92 | data_dir = os.path.join(data_dir, dataset_name) 93 | fd = open(os.path.join(data_dir,'train-images-idx3-ubyte')) 94 | loaded = np.fromfile(file=fd,dtype=np.uint8) 95 | 96 | trX = loaded[16:].reshape((60000,28,28,1)).astype(np.float) 97 | 98 | fd = open(os.path.join(data_dir,'train-labels-idx1-ubyte')) 99 | loaded = np.fromfile(file=fd,dtype=np.uint8) 100 | trY = loaded[8:].reshape((60000)).astype(np.float) 101 | 102 | fd = open(os.path.join(data_dir,'t10k-images-idx3-ubyte')) 103 | loaded = np.fromfile(file=fd,dtype=np.uint8) 104 | teX = loaded[16:].reshape((10000,28,28,1)).astype(np.float) 105 | 106 | fd = open(os.path.join(data_dir,'t10k-labels-idx1-ubyte')) 107 | loaded = np.fromfile(file=fd,dtype=np.uint8) 108 | teY = loaded[8:].reshape((10000)).astype(np.float) 109 | 110 | trY = np.asarray(trY) 111 | teY = np.asarray(teY) 112 | 113 | X = np.concatenate((trX, teX), axis=0) 114 | y = np.concatenate((trY, teY), axis=0).astype(np.int) 115 | 116 | seed = 547 117 | np.random.seed(seed) 118 | np.random.shuffle(X) 119 | np.random.seed(seed) 120 | np.random.shuffle(y) 121 | 122 | y_vec = np.zeros((len(y), self.y_dim), dtype=np.float) 123 | for i, label in enumerate(y): 124 | y_vec[i,y[i]] = 1.0 125 | 126 | return X/255.,y_vec 127 | 128 | 129 | def partition_dataset(data, labels, nb_teachers, teacher_id): 130 | """ 131 | Simple partitioning algorithm that returns the right portion of the data 132 | needed by a given teacher out of a certain nb of teachers 133 | :param data: input data to be partitioned 134 | :param labels: output data to be partitioned 135 | :param nb_teachers: number of teachers in the ensemble (affects size of each 136 | partition) 137 | :param teacher_id: id of partition to retrieve 138 | :return: 139 | """ 140 | 141 | # Sanity check 142 | assert(len(data) == len(labels)) 143 | assert(int(teacher_id) < int(nb_teachers)) 144 | 145 | # This will floor the possible number of batches 146 | batch_len = int(len(data) / nb_teachers) 147 | 148 | # Compute start, end indices of partition 149 | start = teacher_id * batch_len 150 | end = (teacher_id+1) * batch_len 151 | 152 | # Slice partition off 153 | partition_data = data[start:end] 154 | partition_labels = labels[start:end] 155 | 156 | return partition_data, partition_labels 157 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import scipy.misc 3 | import numpy as np 4 | 5 | from model import DCGAN 6 | from utils import pp, visualize, to_json, show_all_variables, mkdir 7 | 8 | import tensorflow as tf 9 | import argparse 10 | from gen_data import batch2str 11 | import sys 12 | import pickle 13 | 14 | # os.environ["CUDA_VISIBLE_DEVICES"] = "1" 15 | 16 | flags = tf.app.flags 17 | flags.DEFINE_integer("epoch", 1000, "Epoch for training teacher models") 18 | flags.DEFINE_integer("g_epoch", 500, "Epoch for training the student models") 19 | flags.DEFINE_float("learning_rate", 1e-3, "Learning rate of for adam") 20 | flags.DEFINE_float("beta1", 0.5, "Momentum term of adam [0.5]") 21 | flags.DEFINE_float("train_size", np.inf, "The size of train images [np.inf]") 22 | flags.DEFINE_integer("batch_size", 30, "The size of batch images [64]") 23 | flags.DEFINE_integer("input_height", 32, "The size of image to use (will be center cropped). [108]") 24 | flags.DEFINE_integer("input_width", 32, 25 | "The size of image to use (will be center cropped). If None, same value as input_height [None]") 26 | flags.DEFINE_integer("output_height", 32, "The size of the output images to produce [64]") 27 | flags.DEFINE_integer("output_width", 32, 28 | "The size of the output images to produce. If None, same value as output_height [None]") 29 | flags.DEFINE_string("dataset", "slt", "The name of dataset [cinic, celebA, mnist, lsun, fire-small]") 30 | flags.DEFINE_string("checkpoint_dir", "checkpoint", "Directory name to save the checkpoints [checkpoint]") 31 | flags.DEFINE_string("checkpoint_name", "checkpoint", "checkpoint model name [checkpoint]") 32 | 33 | flags.DEFINE_string("data_dir", "../../data", "Root directory of dataset [data]") 34 | flags.DEFINE_string("sample_dir", "samples", "Directory name to save the image samples [samples]") 35 | flags.DEFINE_boolean("train", False, "True for training, False for testing [False]") 36 | flags.DEFINE_boolean("pretrain", True, "True for loading the pretrained models, False for not load [True]") 37 | flags.DEFINE_boolean("load_d", True, 38 | "True for loading the pretrained models w/ discriminator, False for not load [True]") 39 | flags.DEFINE_boolean("crop", False, "True for cropping") 40 | flags.DEFINE_integer("orders", 200, "rdp orders") 41 | flags.DEFINE_integer("proj_mat", 1, "#/ projection mat") 42 | flags.DEFINE_integer("z_dim", 100, "#/ z dim") 43 | flags.DEFINE_integer("y_dim", 10, "#/ y dim") 44 | flags.DEFINE_boolean("tanh", False, "Use tanh as activation func") 45 | 46 | flags.DEFINE_boolean("random_proj", True, "Apply pca for gradient aggregation ") 47 | flags.DEFINE_boolean("simple_gan", False, "Use fc to build GAN") 48 | flags.DEFINE_boolean("mean_kernel", False, "Apply Mean Kernel for gradient agggregation") 49 | flags.DEFINE_boolean("signsgd", False, "Apply sign sgd for gradient agggregation") 50 | flags.DEFINE_boolean("signsgd_nothresh", False, "Apply sign sgd for gradient agggregation") 51 | flags.DEFINE_boolean("klevelsgd", False, "Apply klevel sgd for gradient agggregation") 52 | flags.DEFINE_boolean("sketchsgd", False, "Apply sketch sgd for gradient agggregation") 53 | flags.DEFINE_boolean("signsgd_dept", False, "Apply sign sgd for gradient agggregation with data dependent bound") 54 | flags.DEFINE_boolean("stochastic", False, "Apply stochastic sign sgd for gradient agggregation") 55 | flags.DEFINE_integer("pretrain_teacher", 0, "Pretrain teacher for epochs") 56 | flags.DEFINE_boolean("save_vote", False, "Save voting results") 57 | flags.DEFINE_boolean("pca", False, "Apply pca for gradient aggregation ") 58 | flags.DEFINE_boolean("non_private", False, "Do not apply differential privacy") 59 | flags.DEFINE_boolean("increasing_dim", False, "Increase the projection dimension for each epoch") 60 | flags.DEFINE_boolean("wgan", False, "Train wgan") 61 | flags.DEFINE_boolean("small", False, "Use a smaller discriminator") 62 | flags.DEFINE_float("sigma", 2000.0, "Scale of gaussian noise for gradient aggregation") 63 | flags.DEFINE_float("sigma_thresh", 4500.0, "Scale of gaussian noise for thresh gnmax") 64 | flags.DEFINE_float("pca_sigma", 1.0, "Scale of gaussian noise for dp pca") 65 | flags.DEFINE_float("step_size", 1e-4, "Step size for gradient aggregation") 66 | flags.DEFINE_float("delta", 1e-5, "delta for differential privacy") 67 | flags.DEFINE_integer("g_step", 1, "steps of the generator") 68 | flags.DEFINE_integer("d_step", 1, "steps of the discriminator") 69 | flags.DEFINE_integer("pca_dim", 10, "principal dimensions for pca") 70 | flags.DEFINE_float("thresh", 0.5, "threshhold for threshgmax") 71 | flags.DEFINE_float("max_eps", 1, "maximum epsilon") 72 | flags.DEFINE_float("max_grad", 0, "maximum gradient for signsgd aggregation") 73 | flags.DEFINE_boolean("random_label", False, "random labels for training data, only used when pretraining some models") 74 | flags.DEFINE_boolean("shuffle", True, "Evenly distribute dataset") 75 | flags.DEFINE_boolean("save_epoch", False, "Save each epoch per 0.1 eps") 76 | flags.DEFINE_integer("batch_teachers", 1, "Number of teacher models in one batch") 77 | flags.DEFINE_integer("teachers_batch", 1, "Number of batch") 78 | flags.DEFINE_integer("topk", 50, "Number of top k gradients") 79 | flags.DEFINE_integer("klevel", 4, "Levels of gradient quantization") 80 | flags.DEFINE_string("teacher_dir", "teacher", "Directory name to save the teacher [teacher]") 81 | flags.DEFINE_string("generator_dir", "generator", "Directory name to save the generator") 82 | flags.DEFINE_string("loss", "l1", "AE reconstruction loss") 83 | flags.DEFINE_string("ae", "", "AE model name") 84 | flags.DEFINE_boolean("train_ae", False, "Train ae") 85 | flags.DEFINE_boolean("finetune_ae", False, "Finetune ae") 86 | flags.DEFINE_integer("sample_step", 10, "Number of teacher models in one batch") 87 | flags.DEFINE_integer("hid_dim", 512, "Dimmension of hidden dim") 88 | FLAGS = flags.FLAGS 89 | 90 | 91 | def main(_): 92 | pp.pprint(flags.FLAGS.flag_values_dict()) 93 | 94 | if not os.path.exists(FLAGS.checkpoint_dir): 95 | os.makedirs(FLAGS.checkpoint_dir) 96 | if not os.path.exists(FLAGS.sample_dir): 97 | os.makedirs(FLAGS.sample_dir) 98 | 99 | # gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333) 100 | run_config = tf.ConfigProto() 101 | run_config.gpu_options.allow_growth = True 102 | 103 | if FLAGS.thresh == 0: 104 | thresh = None 105 | else: 106 | thresh = FLAGS.thresh 107 | 108 | if FLAGS.wgan: 109 | FLAGS.learning_rate = 5e-5 110 | FLAGS.step_size = 5e-4 111 | 112 | with tf.Session(config=run_config) as sess: 113 | 114 | dcgan = DCGAN( 115 | sess, 116 | batch_size=FLAGS.batch_size, 117 | sample_num=FLAGS.batch_size, 118 | y_dim=FLAGS.y_dim, 119 | z_dim=FLAGS.z_dim, 120 | dataset_name=FLAGS.dataset, 121 | crop=FLAGS.crop, 122 | checkpoint_dir=FLAGS.checkpoint_dir, 123 | sample_dir=FLAGS.sample_dir, 124 | data_dir=FLAGS.data_dir, 125 | # parameters to tune 126 | batch_teachers=FLAGS.batch_teachers, 127 | pca=FLAGS.pca, 128 | random_proj=FLAGS.random_proj, 129 | thresh=thresh, 130 | dp_delta=FLAGS.delta, 131 | pca_dim=FLAGS.pca_dim, 132 | teachers_batch=FLAGS.teachers_batch, 133 | teacher_dir=os.path.join(FLAGS.checkpoint_dir, FLAGS.teacher_dir), 134 | generator_dir=FLAGS.generator_dir, 135 | non_private=FLAGS.non_private, 136 | input_height=FLAGS.input_height, 137 | input_width=FLAGS.input_width, 138 | output_height=FLAGS.output_height, 139 | output_width=FLAGS.output_width, 140 | wgan=FLAGS.wgan, 141 | small=FLAGS.small, 142 | config=FLAGS 143 | ) 144 | 145 | show_all_variables() 146 | 147 | if FLAGS.train_ae and FLAGS.ae: 148 | dcgan.train_ae() 149 | elif FLAGS.finetune_ae and FLAGS.ae: 150 | dcgan.finetune_ae() 151 | else: 152 | if FLAGS.train: 153 | if FLAGS.ae: 154 | pass 155 | else: 156 | epsilon, delta = dcgan.train_together(FLAGS) 157 | filename = '%.2fepsilon-%.2fdelta.data' % (epsilon, delta) 158 | else: 159 | if not dcgan.load(FLAGS.checkpoint_dir, FLAGS.checkpoint_name)[0]: 160 | raise Exception("[!] Train a model first, then run test mode") 161 | filename = 'private.data' 162 | 163 | outpath = os.path.join(FLAGS.checkpoint_dir, FLAGS.sample_dir) 164 | if not os.path.exists(outpath): 165 | os.makedirs(outpath) 166 | 167 | outfile = os.path.join(outpath, filename) 168 | n_batch = 100000 // FLAGS.batch_size + 1 169 | data = dcgan.gen_data(n_batch) 170 | data = data[:100000] 171 | import joblib 172 | 173 | joblib.dump(data, outfile) 174 | 175 | 176 | if __name__ == '__main__': 177 | tf.app.run() 178 | -------------------------------------------------------------------------------- /mnist_cnn_icp_eval.py: -------------------------------------------------------------------------------- 1 | import gzip 2 | import os 3 | 4 | # GPUID = 0 5 | # os.environ["CUDA_VISIBLE_DEVICES"] = str(GPUID) 6 | 7 | from scipy import ndimage 8 | from six.moves import urllib 9 | import numpy as np 10 | import tensorflow as tf 11 | import tensorflow.contrib.slim as slim 12 | import math 13 | import sys 14 | import scipy.io 15 | 16 | 17 | import pdb 18 | 19 | print ("PACKAGES LOADED") 20 | 21 | 22 | 23 | def CNN(inputs, _is_training=True): 24 | x = tf.reshape(inputs, [-1, 28, 28, 1]) 25 | batch_norm_params = {'is_training': _is_training, 'decay': 0.9, 'updates_collections': None} 26 | net = slim.conv2d(x, 32, [5, 5], padding='SAME' 27 | , activation_fn = tf.nn.relu 28 | , weights_initializer = tf.truncated_normal_initializer(stddev=0.01) 29 | , normalizer_fn = slim.batch_norm 30 | , normalizer_params = batch_norm_params 31 | , scope='conv1') 32 | net = slim.max_pool2d(net, [2, 2], scope='pool1') 33 | net = slim.conv2d(net, 64, [5, 5], scope='conv2') 34 | net = slim.max_pool2d(net, [2, 2], scope='pool2') 35 | net = slim.flatten(net, scope='flatten3') 36 | net = slim.fully_connected(net, 1024 37 | , activation_fn = tf.nn.relu 38 | , weights_initializer = tf.truncated_normal_initializer(stddev=0.01) 39 | , normalizer_fn = slim.batch_norm 40 | , normalizer_params = batch_norm_params 41 | , scope='fc4') 42 | net = slim.dropout(net, keep_prob=0.7, is_training=_is_training, scope='dropout4') 43 | out = slim.fully_connected(net, 10, activation_fn=None, normalizer_fn=None, scope='fco') 44 | return out 45 | 46 | 47 | # DATA URL 48 | SOURCE_URL = 'http://yann.lecun.com/exdb/mnist/' 49 | DATA_DIRECTORY = "data" 50 | # PARAMETERS FOR MNIST 51 | IMAGE_SIZE = 28 52 | NUM_CHANNELS = 1 53 | PIXEL_DEPTH = 255 54 | NUM_LABELS = 10 55 | VALIDATION_SIZE = 5000 # Size of the validation set. 56 | 57 | # DOWNLOAD MNIST DATA, IF NECESSARY 58 | def maybe_download(filename): 59 | if not tf.gfile.Exists(DATA_DIRECTORY): 60 | tf.gfile.MakeDirs(DATA_DIRECTORY) 61 | filepath = os.path.join(DATA_DIRECTORY, filename) 62 | if not tf.gfile.Exists(filepath): 63 | filepath, _ = urllib.request.urlretrieve(SOURCE_URL + filename, filepath) 64 | with tf.gfile.GFile(filepath) as f: 65 | size = f.size() 66 | print('Successfully downloaded', filename, size, 'bytes.') 67 | return filepath 68 | 69 | # EXTRACT IMAGES 70 | def extract_data(filename, num_images): 71 | with gzip.open(filename) as bytestream: 72 | bytestream.read(16) 73 | buf = bytestream.read(IMAGE_SIZE * IMAGE_SIZE * num_images * NUM_CHANNELS) 74 | data = np.frombuffer(buf, dtype=np.uint8).astype(np.float32) 75 | data = (data - (PIXEL_DEPTH / 2.0)) / PIXEL_DEPTH # -0.5~0.5 76 | data = data.reshape(num_images, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS) 77 | data = np.reshape(data, [num_images, -1]) 78 | return data # [image index, y, x, channels] 79 | 80 | # EXTRACT LABELS 81 | def extract_labels(filename, num_images): 82 | with gzip.open(filename) as bytestream: 83 | bytestream.read(8) 84 | buf = bytestream.read(1 * num_images) 85 | labels = np.frombuffer(buf, dtype=np.uint8).astype(np.int64) 86 | num_labels_data = len(labels) 87 | one_hot_encoding = np.zeros((num_labels_data,NUM_LABELS)) 88 | one_hot_encoding[np.arange(num_labels_data),labels] = 1 89 | one_hot_encoding = np.reshape(one_hot_encoding, [-1, NUM_LABELS]) 90 | return one_hot_encoding 91 | 92 | # AUGMENT TRAINING DATA 93 | def expend_training_data(images, labels): 94 | expanded_images = [] 95 | expanded_labels = [] 96 | j = 0 # counter 97 | for x, y in zip(images, labels): 98 | j = j+1 99 | # APPEND ORIGINAL DATA 100 | expanded_images.append(x) 101 | expanded_labels.append(y) 102 | # ASSUME MEDIAN COLOR TO BE BACKGROUND COLOR 103 | bg_value = np.median(x) # this is regarded as background's value 104 | image = np.reshape(x, (-1, 28)) 105 | 106 | for i in range(4): 107 | # ROTATE IMAGE 108 | angle = np.random.randint(-15,15,1) 109 | new_img = ndimage.rotate(image,angle,reshape=False, cval=bg_value) 110 | # SHIFT IAMGE 111 | shift = np.random.randint(-2, 2, 2) 112 | new_img_ = ndimage.shift(new_img,shift, cval=bg_value) 113 | # ADD TO THE LIST 114 | expanded_images.append(np.reshape(new_img_, 784)) 115 | expanded_labels.append(y) 116 | expanded_train_total_data = np.concatenate((expanded_images, expanded_labels), axis=1) 117 | np.random.shuffle(expanded_train_total_data) 118 | return expanded_train_total_data 119 | 120 | # PREPARE MNIST DATA 121 | def prepare_MNIST_data(use_data_augmentation=True): 122 | # Get the data. 123 | train_data_filename = maybe_download('train-images-idx3-ubyte.gz') 124 | train_labels_filename = maybe_download('train-labels-idx1-ubyte.gz') 125 | test_data_filename = maybe_download('t10k-images-idx3-ubyte.gz') 126 | test_labels_filename = maybe_download('t10k-labels-idx1-ubyte.gz') 127 | train_data = extract_data(train_data_filename, 60000) 128 | train_labels = extract_labels(train_labels_filename, 60000) 129 | test_data = extract_data(test_data_filename, 10000) 130 | test_labels = extract_labels(test_labels_filename, 10000) 131 | validation_data = train_data[:VALIDATION_SIZE, :] 132 | validation_labels = train_labels[:VALIDATION_SIZE,:] 133 | train_data = train_data[VALIDATION_SIZE:, :] 134 | train_labels = train_labels[VALIDATION_SIZE:,:] 135 | if use_data_augmentation: 136 | train_total_data = expend_training_data(train_data, train_labels) 137 | else: 138 | train_total_data = np.concatenate((train_data, train_labels), axis=1) 139 | train_size = train_total_data.shape[0] 140 | return train_total_data, train_size, validation_data, validation_labels, test_data, test_labels 141 | 142 | 143 | 144 | # CONFIGURATION 145 | # MODEL_DIRECTORY = "./model/model.ckpt" 146 | MODEL_DIRECTORY = "../model/model.ckpt" 147 | LOGS_DIRECTORY = "logs/train" 148 | training_epochs = 10 149 | TRAIN_BATCH_SIZE = 50 150 | display_step = 500 151 | validation_step = 500 152 | TEST_BATCH_SIZE = 5000 153 | 154 | 155 | # PREPARE MNIST DATA 156 | batch_size = TRAIN_BATCH_SIZE # BATCH SIZE (50) 157 | num_labels = NUM_LABELS # NUMBER OF LABELS (10) 158 | # train_total_data, train_size, validation_data, validation_labels \ 159 | # , test_data, test_labels = prepare_MNIST_data(False) 160 | # PRINT FUNCTION 161 | def print_np(x, str): 162 | print (" TYPE AND SHAPE OF [%18s ] ARE %s and %14s" 163 | % (str, type(x), x.shape,)) 164 | # print_np(train_total_data, 'train_total_data') 165 | # print_np(validation_data, 'validation_data') 166 | # print_np(validation_labels, 'validation_labels') 167 | # print_np(test_data, 'test_data') 168 | # print_np(test_labels, 'test_labels') 169 | ################################################################################ 170 | 171 | 172 | 173 | # DEFINE MODEL 174 | # PLACEHOLDERS 175 | x = tf.placeholder(tf.float32, [None, 784]) 176 | y_ = tf.placeholder(tf.float32, [None, 10]) #answer 177 | is_training = tf.placeholder(tf.bool, name='MODE') 178 | # CONVOLUTIONAL NEURAL NETWORK MODEL 179 | y = CNN(x, is_training) 180 | # DEFINE LOSS 181 | with tf.name_scope("LOSS"): 182 | loss = slim.losses.softmax_cross_entropy(y, y_) 183 | # DEFINE ACCURACY 184 | with tf.name_scope("ACC"): 185 | correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1)) 186 | accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 187 | 188 | saver = tf.train.Saver() 189 | 190 | 191 | # OPEN SESSION 192 | config = tf.ConfigProto() 193 | config.gpu_options.allow_growth=True 194 | # config.gpu_options.per_process_gpu_memory_fraction = 0.2 195 | sess = tf.Session(config=config) 196 | sess.run(tf.global_variables_initializer(), feed_dict={is_training: False}) 197 | 198 | 199 | def softmax(x): 200 | """Compute softmax values for each sets of scores in x.""" 201 | e_x = np.exp(x - np.max(x)) 202 | return e_x / np.expand_dims(e_x.sum(axis=1), axis=1) # only difference 203 | 204 | # RESTORE SAVED NETWORK 205 | saver.restore(sess, MODEL_DIRECTORY) 206 | print("model restored") 207 | 208 | from config import * 209 | import joblib 210 | 211 | args = parse_arguments() 212 | 213 | # folders for generated images 214 | result_folder = './ali_results/' 215 | 216 | icp = [] 217 | try: 218 | data = joblib.load(args.exp_name) 219 | except: 220 | from tqdm import tqdm 221 | 222 | data = np.zeros((100000, 794)) 223 | dim = 0 224 | for i in tqdm(range(10)): 225 | data_x = joblib.load(args.exp_name + f'-{i}.pkl') 226 | data[dim: dim + len(data_x)] = data_x 227 | dim += len(data_x) 228 | print(data.shape) 229 | if 'test' not in args.exp_name: 230 | test_x = data[:, :784] 231 | test_x -= 0.5 232 | 233 | for k in range(3): 234 | k = k + 1 235 | # result_folder = '../GAN/ali_bigan/ali_shell_results/' 236 | # mat = scipy.io.loadmat(result_folder+ '2_2_2_256_256_1024.mat' ) 237 | # mat = scipy.io.loadmat(result_folder+ '{}.mat'.format(str(k).zfill(3))) 238 | from sklearn.utils import shuffle 239 | test_x = shuffle(test_x) 240 | # label = label.reshape((label.shape[0], nb_classes), order='F') 241 | test_data = test_x.reshape(test_x.shape[0], 784) 242 | 243 | # test_data = mat['images'] 244 | # pdb.set_trace() 245 | # COMPUTE ACCURACY FOR TEST DATA 246 | batch_size = 512 247 | test_size = test_data.shape[0] 248 | total_batch = int(test_size / batch_size) 249 | acc_buffer = [] 250 | preds = [] 251 | for i in range(total_batch): 252 | offset = (i * batch_size) % (test_size) 253 | batch_xs = test_data[offset:(offset + batch_size), :] 254 | y_final = sess.run(y, feed_dict={x: batch_xs, is_training: False}) 255 | pred_softmax = softmax(y_final) 256 | preds.append(pred_softmax) 257 | 258 | 259 | preds = np.concatenate(preds, 0) 260 | scores = [] 261 | splits = 10 262 | for i in range(splits): 263 | part = preds[(i * preds.shape[0] // splits):((i + 1) * preds.shape[0] // splits), :] 264 | kl = part * (np.log(part) - np.log(np.expand_dims(np.mean(part, 0), 0))) 265 | kl = np.mean(np.sum(kl, 1)) 266 | scores.append(np.exp(kl)) 267 | 268 | icp.append((np.mean(scores) , np.std(scores))) 269 | print("Inception score is: %.4f, %.4f" % (np.mean(scores) , np.std(scores))) 270 | 271 | # scipy.io.savemat('ali_inception_50.mat', mdict={'icp': icp}) 272 | joblib.dump(icp, 'inception.pkl') 273 | -------------------------------------------------------------------------------- /ops.py: -------------------------------------------------------------------------------- 1 | import math 2 | import numpy as np 3 | import tensorflow as tf 4 | 5 | from tensorflow.python.framework import ops 6 | 7 | from utils import * 8 | 9 | try: 10 | image_summary = tf.image_summary 11 | scalar_summary = tf.scalar_summary 12 | histogram_summary = tf.histogram_summary 13 | merge_summary = tf.merge_summary 14 | SummaryWriter = tf.train.SummaryWriter 15 | except: 16 | image_summary = tf.summary.image 17 | scalar_summary = tf.summary.scalar 18 | histogram_summary = tf.summary.histogram 19 | merge_summary = tf.summary.merge 20 | SummaryWriter = tf.summary.FileWriter 21 | 22 | if "concat_v2" in dir(tf): 23 | def concat(tensors, axis, *args, **kwargs): 24 | return tf.concat_v2(tensors, axis, *args, **kwargs) 25 | else: 26 | def concat(tensors, axis, *args, **kwargs): 27 | return tf.concat(tensors, axis, *args, **kwargs) 28 | 29 | class batch_norm(object): 30 | def __init__(self, epsilon=1e-5, momentum = 0.9, name="batch_norm"): 31 | with tf.variable_scope(name): 32 | self.epsilon = epsilon 33 | self.momentum = momentum 34 | self.name = name 35 | self.bn = None 36 | 37 | def __call__(self, x, train=True): 38 | return tf.contrib.layers.batch_norm(x, 39 | decay=self.momentum, 40 | updates_collections=None, 41 | epsilon=self.epsilon, 42 | scale=True, 43 | is_training=train, 44 | scope=self.name) 45 | 46 | def conv_cond_concat(x, y): 47 | """Concatenate conditioning vector on feature map axis.""" 48 | x_shapes = x.get_shape() 49 | y_shapes = y.get_shape() 50 | return concat([ 51 | x, y*tf.ones([x_shapes[0], x_shapes[1], x_shapes[2], y_shapes[3]])], 3) 52 | 53 | def conv2d(input_, output_dim, 54 | k_h=5, k_w=5, d_h=2, d_w=2, stddev=0.02, 55 | name="conv2d"): 56 | with tf.variable_scope(name): 57 | w = tf.get_variable('w', [k_h, k_w, input_.get_shape()[-1], output_dim], 58 | initializer=tf.truncated_normal_initializer(stddev=stddev)) 59 | conv = tf.nn.conv2d(input_, w, strides=[1, d_h, d_w, 1], padding='SAME') 60 | 61 | biases = tf.get_variable('biases', [output_dim], initializer=tf.constant_initializer(0.0)) 62 | conv = tf.reshape(tf.nn.bias_add(conv, biases), conv.get_shape()) 63 | 64 | return conv 65 | 66 | def deconv2d(input_, output_shape, 67 | k_h=5, k_w=5, d_h=2, d_w=2, stddev=0.02, 68 | name="deconv2d", with_w=False): 69 | with tf.variable_scope(name): 70 | # filter : [height, width, output_channels, in_channels] 71 | w = tf.get_variable('w', [k_h, k_w, output_shape[-1], input_.get_shape()[-1]], 72 | initializer=tf.random_normal_initializer(stddev=stddev)) 73 | 74 | try: 75 | deconv = tf.nn.conv2d_transpose(input_, w, output_shape=output_shape, 76 | strides=[1, d_h, d_w, 1]) 77 | 78 | # Support for verisons of TensorFlow before 0.7.0 79 | except AttributeError: 80 | deconv = tf.nn.deconv2d(input_, w, output_shape=output_shape, 81 | strides=[1, d_h, d_w, 1]) 82 | 83 | biases = tf.get_variable('biases', [output_shape[-1]], initializer=tf.constant_initializer(0.0)) 84 | deconv = tf.reshape(tf.nn.bias_add(deconv, biases), deconv.get_shape()) 85 | 86 | if with_w: 87 | return deconv, w, biases 88 | else: 89 | return deconv 90 | 91 | def lrelu(x, leak=0.2, name="lrelu"): 92 | return tf.maximum(x, leak*x) 93 | 94 | def linear(input_, output_size, scope=None, stddev=0.02, bias_start=0.0, with_w=False): 95 | shape = input_.get_shape().as_list() 96 | 97 | with tf.variable_scope(scope or "Linear"): 98 | try: 99 | matrix = tf.get_variable("Matrix", [shape[1], output_size], tf.float32, 100 | tf.random_normal_initializer(stddev=stddev)) 101 | except ValueError as err: 102 | msg = "NOTE: Usually, this is due to an issue with the image dimensions. Did you correctly set '--crop' or '--input_height' or '--output_height'?" 103 | err.args = err.args + (msg,) 104 | raise 105 | bias = tf.get_variable("bias", [output_size], 106 | initializer=tf.constant_initializer(bias_start)) 107 | if with_w: 108 | return tf.matmul(input_, matrix) + bias, matrix, bias 109 | else: 110 | return tf.matmul(input_, matrix) + bias 111 | -------------------------------------------------------------------------------- /pate_core.py: -------------------------------------------------------------------------------- 1 | 2 | # core method from 'Scalable Private Learning with PATE' 3 | 4 | # Copyright 2017 The 'Scalable Private Learning with PATE' Authors All Rights Reserved. 5 | # 6 | # Licensed under the Apache License, Version 2.0 (the "License"); 7 | # you may not use this file except in compliance with the License. 8 | # You may obtain a copy of the License at 9 | # 10 | # http://www.apache.org/licenses/LICENSE-2.0 11 | # 12 | # Unless required by applicable law or agreed to in writing, software 13 | # distributed under the License is distributed on an "AS IS" BASIS, 14 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 15 | # See the License for the specific language governing permissions and 16 | # limitations under the License. 17 | # ============================================================================== 18 | 19 | """Core functions for RDP analysis in PATE framework. 20 | 21 | This library comprises the core functions for doing differentially private 22 | analysis of the PATE architecture and its various Noisy Max and other 23 | mechanisms. 24 | """ 25 | from __future__ import absolute_import 26 | from __future__ import division 27 | from __future__ import print_function 28 | 29 | import math 30 | 31 | from absl import app 32 | import numpy as np 33 | import scipy.stats 34 | 35 | 36 | def _logaddexp(x): 37 | """Addition in the log space. Analogue of numpy.logaddexp for a list.""" 38 | m = max(x) 39 | return m + math.log(sum(np.exp(x - m))) 40 | 41 | 42 | def _log1mexp(x): 43 | """Numerically stable computation of log(1-exp(x)).""" 44 | if x < -1: 45 | return math.log1p(-math.exp(x)) 46 | elif x < 0: 47 | return math.log(-math.expm1(x)) 48 | elif x == 0: 49 | return -np.inf 50 | else: 51 | raise ValueError("Argument must be non-positive.") 52 | 53 | 54 | def compute_eps_from_delta(orders, rdp, delta): 55 | """Translates between RDP and (eps, delta)-DP. 56 | 57 | Args: 58 | orders: A list (or a scalar) of orders. 59 | rdp: A list of RDP guarantees (of the same length as orders). 60 | delta: Target delta. 61 | 62 | Returns: 63 | Pair of (eps, optimal_order). 64 | 65 | Raises: 66 | ValueError: If input is malformed. 67 | """ 68 | if len(orders) != len(rdp): 69 | raise ValueError("Input lists must have the same length.") 70 | eps = np.array(rdp) - math.log(delta) / (np.array(orders) - 1) 71 | idx_opt = np.argmin(eps) 72 | return eps[idx_opt], orders[idx_opt] 73 | 74 | 75 | ##################### 76 | # RDP FOR THE GNMAX # 77 | ##################### 78 | 79 | 80 | def compute_logq_gaussian(counts, sigma): 81 | """Returns an upper bound on ln Pr[outcome != argmax] for GNMax. 82 | 83 | Implementation of Proposition 7. 84 | 85 | Args: 86 | counts: A numpy array of scores. 87 | sigma: The standard deviation of the Gaussian noise in the GNMax mechanism. 88 | 89 | Returns: 90 | logq: Natural log of the probability that outcome is different from argmax. 91 | """ 92 | n = len(counts) 93 | variance = sigma**2 94 | idx_max = np.argmax(counts) 95 | counts_normalized = counts[idx_max] - counts 96 | counts_rest = counts_normalized[np.arange(n) != idx_max] # exclude one index 97 | # Upper bound q via a union bound rather than a more precise calculation. 98 | logq = _logaddexp( 99 | scipy.stats.norm.logsf(counts_rest, scale=math.sqrt(2 * variance))) 100 | print("log q:", logq) 101 | # A sketch of a more accurate estimate, which is currently disabled for two 102 | # reasons: 103 | # 1. Numerical instability; 104 | # 2. Not covered by smooth sensitivity analysis. 105 | # from statsmodels.sandbox.distributions import extras 106 | # covariance = variance * (np.ones((n - 1, n - 1)) + np.identity(n - 1)) 107 | # from scipy.stats import multivariate_normal 108 | # print(multivariate_normal.cdf( 109 | # counts_rest, np.zeros(n - 1), covariance, maxpts=1e12)) 110 | # logq = np.log1p(-multivariate_normal.cdf( 111 | # counts_rest, np.zeros(n - 1), covariance, maxpts=1e12)) 112 | # print("log q 2:", logq) 113 | # print("another:", math.log(1 - (1 / n))) 114 | return min(logq, math.log(1 - (1 / n))) 115 | 116 | 117 | def rdp_data_independent_gaussian(sigma, orders): 118 | """Computes a data-independent RDP curve for GNMax. 119 | 120 | Implementation of Proposition 8. 121 | 122 | Args: 123 | sigma: Standard deviation of Gaussian noise. 124 | orders: An array_like list of Renyi orders. 125 | 126 | Returns: 127 | Upper bound on RPD for all orders. A scalar if orders is a scalar. 128 | 129 | Raises: 130 | ValueError: If the input is malformed. 131 | """ 132 | if sigma < 0 or np.any(orders <= 1): # not defined for alpha=1 133 | raise ValueError("Inputs are malformed.") 134 | 135 | variance = sigma**2 136 | if np.isscalar(orders): 137 | return orders / variance 138 | else: 139 | return np.atleast_1d(orders) / variance 140 | 141 | 142 | def rdp_gaussian(logq, sigma, orders): 143 | """Bounds RDP from above of GNMax given an upper bound on q (Theorem 6). 144 | 145 | Args: 146 | logq: Natural logarithm of the probability of a non-argmax outcome. 147 | sigma: Standard deviation of Gaussian noise. 148 | orders: An array_like list of Renyi orders. 149 | 150 | Returns: 151 | Upper bound on RPD for all orders. A scalar if orders is a scalar. 152 | 153 | Raises: 154 | ValueError: If the input is malformed. 155 | """ 156 | if logq > 0 or sigma < 0 or np.any(orders <= 1): # not defined for alpha=1 157 | raise ValueError("Inputs are malformed.") 158 | 159 | if np.isneginf(logq): # If the mechanism's output is fixed, it has 0-DP. 160 | if np.isscalar(orders): 161 | return 0. 162 | else: 163 | return np.full_like(orders, 0., dtype=np.float) 164 | 165 | variance = sigma**2 166 | 167 | # Use two different higher orders: mu_hi1 and mu_hi2 computed according to 168 | # Proposition 10. 169 | mu_hi2 = math.sqrt(variance * -logq) 170 | mu_hi1 = mu_hi2 + 1 171 | 172 | orders_vec = np.atleast_1d(orders) 173 | 174 | ret = orders_vec / variance # baseline: data-independent bound 175 | 176 | # Filter out entries where data-dependent bound does not apply. 177 | mask = np.logical_and(mu_hi1 > orders_vec, mu_hi2 > 1) 178 | print('mask len: ', np.sum(mask)) 179 | 180 | rdp_hi1 = mu_hi1 / variance 181 | rdp_hi2 = mu_hi2 / variance 182 | 183 | log_a2 = (mu_hi2 - 1) * rdp_hi2 184 | 185 | # Make sure q is in the increasing wrt q range and A is positive. 186 | if (np.any(mask) and logq <= log_a2 - mu_hi2 * 187 | (math.log(1 + 1 / (mu_hi1 - 1)) + math.log(1 + 1 / (mu_hi2 - 1))) and 188 | -logq > rdp_hi2): 189 | # Use log1p(x) = log(1 + x) to avoid catastrophic cancellations when x ~ 0. 190 | log1q = _log1mexp(logq) # log1q = log(1-q) 191 | log_a = (orders - 1) * ( 192 | log1q - _log1mexp((logq + rdp_hi2) * (1 - 1 / mu_hi2))) 193 | log_b = (orders - 1) * (rdp_hi1 - logq / (mu_hi1 - 1)) 194 | 195 | # Use logaddexp(x, y) = log(e^x + e^y) to avoid overflow for large x, y. 196 | log_s = np.logaddexp(log1q + log_a, logq + log_b) 197 | print('choose data dependent bound: ', np.sum((ret > log_s / (orders - 1))[mask])) 198 | ret[mask] = np.minimum(ret, log_s / (orders - 1))[mask] 199 | else: 200 | print("Condition not satisfied:") 201 | # print("np.any(mask): ", np.any(mask)) 202 | # print("logq <= log_a2 - mu_hi2 * (math.log(1 + 1 / (mu_hi1 - 1)) + math.log(1 + 1 / (mu_hi2 - 1))):", logq <= log_a2 - mu_hi2 * (math.log(1 + 1 / (mu_hi1 - 1)) + math.log(1 + 1 / (mu_hi2 - 1)))) 203 | # print("-logq > rdp_hi2:", -logq > rdp_hi2) 204 | 205 | assert np.all(ret >= 0) 206 | 207 | if np.isscalar(orders): 208 | return np.asscalar(ret) 209 | else: 210 | return ret 211 | 212 | 213 | def double_rdp_gaussian(logq, sigma, orders): 214 | """Bounds RDP from above of GNMax given an upper bound on q (Theorem 6). 215 | 216 | Args: 217 | logq: Natural logarithm of the probability of a non-argmax outcome. 218 | sigma: Standard deviation of Gaussian noise. 219 | orders: An array_like list of Renyi orders. 220 | 221 | Returns: 222 | Upper bound on RPD for all orders. A scalar if orders is a scalar. 223 | 224 | Raises: 225 | ValueError: If the input is malformed. 226 | """ 227 | if logq > 0 or sigma < 0 or np.any(orders <= 1): # not defined for alpha=1 228 | raise ValueError("Inputs are malformed.") 229 | 230 | if np.isneginf(logq): # If the mechanism's output is fixed, it has 0-DP. 231 | if np.isscalar(orders): 232 | return 0. 233 | else: 234 | return np.full_like(orders, 0., dtype=np.float) 235 | 236 | variance = sigma**2 237 | 238 | # Use two different higher orders: mu_hi1 and mu_hi2 computed according to 239 | # Proposition 10. 240 | mu_hi2 = math.sqrt(variance * -logq) 241 | mu_hi1 = mu_hi2 + 1 242 | 243 | orders_vec = np.atleast_1d(orders) 244 | 245 | ret = orders_vec / variance # baseline: data-independent bound 246 | ret1 = np.array(ret) 247 | 248 | # Filter out entries where data-dependent bound does not apply. 249 | mask = np.logical_and(mu_hi1 > orders_vec, mu_hi2 > 1) 250 | print('mask len: ', np.sum(mask)) 251 | 252 | rdp_hi1 = mu_hi1 / variance 253 | rdp_hi2 = mu_hi2 / variance 254 | 255 | log_a2 = (mu_hi2 - 1) * rdp_hi2 256 | 257 | # Make sure q is in the increasing wrt q range and A is positive. 258 | if (np.any(mask) and logq <= log_a2 - mu_hi2 * 259 | (math.log(1 + 1 / (mu_hi1 - 1)) + math.log(1 + 1 / (mu_hi2 - 1))) and 260 | -logq > rdp_hi2): 261 | # Use log1p(x) = log(1 + x) to avoid catastrophic cancellations when x ~ 0. 262 | log1q = _log1mexp(logq) # log1q = log(1-q) 263 | log_a = (orders - 1) * ( 264 | log1q - _log1mexp((logq + rdp_hi2) * (1 - 1 / mu_hi2))) 265 | log_b = (orders - 1) * (rdp_hi1 - logq / (mu_hi1 - 1)) 266 | 267 | # Use logaddexp(x, y) = log(e^x + e^y) to avoid overflow for large x, y. 268 | log_s = np.logaddexp(log1q + log_a, logq + log_b) 269 | print('choose data dependent bound: ', np.sum((ret > log_s / (orders - 1))[mask])) 270 | # always use data-dependent bound without comparison 271 | ret1[mask] = (log_s / (orders - 1))[mask] 272 | else: 273 | print("Condition not satisfied") 274 | 275 | 276 | assert np.all(ret >= 0) 277 | 278 | if np.isscalar(orders): 279 | return np.asscalar(ret1), np.asscalar(ret) 280 | else: 281 | return ret1, ret 282 | 283 | 284 | def is_data_independent_always_opt_gaussian(num_teachers, num_classes, sigma, 285 | orders): 286 | """Tests whether data-ind bound is always optimal for GNMax. 287 | 288 | Args: 289 | num_teachers: Number of teachers. 290 | num_classes: Number of classes. 291 | sigma: Standard deviation of the Gaussian noise. 292 | orders: An array_like list of Renyi orders. 293 | 294 | Returns: 295 | Boolean array of length |orders| (a scalar if orders is a scalar). True if 296 | the data-independent bound is always the same as the data-dependent bound. 297 | 298 | """ 299 | unanimous = np.array([num_teachers] + [0] * (num_classes - 1)) 300 | logq = compute_logq_gaussian(unanimous, sigma) 301 | 302 | rdp_dep = rdp_gaussian(logq, sigma, orders) 303 | rdp_ind = rdp_data_independent_gaussian(sigma, orders) 304 | return np.isclose(rdp_dep, rdp_ind) 305 | 306 | 307 | ################################### 308 | # RDP FOR THE THRESHOLD MECHANISM # 309 | ################################### 310 | 311 | 312 | def compute_logpr_answered(t, sigma, counts): 313 | """Computes log of the probability that a noisy threshold is crossed. 314 | 315 | Args: 316 | t: The threshold. 317 | sigma: The stdev of the Gaussian noise added to the threshold. 318 | counts: An array of votes. 319 | 320 | Returns: 321 | Natural log of the probability that max is larger than a noisy threshold. 322 | """ 323 | # Compared to the paper, max(counts) is rounded to the nearest integer. This 324 | # is done to facilitate computation of smooth sensitivity for the case of 325 | # the interactive mechanism, where votes are not necessarily integer. 326 | return scipy.stats.norm.logsf(t - round(max(counts)), scale=sigma) 327 | 328 | 329 | def compute_rdp_data_independent_threshold(sigma, orders): 330 | # The input to the threshold mechanism has stability 1, compared to 331 | # GNMax, which has stability = 2. Hence the sqrt(2) factor below. 332 | return rdp_data_independent_gaussian(2**.5 * sigma, orders) 333 | 334 | 335 | def compute_rdp_threshold(log_pr_answered, sigma, orders): 336 | logq = min(log_pr_answered, _log1mexp(log_pr_answered)) 337 | # The input to the threshold mechanism has stability 1, compared to 338 | # GNMax, which has stability = 2. Hence the sqrt(2) factor below. 339 | return rdp_gaussian(logq, 2**.5 * sigma, orders) 340 | 341 | 342 | def is_data_independent_always_opt_threshold(num_teachers, threshold, sigma, 343 | orders): 344 | """Tests whether data-ind bound is always optimal for the threshold mechanism. 345 | 346 | Args: 347 | num_teachers: Number of teachers. 348 | threshold: The cut-off threshold. 349 | sigma: Standard deviation of the Gaussian noise. 350 | orders: An array_like list of Renyi orders. 351 | 352 | Returns: 353 | Boolean array of length |orders| (a scalar if orders is a scalar). True if 354 | the data-independent bound is always the same as the data-dependent bound. 355 | """ 356 | 357 | # Since the data-dependent bound depends only on max(votes), it suffices to 358 | # check whether the data-dependent bounds are better than data-independent 359 | # bounds in the extreme cases when max(votes) is minimal or maximal. 360 | # For both Confident GNMax and Interactive GNMax it holds that 361 | # 0 <= max(votes) <= num_teachers. 362 | # The upper bound is trivial in both cases. 363 | # The lower bound is trivial for Confident GNMax (and a stronger one, based on 364 | # the pigeonhole principle, is possible). 365 | # For Interactive GNMax (Algorithm 2), the lower bound follows from the 366 | # following argument. Since the votes vector is the difference between the 367 | # actual teachers' votes and the student's baseline, we need to argue that 368 | # max(n_j - M * p_j) >= 0. 369 | # The bound holds because sum_j n_j = sum M * p_j = M. Thus, 370 | # sum_j (n_j - M * p_j) = 0, and max_j (n_j - M * p_j) >= 0 as needed. 371 | logq1 = compute_logpr_answered(threshold, sigma, [0]) 372 | logq2 = compute_logpr_answered(threshold, sigma, [num_teachers]) 373 | 374 | rdp_dep1 = compute_rdp_threshold(logq1, sigma, orders) 375 | rdp_dep2 = compute_rdp_threshold(logq2, sigma, orders) 376 | 377 | rdp_ind = compute_rdp_data_independent_threshold(sigma, orders) 378 | return np.isclose(rdp_dep1, rdp_ind) and np.isclose(rdp_dep2, rdp_ind) 379 | 380 | 381 | ############################# 382 | # RDP FOR THE LAPLACE NOISE # 383 | ############################# 384 | 385 | 386 | def compute_logq_laplace(counts, lmbd): 387 | """Computes an upper bound on log Pr[outcome != argmax] for LNMax. 388 | 389 | Args: 390 | counts: A list of scores. 391 | lmbd: The lambda parameter of the Laplace distribution ~exp(-|x| / lambda). 392 | 393 | Returns: 394 | logq: Natural log of the probability that outcome is different from argmax. 395 | """ 396 | # For noisy max, we only get an upper bound via the union bound. See Lemma 4 397 | # in https://arxiv.org/abs/1610.05755. 398 | # 399 | # Pr[ j beats i*] = (2+gap(j,i*))/ 4 exp(gap(j,i*) 400 | # proof at http://mathoverflow.net/questions/66763/ 401 | 402 | idx_max = np.argmax(counts) 403 | counts_normalized = (counts - counts[idx_max]) / lmbd 404 | counts_rest = np.array( 405 | [counts_normalized[i] for i in range(len(counts)) if i != idx_max]) 406 | 407 | logq = _logaddexp(np.log(2 - counts_rest) + math.log(.25) + counts_rest) 408 | 409 | return min(logq, math.log(1 - (1 / len(counts)))) 410 | 411 | 412 | def rdp_pure_eps(logq, pure_eps, orders): 413 | """Computes the RDP value given logq and pure privacy eps. 414 | 415 | Implementation of https://arxiv.org/abs/1610.05755, Theorem 3. 416 | 417 | The bound used is the min of three terms. The first term is from 418 | https://arxiv.org/pdf/1605.02065.pdf. 419 | The second term is based on the fact that when event has probability (1-q) for 420 | q close to zero, q can only change by exp(eps), which corresponds to a 421 | much smaller multiplicative change in (1-q) 422 | The third term comes directly from the privacy guarantee. 423 | 424 | Args: 425 | logq: Natural logarithm of the probability of a non-optimal outcome. 426 | pure_eps: eps parameter for DP 427 | orders: array_like list of moments to compute. 428 | 429 | Returns: 430 | Array of upper bounds on rdp (a scalar if orders is a scalar). 431 | """ 432 | orders_vec = np.atleast_1d(orders) 433 | q = math.exp(logq) 434 | log_t = np.full_like(orders_vec, np.inf) 435 | if q <= 1 / (math.exp(pure_eps) + 1): 436 | logt_one = math.log1p(-q) + ( 437 | math.log1p(-q) - _log1mexp(pure_eps + logq)) * ( 438 | orders_vec - 1) 439 | logt_two = logq + pure_eps * (orders_vec - 1) 440 | log_t = np.logaddexp(logt_one, logt_two) 441 | 442 | ret = np.minimum( 443 | np.minimum(0.5 * pure_eps * pure_eps * orders_vec, 444 | log_t / (orders_vec - 1)), pure_eps) 445 | if np.isscalar(orders): 446 | return np.asscalar(ret) 447 | else: 448 | return ret 449 | 450 | 451 | def main(argv): 452 | del argv # Unused. 453 | 454 | 455 | if __name__ == "__main__": 456 | app.run(main) 457 | -------------------------------------------------------------------------------- /rdp_utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import math 3 | import sys 4 | 5 | import torch 6 | from sklearn.preprocessing import normalize 7 | from pate_core import * 8 | from numpy import linalg as LA 9 | 10 | EPS = sys.float_info.epsilon 11 | 12 | 13 | # Algorithm 1 in 'Scalable Private Learning with PATE' 14 | def gnmax_thresh_aggregator(counts, thresh_cnt, sigma_thresh, sigma, orders): 15 | log_pr_answered = compute_logpr_answered(thresh_cnt, sigma_thresh, counts) 16 | rdp_budget = compute_rdp_threshold(log_pr_answered, sigma_thresh, orders) 17 | # print("Threshold budget:" + str(rdp_budget)) 18 | 19 | if np.random.normal(np.max(counts), sigma_thresh) >= thresh_cnt: 20 | logq = compute_logq_gaussian(counts, sigma) 21 | res = np.argmax(np.random.normal(counts, sigma)) 22 | g_rdp_budget = rdp_gaussian(logq, sigma, orders) 23 | rdp_budget += g_rdp_budget 24 | else: 25 | # do not return result if teacher models do not agree 26 | res = -1 27 | 28 | return res, rdp_budget 29 | 30 | 31 | def gnmax_aggregator(counts, sigma, orders): 32 | logq = compute_logq_gaussian(counts, sigma) 33 | dir_index = np.argmax(np.random.normal(counts, sigma)) 34 | rdp_budget = rdp_gaussian(logq, sigma, orders) 35 | return dir_index, rdp_budget 36 | 37 | 38 | def rdp_percentile(arr_list, q, orders, vmin, vmax, lmbd, axis=0): 39 | arr_length = len(arr_list) 40 | arr_size = arr_list[0].size 41 | input_shape = arr_list[0].shape 42 | arr_reshaped = np.vstack([arr.reshape([1, arr_size]) for arr in arr_list]) 43 | 44 | arr_ordered = np.sort(arr_reshaped, axis=0) 45 | arr_ordered = arr_ordered.clip(min=vmin, max=vmax) 46 | 47 | arr_ordered_new = np.vstack([np.ones([1, arr_size]) * vmin, arr_ordered, np.ones([1, arr_size]) * vmax]) 48 | arr_ordered_new[np.abs(arr_ordered_new) < sys.float_info.epsilon] = 0 49 | 50 | n_teachers, n_feature = arr_reshaped.shape 51 | arr_prob = np.zeros([n_teachers + 1, n_feature]) 52 | 53 | for i in range(arr_length + 1): 54 | diff = arr_ordered_new[i + 1, :] - arr_ordered_new[i, :] 55 | diff = diff.clip(min=0) 56 | arr_prob[i] = diff * np.exp(-0.5 / lmbd * abs(i - q / 100 * arr_length)) 57 | # arr_prob[i] = np.exp(np.log(diff) - 0.5/lmbd * abs(i - q/100 * arr_length)) 58 | 59 | # arr_prob = normalize(arr_prob, norm='l1', axis=0) 60 | 61 | if np.min(arr_prob) < 0: 62 | print(arr_prob) 63 | exit() 64 | 65 | low = np.zeros([1, arr_size]) 66 | high = np.zeros([1, arr_size]) 67 | 68 | for i in range(arr_size): 69 | prob = arr_prob[:, i] / np.sum(arr_prob[:, i]) 70 | rindex = np.random.choice(arr_length + 1, p=prob) 71 | # print(rindex) 72 | 73 | low[0, i] = arr_ordered_new[rindex, i] 74 | high[0, i] = arr_ordered_new[rindex + 1, i] 75 | 76 | output_q = np.random.uniform(low=low, high=high, size=[1, arr_size]) 77 | output_q = output_q.reshape(input_shape) 78 | 79 | rdp_budget = arr_size * np.multiply( 80 | 1 / (orders - 1), 81 | np.log( 82 | np.multiply(np.divide(orders, 2 * orders - 1), np.exp((orders - 1) / lmbd)) \ 83 | + np.multiply(np.divide(orders - 1, 2 * orders - 1), np.exp(-orders / lmbd)) 84 | ) 85 | ) 86 | 87 | return output_q, rdp_budget 88 | 89 | 90 | def rdp_winsorized_mean(arr_list, step_size, sigma_mean, sigma_percentile, orders, pca_mat=None): 91 | vmin = -step_size 92 | vmax = step_size 93 | 94 | flatten_arr = np.asarray([arr.flatten() for arr in arr_list]) 95 | n_teachers, n_features = flatten_arr.shape 96 | 97 | if pca_mat is not None: 98 | # project to principal components 99 | flatten_arr = np.matmul(flatten_arr, pca_mat) 100 | n_features = flatten_arr.shape[1] 101 | 102 | q25, q25_budget = rdp_percentile(flatten_arr, 25, orders, vmin=vmin, vmax=vmax, lmbd=sigma_percentile) 103 | q75, q75_budget = rdp_percentile(flatten_arr, 75, orders, vmin=vmin, vmax=vmax, lmbd=sigma_percentile) 104 | 105 | arr_mean = np.mean(flatten_arr.clip(min=q25, max=q75), axis=0) 106 | 107 | arr_mean[np.sign(q75) != np.sign(q25)] = 0 108 | 109 | # when 75 percentile is smaller, update the model with the average of 75 and 25 percentile 110 | # quantile_mean = (q75 + q25) / 2 111 | arr_mean[q75 < q25] = 0 112 | 113 | update_index = np.nonzero(np.logical_and(np.sign(q75) == np.sign(q25), q75 > q25)) 114 | q_range = q75 - q25 115 | 116 | sensitivity = LA.norm(q_range[update_index] / len(arr_list)) 117 | 118 | gaussian_noise, mean_budget = gaussian_rdp(arr_mean[update_index], sensitivity, orders, sigma_mean) 119 | arr_mean[update_index] += gaussian_noise 120 | arr_mean[update_index] = arr_mean[update_index].clip(min=q25[update_index], max=q75[update_index]) 121 | 122 | # for testing only 123 | # update_ratio = gaussian_noise.size / arr_mean.size 124 | # print("Update ratio: %.8f, norm: %.8f" % (update_ratio, sensitivity)) 125 | 126 | rdp_budget = q25_budget + q75_budget + mean_budget 127 | 128 | if pca_mat is not None: 129 | # project res direction back to original axis 130 | arr_mean = np.matmul(arr_mean, np.transpose(pca_mat)) 131 | 132 | return arr_mean.reshape(arr_list[0].shape), rdp_budget 133 | 134 | 135 | def gradient_voting_nonprivate(output_list, step_size, nbins=10): 136 | n = len(output_list) 137 | flatten_arr = np.asarray([arr.flatten() for arr in output_list]) 138 | n_teachers, n_features = flatten_arr.shape 139 | 140 | flatten_arr = flatten_arr.clip(min=-step_size, max=step_size) 141 | 142 | bins = np.arange(-step_size, step_size, (step_size * 2 / nbins)) 143 | bins = np.hstack([bins, step_size]) 144 | result = np.zeros([1, n_features]) 145 | 146 | for i in range(n_features): 147 | votes_arr, _ = np.histogram(flatten_arr[:, i], bins) 148 | res_idx = np.argmax(votes_arr) 149 | result[:, i] = (bins[res_idx] + bins[res_idx + 1]) / 2 150 | 151 | return result.reshape(output_list[0].shape) 152 | 153 | 154 | def gradient_voting_rdp(output_list, step_size, sigma, sigma_thresh, orders, pca_mat=None, nbins=10, thresh=0.9): 155 | import time 156 | st = time.time() 157 | n = len(output_list) 158 | use_gpu = False # turn it on if you are running a huge matrix and the bottleneck lies on CPU matmul 159 | if use_gpu: 160 | # have to use torch==1.2.0 and torchvision==0.4.0 to run tensorflow-gpu==1.4.0 161 | import torch 162 | flatten_arr = torch.tensor([arr.flatten() for arr in output_list], device='cuda:0') 163 | else: 164 | flatten_arr = np.asarray([arr.flatten() for arr in output_list]) 165 | n_teachers, n_features = flatten_arr.shape 166 | 167 | if pca_mat is not None: 168 | # project to principal components 169 | if use_gpu: 170 | pca_mat_tensor = torch.from_numpy(pca_mat).float().to('cuda:0') 171 | flatten_arr = torch.matmul(flatten_arr, pca_mat_tensor) 172 | flatten_arr = flatten_arr.cpu().numpy() 173 | else: 174 | flatten_arr = np.matmul(flatten_arr, pca_mat) 175 | n_features = flatten_arr.shape[1] 176 | 177 | flatten_arr = flatten_arr.clip(min=-step_size, max=step_size) 178 | 179 | bins = np.arange(-step_size, step_size, (step_size * 2 / nbins)) 180 | bins = np.hstack([bins, step_size]) 181 | result = np.zeros([1, n_features]) 182 | 183 | rdp_budget = 0 184 | skipped_cnt = 0 185 | for i in range(n_features): 186 | votes_arr, _ = np.histogram(flatten_arr[:, i], bins) 187 | print(votes_arr) 188 | res_idx, cur_budget = gnmax_thresh_aggregator(votes_arr, thresh * n_teachers, sigma_thresh, sigma, orders) 189 | rdp_budget += cur_budget 190 | if res_idx < 0: 191 | skipped_cnt += 1 192 | else: 193 | result[:, i] = (bins[res_idx] + bins[res_idx + 1]) / 2 194 | print("Skipped %d feaatures out of %d" % (skipped_cnt, n_features)) 195 | 196 | 197 | if pca_mat is not None: 198 | # project res direction back to original axis 199 | result = np.matmul(result, np.transpose(pca_mat)) 200 | return result.reshape(output_list[0].shape), rdp_budget 201 | 202 | 203 | def convert2topk(grad, topk): 204 | """ 205 | :param grad: original gradient 206 | :param topk: topk we want to choose 207 | :return: voting array (+1/0/-1) 208 | """ 209 | topk_ind = np.argpartition(np.abs(grad), -topk)[:, -topk:] 210 | votes = np.zeros_like(grad) 211 | sign_grad = np.sign(grad) 212 | for i in range(len(grad)): 213 | votes[i, topk_ind[i]] = 1 214 | votes[i] = sign_grad[i] * votes[i] 215 | return votes 216 | 217 | 218 | def convert2topk_gpu(grad, topk): 219 | """ 220 | :param grad: sign grad (torch.tensor.cuda()) 221 | :param topk: topk value (int) 222 | :return: voted sign grad (torch.tensor.cuda()) 223 | """ 224 | topk_ind = torch.topk(torch.abs(grad), k=topk)[1] 225 | sign_grad = torch.sign(grad) 226 | votes = torch.zeros_like(grad).cuda() 227 | votes.scatter_(1, topk_ind, 1) 228 | votes = sign_grad * votes 229 | print(votes.type()) 230 | return votes 231 | 232 | def stachastic_convert2topk(grad, topk, b=None): 233 | abs_grad = np.abs(grad) 234 | topk_ind = np.argpartition(abs_grad, -topk)[:, -topk:] 235 | if b is None: 236 | # b = np.max(abs_grad, axis=0) ## DP proof assumes all the teachers to be independent 237 | b = np.max(abs_grad, axis=1) 238 | else: 239 | b = np.max(abs_grad, axis=1).clip(max=b) 240 | prob = 1/2 + (grad.T / b).T / 2 # prob of positive sign 241 | rand = np.random.rand(*prob.shape) 242 | sign_grad = np.ones_like(grad) 243 | sign_grad[rand > prob] = -1 244 | 245 | votes = np.zeros_like(grad) 246 | for i in range(len(votes)): 247 | votes[i, topk_ind[i]] = 1 248 | votes[i] = sign_grad[i] * votes[i] 249 | return votes 250 | 251 | 252 | def stochastic_klevel(grad, b, k_level): 253 | from scipy.stats import special_ortho_group 254 | from tqdm import tqdm 255 | # for i in tqdm(range(len(grad))): ## extremely slow (1h to finish on CPU) 256 | # clipped_grad = grad[i] # clipping factor = 1 257 | # rotation_matrix= special_ortho_group.rvs(len(grad[i])) 258 | # grad[i] = rotation_matrix @ clipped_grad # gradient rotation 259 | 260 | interval = 2 * b / (k_level - 1) 261 | lower = -b 262 | lower_grad = - k_level / 2 263 | rand = np.random.rand(*grad.shape) 264 | votes = np.zeros_like(grad) 265 | 266 | for i in range(1, k_level): 267 | upper = lower + interval 268 | upper_grad = lower_grad + 1 269 | if i == 1: 270 | mask = (grad <= upper) 271 | elif i == k_level - 1: 272 | mask = (grad >= lower) 273 | else: 274 | mask = (grad <= upper) & (grad >= lower) 275 | 276 | print(f"level {i}: {np.sum(mask)}") 277 | print(f"lower_grad : {lower_grad}") 278 | print(f"interval: {interval}") 279 | prob = (grad[mask] - lower) / (upper - lower) 280 | prob_grad = np.full_like(prob, lower_grad) 281 | prob_grad[rand[mask] <= prob] = upper_grad 282 | votes[mask] = prob_grad 283 | 284 | lower = upper 285 | lower_grad = upper_grad 286 | 287 | return votes 288 | 289 | def ablation_test_on_alpha_k(grad, topk): 290 | abs_grad = torch.abs(grad) 291 | topk_ind = torch.topk(abs_grad, k=topk)[1] 292 | 293 | votes = torch.zeros_like(grad).cuda() 294 | votes.scatter_(1, topk_ind, 1) 295 | masked_topk_grad = grad * votes 296 | 297 | residual_grad_norm = torch.norm(grad - masked_topk_grad, dim=1) 298 | grad_norm = torch.norm(grad, dim=1) 299 | 300 | alpha_k = 1 - residual_grad_norm / grad_norm 301 | return alpha_k.cpu().numpy() 302 | 303 | def ablation_test_on_different_k(output_list, ckpt_dir='', epoch=0): 304 | grad = torch.tensor([arr.flatten() for arr in output_list]).cuda() 305 | dim = grad.shape[-1] 306 | alpha_k = [] 307 | for i in range(5): 308 | topk = int(dim / 2**(i)) 309 | alpha_k.append(ablation_test_on_alpha_k(grad, topk)) 310 | alpha_k = np.vstack(alpha_k) 311 | import joblib 312 | import os 313 | save_dir = os.path.join(ckpt_dir, 'alpha_k_epoch_' + str(epoch) + '.pkl') 314 | joblib.dump(alpha_k, save_dir) 315 | 316 | def stochastic_klevel_gpu(grad, b, k_level): 317 | # from scipy.stats import special_ortho_group 318 | # rotation_matrix = torch.tensor(special_ortho_group.rvs(len(grad[0])), dtype=torch.float32).cuda() 319 | # grad = torch.matmul(rotation_matrix, grad.T).T # gradient rotation 320 | 321 | interval = 2 * b / (k_level - 1) 322 | lower = -b 323 | lower_grad = - k_level / 2 324 | rand = torch.rand_like(grad).cuda() 325 | votes = torch.zeros_like(grad).cuda() 326 | 327 | for i in range(1, k_level): 328 | upper = lower + interval 329 | upper_grad = lower_grad + 1 330 | if i == 1: 331 | mask = (grad <= upper) 332 | elif i == k_level - 1: 333 | mask = (grad >= lower) 334 | else: 335 | mask = (grad <= upper) & (grad >= lower) 336 | 337 | print(f"level {i}: {torch.sum(mask)}") 338 | print(f"lower_grad : {lower_grad}") 339 | print(f"interval: {interval}") 340 | prob = (grad[mask] - lower) / (upper - lower) 341 | prob_grad = torch.full_like(prob, lower_grad) 342 | prob_grad[rand[mask] <= prob] = upper_grad 343 | votes[mask] = prob_grad 344 | 345 | lower = upper 346 | lower_grad = upper_grad 347 | 348 | return votes 349 | 350 | 351 | def stachastic_convert2topk_gpu(grad, topk, b=None): 352 | """ 353 | :param grad: sign grad (torch.tensor.cuda()) 354 | :param topk: topk value (int) 355 | :return: voted sign grad (torch.tensor.cuda()) 356 | """ 357 | abs_grad = torch.abs(grad) 358 | topk_ind = torch.topk(abs_grad, k=topk)[1] 359 | if b is None: 360 | b = torch.max(abs_grad, dim=1)[0] 361 | else: 362 | b = torch.max(abs_grad, dim=1)[0].clamp(max=b) 363 | 364 | prob = 1/2 + (grad.T / b).T / 2 # prob of positive sign 365 | rand = torch.rand_like(prob).cuda() 366 | sign_grad = torch.ones_like(grad).cuda() 367 | sign_grad[rand > prob] = -1 368 | 369 | votes = torch.zeros_like(grad).cuda() 370 | votes.scatter_(1, topk_ind, 1) 371 | votes = sign_grad * votes 372 | 373 | sign_sgd = torch.sign(grad) 374 | biased_votes = torch.zeros_like(grad).cuda() 375 | biased_votes.scatter_(1, topk_ind, 1) 376 | biased_votes = sign_sgd * biased_votes 377 | print("prob topk", prob[0][topk_ind[0]]) 378 | print("grad topk", grad[0][topk_ind[0]]) 379 | print("sto sign topk", votes[0][topk_ind[0]]) 380 | print("biased sign topk", biased_votes[0][topk_ind[0]]) 381 | print("not agreed all sign:", torch.sum(votes != biased_votes)) 382 | print("not agreed one sign:", torch.sum(votes[0][topk_ind[0]] != biased_votes[0][topk_ind[0]])) 383 | return votes 384 | 385 | 386 | def stochastic_sketch_topk_gpu(grad, topk, b=None): 387 | """ 388 | :param grad: sign grad (torch.tensor.cuda()) 389 | :param topk: topk value (int) 390 | :return: voted sign grad (torch.tensor.cuda()) 391 | """ 392 | abs_grad = torch.abs(grad) 393 | if b is None: 394 | b = torch.max(abs_grad, dim=1)[0] 395 | else: 396 | b = torch.max(abs_grad, dim=1)[0].clamp(max=b) 397 | 398 | prob = 1/2 + (grad.T / b).T / 2 # prob of positive sign 399 | rand = torch.rand_like(prob).cuda() 400 | sign_grad = torch.ones_like(grad).cuda() 401 | sign_grad[rand > prob] = -1 402 | 403 | d = sign_grad.shape[1] 404 | c = 10000 405 | r = 20 406 | from csvec import CSVec 407 | sketch = CSVec(d, c, r) 408 | for grad in sign_grad: 409 | sketch.accumulateVec(grad) 410 | votes = sketch.unSketch(topk) 411 | print(votes.shape) 412 | return votes 413 | 414 | 415 | def signsgd_aggregate(output_list, sigma, orders, topk, beta=0.1, alpha=1e-3, stochastic=False, b=None): 416 | use_gpu = True 417 | if not use_gpu: 418 | nteacher = len(output_list) 419 | flatten_grad = np.asarray([arr.flatten() for arr in output_list]) 420 | if stochastic: 421 | voted_arr = np.sum(stachastic_convert2topk(flatten_grad, topk, b=b), axis=0) 422 | else: 423 | voted_arr = np.sum(convert2topk(flatten_grad, topk), axis=0) 424 | else: 425 | nteacher = len(output_list) 426 | flatten_grad = torch.tensor([arr.flatten() for arr in output_list]).cuda() 427 | if stochastic: 428 | voted_arr = torch.sum(stachastic_convert2topk_gpu(flatten_grad, topk, b=b), dim=0).cpu().numpy() 429 | else: 430 | voted_arr = torch.sum(convert2topk_gpu(flatten_grad, topk), dim=0).cpu().numpy() 431 | 432 | voted_arr = np.random.normal(voted_arr, sigma) 433 | logq = compute_logq_gaussian(voted_arr, sigma) 434 | rdp_budget = rdp_gaussian(logq, sigma / ((2*topk) ** 0.5), orders) 435 | sign_grad = np.zeros_like(voted_arr) 436 | 437 | sign_grad[voted_arr > beta * nteacher] = 1 438 | sign_grad[voted_arr < -beta * nteacher] = -1 439 | print("Agreed Dimension: " + str(np.sum(abs(sign_grad)))) 440 | 441 | return alpha * sign_grad.reshape(output_list[0].shape), rdp_budget 442 | 443 | 444 | def signsgd_aggregate_no_thresh(output_list, sigma, orders, topk, beta=0.1, alpha=1e-3, stochastic=False, b=None): 445 | use_gpu = True 446 | if not use_gpu: 447 | nteacher = len(output_list) 448 | flatten_grad = np.asarray([arr.flatten() for arr in output_list]) 449 | if stochastic: 450 | voted_arr = np.sum(stachastic_convert2topk(flatten_grad, topk, b=b), axis=0) 451 | else: 452 | voted_arr = np.sum(convert2topk(flatten_grad, topk), axis=0) 453 | else: 454 | nteacher = len(output_list) 455 | flatten_grad = torch.tensor([arr.flatten() for arr in output_list]).cuda() 456 | if stochastic: 457 | voted_arr = torch.sum(stachastic_convert2topk_gpu(flatten_grad, topk, b=b), dim=0).cpu().numpy() 458 | else: 459 | voted_arr = torch.sum(convert2topk_gpu(flatten_grad, topk), dim=0).cpu().numpy() 460 | 461 | # noise, rdp_budget2 = gaussian_rdp(voted_arr, (4 * topk) **0.5, orders, sigma) 462 | # print("before adding noise:", voted_arr) 463 | voted_arr = np.random.normal(voted_arr, sigma) 464 | # print("after adding noise:", voted_arr) 465 | # voted_arr += noise 466 | logq = compute_logq_gaussian(voted_arr, sigma) 467 | ## l2-sensitivity is (4k)**0.5, while GNMax sensitivity is 2**0.5. Hence the factor is 2k**0.5 468 | rdp_budget = rdp_gaussian(logq, sigma / ((2*topk) ** 0.5), orders) 469 | # print(rdp_budget) 470 | # sign_grad = np.zeros_like(voted_arr) 471 | 472 | # sign_grad[voted_arr > beta * nteacher] = 1 473 | # sign_grad[voted_arr < -beta * nteacher] = -1 474 | # print("Agreed Dimension: " + str(np.sum(abs(sign_grad)))) 475 | 476 | return alpha * voted_arr.reshape(output_list[0].shape) / topk, rdp_budget 477 | 478 | 479 | 480 | def sketchtopk_aggregate(output_list, sigma, orders, topk, beta=0.1, alpha=1e-3, stochastic=False, b=None): 481 | nteacher = len(output_list) 482 | flatten_grad = torch.tensor([arr.flatten() for arr in output_list]).cuda() 483 | voted_arr = stochastic_sketch_topk_gpu(flatten_grad, topk, b=b).cpu().numpy() 484 | 485 | # noise, rdp_budget2 = gaussian_rdp(voted_arr, (4 * topk) **0.5, orders, sigma) 486 | # print("before adding noise:", voted_arr) 487 | voted_arr = np.random.normal(voted_arr, sigma) 488 | # print("after adding noise:", voted_arr) 489 | # voted_arr += noise 490 | logq = compute_logq_gaussian(voted_arr, sigma) 491 | ## l2-sensitivity is (4k)**0.5, while GNMax sensitivity is 2**0.5. Hence the factor is 2k**0.5 492 | rdp_budget = rdp_gaussian(logq, sigma / ((2*topk) ** 0.5), orders) 493 | # print(rdp_budget) 494 | sign_grad = np.zeros_like(voted_arr) 495 | 496 | sign_grad[voted_arr > beta * nteacher] = 1 497 | sign_grad[voted_arr < -beta * nteacher] = -1 498 | print("Agreed Dimension: " + str(np.sum(abs(sign_grad)))) 499 | 500 | return alpha * sign_grad.reshape(output_list[0].shape), rdp_budget 501 | 502 | def k_level_sgd_aggregate(output_list, sigma, orders, k_level, beta=0.1, alpha=1e-3, b=None): 503 | use_gpu = False 504 | if not use_gpu: 505 | nteacher = len(output_list) 506 | flatten_grad = np.asarray([arr.flatten() for arr in output_list]) 507 | voted_arr = np.sum(stochastic_klevel(flatten_grad, k_level=k_level, b=b), axis=0) 508 | else: 509 | nteacher = len(output_list) 510 | flatten_grad = torch.tensor([arr.flatten() for arr in output_list]).cuda() 511 | voted_arr = torch.sum(stochastic_klevel_gpu(flatten_grad, k_level=k_level, b=b), dim=0).cpu().numpy() 512 | 513 | # noise, rdp_budget2 = gaussian_rdp(voted_arr, (4 * topk) **0.5, orders, sigma) 514 | # print("before adding noise:", voted_arr) 515 | voted_arr = np.random.normal(voted_arr, sigma) 516 | # print("after adding noise:", voted_arr) 517 | # voted_arr += noise 518 | logq = compute_logq_gaussian(voted_arr, sigma) 519 | dim = voted_arr.shape[0] 520 | ## l2-sensitivity is (k_level^2 * dim)**0.5, while GNMax sensitivity is 2**0.5. Hence the factor is 2k**0.5 521 | rdp_budget = rdp_gaussian(logq, sigma / ((k_level**2 * dim) ** 0.5), orders) 522 | # print(rdp_budget) 523 | sign_grad = np.zeros_like(voted_arr) 524 | 525 | sign_grad[voted_arr > beta * nteacher] = 1 526 | sign_grad[voted_arr < -beta * nteacher] = -1 527 | print("Agreed Dimension: " + str(np.sum(abs(sign_grad)))) 528 | 529 | return alpha * sign_grad.reshape(output_list[0].shape), rdp_budget 530 | 531 | 532 | def signsgd_aggregate_dept(output_list, sigma, orders, topk, beta=0.1, alpha=1e-3, stochastic=False, b=None): 533 | use_gpu = True 534 | if not use_gpu: 535 | nteacher = len(output_list) 536 | flatten_grad = np.asarray([arr.flatten() for arr in output_list]) 537 | if stochastic: 538 | voted_arr = np.sum(stachastic_convert2topk(flatten_grad, topk, b=b), axis=0) 539 | else: 540 | voted_arr = np.sum(convert2topk(flatten_grad, topk), axis=0) 541 | else: 542 | nteacher = len(output_list) 543 | flatten_grad = torch.tensor([arr.flatten() for arr in output_list]).cuda() 544 | if stochastic: 545 | voted_arr = torch.sum(stachastic_convert2topk_gpu(flatten_grad, topk, b=b), dim=0).cpu().numpy() 546 | else: 547 | voted_arr = torch.sum(convert2topk_gpu(flatten_grad, topk), dim=0).cpu().numpy() 548 | 549 | # noise, rdp_budget2 = gaussian_rdp(voted_arr, (4 * topk) **0.5, orders, sigma) 550 | # print("before adding noise:", voted_arr) 551 | voted_arr = np.random.normal(voted_arr, sigma) 552 | # print("after adding noise:", voted_arr) 553 | # voted_arr += noise 554 | logq = compute_logq_gaussian(voted_arr, sigma) 555 | ## l2-sensitivity is (4k)**0.5, while GNMax sensitivity is 2**0.5. Hence the factor is 2k**0.5 556 | # rdp_budget = rdp_gaussian(logq, sigma / ((2*topk) ** 0.5), orders) 557 | rdp_budget, dept_rdp_budget = double_rdp_gaussian(logq, sigma / ((2*topk) ** 0.5), orders) 558 | # print(rdp_budget) 559 | sign_grad = np.zeros_like(voted_arr) 560 | 561 | sign_grad[voted_arr > beta * nteacher] = 1 562 | sign_grad[voted_arr < -beta * nteacher] = -1 563 | print("Agreed Dimension: " + str(np.sum(abs(sign_grad)))) 564 | 565 | return alpha * sign_grad.reshape(output_list[0].shape), rdp_budget, dept_rdp_budget 566 | 567 | 568 | 569 | def gradient_voting_rdp_multiproj(output_list, step_size, sigma, sigma_thresh, orders, pca_mats=None, nbins=10, thresh=0.9): 570 | n = len(output_list) 571 | flatten_arr = np.asarray([arr.flatten() for arr in output_list]) 572 | n_teachers, n_features = flatten_arr.shape 573 | print("flatten arr shape", flatten_arr.shape) 574 | 575 | if pca_mats is not None: 576 | # project to principal components 577 | split_flatten_arr = np.split(flatten_arr, len(pca_mats), axis=1) 578 | reduced_flatten_arr = [] 579 | for pca_mat, arr in zip(pca_mats, split_flatten_arr): 580 | print("arr shape", arr.shape) 581 | print("pca shape", pca_mat.shape) 582 | arr = np.matmul(arr, pca_mat) 583 | reduced_flatten_arr.append(arr) 584 | flatten_arr = np.concatenate(reduced_flatten_arr, axis=1) 585 | n_features = flatten_arr.shape[1] 586 | 587 | flatten_arr = flatten_arr.clip(min=-step_size, max=step_size) 588 | 589 | bins = np.arange(-step_size, step_size, (step_size * 2 / nbins)) 590 | bins = np.hstack([bins, step_size]) 591 | result = np.zeros([1, n_features]) 592 | 593 | rdp_budget = 0 594 | skipped_cnt = 0 595 | for i in range(n_features): 596 | votes_arr, _ = np.histogram(flatten_arr[:, i], bins) 597 | print(votes_arr) 598 | res_idx, cur_budget = gnmax_thresh_aggregator(votes_arr, thresh * n_teachers, sigma_thresh, sigma, orders) 599 | rdp_budget += cur_budget 600 | if res_idx < 0: 601 | skipped_cnt += 1 602 | else: 603 | result[:, i] = (bins[res_idx] + bins[res_idx + 1]) / 2 604 | 605 | print("Skipped %d feaatures out of %d" % (skipped_cnt, n_features)) 606 | 607 | if pca_mat is not None: 608 | # project res direction back to original axis 609 | split_results = np.split(result, len(pca_mats), axis=1) 610 | final_results = [] 611 | for split_result, pca_mat in zip(split_results, pca_mats): 612 | final_results.append(np.matmul(split_result, np.transpose(pca_mat))) 613 | final_results = np.concatenate(final_results, axis=1) 614 | return final_results.reshape(output_list[0].shape), rdp_budget 615 | 616 | 617 | def gradient_sign_rdp(output_list, step_size, sigma, sigma_thresh, orders, pca_mat=None, thresh=0.9): 618 | n = len(output_list) 619 | flatten_arr = np.asarray([arr.flatten() for arr in output_list]) 620 | n_teachers, n_features = flatten_arr.shape 621 | 622 | if pca_mat is not None: 623 | # project to principal components 624 | flatten_arr = np.matmul(flatten_arr, pca_mat) 625 | n_features = flatten_arr.shape[1] 626 | 627 | # first line for positive votes, second line for negative votes 628 | votes_arr = np.zeros([2, n_features]) 629 | votes_sign = np.sign(flatten_arr) 630 | # counts for positive votes 631 | votes_arr[0, :] = np.sum(votes_sign[votes_sign > 0], axis=0) 632 | # counts for negative votes 633 | votes_arr[1, :] = -np.sum(votes_sign[votes_sign < 0], axis=0) 634 | 635 | res_dir = np.zeros([1, n_features]) 636 | 637 | rdp_budget = 0 638 | skipped_cnt = 0 639 | for i in range(n_features): 640 | dir_index, cur_budget = gnmax_thresh_aggregator(votes_arr[:, i], thresh * n_teachers, sigma_thresh, sigma, 641 | orders) 642 | if dir_index == 0: 643 | res_dir[0, i] = step_size 644 | elif dir_index == 1: 645 | res_dir[0, i] = -step_size 646 | else: 647 | skipped_cnt += 1 648 | rdp_budget += cur_budget 649 | 650 | print("Skipped %d feaatures out of %d" % (skipped_cnt, n_features)) 651 | 652 | if pca_mat is not None: 653 | # project res direction back to original axis 654 | res_dir = np.matmul(res_dir, np.transpose(pca_mat)) 655 | 656 | return res_dir.reshape(output_list[0].shape), rdp_budget 657 | 658 | 659 | def gradient_rdp(output_list, step_size, sigma, orders, pca_mat=None, thresh=None, sigma_thresh=1): 660 | n = len(output_list) 661 | flatten_arr = np.asarray([arr.flatten() for arr in output_list]) 662 | n_teachers, n_features = flatten_arr.shape 663 | 664 | if pca_mat is not None: 665 | # project to principal components 666 | flatten_arr = np.matmul(flatten_arr, pca_mat) 667 | n_features = flatten_arr.shape[1] 668 | 669 | # first half votes for positive direction, second half votes for negative direction 670 | votes_arr = np.zeros([n_teachers, n_features * 2]) 671 | max_index = np.argmax(np.abs(flatten_arr), axis=1) 672 | 673 | for i in range(n_teachers): 674 | if flatten_arr[i, max_index[i]] > 0: 675 | votes_arr[i, max_index[i]] = 1 676 | else: 677 | votes_arr[i, max_index[i] + n_features] = 1 678 | 679 | votes_count = np.sum(votes_arr, axis=0) 680 | 681 | if thresh is None: 682 | dir_index, rdp_budget = gnmax_aggregator(votes_count, sigma, orders) 683 | else: 684 | dir_index, rdp_budget = gnmax_thresh_aggregator(votes_count, thresh * n_teachers, sigma_thresh, sigma, orders) 685 | 686 | max_votes = np.max(votes_count) 687 | selected_votes = votes_count[dir_index] 688 | # print("Max cnt: %d, selected cnt: %d" % (max_votes, selected_votes)) 689 | 690 | res_dir = np.zeros([1, n_features]) 691 | 692 | if dir_index < n_features and dir_index >= 0: 693 | res_dir[0, dir_index] = step_size 694 | elif dir_index >= n_features: 695 | res_dir[0, dir_index - n_features] = -step_size 696 | else: 697 | print("Teachers don't agree. Skip...") 698 | 699 | if pca_mat is not None: 700 | # project res direction back to original axis 701 | res_dir = np.matmul(res_dir, np.transpose(pca_mat)) 702 | 703 | return res_dir.reshape(output_list[0].shape), rdp_budget 704 | 705 | 706 | def gaussian_rdp(arr, sensitivity, orders, sigma): 707 | gaussian_noise = np.random.normal(loc=np.zeros(arr.shape), scale=sigma * sensitivity, size=arr.shape) 708 | 709 | # Table 2 @ https://arxiv.org/pdf/1702.07476.pdf 710 | rdp_budget = [o / ((2 * sigma) ** 2) for o in orders] 711 | 712 | return gaussian_noise, rdp_budget 713 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | torch~=1.4.0 2 | numpy~=1.18.4 3 | tensorflow-gpu==1.14.0 4 | scipy~=1.4.1 5 | six~=1.12.0 6 | torchvision~=0.5.0 7 | Keras~=2.3.1 8 | scikit-learn~=0.21.3 9 | Pillow~=7.0.0 10 | requests~=2.22.0 11 | tqdm~=4.46.0 12 | joblib~=0.13.2 -------------------------------------------------------------------------------- /temp.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 The TensorFlow Authors. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ============================================================================== 15 | """Self-diagnosis script for TensorBoard. 16 | 17 | Instructions: Save this script to your local machine, then execute it in 18 | the same environment (virtualenv, Conda, etc.) from which you normally 19 | run TensorBoard. Read the output and follow the directions. 20 | """ 21 | 22 | from __future__ import absolute_import 23 | from __future__ import division 24 | from __future__ import print_function 25 | 26 | # This script may only depend on the Python standard library. It is not 27 | # built with Bazel and should not assume any third-party dependencies. 28 | import collections 29 | import errno 30 | import functools 31 | import hashlib 32 | import inspect 33 | import logging 34 | import os 35 | import pipes 36 | import shlex 37 | import socket 38 | import subprocess 39 | import sys 40 | import tempfile 41 | import textwrap 42 | import traceback 43 | 44 | 45 | # A *check* is a function (of no arguments) that performs a diagnostic, 46 | # writes log messages, and optionally yields suggestions. Each check 47 | # runs in isolation; exceptions will be caught and reported. 48 | CHECKS = [] 49 | 50 | 51 | # A suggestion to the end user. 52 | # headline (str): A short description, like "Turn it off and on 53 | # again". Should be imperative with no trailing punctuation. May 54 | # contain inline Markdown. 55 | # description (str): A full enumeration of the steps that the user 56 | # should take to accept the suggestion. Within this string, prose 57 | # should be formatted with `reflow`. May contain Markdown. 58 | Suggestion = collections.namedtuple("Suggestion", ("headline", "description")) 59 | 60 | 61 | def check(fn): 62 | """Decorator to register a function as a check. 63 | 64 | Checks are run in the order in which they are registered. 65 | 66 | Args: 67 | fn: A function that takes no arguments and either returns `None` or 68 | returns a generator of `Suggestion`s. (The ability to return 69 | `None` is to work around the awkwardness of defining empty 70 | generator functions in Python.) 71 | 72 | Returns: 73 | A wrapped version of `fn` that returns a generator of `Suggestion`s. 74 | """ 75 | 76 | @functools.wraps(fn) 77 | def wrapper(): 78 | result = fn() 79 | return iter(()) if result is None else result 80 | 81 | CHECKS.append(wrapper) 82 | return wrapper 83 | 84 | 85 | def reflow(paragraph): 86 | return textwrap.fill(textwrap.dedent(paragraph).strip()) 87 | 88 | 89 | def pip(args): 90 | """Invoke command-line Pip with the specified args. 91 | 92 | Returns: 93 | A bytestring containing the output of Pip. 94 | """ 95 | # Suppress the Python 2.7 deprecation warning. 96 | PYTHONWARNINGS_KEY = "PYTHONWARNINGS" 97 | old_pythonwarnings = os.environ.get(PYTHONWARNINGS_KEY) 98 | new_pythonwarnings = "%s%s" % ( 99 | "ignore:DEPRECATION", 100 | ",%s" % old_pythonwarnings if old_pythonwarnings else "", 101 | ) 102 | command = [sys.executable, "-m", "pip", "--disable-pip-version-check"] 103 | command.extend(args) 104 | try: 105 | os.environ[PYTHONWARNINGS_KEY] = new_pythonwarnings 106 | return subprocess.check_output(command) 107 | finally: 108 | if old_pythonwarnings is None: 109 | del os.environ[PYTHONWARNINGS_KEY] 110 | else: 111 | os.environ[PYTHONWARNINGS_KEY] = old_pythonwarnings 112 | 113 | 114 | def which(name): 115 | """Return the path to a binary, or `None` if it's not on the path. 116 | 117 | Returns: 118 | A bytestring. 119 | """ 120 | binary = "where" if os.name == "nt" else "which" 121 | try: 122 | return subprocess.check_output([binary, name]) 123 | except subprocess.CalledProcessError: 124 | return None 125 | 126 | 127 | def sgetattr(attr, default): 128 | """Get an attribute off the `socket` module, or use a default.""" 129 | sentinel = object() 130 | result = getattr(socket, attr, sentinel) 131 | if result is sentinel: 132 | print("socket.%s does not exist" % attr) 133 | return default 134 | else: 135 | print("socket.%s = %r" % (attr, result)) 136 | return result 137 | 138 | 139 | @check 140 | def autoidentify(): 141 | """Print the Git hash of this version of `diagnose_tensorboard.py`. 142 | 143 | Given this hash, use `git cat-file blob HASH` to recover the 144 | relevant version of the script. 145 | """ 146 | module = sys.modules[__name__] 147 | try: 148 | source = inspect.getsource(module).encode("utf-8") 149 | except TypeError: 150 | logging.info("diagnose_tensorboard.py source unavailable") 151 | else: 152 | # Git inserts a length-prefix before hashing; cf. `git-hash-object`. 153 | blob = b"blob %d\0%s" % (len(source), source) 154 | hash = hashlib.sha1(blob).hexdigest() 155 | logging.info("diagnose_tensorboard.py version %s", hash) 156 | 157 | 158 | @check 159 | def general(): 160 | logging.info("sys.version_info: %s", sys.version_info) 161 | logging.info("os.name: %s", os.name) 162 | na = type("N/A", (object,), {"__repr__": lambda self: "N/A"}) 163 | logging.info( 164 | "os.uname(): %r", 165 | getattr(os, "uname", na)(), 166 | ) 167 | logging.info( 168 | "sys.getwindowsversion(): %r", 169 | getattr(sys, "getwindowsversion", na)(), 170 | ) 171 | 172 | 173 | @check 174 | def package_management(): 175 | conda_meta = os.path.join(sys.prefix, "conda-meta") 176 | logging.info("has conda-meta: %s", os.path.exists(conda_meta)) 177 | logging.info("$VIRTUAL_ENV: %r", os.environ.get("VIRTUAL_ENV")) 178 | 179 | 180 | @check 181 | def installed_packages(): 182 | freeze = pip(["freeze", "--all"]).decode("utf-8").splitlines() 183 | packages = {line.split("==")[0]: line for line in freeze} 184 | packages_set = frozenset(packages) 185 | 186 | # For each of the following families, expect exactly one package to be 187 | # installed. 188 | expect_unique = [ 189 | frozenset( 190 | [ 191 | "tensorboard", 192 | "tb-nightly", 193 | "tensorflow-tensorboard", 194 | ] 195 | ), 196 | frozenset( 197 | [ 198 | "tensorflow", 199 | "tensorflow-gpu", 200 | "tf-nightly", 201 | "tf-nightly-2.0-preview", 202 | "tf-nightly-gpu", 203 | "tf-nightly-gpu-2.0-preview", 204 | ] 205 | ), 206 | frozenset( 207 | [ 208 | "tensorflow-estimator", 209 | "tensorflow-estimator-2.0-preview", 210 | "tf-estimator-nightly", 211 | ] 212 | ), 213 | ] 214 | 215 | found_conflict = False 216 | for family in expect_unique: 217 | actual = family & packages_set 218 | for package in actual: 219 | logging.info("installed: %s", packages[package]) 220 | if len(actual) == 0: 221 | logging.warning("no installation among: %s", sorted(family)) 222 | elif len(actual) > 1: 223 | logging.warning("conflicting installations: %s", sorted(actual)) 224 | found_conflict = True 225 | 226 | if found_conflict: 227 | preamble = reflow( 228 | """ 229 | Conflicting package installations found. Depending on the order 230 | of installations and uninstallations, behavior may be undefined. 231 | Please uninstall ALL versions of TensorFlow and TensorBoard, 232 | then reinstall ONLY the desired version of TensorFlow, which 233 | will transitively pull in the proper version of TensorBoard. (If 234 | you use TensorBoard without TensorFlow, just reinstall the 235 | appropriate version of TensorBoard directly.) 236 | """ 237 | ) 238 | packages_to_uninstall = sorted( 239 | frozenset().union(*expect_unique) & packages_set 240 | ) 241 | commands = [ 242 | "pip uninstall %s" % " ".join(packages_to_uninstall), 243 | "pip install tensorflow # or `tensorflow-gpu`, or `tf-nightly`, ...", 244 | ] 245 | message = "%s\n\nNamely:\n\n%s" % ( 246 | preamble, 247 | "\n".join("\t%s" % c for c in commands), 248 | ) 249 | yield Suggestion("Fix conflicting installations", message) 250 | 251 | wit_version = packages.get("tensorboard-plugin-wit") 252 | if wit_version == "tensorboard-plugin-wit==1.6.0.post2": 253 | # This is only incompatible with TensorBoard prior to 2.2.0, but 254 | # we just issue a blanket warning so that we don't have to pull 255 | # in a `pkg_resources` dep to parse the version. 256 | preamble = reflow( 257 | """ 258 | Versions of the What-If Tool (`tensorboard-plugin-wit`) 259 | prior to 1.6.0.post3 are incompatible with some versions of 260 | TensorBoard. Please upgrade this package to the latest 261 | version to resolve any startup errors: 262 | """ 263 | ) 264 | command = "pip install -U tensorboard-plugin-wit" 265 | message = "%s\n\n\t%s" % (preamble, command) 266 | yield Suggestion("Upgrade `tensorboard-plugin-wit`", message) 267 | 268 | 269 | @check 270 | def tensorboard_python_version(): 271 | from tensorboard import version 272 | 273 | logging.info("tensorboard.version.VERSION: %r", version.VERSION) 274 | 275 | 276 | @check 277 | def tensorflow_python_version(): 278 | import tensorflow as tf 279 | 280 | logging.info("tensorflow.__version__: %r", tf.__version__) 281 | logging.info("tensorflow.__git_version__: %r", tf.__git_version__) 282 | 283 | 284 | @check 285 | def tensorboard_binary_path(): 286 | logging.info("which tensorboard: %r", which("tensorboard")) 287 | 288 | 289 | @check 290 | def addrinfos(): 291 | sgetattr("has_ipv6", None) 292 | family = sgetattr("AF_UNSPEC", 0) 293 | socktype = sgetattr("SOCK_STREAM", 0) 294 | protocol = 0 295 | flags_loopback = sgetattr("AI_ADDRCONFIG", 0) 296 | flags_wildcard = sgetattr("AI_PASSIVE", 0) 297 | 298 | hints_loopback = (family, socktype, protocol, flags_loopback) 299 | infos_loopback = socket.getaddrinfo(None, 0, *hints_loopback) 300 | print("Loopback flags: %r" % (flags_loopback,)) 301 | print("Loopback infos: %r" % (infos_loopback,)) 302 | 303 | hints_wildcard = (family, socktype, protocol, flags_wildcard) 304 | infos_wildcard = socket.getaddrinfo(None, 0, *hints_wildcard) 305 | print("Wildcard flags: %r" % (flags_wildcard,)) 306 | print("Wildcard infos: %r" % (infos_wildcard,)) 307 | 308 | 309 | @check 310 | def readable_fqdn(): 311 | # May raise `UnicodeDecodeError` for non-ASCII hostnames: 312 | # https://github.com/tensorflow/tensorboard/issues/682 313 | try: 314 | logging.info("socket.getfqdn(): %r", socket.getfqdn()) 315 | except UnicodeDecodeError as e: 316 | try: 317 | binary_hostname = subprocess.check_output(["hostname"]).strip() 318 | except subprocess.CalledProcessError: 319 | binary_hostname = b"" 320 | is_non_ascii = not all( 321 | 0x20 322 | <= (ord(c) if not isinstance(c, int) else c) 323 | <= 0x7E # Python 2 324 | for c in binary_hostname 325 | ) 326 | if is_non_ascii: 327 | message = reflow( 328 | """ 329 | Your computer's hostname, %r, contains bytes outside of the 330 | printable ASCII range. Some versions of Python have trouble 331 | working with such names (https://bugs.python.org/issue26227). 332 | Consider changing to a hostname that only contains printable 333 | ASCII bytes. 334 | """ 335 | % (binary_hostname,) 336 | ) 337 | yield Suggestion("Use an ASCII hostname", message) 338 | else: 339 | message = reflow( 340 | """ 341 | Python can't read your computer's hostname, %r. This can occur 342 | if the hostname contains non-ASCII bytes 343 | (https://bugs.python.org/issue26227). Consider changing your 344 | hostname, rebooting your machine, and rerunning this diagnosis 345 | script to see if the problem is resolved. 346 | """ 347 | % (binary_hostname,) 348 | ) 349 | yield Suggestion("Use a simpler hostname", message) 350 | raise e 351 | 352 | 353 | @check 354 | def stat_tensorboardinfo(): 355 | # We don't use `manager._get_info_dir`, because (a) that requires 356 | # TensorBoard, and (b) that creates the directory if it doesn't exist. 357 | path = os.path.join(tempfile.gettempdir(), ".tensorboard-info") 358 | logging.info("directory: %s", path) 359 | try: 360 | stat_result = os.stat(path) 361 | except OSError as e: 362 | if e.errno == errno.ENOENT: 363 | # No problem; this is just fine. 364 | logging.info(".tensorboard-info directory does not exist") 365 | return 366 | else: 367 | raise 368 | logging.info("os.stat(...): %r", stat_result) 369 | logging.info("mode: 0o%o", stat_result.st_mode) 370 | if stat_result.st_mode & 0o777 != 0o777: 371 | preamble = reflow( 372 | """ 373 | The ".tensorboard-info" directory was created by an old version 374 | of TensorBoard, and its permissions are not set correctly; see 375 | issue #2010. Change that directory to be world-accessible (may 376 | require superuser privilege): 377 | """ 378 | ) 379 | # This error should only appear on Unices, so it's okay to use 380 | # Unix-specific utilities and shell syntax. 381 | quote = getattr(shlex, "quote", None) or pipes.quote # Python <3.3 382 | command = "chmod 777 %s" % quote(path) 383 | message = "%s\n\n\t%s" % (preamble, command) 384 | yield Suggestion('Fix permissions on "%s"' % path, message) 385 | 386 | 387 | @check 388 | def source_trees_without_genfiles(): 389 | roots = list(sys.path) 390 | if "" not in roots: 391 | # Catch problems that would occur in a Python interactive shell 392 | # (where `""` is prepended to `sys.path`) but not when 393 | # `diagnose_tensorboard.py` is run as a standalone script. 394 | roots.insert(0, "") 395 | 396 | def has_tensorboard(root): 397 | return os.path.isfile(os.path.join(root, "tensorboard", "__init__.py")) 398 | 399 | def has_genfiles(root): 400 | sample_genfile = os.path.join("compat", "proto", "summary_pb2.py") 401 | return os.path.isfile(os.path.join(root, "tensorboard", sample_genfile)) 402 | 403 | def is_bad(root): 404 | return has_tensorboard(root) and not has_genfiles(root) 405 | 406 | tensorboard_roots = [root for root in roots if has_tensorboard(root)] 407 | bad_roots = [root for root in roots if is_bad(root)] 408 | 409 | logging.info( 410 | "tensorboard_roots (%d): %r; bad_roots (%d): %r", 411 | len(tensorboard_roots), 412 | tensorboard_roots, 413 | len(bad_roots), 414 | bad_roots, 415 | ) 416 | 417 | if bad_roots: 418 | if bad_roots == [""]: 419 | message = reflow( 420 | """ 421 | Your current directory contains a `tensorboard` Python package 422 | that does not include generated files. This can happen if your 423 | current directory includes the TensorBoard source tree (e.g., 424 | you are in the TensorBoard Git repository). Consider changing 425 | to a different directory. 426 | """ 427 | ) 428 | else: 429 | preamble = reflow( 430 | """ 431 | Your Python path contains a `tensorboard` package that does 432 | not include generated files. This can happen if your current 433 | directory includes the TensorBoard source tree (e.g., you are 434 | in the TensorBoard Git repository). The following directories 435 | from your Python path may be problematic: 436 | """ 437 | ) 438 | roots = [] 439 | realpaths_seen = set() 440 | for root in bad_roots: 441 | label = repr(root) if root else "current directory" 442 | realpath = os.path.realpath(root) 443 | if realpath in realpaths_seen: 444 | # virtualenvs on Ubuntu install to both `lib` and `local/lib`; 445 | # explicitly call out such duplicates to avoid confusion. 446 | label += " (duplicate underlying directory)" 447 | realpaths_seen.add(realpath) 448 | roots.append(label) 449 | message = "%s\n\n%s" % ( 450 | preamble, 451 | "\n".join(" - %s" % s for s in roots), 452 | ) 453 | yield Suggestion( 454 | "Avoid `tensorboard` packages without genfiles", message 455 | ) 456 | 457 | 458 | # Prefer to include this check last, as its output is long. 459 | @check 460 | def full_pip_freeze(): 461 | logging.info( 462 | "pip freeze --all:\n%s", pip(["freeze", "--all"]).decode("utf-8") 463 | ) 464 | 465 | 466 | def set_up_logging(): 467 | # Manually install handlers to prevent TensorFlow from stomping the 468 | # default configuration if it's imported: 469 | # https://github.com/tensorflow/tensorflow/issues/28147 470 | logger = logging.getLogger() 471 | logger.setLevel(logging.INFO) 472 | handler = logging.StreamHandler(sys.stdout) 473 | handler.setFormatter(logging.Formatter("%(levelname)s: %(message)s")) 474 | logger.addHandler(handler) 475 | 476 | 477 | def main(): 478 | set_up_logging() 479 | 480 | print("### Diagnostics") 481 | print() 482 | 483 | print("
") 484 | print("Diagnostics output") 485 | print() 486 | 487 | markdown_code_fence = "``````" # seems likely to be sufficient 488 | print(markdown_code_fence) 489 | suggestions = [] 490 | for (i, check) in enumerate(CHECKS): 491 | if i > 0: 492 | print() 493 | print("--- check: %s" % check.__name__) 494 | try: 495 | suggestions.extend(check()) 496 | except Exception: 497 | traceback.print_exc(file=sys.stdout) 498 | pass 499 | print(markdown_code_fence) 500 | print() 501 | print("
") 502 | 503 | for suggestion in suggestions: 504 | print() 505 | print("### Suggestion: %s" % suggestion.headline) 506 | print() 507 | print(suggestion.description) 508 | 509 | print() 510 | print("### Next steps") 511 | print() 512 | if suggestions: 513 | print( 514 | reflow( 515 | """ 516 | Please try each suggestion enumerated above to determine whether 517 | it solves your problem. If none of these suggestions works, 518 | please copy ALL of the above output, including the lines 519 | containing only backticks, into your GitHub issue or comment. Be 520 | sure to redact any sensitive information. 521 | """ 522 | ) 523 | ) 524 | else: 525 | print( 526 | reflow( 527 | """ 528 | No action items identified. Please copy ALL of the above output, 529 | including the lines containing only backticks, into your GitHub 530 | issue or comment. Be sure to redact any sensitive information. 531 | """ 532 | ) 533 | ) 534 | 535 | 536 | if __name__ == "__main__": 537 | main() -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Some codes from https://github.com/Newmu/dcgan_code 3 | """ 4 | from __future__ import division 5 | import math 6 | import json 7 | import random 8 | import pprint 9 | import scipy.misc 10 | import numpy as np 11 | from time import gmtime, strftime 12 | from six.moves import xrange 13 | import os 14 | 15 | import tensorflow as tf 16 | import tensorflow.contrib.slim as slim 17 | 18 | pp = pprint.PrettyPrinter() 19 | 20 | get_stddev = lambda x, k_h, k_w: 1/math.sqrt(k_w*k_h*x.get_shape()[-1]) 21 | 22 | def mkdir(dir_name): 23 | dirs = dir_name.split('/') 24 | 25 | cur_dir = '' 26 | for d in dirs: 27 | cur_dir = os.path.join(cur_dir, d) 28 | if not os.path.exists(cur_dir): 29 | print("mkdir %s" % cur_dir) 30 | os.mkdir(cur_dir) 31 | else: 32 | print("%s exists" % cur_dir) 33 | 34 | def show_all_variables(): 35 | model_vars = tf.trainable_variables() 36 | slim.model_analyzer.analyze_vars(model_vars, print_info=True) 37 | 38 | def get_image(image_path, input_height, input_width, 39 | resize_height=64, resize_width=64, 40 | crop=True, grayscale=False): 41 | image = imread(image_path, grayscale) 42 | return transform(image, input_height, input_width, 43 | resize_height, resize_width, crop) 44 | 45 | def save_images(images, size, image_path): 46 | return imsave(inverse_transform(images), size, image_path) 47 | 48 | def imread(path, grayscale = False): 49 | if (grayscale): 50 | return scipy.misc.imread(path, flatten = True).astype(np.float) 51 | else: 52 | return scipy.misc.imread(path).astype(np.float) 53 | 54 | def merge_images(images, size): 55 | return inverse_transform(images) 56 | 57 | def merge(images, size): 58 | h, w = images.shape[1], images.shape[2] 59 | if (images.shape[3] in (3,4)): 60 | c = images.shape[3] 61 | img = np.zeros((h * size[0], w * size[1], c)) 62 | for idx, image in enumerate(images): 63 | i = idx % size[1] 64 | j = idx // size[1] 65 | img[j * h:j * h + h, i * w:i * w + w, :] = image 66 | return img 67 | elif images.shape[3]==1: 68 | img = np.zeros((h * size[0], w * size[1])) 69 | for idx, image in enumerate(images): 70 | i = idx % size[1] 71 | j = idx // size[1] 72 | img[j * h:j * h + h, i * w:i * w + w] = image[:,:,0] 73 | return img 74 | else: 75 | raise ValueError('in merge(images,size) images parameter ' 76 | 'must have dimensions: HxW or HxWx3 or HxWx4') 77 | 78 | def imsave(images, size, path): 79 | image = np.squeeze(merge(images, size)) 80 | return scipy.misc.imsave(path, image) 81 | 82 | def center_crop(x, crop_h, crop_w, 83 | resize_h=64, resize_w=64): 84 | if crop_w is None: 85 | crop_w = crop_h 86 | h, w = x.shape[:2] 87 | j = int(round((h - crop_h)/2.)) 88 | i = int(round((w - crop_w)/2.)) 89 | return scipy.misc.imresize( 90 | x[j:j+crop_h, i:i+crop_w], [resize_h, resize_w]) 91 | 92 | def transform(image, input_height, input_width, 93 | resize_height=64, resize_width=64, crop=True): 94 | if crop: 95 | cropped_image = center_crop( 96 | image, input_height, input_width, 97 | resize_height, resize_width) 98 | else: 99 | cropped_image = scipy.misc.imresize(image, [resize_height, resize_width]) 100 | return np.array(cropped_image)/127.5 - 1. 101 | 102 | def inverse_transform(images): 103 | return (images+1.)/2. 104 | 105 | def to_json(output_path, *layers): 106 | with open(output_path, "w") as layer_f: 107 | lines = "" 108 | for w, b, bn in layers: 109 | layer_idx = w.name.split('/')[0].split('h')[1] 110 | 111 | B = b.eval() 112 | 113 | if "lin/" in w.name: 114 | W = w.eval() 115 | depth = W.shape[1] 116 | else: 117 | W = np.rollaxis(w.eval(), 2, 0) 118 | depth = W.shape[0] 119 | 120 | biases = {"sy": 1, "sx": 1, "depth": depth, "w": ['%.2f' % elem for elem in list(B)]} 121 | if bn != None: 122 | gamma = bn.gamma.eval() 123 | beta = bn.beta.eval() 124 | 125 | gamma = {"sy": 1, "sx": 1, "depth": depth, "w": ['%.2f' % elem for elem in list(gamma)]} 126 | beta = {"sy": 1, "sx": 1, "depth": depth, "w": ['%.2f' % elem for elem in list(beta)]} 127 | else: 128 | gamma = {"sy": 1, "sx": 1, "depth": 0, "w": []} 129 | beta = {"sy": 1, "sx": 1, "depth": 0, "w": []} 130 | 131 | if "lin/" in w.name: 132 | fs = [] 133 | for w in W.T: 134 | fs.append({"sy": 1, "sx": 1, "depth": W.shape[0], "w": ['%.2f' % elem for elem in list(w)]}) 135 | 136 | lines += """ 137 | var layer_%s = { 138 | "layer_type": "fc", 139 | "sy": 1, "sx": 1, 140 | "out_sx": 1, "out_sy": 1, 141 | "stride": 1, "pad": 0, 142 | "out_depth": %s, "in_depth": %s, 143 | "biases": %s, 144 | "gamma": %s, 145 | "beta": %s, 146 | "filters": %s 147 | };""" % (layer_idx.split('_')[0], W.shape[1], W.shape[0], biases, gamma, beta, fs) 148 | else: 149 | fs = [] 150 | for w_ in W: 151 | fs.append({"sy": 5, "sx": 5, "depth": W.shape[3], "w": ['%.2f' % elem for elem in list(w_.flatten())]}) 152 | 153 | lines += """ 154 | var layer_%s = { 155 | "layer_type": "deconv", 156 | "sy": 5, "sx": 5, 157 | "out_sx": %s, "out_sy": %s, 158 | "stride": 2, "pad": 1, 159 | "out_depth": %s, "in_depth": %s, 160 | "biases": %s, 161 | "gamma": %s, 162 | "beta": %s, 163 | "filters": %s 164 | };""" % (layer_idx, 2**(int(layer_idx)+2), 2**(int(layer_idx)+2), 165 | W.shape[0], W.shape[3], biases, gamma, beta, fs) 166 | layer_f.write(" ".join(lines.replace("'","").split())) 167 | 168 | def make_gif(images, fname, duration=2, true_image=False): 169 | import moviepy.editor as mpy 170 | 171 | def make_frame(t): 172 | try: 173 | x = images[int(len(images)/duration*t)] 174 | except: 175 | x = images[-1] 176 | 177 | if true_image: 178 | return x.astype(np.uint8) 179 | else: 180 | return ((x+1)/2*255).astype(np.uint8) 181 | 182 | clip = mpy.VideoClip(make_frame, duration=duration) 183 | clip.write_gif(fname, fps = len(images) / duration) 184 | 185 | def visualize(sess, dcgan, config, option): 186 | image_frame_dim = int(math.ceil(config.batch_size**.5)) 187 | if option == 0: 188 | z_sample = np.random.uniform(-0.5, 0.5, size=(config.batch_size, dcgan.z_dim)) 189 | samples = sess.run(dcgan.sampler, feed_dict={dcgan.z: z_sample}) 190 | save_images(samples, [image_frame_dim, image_frame_dim], './samples/test_%s.png' % strftime("%Y-%m-%d-%H-%M-%S", gmtime())) 191 | elif option == 1: 192 | values = np.arange(0, 1, 1./config.batch_size) 193 | for idx in xrange(dcgan.z_dim): 194 | print(" [*] %d" % idx) 195 | z_sample = np.random.uniform(-1, 1, size=(config.batch_size , dcgan.z_dim)) 196 | for kdx, z in enumerate(z_sample): 197 | z[idx] = values[kdx] 198 | 199 | if config.dataset == "mnist": 200 | y = np.random.choice(10, config.batch_size) 201 | y_one_hot = np.zeros((config.batch_size, 10)) 202 | y_one_hot[np.arange(config.batch_size), y] = 1 203 | 204 | samples = sess.run(dcgan.sampler, feed_dict={dcgan.z: z_sample, dcgan.y: y_one_hot}) 205 | else: 206 | samples = sess.run(dcgan.sampler, feed_dict={dcgan.z: z_sample}) 207 | 208 | save_images(samples, [image_frame_dim, image_frame_dim], './samples/test_arange_%s.png' % (idx)) 209 | elif option == 2: 210 | values = np.arange(0, 1, 1./config.batch_size) 211 | for idx in [random.randint(0, dcgan.z_dim - 1) for _ in xrange(dcgan.z_dim)]: 212 | print(" [*] %d" % idx) 213 | z = np.random.uniform(-0.2, 0.2, size=(dcgan.z_dim)) 214 | z_sample = np.tile(z, (config.batch_size, 1)) 215 | #z_sample = np.zeros([config.batch_size, dcgan.z_dim]) 216 | for kdx, z in enumerate(z_sample): 217 | z[idx] = values[kdx] 218 | 219 | if config.dataset == "mnist": 220 | y = np.random.choice(10, config.batch_size) 221 | y_one_hot = np.zeros((config.batch_size, 10)) 222 | y_one_hot[np.arange(config.batch_size), y] = 1 223 | 224 | samples = sess.run(dcgan.sampler, feed_dict={dcgan.z: z_sample, dcgan.y: y_one_hot}) 225 | else: 226 | samples = sess.run(dcgan.sampler, feed_dict={dcgan.z: z_sample}) 227 | 228 | try: 229 | make_gif(samples, './samples/test_gif_%s.gif' % (idx)) 230 | except: 231 | save_images(samples, [image_frame_dim, image_frame_dim], './samples/test_%s.png' % strftime("%Y-%m-%d-%H-%M-%S", gmtime())) 232 | elif option == 3: 233 | values = np.arange(0, 1, 1./config.batch_size) 234 | for idx in xrange(dcgan.z_dim): 235 | print(" [*] %d" % idx) 236 | z_sample = np.zeros([config.batch_size, dcgan.z_dim]) 237 | for kdx, z in enumerate(z_sample): 238 | z[idx] = values[kdx] 239 | 240 | samples = sess.run(dcgan.sampler, feed_dict={dcgan.z: z_sample}) 241 | make_gif(samples, './samples/test_gif_%s.gif' % (idx)) 242 | elif option == 4: 243 | image_set = [] 244 | values = np.arange(0, 1, 1./config.batch_size) 245 | 246 | for idx in xrange(dcgan.z_dim): 247 | print(" [*] %d" % idx) 248 | z_sample = np.zeros([config.batch_size, dcgan.z_dim]) 249 | for kdx, z in enumerate(z_sample): z[idx] = values[kdx] 250 | 251 | image_set.append(sess.run(dcgan.sampler, feed_dict={dcgan.z: z_sample})) 252 | make_gif(image_set[-1], './samples/test_gif_%s.gif' % (idx)) 253 | 254 | new_image_set = [merge(np.array([images[idx] for images in image_set]), [10, 10]) \ 255 | for idx in range(64) + range(63, -1, -1)] 256 | make_gif(new_image_set, './samples/test_gif_merged.gif', duration=8) 257 | 258 | 259 | def image_manifold_size(num_images): 260 | manifold_h = int(np.floor(np.sqrt(num_images))) 261 | manifold_w = int(np.ceil(np.sqrt(num_images))) 262 | assert manifold_h * manifold_w == num_images 263 | return manifold_h, manifold_w 264 | --------------------------------------------------------------------------------