├── .gitignore ├── README.md ├── experiments ├── __init__.py ├── classification_ensembles.py ├── classification_imagenet.py ├── classification_ue.py ├── classification_ue_ood.py ├── configs.py ├── data │ ├── .gitignore │ ├── al │ │ └── .gitkeep │ ├── detector │ │ └── .gitkeep │ ├── models │ │ └── .gitkeep │ ├── ood │ │ └── .gitkeep │ ├── regression │ │ └── .gitkeep │ └── xor │ │ └── .gitkeep ├── deprecated │ ├── __init__.py │ ├── active_learning_debug.py │ ├── active_learning_mnist.py │ ├── active_learning_svhn.py │ ├── al_rosenbrock_experiment.py │ ├── classification_active_learning.py │ ├── classification_active_learning_fasetai.py │ ├── classification_bench_masks.ipynb │ ├── classification_error_detection.py │ ├── classification_error_detection_fastai.py │ ├── classification_image.py │ ├── classification_ood_detection.py │ ├── classification_visual.ipynb │ ├── classification_xor.ipynb │ ├── dpp.ipynb │ ├── ensemble-datasets.ipynb │ ├── ensemble-masks.ipynb │ ├── ensemble_debug.py │ ├── ensembles.ipynb │ ├── ensembles_2.ipynb │ ├── ensembles_3.ipynb │ ├── ensembles_4.ipynb │ ├── ensembles_5.ipynb │ ├── move.py │ ├── plot_df.py │ ├── plot_df_al.py │ ├── print_confidence_accuracy_multi_ack.py │ ├── print_histogram.py │ ├── print_it.py │ ├── print_ll.py │ ├── regression_2_prfm2.ipynb │ ├── regression_3_dolan-more.ipynb │ ├── regression_active_learning.ipynb │ ├── regression_bench_ue.ipynb │ ├── regression_dm_big_exper_train.ipynb │ ├── regression_dm_dolan-more.ipynb │ ├── regression_dm_produce_results_from_models.ipynb │ ├── regression_masks.ipynb │ ├── regression_visual-circles.ipynb │ ├── regression_visual.ipynb │ ├── small_eigen_debug.ipynb │ ├── uq_proc (2).ipynb │ └── utils │ │ ├── __init__.py │ │ ├── data.py │ │ └── fastai.py ├── logs │ └── .gitignore ├── models.py ├── move_chest.py ├── print_confidence_accuracy.py ├── print_ood.py ├── regression_1_big_exper_train-clean.ipynb ├── regression_2_ll_on_trained_models.ipynb ├── regression_3_ood_w_training.ipynb └── visual_datasets.py ├── figures ├── 2d_toy.png ├── active_learning_mnist.png ├── benchmark.png ├── convergence.png ├── dolan_acc_ens.png ├── dolan_acc_single.png ├── dpp_ring_contour.png ├── error_detector_cifar.png ├── error_detector_mnist.png ├── error_detector_svhn.png ├── ood_mnist.png └── ring_results.png └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | .ipynb_checkpoints 3 | *.pyc 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Uncertainty estimation via decorrelation and DPP 2 | Code for paper "Dropout Strikes Back: Improved Uncertainty 3 | Estimation via Diversity Sampling" by Kirill Fedyanin, Evgenii Tsymbalov, Maxim Panov published in [Recent Trends in Analysis of Images, Social Networks and Texts](https://link.springer.com/chapter/10.1007/978-3-031-15168-2_11) by Springer. You can cite this paper by bibtex provided below. 4 | 5 | Main code with implemented methods (DPP, k-DPP, leverages masks for dropout) are in our [alpaca library](https://github.com/stat-ml/alpaca) 6 | 7 | 8 | ### Motivation 9 | 10 | 11 | For regression tasks, it could be useful to know not just a prediction but also a confidence interval. It's hard to do it in the close form for deep learning, but to estimate, you can you so-called [ensembles](https://arxiv.org/abs/1612.01474) of several models. To avoid training and keeping several models, you could use [monte-carlo dropout](https://arxiv.org/abs/1506.02142) on inference. 12 | 13 | To use MC dropout, you need multiple forward passes; converging requires tens or even hundreds of forward passes. 14 | 15 |

16 | 17 |

18 | 19 | We propose to force the diversity of forward passes by hiring determinantal point processes. See how it improves the log-likelihood metric across various UCI datasets for the different numbers of stochastic passes T = 10, 30, 100. 20 | 21 | ![Benchmarks on UCI datasets](figures/benchmark.png) 22 | 23 | 24 | ### Paper 25 | 26 | You can read full paper here [https://link.springer.com/chapter/10.1007/978-3-031-15168-2_11](https://link.springer.com/chapter/10.1007/978-3-031-15168-2_11) 27 | 28 | For the citation, please use 29 | 30 | ```bibtex 31 | @InProceedings{Fedyanin2021DropoutSB, 32 | author="Fedyanin, Kirill 33 | and Tsymbalov, Evgenii 34 | and Panov, Maxim", 35 | title="Dropout Strikes Back: Improved Uncertainty Estimation via Diversity Sampling", 36 | booktitle="Recent Trends in Analysis of Images, Social Networks and Texts", 37 | year="2022", 38 | publisher="Springer International Publishing", 39 | pages="125--137", 40 | isbn="978-3-031-15168-2" 41 | } 42 | ``` 43 | 44 | 45 | ## Install dependency 46 | ``` 47 | pip install -r requirements.txt 48 | ``` 49 | 50 | ## Regression 51 | To get the experiment results from the paper, run the following notebooks 52 | - `experiments/regression_1_big_exper_train-clean.ipynb` to train the models 53 | - `experiments/regression_2_ll_on_trained_models.ipynb` to get the ll values for different datasets 54 | - `experiments/regression_3_ood_w_training.ipynb` for the OOD experiments 55 | 56 | ## Classification 57 | 58 | From the experiment folder run the following scripts. They goes in pairs, first script trains models and estimate the uncertainty, second just print the results. 59 | 60 | #### Accuracy experiment on MNIST 61 | ```bash 62 | python classification_ue.py mnist 63 | python print_confidence_accuracy.py mnist 64 | ``` 65 | #### Accuracy experiment on CIFAR 66 | ```bash 67 | python classification_ue.py cifar 68 | python print_confidence_accuracy.py cifar 69 | ``` 70 | #### Accuracy experiment on ImageNet 71 | For the imagenet you need to manually download validation dataset (version ILSVRC2012) and put images to the `experiments/data/imagenet/valid` folder 72 | ```bash 73 | python classification_imagenet.py 74 | python print_confidence_accuracy.py imagenet 75 | ``` 76 | #### OOD experiment on MNIST 77 | ```bash 78 | python classification_ue_ood.py mnist 79 | python print_ood.py mnist 80 | ``` 81 | #### OOD experiment on CIFAR 82 | ```bash 83 | python classification_ue_ood.py cifar 84 | python print_ood.py cifar 85 | ``` 86 | #### OOD experiment on ImageNet 87 | ```bash 88 | python classification_imagenet.py --ood 89 | python print_ood.py imagenet 90 | ``` 91 | 92 | You can change the uncertainty estimation function for mnist/cifar by adding `-a=var_ratio` or `-a=max_prob` keys to the scripts. 93 | -------------------------------------------------------------------------------- /experiments/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/__init__.py -------------------------------------------------------------------------------- /experiments/classification_ensembles.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pickle 3 | from argparse import ArgumentParser 4 | from collections import OrderedDict 5 | from copy import deepcopy 6 | from pathlib import Path 7 | 8 | import numpy as np 9 | import torch 10 | from sklearn.model_selection import train_test_split 11 | from catalyst.dl import SupervisedRunner 12 | from catalyst.dl.callbacks import AccuracyCallback, EarlyStoppingCallback 13 | from catalyst.utils import set_global_seed 14 | 15 | from alpaca.uncertainty_estimator.bald import bald as bald_score 16 | 17 | from configs import base_config, experiment_config 18 | from deprecated.classification_active_learning import loader 19 | from classification_ue_ood import get_data as get_data_ood 20 | from classification_ue import get_data, train 21 | 22 | 23 | def parse_arguments(): 24 | parser = ArgumentParser() 25 | parser.add_argument('name') 26 | parser.add_argument('--acquisition', '-a', type=str, default='bald') 27 | parser.add_argument('--ood', dest='ood', action='store_true') 28 | args = parser.parse_args() 29 | 30 | config = deepcopy(base_config) 31 | config.update(experiment_config[args.name]) 32 | config['name'] = args.name 33 | config['acquisition'] = args.acquisition 34 | config['ood'] = args.ood 35 | 36 | return config 37 | 38 | 39 | def bench_uncertainty(ensemble, x_val_tensor, y_val, config): 40 | probabilities, max_prob, bald, var_ratio_ue = ensemble.estimate(x_val_tensor) 41 | uncertainties = { 42 | 'ensemble_max_prob': np.array(max_prob), 43 | 'ensemble_bald': np.array(bald), 44 | 'ensemble_var_ratio': np.array(var_ratio_ue) 45 | } 46 | record = { 47 | 'y_val': y_val, 48 | 'uncertainties': uncertainties, 49 | 'probabilities': probabilities, 50 | 'estimators': list(uncertainties.keys()), 51 | } 52 | 53 | ood_str = '_ood' if config['ood'] else '' 54 | with open(ensemble.logdir / f"ue_ensemble{ood_str}.pickle", 'wb') as f: 55 | pickle.dump(record, f) 56 | 57 | 58 | class Ensemble: 59 | def __init__(self, logbase, n_models, config, loaders, start_i=0): 60 | self.models = [] 61 | self.n_models = n_models 62 | self.logdir = Path(f"{logbase}/{config['name']}_{start_i}") 63 | 64 | for i in range(start_i, start_i + n_models): 65 | set_global_seed(i + 42) 66 | logdir = Path(f"{logbase}/{config['name']}_{i}") 67 | print(logdir) 68 | 69 | possible_checkpoint = logdir / 'checkpoints' / 'best.pth' 70 | if os.path.exists(possible_checkpoint): 71 | checkpoint = possible_checkpoint 72 | else: 73 | checkpoint = None 74 | 75 | self.models.append(train(config, loaders, logdir, checkpoint)) 76 | 77 | def estimate(self, X_pool): 78 | mcd_realizations = torch.zeros((len(X_pool), self.n_models, 10)) 79 | 80 | with torch.no_grad(): 81 | for i, model in enumerate(self.models): 82 | model.cuda() 83 | prediction = model(X_pool.cuda()) 84 | prediction = prediction.to('cpu') 85 | # mcd_realizations.append(prediction) 86 | mcd_realizations[:, i, :] = torch.softmax(prediction, dim=-1) 87 | 88 | # mcd_realizations = torch.cat(mcd_realizations, dim=0) 89 | probabilities = mcd_realizations.mean(dim=1) 90 | max_class = torch.argmax(probabilities, dim=-1) 91 | max_prob_ens = mcd_realizations[np.arange(len(X_pool)), :, max_class] 92 | max_prob_ue = -max_prob_ens.mean(dim=-1) + 1 93 | bald = bald_score(mcd_realizations.numpy()) 94 | var_ratio_ue = var_ratio_score(mcd_realizations, max_class, self.n_models) 95 | return probabilities, max_prob_ue, bald, var_ratio_ue 96 | 97 | 98 | def var_ratio_score(mcd_realizations, max_class, n_models): 99 | reps = max_class.unsqueeze(dim=-1).repeat(1, n_models) 100 | tops = torch.argmax(mcd_realizations, axis=-1) 101 | incorrects = (reps != tops).float() 102 | ue = torch.sum(incorrects, dim=-1) / n_models 103 | return ue 104 | 105 | 106 | if __name__ == '__main__': 107 | config = parse_arguments() 108 | set_global_seed(42) 109 | logbase = 'logs/classification_ensembles' 110 | n_models = config['n_models'] 111 | 112 | if config['ood']: 113 | loaders, ood_loader, x_train, y_train, x_val, y_val, x_ood, y_ood = get_data_ood(config) 114 | x_ood_tensor = torch.cat([batch[0] for batch in ood_loader]) 115 | for j in range(config['repeats']): 116 | ensemble = Ensemble(logbase, n_models, config, loaders, j * n_models) 117 | bench_uncertainty(ensemble, x_ood_tensor, y_val, config) 118 | else: 119 | loaders, x_train, y_train, x_val, y_val = get_data(config) 120 | x_val_tensor = torch.cat([batch[0] for batch in loaders['valid']]) 121 | 122 | for j in range(config['repeats']): 123 | ensemble = Ensemble(logbase, n_models, config, loaders, j * n_models) 124 | bench_uncertainty(ensemble, x_val_tensor, y_val, config) 125 | 126 | -------------------------------------------------------------------------------- /experiments/classification_imagenet.py: -------------------------------------------------------------------------------- 1 | from time import time 2 | from collections import defaultdict 3 | import os 4 | import pickle 5 | from argparse import ArgumentParser 6 | from copy import deepcopy 7 | from pathlib import Path 8 | 9 | from torch.utils.data import Dataset, DataLoader 10 | from torchvision import transforms 11 | from PIL import Image 12 | 13 | import numpy as np 14 | import torch 15 | from catalyst.utils import set_global_seed 16 | 17 | from alpaca.active_learning.simple_update import entropy 18 | from alpaca.uncertainty_estimator import build_estimator 19 | 20 | from configs import base_config, experiment_config 21 | from models import resnet_dropout 22 | 23 | 24 | def parse_arguments(): 25 | parser = ArgumentParser() 26 | parser.add_argument('--acquisition', '-a', type=str, default='bald') 27 | parser.add_argument('--dataset-folder', type=str, default='data/imagenet') 28 | parser.add_argument('--bs', type=int, default=250) 29 | parser.add_argument('--ood', dest='ood', action='store_true') 30 | args = parser.parse_args() 31 | args.name ='imagenet' 32 | 33 | config = deepcopy(base_config) 34 | config.update(experiment_config[args.name]) 35 | 36 | 37 | if args.ood: 38 | args.dataset_folder = 'data/chest' 39 | 40 | for param in ['name', 'acquisition', 'bs', 'dataset_folder', 'ood']: 41 | config[param] = getattr(args, param) 42 | 43 | 44 | return config 45 | 46 | 47 | def bench_uncertainty(model, val_loader, y_val, acquisition, config, logdir): 48 | ood_str = '_ood' if config['ood'] else '' 49 | logfile = logdir / f"ue{ood_str}_{config['acquisition']}.pickle" 50 | print(logfile) 51 | 52 | probabilities = get_probabilities(model, val_loader).astype(np.single) 53 | 54 | estimators = ['max_prob', 'mc_dropout', 'ht_dpp', 'cov_k_dpp'] 55 | # estimators = ['mc_dropout'] 56 | 57 | print(estimators) 58 | 59 | uncertainties = {} 60 | times = defaultdict(list) 61 | for estimator_name in estimators: 62 | print(estimator_name) 63 | t0 = time() 64 | ue = calc_ue(model, val_loader, probabilities, config['dropout_rate'], estimator_name, nn_runs=config['nn_runs'], acquisition=acquisition) 65 | times[estimator_name].append(time() - t0) 66 | print('time', time() - t0) 67 | uncertainties[estimator_name] = ue 68 | 69 | record = { 70 | 'y_val': y_val, 71 | 'uncertainties': uncertainties, 72 | 'probabilities': probabilities, 73 | 'estimators': estimators, 74 | 'times': times 75 | } 76 | with open(logfile, 'wb') as f: 77 | pickle.dump(record, f) 78 | 79 | return probabilities, uncertainties, estimators 80 | 81 | 82 | def calc_ue(model, val_loader, probabilities, dropout_rate, estimator_type='max_prob', nn_runs=150, acquisition='bald'): 83 | if estimator_type == 'max_prob': 84 | ue = 1 - np.max(probabilities, axis=-1) 85 | elif estimator_type == 'max_entropy': 86 | ue = entropy(probabilities) 87 | else: 88 | acquisition_param = 'var_ratio' if acquisition == 'max_prob' else acquisition 89 | 90 | estimator = build_estimator( 91 | 'bald_masked', model, dropout_mask=estimator_type, num_classes=1000, 92 | nn_runs=nn_runs, keep_runs=False, acquisition=acquisition_param, 93 | dropout_rate=dropout_rate 94 | ) 95 | ue = estimator.estimate(val_loader) 96 | 97 | return ue 98 | 99 | 100 | def get_probabilities(model, loader): 101 | model.eval().cuda() 102 | 103 | results = [] 104 | 105 | with torch.no_grad(): 106 | for i, batch in enumerate(loader): 107 | print(i) 108 | probabilities = torch.softmax(model(batch.cuda()), dim=-1) 109 | results.extend(list(probabilities.cpu().numpy())) 110 | return np.array(results) 111 | 112 | 113 | 114 | image_transforms = transforms.Compose([ 115 | transforms.Resize(256), 116 | transforms.CenterCrop(224), 117 | transforms.ToTensor(), 118 | transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) 119 | ]) 120 | 121 | 122 | class ImageDataset(Dataset): 123 | def __init__(self, folder): 124 | self.folder = Path(folder) 125 | files = sorted(os.listdir(folder)) 126 | self.files = files[:downsample] 127 | 128 | def __len__(self): 129 | return len(self.files) 130 | 131 | def __getitem__(self, idx): 132 | img_path = self.folder / self.files[idx] 133 | image = Image.open(img_path).convert('RGB') 134 | 135 | image = image_transforms(image).double() 136 | 137 | return image 138 | 139 | 140 | def get_data(config): 141 | label_file = Path(config['dataset_folder'])/'val.txt' 142 | 143 | with open(label_file, 'r') as f: 144 | y_val = np.array([int(line.split()[1]) for line in f.readlines()])[:downsample] 145 | 146 | valid_folder = Path(config['dataset_folder']) / 'valid' 147 | 148 | dataset = ImageDataset(valid_folder) 149 | val_loader = DataLoader(dataset, batch_size=config['bs']) 150 | val_loader.shape = (downsample, 3, 224, 224) 151 | 152 | return val_loader, y_val 153 | 154 | 155 | if __name__ == '__main__': 156 | downsample = 50_000 157 | 158 | config = parse_arguments() 159 | print(config) 160 | set_global_seed(42) 161 | val_loader, y_val = get_data(config) 162 | 163 | for i in range(config['repeats']): 164 | set_global_seed(i + 42) 165 | logdir = Path(f"logs/classification/{config['name']}_{i}") 166 | 167 | model = resnet_dropout(dropout_rate=config['dropout_rate']).double() 168 | 169 | bench_uncertainty(model, val_loader, y_val, config['acquisition'], config, logdir) 170 | -------------------------------------------------------------------------------- /experiments/classification_ue.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pickle 3 | from argparse import ArgumentParser 4 | from collections import OrderedDict 5 | from copy import deepcopy 6 | from pathlib import Path 7 | 8 | import numpy as np 9 | import torch 10 | from sklearn.model_selection import train_test_split 11 | from sklearn.metrics import roc_curve, roc_auc_score 12 | from scipy.special import softmax 13 | import matplotlib.pyplot as plt 14 | import seaborn as sns 15 | import pandas as pd 16 | from catalyst.dl import SupervisedRunner 17 | from catalyst.dl.callbacks import AccuracyCallback, EarlyStoppingCallback 18 | from catalyst.utils import set_global_seed 19 | 20 | from alpaca.active_learning.simple_update import entropy 21 | from alpaca.uncertainty_estimator import build_estimator 22 | from alpaca.uncertainty_estimator.masks import DEFAULT_MASKS 23 | 24 | from configs import base_config, experiment_config 25 | from visual_datasets import loader 26 | 27 | 28 | def parse_arguments(): 29 | parser = ArgumentParser() 30 | parser.add_argument('name') 31 | parser.add_argument('--acquisition', '-a', type=str, default='bald') 32 | args = parser.parse_args() 33 | 34 | config = deepcopy(base_config) 35 | config.update(experiment_config[args.name]) 36 | config['name'] = args.name 37 | config['acquisition'] = args.acquisition 38 | 39 | return config 40 | 41 | 42 | def train(config, loaders, logdir, checkpoint=None): 43 | model = config['model_class'](dropout_rate=config['dropout_rate']).double() 44 | 45 | if checkpoint is not None: 46 | model.load_state_dict(torch.load(checkpoint)['model_state_dict']) 47 | model.eval() 48 | else: 49 | criterion = torch.nn.CrossEntropyLoss() 50 | optimizer = torch.optim.Adam(model.parameters()) 51 | callbacks = [AccuracyCallback(num_classes=10), EarlyStoppingCallback(config['patience'])] 52 | 53 | runner = SupervisedRunner() 54 | runner.train( 55 | model, criterion, optimizer, loaders, 56 | logdir=logdir, num_epochs=config['epochs'], verbose=True, 57 | callbacks=callbacks 58 | ) 59 | 60 | return model 61 | 62 | 63 | def bench_uncertainty(model, model_checkpoint, loaders, x_val, y_val, acquisition, config): 64 | runner = SupervisedRunner() 65 | logits = runner.predict_loader(model, loaders['valid']) 66 | probabilities = softmax(logits, axis=-1) 67 | 68 | if config['acquisition'] in ['bald', 'var_ratio']: 69 | estimators = ['mc_dropout', 'ht_dpp', 'ht_k_dpp', 'cov_dpp', 'cov_k_dpp'] 70 | elif config['acquisition'] == 'max_prob': 71 | estimators = ['max_prob', 'mc_dropout', 'ht_dpp', 'ht_k_dpp', 'cov_dpp', 'cov_k_dpp'] 72 | else: 73 | raise ValueError 74 | 75 | print(estimators) 76 | 77 | uncertainties = {} 78 | lls = {} 79 | for estimator_name in estimators: 80 | # try: 81 | print(estimator_name) 82 | ue, ll = calc_ue(model, x_val, y_val, probabilities, estimator_name, nn_runs=config['nn_runs'], acquisition=acquisition) 83 | uncertainties[estimator_name] = ue 84 | lls[estimator_name] = ll 85 | # except Exception as e: 86 | # print(e) 87 | 88 | record = { 89 | 'checkpoint': model_checkpoint, 90 | 'y_val': y_val, 91 | 'uncertainties': uncertainties, 92 | 'probabilities': probabilities, 93 | 'logits': logits, 94 | 'estimators': estimators, 95 | 'lls': lls 96 | } 97 | with open(logdir / f"ue_{config['acquisition']}.pickle", 'wb') as f: 98 | pickle.dump(record, f) 99 | 100 | return probabilities, uncertainties, estimators 101 | 102 | 103 | def log_likelihood(probabilities, y): 104 | try: 105 | ll = np.mean(np.log(probabilities[np.arange(len(probabilities)), y])) 106 | except FloatingPointError: 107 | import ipdb; ipdb.set_trace() 108 | return ll 109 | 110 | 111 | def calc_ue(model, datapoints, y_val, probabilities, estimator_type='max_prob', nn_runs=100, acquisition='bald'): 112 | if estimator_type == 'max_prob': 113 | ue = 1 - np.max(probabilities, axis=-1) 114 | ll = log_likelihood(probabilities, y_val) 115 | elif estimator_type == 'max_entropy': 116 | ue = entropy(probabilities) 117 | ll = log_likelihood(probabilities, y_val) 118 | else: 119 | acquisition_param = 'var_ratio' if acquisition == 'max_prob' else acquisition 120 | 121 | estimator = build_estimator( 122 | 'bald_masked', model, dropout_mask=estimator_type, num_classes=10, 123 | nn_runs=nn_runs, keep_runs=True, acquisition=acquisition_param) 124 | ue = estimator.estimate(torch.DoubleTensor(datapoints).cuda()) 125 | probs = softmax(estimator.last_mcd_runs(), axis=-1) 126 | probs = np.mean(probs, axis=-2) 127 | 128 | ll = log_likelihood(probs, y_val) 129 | 130 | if acquisition == 'max_prob': 131 | ue = 1 - np.max(probs, axis=-1) 132 | 133 | return ue, ll 134 | 135 | 136 | def misclassfication_detection(y_val, probabilities, uncertainties, estimators): 137 | results = [] 138 | predictions = np.argmax(probabilities, axis=-1) 139 | errors = (predictions != y_val) 140 | for estimator_name in estimators: 141 | fpr, tpr, _ = roc_curve(errors, uncertainties[estimator_name]) 142 | roc_auc = roc_auc_score(errors, uncertainties[estimator_name]) 143 | results.append((estimator_name, roc_auc)) 144 | 145 | return results 146 | 147 | 148 | def get_data(config): 149 | x_train, y_train, x_val, y_val, train_tfms = config['prepare_dataset'](config) 150 | 151 | if len(x_train) > config['train_size']: 152 | x_train, _, y_train, _ = train_test_split( 153 | x_train, y_train, train_size=config['train_size'], stratify=y_train 154 | ) 155 | 156 | loaders = OrderedDict({ 157 | 'train': loader(x_train, y_train, config['batch_size'], tfms=train_tfms, train=True), 158 | 'valid': loader(x_val, y_val, config['batch_size']) 159 | }) 160 | return loaders, x_train, y_train, x_val, y_val 161 | 162 | 163 | if __name__ == '__main__': 164 | config = parse_arguments() 165 | set_global_seed(42) 166 | loaders, x_train, y_train, x_val, y_val = get_data(config) 167 | print(y_train[:5]) 168 | 169 | rocaucs = [] 170 | for i in range(config['repeats']): 171 | set_global_seed(i + 42) 172 | logdir = Path(f"logs/classification/{config['name']}_{i}") 173 | print(logdir) 174 | 175 | possible_checkpoint = logdir / 'checkpoints' / 'best.pth' 176 | if os.path.exists(possible_checkpoint): 177 | checkpoint = possible_checkpoint 178 | else: 179 | checkpoint = None 180 | 181 | model = train(config, loaders, logdir, checkpoint) 182 | x_val_tensor = torch.cat([batch[0] for batch in loaders['valid']]) 183 | 184 | bench_uncertainty( 185 | model, checkpoint, loaders, x_val_tensor, y_val, config['acquisition'], config) 186 | 187 | 188 | -------------------------------------------------------------------------------- /experiments/classification_ue_ood.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pickle 3 | from argparse import ArgumentParser 4 | from collections import OrderedDict 5 | from copy import deepcopy 6 | from pathlib import Path 7 | 8 | import numpy as np 9 | import torch 10 | from sklearn.model_selection import train_test_split 11 | from scipy.special import softmax 12 | from catalyst.dl import SupervisedRunner 13 | from catalyst.utils import set_global_seed 14 | 15 | from alpaca.active_learning.simple_update import entropy 16 | from alpaca.uncertainty_estimator import build_estimator 17 | from alpaca.uncertainty_estimator.masks import DEFAULT_MASKS 18 | 19 | from configs import base_config, experiment_ood_config 20 | from visual_datasets import loader 21 | 22 | from classification_ue import train 23 | 24 | 25 | def parse_arguments(): 26 | parser = ArgumentParser() 27 | parser.add_argument('name') 28 | parser.add_argument('--acquisition', '-a', type=str, default='bald') 29 | args = parser.parse_args() 30 | 31 | config = deepcopy(base_config) 32 | config.update(experiment_ood_config[args.name]) 33 | config['name'] = args.name 34 | config['acquisition'] = args.acquisition 35 | 36 | return config 37 | 38 | 39 | def bench_uncertainty(model, model_checkpoint, ood_loader, x_ood, acquisition, nn_runs): 40 | runner = SupervisedRunner() 41 | logits = runner.predict_loader(model, ood_loader) 42 | probabilities = softmax(logits, axis=-1) 43 | 44 | if config['acquisition'] in ['bald', 'var_ratio']: 45 | estimators = ['mc_dropout', 'ht_dpp', 'ht_k_dpp', 'cov_dpp', 'cov_k_dpp'] 46 | elif config['acquisition'] == 'max_prob': 47 | estimators = ['max_prob', 'mc_dropout', 'ht_dpp', 'ht_k_dpp', 'cov_dpp', 'cov_k_dpp'] 48 | else: 49 | raise ValueError 50 | 51 | 52 | uncertainties = {} 53 | for estimator_name in estimators: 54 | print(estimator_name) 55 | ue = calc_ue(model, x_ood, probabilities, estimator_name, nn_runs=nn_runs, acquisition=acquisition) 56 | uncertainties[estimator_name] = ue 57 | print(ue) 58 | 59 | record = { 60 | 'checkpoint': model_checkpoint, 61 | 'uncertainties': uncertainties, 62 | 'probabilities': probabilities, 63 | 'logits': logits, 64 | 'estimators': estimators 65 | } 66 | 67 | file_name = logdir / f"ue_ood_{config['acquisition']}.pickle" 68 | with open(file_name, 'wb') as f: 69 | pickle.dump(record, f) 70 | 71 | return probabilities, uncertainties, estimators 72 | 73 | 74 | def calc_ue(model, datapoints, probabilities, estimator_type='max_prob', nn_runs=100, acquisition='bald'): 75 | if estimator_type == 'max_prob': 76 | ue = 1 - np.max(probabilities, axis=-1) 77 | elif estimator_type == 'max_entropy': 78 | ue = entropy(probabilities) 79 | else: 80 | acquisition_param = 'var_ratio' if acquisition == 'max_prob' else acquisition 81 | 82 | estimator = build_estimator( 83 | 'bald_masked', model, dropout_mask=estimator_type, num_classes=10, 84 | nn_runs=nn_runs, keep_runs=True, acquisition=acquisition_param) 85 | ue = estimator.estimate(torch.DoubleTensor(datapoints).cuda()) 86 | 87 | if acquisition == 'max_prob': 88 | probs = softmax(estimator.last_mcd_runs(), axis=-1) 89 | probs = np.mean(probs, axis=-2) 90 | ue = 1 - np.max(probs, axis=-1) 91 | 92 | return ue 93 | 94 | 95 | def get_data(config): 96 | x_train, y_train, x_val, y_val, train_tfms = config['prepare_dataset'](config) 97 | 98 | if len(x_train) > config['train_size']: 99 | x_train, _, y_train, _ = train_test_split( 100 | x_train, y_train, train_size=config['train_size'], stratify=y_train 101 | ) 102 | 103 | _, _, x_ood, y_ood, _ = config['prepare_dataset'](config) 104 | ood_loader = loader(x_ood, y_ood, config['batch_size']) 105 | 106 | loaders = OrderedDict({ 107 | 'train': loader(x_train, y_train, config['batch_size'], tfms=train_tfms, train=True), 108 | 'valid': loader(x_val, y_val, config['batch_size']), 109 | }) 110 | return loaders, ood_loader, x_train, y_train, x_val, y_val, x_ood, y_ood 111 | 112 | 113 | if __name__ == '__main__': 114 | config = parse_arguments() 115 | print(config) 116 | set_global_seed(42) 117 | loaders, ood_loader, x_train, y_train, x_val, y_val, x_ood, y_ood = get_data(config) 118 | print(y_train[:5]) 119 | 120 | for i in range(config['repeats']): 121 | set_global_seed(i + 42) 122 | logdir = Path(f"logs/classification/{config['name']}_{i}") 123 | print(logdir) 124 | 125 | possible_checkpoint = logdir / 'checkpoints' / 'best.pth' 126 | if os.path.exists(possible_checkpoint): 127 | checkpoint = possible_checkpoint 128 | else: 129 | checkpoint = None 130 | 131 | model = train(config, loaders, logdir, checkpoint) 132 | print(model) 133 | x_ood_tensor = torch.cat([batch[0] for batch in ood_loader]) 134 | 135 | probabilities, uncertainties, estimators = bench_uncertainty( 136 | model, checkpoint, ood_loader, x_ood_tensor, config['acquisition'], config['nn_runs']) 137 | 138 | -------------------------------------------------------------------------------- /experiments/configs.py: -------------------------------------------------------------------------------- 1 | from copy import deepcopy 2 | 3 | from visual_datasets import prepare_cifar, prepare_mnist, prepare_svhn 4 | from models import SimpleConv, StrongConv 5 | 6 | base_config = { 7 | 'train_size': 50_000, 8 | 'val_size': 10_000, 9 | 'prepare_dataset': prepare_mnist, 10 | 'model_class': SimpleConv, 11 | 'epochs': 50, 12 | 'patience': 3, 13 | 'batch_size': 128, 14 | 'repeats': 3, 15 | 'dropout_rate': 0.5, 16 | 'nn_runs': 100 17 | } 18 | 19 | 20 | experiment_config = { 21 | 'mnist': { 22 | 'train_size': 500, 23 | 'dropout_rate': 0.5, 24 | 'n_models': 20 25 | }, 26 | 'cifar': { 27 | 'prepare_dataset': prepare_cifar, 28 | 'model_class': StrongConv, 29 | 'n_models': 20, 30 | }, 31 | 'imagenet': { 32 | 'dropout_rate': 0.5, 33 | 'repeats': 1, 34 | } 35 | } 36 | 37 | experiment_ood_config = { 38 | 'mnist': { 39 | 'train_size': 500, 40 | 'ood_dataset': prepare_mnist 41 | }, 42 | 'svhn': { 43 | 'prepare_dataset': prepare_svhn, 44 | 'model_class': StrongConv, 45 | 'repeats': 3, 46 | }, 47 | 'cifar': { 48 | 'prepare_dataset': prepare_cifar, 49 | 'model_class': StrongConv, 50 | 'repeats': 3 51 | } 52 | } 53 | 54 | 55 | al_config = deepcopy(base_config) 56 | al_config.update({ 57 | 'pool_size': 10_000, 58 | 'val_size': 10_000, 59 | 'repeats': 5, 60 | 'methods': ['random', 'mc_dropout', 'decorrelating_sc', 'dpp', 'ht_dpp', 'k_dpp', 'ht_k_dpp'], 61 | 'steps': 50 62 | }) 63 | 64 | al_experiments = { 65 | 'mnist': { 66 | 'start_size': 200, 67 | 'step_size': 10, 68 | }, 69 | 'cifar': { 70 | 'start_size': 2000, 71 | 'step_size': 30, 72 | 'prepare_dataset': prepare_cifar, 73 | 'model_class': StrongConv, 74 | 'repeats': 3 75 | }, 76 | 'svhn': { 77 | 'start_size': 2000, 78 | 'step_size': 30, 79 | 'prepare_dataset': prepare_svhn, 80 | 'model_class': StrongConv, 81 | 'repeats': 3 82 | } 83 | } 84 | -------------------------------------------------------------------------------- /experiments/data/.gitignore: -------------------------------------------------------------------------------- 1 | *.csv 2 | *.png 3 | *.pkl 4 | -------------------------------------------------------------------------------- /experiments/data/al/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/data/al/.gitkeep -------------------------------------------------------------------------------- /experiments/data/detector/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/data/detector/.gitkeep -------------------------------------------------------------------------------- /experiments/data/models/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/data/models/.gitkeep -------------------------------------------------------------------------------- /experiments/data/ood/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/data/ood/.gitkeep -------------------------------------------------------------------------------- /experiments/data/regression/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/data/regression/.gitkeep -------------------------------------------------------------------------------- /experiments/data/xor/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/data/xor/.gitkeep -------------------------------------------------------------------------------- /experiments/deprecated/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/deprecated/__init__.py -------------------------------------------------------------------------------- /experiments/deprecated/active_learning_debug.py: -------------------------------------------------------------------------------- 1 | import sys 2 | sys.path.append('..') 3 | 4 | import torch 5 | import numpy as np 6 | import matplotlib.pyplot as plt 7 | 8 | from fastai.vision import rand_pad, flip_lr, ImageDataBunch 9 | 10 | from model.cnn import AnotherConv 11 | from dataloader.builder import build_dataset 12 | from deprecated.utils import ImageArrayDS 13 | 14 | # == Place to develop, experiment and debug new methods of active learning == # 15 | 16 | # torch.cuda.set_device(1) 17 | 18 | 19 | total_size = 60_000 20 | val_size = 10_000 21 | start_size = 1_000 22 | step_size = 10 23 | steps = 8 24 | reload = True 25 | nn_runs = 100 26 | 27 | pool_size = 200 28 | 29 | 30 | # Load data 31 | dataset = build_dataset('cifar_10', val_size=val_size) 32 | x_set, y_set = dataset.dataset('train') 33 | x_val, y_val = dataset.dataset('val') 34 | 35 | shape = (-1, 3, 32, 32) 36 | x_set = ((x_set - 128)/128).reshape(shape) 37 | x_val = ((x_val - 128)/128).reshape(shape) 38 | 39 | # x_pool, x_train, y_pool, y_train = train_test_split(x_set, y_set, test_size=start_size, stratify=y_set) 40 | x_train, y_train = x_set, y_set 41 | 42 | train_tfms = [*rand_pad(4, 32), flip_lr(p=0.5)] 43 | train_ds = ImageArrayDS(x_train, y_train, train_tfms) 44 | val_ds = ImageArrayDS(x_val, y_val) 45 | data = ImageDataBunch.create(train_ds, val_ds, bs=256) 46 | 47 | 48 | loss_func = torch.nn.CrossEntropyLoss() 49 | 50 | np.set_printoptions(threshold=sys.maxsize, suppress=True) 51 | 52 | model = AnotherConv() 53 | # model = resnet_masked(pretrained=True) 54 | # model = resnet_linear(pretrained=True, dropout_rate=0.5, freeze=False) 55 | 56 | # learner = Learner(data, model, metrics=accuracy, loss_func=loss_func) 57 | # 58 | # model_path = "experiments/data/model.pt" 59 | # if reload and os.path.exists(model_path): 60 | # model.load_state_dict(torch.load(model_path)) 61 | # else: 62 | # learner.fit(10, 1e-3, wd=0.02) 63 | # torch.save(model.state_dict(), model_path) 64 | # 65 | # images = torch.FloatTensor(x_val)# .to('cuda') 66 | # 67 | # inferencer = Inferencer(model) 68 | # predictions = F.softmax(inferencer(images), dim=1).detach().cpu().numpy()[:10] 69 | 70 | 71 | repeats = 3 72 | methods = ['goy', 'mus', 'cosher'] 73 | from random import random 74 | 75 | results = [] 76 | for _ in range(repeats): 77 | start = 0.1 * random() 78 | for method in methods: 79 | accuracies = [start] 80 | current = start 81 | for i in range(10): 82 | current += 0.1*random() 83 | accuracies.append(current) 84 | records = list(zip(accuracies, range(len(accuracies)), [method] * len(accuracies))) 85 | results.extend(records) 86 | 87 | import pandas as pd 88 | import seaborn as sns 89 | df = pd.DataFrame(results, columns=['accuracy', 'step', 'method']) 90 | sns.lineplot('step', 'accuracy', hue='method', data=df) 91 | plt.show() 92 | 93 | 94 | # idxs = np.argsort(entropies)[::-1][:10] 95 | # print(idxs) 96 | 97 | # mask = build_mask('k_dpp') 98 | # estimator = BaldMasked(inferencer, dropout_mask=mask, num_classes=10, nn_runs=nn_runs) 99 | # estimations = estimator.estimate(images) 100 | # print(estimations) 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | -------------------------------------------------------------------------------- /experiments/deprecated/active_learning_mnist.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import torch 3 | 4 | sys.path.append('..') 5 | from dataloader.builder import build_dataset 6 | from uncertainty_estimator.masks import DEFAULT_MASKS 7 | from active_learning import main 8 | 9 | torch.cuda.set_device(1) 10 | torch.backends.cudnn.benchmark = True 11 | 12 | 13 | def prepare_mnist(config): 14 | dataset = build_dataset('mnist', val_size=config['val_size']) 15 | x_set, y_set = dataset.dataset('train') 16 | x_val, y_val = dataset.dataset('val') 17 | 18 | shape = (-1, 1, 28, 28) 19 | x_set = ((x_set - 128) / 128).reshape(shape) 20 | x_val = ((x_val - 128) / 128).reshape(shape) 21 | 22 | train_tfms = [] 23 | 24 | return x_set, y_set, x_val, y_val, train_tfms 25 | 26 | 27 | mnist_config = { 28 | 'repeats': 5, 29 | 'start_size': 100, 30 | 'step_size': 20, 31 | 'val_size': 10_000, 32 | 'pool_size': 10_000, 33 | 'steps': 30, 34 | 'methods': ['random', 'error_oracle', 'max_entropy', *DEFAULT_MASKS], 35 | 'epochs': 30, 36 | 'patience': 2, 37 | 'model_type': 'simple_conv', 38 | 'nn_runs': 100, 39 | 'batch_size': 32, 40 | 'start_lr': 5e-4, 41 | 'weight_decay': 0.2, 42 | 'prepare_dataset': prepare_mnist, 43 | 'name': 'mnist_beauty_2' 44 | } 45 | 46 | 47 | if __name__ == '__main__': 48 | main(mnist_config) 49 | -------------------------------------------------------------------------------- /experiments/deprecated/active_learning_svhn.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import torch 3 | import numpy as np 4 | from fastai.vision import (rand_pad, flip_lr) 5 | 6 | sys.path.append('..') 7 | from dataloader.builder import build_dataset 8 | from uncertainty_estimator.masks import DEFAULT_MASKS 9 | from active_learning import main 10 | 11 | torch.cuda.set_device(1) 12 | torch.backends.cudnn.benchmark = True 13 | 14 | 15 | def prepare_svhn(config): 16 | dataset = build_dataset('svhn', val_size=config['val_size']) 17 | x_set, y_set = dataset.dataset('train') 18 | x_val, y_val = dataset.dataset('val') 19 | y_set[y_set == 10] = 0 20 | y_val[y_val == 10] = 0 21 | 22 | shape = (-1, 32, 32, 3) 23 | x_set = ((x_set - 128) / 128).reshape(shape) 24 | x_val = ((x_val - 128) / 128).reshape(shape) 25 | x_set = np.rollaxis(x_set, 3, 1) 26 | x_val = np.rollaxis(x_val, 3, 1) 27 | 28 | train_tfms = [*rand_pad(4, 32), flip_lr(p=0.5)] # Transformation to augment images 29 | 30 | return x_set, y_set, x_val, y_val, train_tfms 31 | 32 | 33 | svhn_config = { 34 | 'repeats': 3, 35 | 'start_size': 5_000, 36 | 'step_size': 50, 37 | 'val_size': 10_000, 38 | 'pool_size': 12_000, 39 | 'steps': 30, 40 | 'methods': ['random', 'error_oracle', 'max_entropy', *DEFAULT_MASKS], 41 | 'epochs': 30, 42 | 'patience': 2, 43 | 'model_type': 'resnet', 44 | 'nn_runs': 100, 45 | 'batch_size': 256, 46 | 'start_lr': 5e-4, 47 | 'weight_decay': 0.2, 48 | 'prepare_dataset': prepare_svhn, 49 | 'name': 'svhn' 50 | } 51 | 52 | experiment = 4 53 | 54 | if experiment == 2: 55 | svhn_config['pool_size'] = 5000 56 | svhn_config['start_size'] = 2000 57 | svhn_config['step_size'] = 20 58 | svhn_config['model'] = 'conv' 59 | elif experiment == 3: 60 | svhn_config['pool_size'] = 20_000 61 | svhn_config['start_size'] = 10_000 62 | svhn_config['step_size'] = 100 63 | svhn_config['model'] = 'resent' 64 | elif experiment == 4: 65 | svhn_config['pool_size'] = 5_000 66 | svhn_config['start_size'] = 1_000 67 | svhn_config['step_size'] = 30 68 | svhn_config['model'] = 'resent' 69 | svhn_config['repeats'] = 10 70 | 71 | 72 | if __name__ == '__main__': 73 | main(svhn_config) 74 | -------------------------------------------------------------------------------- /experiments/deprecated/al_rosenbrock_experiment.py: -------------------------------------------------------------------------------- 1 | import random 2 | import argparse 3 | 4 | import numpy as np 5 | import matplotlib.pyplot as plt 6 | import torch 7 | 8 | from model.mlp import MLP 9 | from dataloader.rosen import RosenData 10 | from uncertainty_estimator.nngp import NNGPRegression 11 | from uncertainty_estimator.mcdue import MCDUE 12 | from uncertainty_estimator.random_estimator import RandomEstimator 13 | from sample_selector.eager import EagerSampleSelector 14 | from oracle.identity import IdentityOracle 15 | from active_learning.al_trainer import ALTrainer 16 | 17 | 18 | def run_experiment(config): 19 | """ 20 | Run active learning for the 10D rosenbrock function data 21 | It starts from small train dataset and then extends it with points from pool 22 | 23 | We compare three sampling methods: 24 | - Random datapoints 25 | - Points with highest uncertainty by MCDUE 26 | - Points with highest uncertainty by NNGP (proposed method) 27 | """ 28 | rmses = {} 29 | 30 | for estimator_name in config['estimators']: 31 | print("\nEstimator:", estimator_name) 32 | 33 | # load data 34 | 35 | rosen = RosenData( 36 | config['n_dim'], config['data_size'], config['data_split'], 37 | use_cache=config['use_cache']) 38 | x_train, y_train = rosen.dataset('train') 39 | x_val, y_val = rosen.dataset('train') 40 | x_pool, y_pool = rosen.dataset('pool') 41 | 42 | # Build neural net and set random seed 43 | set_random(config['random_seed']) 44 | model = MLP(config['layers']) 45 | 46 | estimator = build_estimator(estimator_name, model) # to estimate uncertainties 47 | oracle = IdentityOracle(y_pool) # generate y for X from pool 48 | sampler = EagerSampleSelector() # sample X and y from pool by uncertainty estimations 49 | 50 | # Active learning training 51 | trainer = ALTrainer( 52 | model, estimator, sampler, oracle, config['al_iterations'], 53 | config['update_size'], verbose=config['verbose']) 54 | rmses[estimator_name] = trainer.train(x_train, y_train, x_val, y_val, x_pool) 55 | 56 | visualize(rmses) 57 | 58 | 59 | def set_random(random_seed): 60 | # Setting seeds for reproducibility 61 | if random_seed is not None: 62 | torch.manual_seed(random_seed) 63 | np.random.seed(random_seed) 64 | random.seed(random_seed) 65 | 66 | 67 | def build_estimator(name, model): 68 | if name == 'nngp': 69 | estimator = NNGPRegression(model) 70 | elif name == 'random': 71 | estimator = RandomEstimator() 72 | elif name == 'mcdue': 73 | estimator = MCDUE(model) 74 | else: 75 | raise ValueError("Wrong estimator name") 76 | return estimator 77 | 78 | 79 | def visualize(rmses): 80 | print(rmses) 81 | plt.figure(figsize=(12, 9)) 82 | plt.xlabel('Active learning iteration') 83 | plt.ylabel('Validation RMSE') 84 | for estimator_name, rmse in rmses.items(): 85 | plt.plot(rmse, label=estimator_name, marker='.') 86 | 87 | plt.title('RMS Error by active learning iterations') 88 | plt.legend() 89 | 90 | plt.show() 91 | 92 | 93 | def parse_arguments(): 94 | parser = argparse.ArgumentParser(description='Change experiment parameters') 95 | parser.add_argument( 96 | '--estimators', choices=['nngp', 'mcdue', 'random'], default=['nngp', 'mcdue', 'random'], 97 | nargs='+', help='Estimator types for the experiment') 98 | parser.add_argument( 99 | '--random-seed', type=int, default=None, 100 | help='Set the seed to make result reproducible') 101 | parser.add_argument( 102 | '--n-dim', type=int, default=10, help='Rosenbrock function dimentions') 103 | parser.add_argument( 104 | '--data-size', type=int, default=2000, help='Size of dataset') 105 | parser.add_argument( 106 | '--data-split', type=int, default=[0.1, 0.1, 0.1, 0.7], help='Size of dataset') 107 | parser.add_argument( 108 | '--update-size', type=int, default=100, 109 | help='Amount of samples to take from pool per iteration') 110 | parser.add_argument( 111 | '--al-iterations', '-i', type=int, default=10, help='Number of learning iterations') 112 | parser.add_argument('--verbose', action='store_true') 113 | parser.add_argument( 114 | '--no-use-cache', dest='use_cache', action='store_false', 115 | help='To generate new sample points for rosenbrock function') 116 | parser.add_argument( 117 | '--layers', type=int, nargs='+', default=[10, 128, 64, 32, 1], 118 | help='Size of the layers in neural net') 119 | 120 | return vars(parser.parse_args()) 121 | 122 | 123 | if __name__ == '__main__': 124 | config = parse_arguments() 125 | run_experiment(config) 126 | 127 | 128 | config = { 129 | 'estimators': ['nngp', 'mcdue', 'random'], 130 | 'random_seed': None, 131 | 'n_dim': 9, 132 | 'data_size': 2400, 133 | 'data_split': [0.2, 0.1, 0.1, 0.6], 134 | 'update_size': 99, 135 | 'al_iterations': 9, 136 | 'verbose': False, 137 | 'use_cache': True, 138 | 'layers': [9, 128, 64, 32, 1] 139 | } 140 | -------------------------------------------------------------------------------- /experiments/deprecated/classification_active_learning.py: -------------------------------------------------------------------------------- 1 | from argparse import ArgumentParser 2 | from collections import OrderedDict 3 | from copy import deepcopy 4 | from pathlib import Path 5 | 6 | import numpy as np 7 | import torch 8 | from torch.utils.data import DataLoader 9 | from sklearn.model_selection import train_test_split 10 | import matplotlib.pyplot as plt 11 | import seaborn as sns 12 | import pandas as pd 13 | from catalyst.dl import SupervisedRunner 14 | from catalyst.dl.callbacks import AccuracyCallback, EarlyStoppingCallback 15 | from catalyst.utils import set_global_seed 16 | 17 | from alpaca.active_learning.simple_update import update_set 18 | 19 | from configs import al_config, al_experiments 20 | from visual_datasets import ImageDataset 21 | 22 | 23 | def parse_arguments(): 24 | parser = ArgumentParser() 25 | parser.add_argument('name') 26 | name = parser.parse_args().name 27 | config = deepcopy(al_config) 28 | config.update(al_experiments[name]) 29 | config['name'] = name 30 | 31 | return config 32 | 33 | 34 | def active_train(config, i): 35 | set_global_seed(i + 42) 36 | # Load data 37 | x_set, y_set, x_val, y_val, train_tfms = config['prepare_dataset'](config) 38 | print(x_set.shape) 39 | 40 | # Initial data split 41 | x_set, x_train_init, y_set, y_train_init = train_test_split(x_set, y_set, test_size=config['start_size'], stratify=y_set) 42 | _, x_pool_init, _, y_pool_init = train_test_split(x_set, y_set, test_size=config['pool_size'], stratify=y_set) 43 | 44 | model_init = config['model_class']().double() 45 | val_accuracy = [] 46 | # Active learning 47 | for method in config['methods']: 48 | model = deepcopy(model_init) 49 | print(f"== {method} ==") 50 | logdir = f"logs/al/{config['name']}_{method}_{i}" 51 | x_pool, y_pool = np.copy(x_pool_init), np.copy(y_pool_init) 52 | x_train, y_train = np.copy(x_train_init), np.copy(y_train_init) 53 | 54 | criterion = torch.nn.CrossEntropyLoss() 55 | optimizer = torch.optim.Adam(model.parameters()) 56 | 57 | accuracies = [] 58 | 59 | for j in range(config['steps']): 60 | print(f"Step {j+1}, train size: {len(x_train)}") 61 | 62 | loaders = get_loaders(x_train, y_train, x_val, y_val, config['batch_size'], train_tfms) 63 | 64 | runner = SupervisedRunner() 65 | callbacks = [AccuracyCallback(num_classes=10), EarlyStoppingCallback(config['patience'])] 66 | runner.train( 67 | model, criterion, optimizer, loaders, 68 | logdir=logdir, num_epochs=config['epochs'], verbose=False, 69 | callbacks=callbacks 70 | ) 71 | 72 | accuracies.append(runner.state.best_valid_metrics['accuracy01']) 73 | 74 | if i != config['steps'] - 1: 75 | samples = next(iter(loader(x_pool, y_pool, batch_size=len(x_pool))))[0].cuda() 76 | print('Samples!', samples.shape) 77 | x_pool, x_train, y_pool, y_train = update_set( 78 | x_pool, x_train, y_pool, y_train, config['step_size'], 79 | method=method, model=model, samples=samples) 80 | print('Metric', accuracies) 81 | 82 | records = list(zip(accuracies, range(len(accuracies)), [method] * len(accuracies))) 83 | val_accuracy.extend(records) 84 | 85 | return val_accuracy 86 | 87 | 88 | def get_loaders(x_train, y_train, x_val, y_val, batch_size, tfms): 89 | loaders = OrderedDict({ 90 | 'train': loader(x_train, y_train, batch_size, tfms=tfms, train=True), 91 | 'valid': loader(x_val, y_val, batch_size) 92 | }) 93 | return loaders 94 | 95 | 96 | # Set initial datas 97 | def loader(x, y, batch_size=128, tfms=None, train=False): 98 | # ds = TensorDataset(torch.DoubleTensor(x), torch.LongTensor(y)) 99 | ds = ImageDataset(x, y, train=train, tfms=tfms) 100 | _loader = DataLoader(ds, batch_size=batch_size, num_workers=4, shuffle=train) 101 | return _loader 102 | 103 | 104 | def plot_metric(metrics, config, title=None): 105 | df = pd.DataFrame(metrics, columns=['Accuracy', 'Step', 'Method']) 106 | 107 | filename = f"ht_{config['name']}_{config['start_size']}_{config['step_size']}" 108 | dir = Path(__file__).parent.absolute() / 'data' / 'al' 109 | df.to_csv(dir / (filename + '.csv')) 110 | 111 | try: 112 | sns.lineplot('Step', 'Accuracy', hue='Method', data=df) 113 | plt.legend(loc='upper left') 114 | plt.figure(figsize=(8, 6)) 115 | default_title = f"Validation accuracy, start size {config['start_size']}, " 116 | default_title += f"step size {config['step_size']}" 117 | title = title or default_title 118 | plt.title(title) 119 | file = dir / filename 120 | plt.savefig(file) 121 | except Exception as e: 122 | print(e) 123 | 124 | 125 | if __name__ == '__main__': 126 | config = parse_arguments() 127 | results = [] 128 | for i in range(config['repeats']): 129 | print(f"======{i}======") 130 | accuracies = active_train(config, i) 131 | results.extend(accuracies) 132 | 133 | print(results) 134 | 135 | plot_metric(results, config) 136 | 137 | -------------------------------------------------------------------------------- /experiments/deprecated/classification_active_learning_fasetai.py: -------------------------------------------------------------------------------- 1 | import random 2 | 3 | import numpy as np 4 | import matplotlib.pyplot as plt 5 | from sklearn.model_selection import train_test_split 6 | import seaborn as sns 7 | import pandas as pd 8 | 9 | import torch 10 | from torch import nn 11 | 12 | from fastai.vision import (ImageDataBunch, Learner, accuracy) 13 | from fastai.callbacks import EarlyStoppingCallback 14 | 15 | from alpaca.model.cnn import AnotherConv, SimpleConv 16 | from alpaca.model.resnet import resnet_masked 17 | from alpaca.active_learning.simple_update import update_set 18 | from deprecated.utils import ImageArrayDS 19 | from visual_datasets import prepare_mnist, prepare_cifar, prepare_svhn 20 | 21 | 22 | """ 23 | Active learning experiment for computer vision tasks (MNIST, CIFAR, SVHN) 24 | """ 25 | 26 | 27 | SEED = 43 28 | torch.manual_seed(SEED) 29 | np.random.seed(SEED) 30 | random.seed(SEED) 31 | 32 | if torch.cuda.is_available(): 33 | torch.cuda.set_device(0) 34 | torch.backends.cudnn.deterministic = True 35 | torch.backends.cudnn.benchmark = False 36 | device = 'cuda' 37 | else: 38 | device = 'cpu' 39 | 40 | experiment_label = 'ht' 41 | 42 | 43 | def main(config): 44 | # Load data 45 | x_set, y_set, x_val, y_val, train_tfms = config['prepare_dataset'](config) 46 | 47 | val_accuracy = [] 48 | for _ in range(config['repeats']): # more repЕсли кнопочный, то есть вот прямо бабушкофон. Ещё его можно забрать из пассажа на фрунзе.eats for robust results 49 | # Initial data split 50 | x_set, x_train_init, y_set, y_train_init = train_test_split(x_set, y_set, test_size=config['start_size'], stratify=y_set) 51 | _, x_pool_init, _, y_pool_init = train_test_split(x_set, y_set, test_size=config['pool_size'], stratify=y_set) 52 | 53 | # Active learning 54 | for method in config['methods']: 55 | print(f"== {method} ==") 56 | x_pool, y_pool = np.copy(x_pool_init), np.copy(y_pool_init) 57 | x_train, y_train = np.copy(x_train_init), np.copy(y_train_init) 58 | 59 | model = build_model(config['model_type']) 60 | accuracies = [] 61 | 62 | for i in range(config['steps']): 63 | print(f"Step {i+1}, train size: {len(x_train)}") 64 | 65 | learner = train_classifier(model, config, x_train, y_train, x_val, y_val, train_tfms) 66 | accuracies.append(learner.recorder.metrics[-1][0].item()) 67 | 68 | if i != config['steps'] - 1: 69 | x_pool, x_train, y_pool, y_train = update_set( 70 | x_pool, x_train, y_pool, y_train, config['step_size'], method=method, model=model) 71 | 72 | records = list(zip(accuracies, range(len(accuracies)), [method] * len(accuracies))) 73 | val_accuracy.extend(records) 74 | 75 | # Display results 76 | try: 77 | plot_metric(val_accuracy, config) 78 | except: 79 | import ipdb; ipdb.set_trace() 80 | 81 | 82 | def train_classifier(model, config, x_train, y_train, x_val, y_val, train_tfms=None): 83 | loss_func = torch.nn.CrossEntropyLoss() 84 | 85 | if train_tfms is None: 86 | train_tfms = [] 87 | train_ds = ImageArrayDS(x_train, y_train, train_tfms) 88 | val_ds = ImageArrayDS(x_val, y_val) 89 | data = ImageDataBunch.create(train_ds, val_ds, bs=config['batch_size']) 90 | 91 | callbacks = [partial(EarlyStoppingCallback, min_delta=1e-3, patience=config['patience'])] 92 | learner = Learner(data, model, metrics=accuracy, loss_func=loss_func, callback_fns=callbacks) 93 | learner.fit(config['epochs'], config['start_lr'], wd=config['weight_decay']) 94 | 95 | return learner 96 | 97 | 98 | def plot_metric(metrics, config, title=None): 99 | plt.figure(figsize=(8, 6)) 100 | default_title = f"Validation accuracy, start size {config['start_size']}, " 101 | default_title += f"step size {config['step_size']}, model {config['model_type']}" 102 | title = title or default_title 103 | plt.title(title) 104 | 105 | df = pd.DataFrame(metrics, columns=['Accuracy', 'Step', 'Method']) 106 | sns.lineplot('Step', 'Accuracy', hue='Method', data=df) 107 | # plt.legend(loc='upper left') 108 | 109 | filename = f"{experiment_label}_{config['name']}_{config['model_type']}_{config['start_size']}_{config['step_size']}" 110 | dir = Path(__file__).parent.absolute() / 'data' / 'al' 111 | file = dir / filename 112 | plt.savefig(file) 113 | df.to_csv(dir / (filename + '.csv')) 114 | # plt.show() 115 | 116 | 117 | class Model21(nn.Module): 118 | def __init__(self): 119 | super().__init__() 120 | base = 16 121 | self.conv = nn.Sequential( 122 | nn.Conv2d(3, base, 3, padding=1, bias=False), 123 | nn.BatchNorm2d(base), 124 | nn.CELU(), 125 | nn.Conv2d(base, base, 3, padding=1, bias=False), 126 | nn.CELU(), 127 | nn.MaxPool2d(2, 2), 128 | nn.Dropout2d(0.2), 129 | nn.Conv2d(base, 2*base, 3, padding=1, bias=False), 130 | nn.BatchNorm2d(2*base), 131 | nn.CELU(), 132 | nn.Conv2d(2 * base, 2 * base, 3, padding=1, bias=False), 133 | nn.CELU(), 134 | nn.Dropout2d(0.3), 135 | nn.MaxPool2d(2, 2), 136 | nn.Conv2d(2*base, 4*base, 3, padding=1, bias=False), 137 | nn.BatchNorm2d(4*base), 138 | nn.CELU(), 139 | nn.Conv2d(4*base, 4*base, 3, padding=1, bias=False), 140 | nn.CELU(), 141 | nn.Dropout2d(0.4), 142 | nn.MaxPool2d(2, 2), 143 | ) 144 | self.linear_size = 8 * 8 * base 145 | self.linear = nn.Sequential( 146 | nn.Linear(self.linear_size, 8*base), 147 | nn.CELU(), 148 | ) 149 | self.dropout = nn.Dropout(0.3) 150 | self.fc = nn.Linear(8*base, 10) 151 | 152 | def forward(self, x, dropout_rate=0.5, dropout_mask=None): 153 | x = self.conv(x) 154 | x = x.reshape(-1, self.linear_size) 155 | x = self.linear(x) 156 | if dropout_mask is None: 157 | x = self.dropout(x) 158 | else: 159 | x = x * dropout_mask(x, 0.3, 0) 160 | return self.fc(x) 161 | 162 | 163 | def build_model(model_type): 164 | if model_type == 'conv': 165 | model = AnotherConv() 166 | elif model_type == 'resnet': 167 | model = resnet_masked(pretrained=True) 168 | elif model_type == 'simple_conv': 169 | model = SimpleConv() 170 | elif model_type == 'strong_conv': 171 | print('piu!') 172 | model = Model21() 173 | return model 174 | 175 | 176 | config_cifar = { 177 | 'val_size': 10_000, 178 | 'pool_size': 15_000, 179 | 'start_size': 7_000, 180 | 'step_size': 50, 181 | 'steps': 50, 182 | # 'methods': ['random', 'error_oracle', 'max_entropy', *DEFAULT_MASKS, 'ht_dpp'], 183 | 'methods': ['random', 'error_oracle', 'max_entropy', 'mc_dropout', 'ht_dpp'], 184 | 'epochs': 30, 185 | 'patience': 2, 186 | 'model_type': 'strong_conv', 187 | 'repeats': 3, 188 | 'nn_runs': 100, 189 | 'batch_size': 128, 190 | 'start_lr': 5e-4, 191 | 'weight_decay': 0.0002, 192 | 'prepare_dataset': prepare_cifar, 193 | 'name': 'cifar' 194 | } 195 | 196 | config_svhn = deepcopy(config_cifar) 197 | config_svhn.update({ 198 | 'prepare_dataset': prepare_svhn, 199 | 'name': 'svhn', 200 | 'model_type': 'strong_conv', 201 | 'repeats': 3, 202 | 'epochs': 30, 203 | 'methods': ['random', 'error_oracle', 'max_entropy', 'mc_dropout', 'ht_dpp'] 204 | }) 205 | 206 | config_mnist = deepcopy(config_cifar) 207 | config_mnist.update({ 208 | 'start_size': 100, 209 | 'step_size': 20, 210 | 'model_type': 'simple_conv', 211 | 'methods': ['ht_dpp'], 212 | 'prepare_dataset': prepare_mnist, 213 | 'batch_size': 32, 214 | 'name': 'mnist', 215 | 'steps': 50 216 | }) 217 | 218 | # configs = [config_mnist, config_cifar, config_svhn] 219 | 220 | configs = [config_mnist] 221 | 222 | if __name__ == '__main__': 223 | for config in configs: 224 | print(config) 225 | main(config) -------------------------------------------------------------------------------- /experiments/deprecated/classification_error_detection.py: -------------------------------------------------------------------------------- 1 | from argparse import ArgumentParser 2 | from collections import OrderedDict, defaultdict 3 | from copy import deepcopy 4 | from pathlib import Path 5 | 6 | import numpy as np 7 | import torch 8 | from torch.utils.data import TensorDataset, DataLoader 9 | from sklearn.model_selection import train_test_split 10 | from sklearn.preprocessing import StandardScaler 11 | from sklearn.metrics import roc_curve, roc_auc_score 12 | from scipy.special import softmax 13 | import matplotlib.pyplot as plt 14 | import seaborn as sns 15 | import pandas as pd 16 | from catalyst.dl import SupervisedRunner 17 | from catalyst.dl.callbacks import AccuracyCallback, EarlyStoppingCallback 18 | from catalyst.utils import set_global_seed 19 | 20 | from alpaca.active_learning.simple_update import entropy 21 | from alpaca.uncertainty_estimator import build_estimator 22 | 23 | from configs import base_config, experiment_config 24 | 25 | 26 | def parse_arguments(): 27 | parser = ArgumentParser() 28 | parser.add_argument('name') 29 | name = parser.parse_args().name 30 | 31 | config = deepcopy(base_config) 32 | config.update(experiment_config[name]) 33 | config['name'] = name 34 | 35 | return config 36 | 37 | 38 | def train(config, loaders, logdir, checkpoint=None): 39 | model = config['model_class']().double() 40 | 41 | if checkpoint is not None: 42 | model.load_state_dict(torch.load(checkpoint)['model_state_dict']) 43 | model.eval() 44 | else: 45 | criterion = torch.nn.CrossEntropyLoss() 46 | optimizer = torch.optim.Adam(model.parameters()) 47 | callbacks = [AccuracyCallback(num_classes=10), EarlyStoppingCallback(config['patience'])] 48 | 49 | runner = SupervisedRunner() 50 | runner.train( 51 | model, criterion, optimizer, loaders, 52 | logdir=logdir, num_epochs=config['epochs'], verbose=False, 53 | callbacks=callbacks 54 | ) 55 | 56 | return model 57 | 58 | 59 | def bench_error_detection(model, estimators, loaders, x_val, y_val): 60 | runner = SupervisedRunner() 61 | logits = runner.predict_loader(model, loaders['valid']) 62 | probabilities = softmax(logits, axis=-1) 63 | 64 | estimators = [ 65 | # 'max_entropy', 'max_prob', 66 | 'mc_dropout', 'decorrelating_sc', 67 | 'dpp', 'ht_dpp', 'k_dpp', 'ht_k_dpp'] 68 | 69 | uncertainties = {} 70 | for estimator_name in estimators: 71 | print(estimator_name) 72 | ue = calc_ue(model, x_val, probabilities, estimator_name, nn_runs=150) 73 | uncertainties[estimator_name] = ue 74 | 75 | predictions = np.argmax(probabilities, axis=-1) 76 | errors = (predictions != y_val).astype(np.int) 77 | 78 | results = [] 79 | for estimator_name in estimators: 80 | fpr, tpr, _ = roc_curve(errors, uncertainties[estimator_name]) 81 | roc_auc = roc_auc_score(errors, uncertainties[estimator_name]) 82 | results.append((estimator_name, roc_auc)) 83 | 84 | return results 85 | 86 | 87 | def get_data(config): 88 | x_train, y_train, x_val, y_val, train_tfms = config['prepare_dataset'](config) 89 | 90 | if len(x_train) > config['train_size']: 91 | x_train, _, y_train, _ = train_test_split( 92 | x_train, y_train, train_size=config['train_size'], stratify=y_train 93 | ) 94 | 95 | loaders = OrderedDict({ 96 | 'train': loader(x_train, y_train, config['batch_size'], shuffle=True), 97 | 'valid': loader(x_val, y_val, config['batch_size']) 98 | }) 99 | return loaders, x_train, y_train, x_val, y_val 100 | 101 | 102 | def calc_ue(model, datapoints, probabilities, estimator_type='max_prob', nn_runs=150): 103 | if estimator_type == 'max_prob': 104 | ue = 1 - probabilities[np.arange(len(probabilities)), np.argmax(probabilities, axis=-1)] 105 | elif estimator_type == 'max_entropy': 106 | ue = entropy(probabilities) 107 | else: 108 | estimator = build_estimator( 109 | 'bald_masked', model, dropout_mask=estimator_type, num_classes=10, 110 | nn_runs=nn_runs, keep_runs=True, acquisition='var_ratio') 111 | ue = estimator.estimate(torch.DoubleTensor(datapoints).cuda()) 112 | return ue 113 | 114 | 115 | # Set initial datas 116 | def loader(x, y, batch_size=128, shuffle=False): 117 | ds = TensorDataset(torch.DoubleTensor(x), torch.LongTensor(y)) 118 | _loader = DataLoader(ds, batch_size=batch_size, num_workers=4, shuffle=shuffle) 119 | return _loader 120 | 121 | 122 | if __name__ == '__main__': 123 | config = parse_arguments() 124 | print(config) 125 | loaders, x_train, y_train, x_val, y_val = get_data(config) 126 | 127 | rocaucs = [] 128 | for i in range(config['repeats']): 129 | set_global_seed(i + 42) 130 | logdir = f"logs/ht/{config['name']}_{i}" 131 | model = train(config, loaders, logdir) 132 | rocaucs.extend(bench_error_detection(model, config, loaders, x_val, y_val)) 133 | 134 | df = pd.DataFrame(rocaucs, columns=['Estimator', 'ROC-AUCs']) 135 | plt.figure(figsize=(9, 6)) 136 | plt.title(f"Error detection for {config['name']}") 137 | sns.boxplot('Estimator', 'ROC-AUCs', data=df) 138 | plt.savefig(f"data/ed/{config['name']}.png", dpi=150) 139 | plt.show() 140 | 141 | df.to_csv(f"logs/{config['name']}_ed.csv") 142 | -------------------------------------------------------------------------------- /experiments/deprecated/classification_error_detection_fastai.py: -------------------------------------------------------------------------------- 1 | from copy import deepcopy 2 | import sys 3 | from functools import partial 4 | import random 5 | 6 | import numpy as np 7 | import matplotlib.pyplot as plt 8 | from sklearn.model_selection import train_test_split 9 | 10 | import torch 11 | import torch.nn.functional as F 12 | from fastai.vision import (ImageDataBunch, Learner, accuracy) 13 | from fastai.callbacks import EarlyStoppingCallback 14 | 15 | from alpaca.uncertainty_estimator import build_estimator 16 | from alpaca.active_learning.simple_update import entropy 17 | 18 | from deprecated.utils import ImageArrayDS 19 | from deprecated.classification_active_learning import build_model 20 | from visual_datasets import prepare_cifar, prepare_mnist, prepare_svhn 21 | 22 | 23 | """ 24 | Experiment to detect errors by uncertainty estimation quantification 25 | It provided on MNIST, CIFAR and SVHN datasets (see config below) 26 | We report results as a boxplot ROC-AUC figure on multiple runs 27 | """ 28 | 29 | label = 'ue_accuracy' 30 | 31 | SEED = 42 32 | torch.manual_seed(SEED) 33 | np.random.seed(SEED) 34 | random.seed(SEED) 35 | 36 | if torch.cuda.is_available(): 37 | torch.cuda.set_device(1) 38 | torch.backends.cudnn.deterministic = True 39 | torch.backends.cudnn.benchmark = False 40 | device = 'cuda' 41 | else: 42 | device = 'cpu' 43 | 44 | 45 | def accuracy_curve(mistake, ue): 46 | accuracy_ = lambda x: 1 - sum(x) / len(x) 47 | thresholds = np.arange(0.1, 1.1, 0.1) 48 | accuracy_by_ue = [accuracy_(mistake[ue < t]) for t in thresholds] 49 | return thresholds, accuracy_by_ue 50 | 51 | 52 | def benchmark_uncertainty(config): 53 | results = [] 54 | plt.figure(figsize=(10, 8)) 55 | for i in range(config['repeats']): 56 | # Load data 57 | x_set, y_set, x_val, y_val, train_tfms = config['prepare_dataset'](config) 58 | 59 | if len(x_set) > config['train_size']: 60 | _, x_train, _, y_train = train_test_split( 61 | x_set, y_set, test_size=config['train_size'], stratify=y_set) 62 | else: 63 | x_train, y_train = x_set, y_set 64 | 65 | train_ds = ImageArrayDS(x_train, y_train, train_tfms) 66 | val_ds = ImageArrayDS(x_val, y_val) 67 | data = ImageDataBunch.create(train_ds, val_ds, bs=config['batch_size']) 68 | 69 | # Train model 70 | loss_func = torch.nn.CrossEntropyLoss() 71 | np.set_printoptions(threshold=sys.maxsize, suppress=True) 72 | 73 | model = build_model(config['model_type']) 74 | callbacks = [partial(EarlyStoppingCallback, min_delta=1e-3, patience=config['patience'])] 75 | learner = Learner(data, model, metrics=accuracy, loss_func=loss_func, callback_fns=callbacks) 76 | learner.fit(config['epochs'], config['start_lr'], wd=config['weight_decay']) 77 | 78 | # Evaluate different estimators 79 | images = torch.FloatTensor(x_val).to(device) 80 | logits = model(images) 81 | probabilities = F.softmax(logits, dim=1).detach().cpu().numpy() 82 | predictions = np.argmax(probabilities, axis=-1) 83 | print('Logits average', np.average(logits)) 84 | 85 | for estimator_name in config['estimators']: 86 | print(estimator_name) 87 | ue = calc_ue(model, images, probabilities, estimator_name, config['nn_runs']) 88 | mistake = 1 - (predictions == y_val).astype(np.int) 89 | 90 | # roc_auc = roc_auc_score(mistake, ue) 91 | # print(estimator_name, roc_auc) 92 | # results.append((estimator_name, roc_auc)) 93 | 94 | # ue_thresholds, ue_accuracy = accuracy_curve(mistake, ue) 95 | # results.extend([(t, accu, estimator_name) for t, accu in zip(ue_thresholds, ue_accuracy)]) 96 | # plt.plot(ue_thresholds, ue_accuracy, label=estimator_name, alpha=0.8) 97 | 98 | # if i == config['repeats'] - 1: 99 | # fpr, tpr, thresholds = roc_curve(mistake, ue, pos_label=1) 100 | # plt.plot(fpr, tpr, label=name, alpha=0.8) 101 | # plt.xlabel('FPR') 102 | # plt.ylabel('TPR') 103 | 104 | # if i == config['repeats'] - 1: 105 | # ue_thresholds, ue_accuracy = accuracy_curve(mistake, ue) 106 | # plt.plot(ue_thresholds, ue_accuracy, label=name, alpha=0.8) 107 | # Plot the results and generate figures 108 | # plt.figure(dpi=150) 109 | # dir = Path(__file__).parent.absolute() / 'data' / 'detector' 110 | # # plt.legend() 111 | # df = pd.DataFrame(results, columns=['ue', 'accuracy', 'estimator_name']) 112 | # 113 | # sns.lineplot('ue', 'accuracy', hue='estimator_name', data=df) 114 | # plt.title(f"Uncertainty vs accuracy, {config['name']}") 115 | # plt.xlabel('Uncertaint thresholds, t') 116 | # plt.ylabel('Accuracy for points where ue(t) < t ') 117 | # file = f"{label}_{config['name']}_{config['train_size']}_{config['nn_runs']}" 118 | # plt.savefig(dir / file, dpi=150) 119 | # plt.show() 120 | 121 | # # Plot the results and generate figures 122 | # dir = Path(__file__).parent.absolute() / 'data' / 'detector' 123 | # plt.title(f"{config['name']} uncertainty ROC") 124 | # plt.legend() 125 | # file = f"var_{label}_roc_{config['name']}_{config['train_size']}_{config['nn_runs']}" 126 | # plt.savefig(dir / file) 127 | # # plt.show() 128 | # 129 | # df = pd.DataFrame(results, columns=['Estimator type', 'ROC-AUC score']) 130 | # df = df.replace('mc_dropout', 'MC dropout') 131 | # df = df.replace('decorrelating_sc', 'decorrelation') 132 | # df = df[df['Estimator type'] != 'k_dpp_noisereg'] 133 | # print(df) 134 | # fig, ax = plt.subplots(figsize=(8, 6)) 135 | # plt.subplots_adjust(left=0.2) 136 | # 137 | # with sns.axes_style('whitegrid'): 138 | # sns.boxplot(data=df, y='ROC-AUC score', x='Estimator type', ax=ax) 139 | # 140 | # ax.yaxis.grid(True) 141 | # ax.xaxis.grid(True) 142 | # 143 | # plt.title(f'{config["name"]} wrong prediction ROC-AUC') 144 | 145 | # file = f"var_{label}_boxplot_{config['name']}_{config['train_size']}_{config['nn_runs']}" 146 | # plt.savefig(dir / file) 147 | # df.to_csv(dir / (file + '.csv')) 148 | 149 | 150 | def calc_ue(model, images, probabilities, estimator_type='max_prob', nn_runs=100): 151 | if estimator_type == 'max_prob': 152 | ue = 1 - probabilities[np.arange(len(probabilities)), np.argmax(probabilities, axis=-1)] 153 | elif estimator_type == 'max_entropy': 154 | ue = entropy(probabilities) 155 | else: 156 | estimator = build_estimator( 157 | 'bald_masked', model, dropout_mask=estimator_type, num_classes=10, 158 | nn_runs=nn_runs, keep_runs=True) 159 | # try: 160 | ue = estimator.estimate(images) 161 | # except: 162 | # import ipdb; ipdb.set_trace() 163 | 164 | print(np.average(estimator.last_mcd_runs())) 165 | 166 | # import ipdb; ipdb.set_trace() 167 | # print(ue[:20]) 168 | return ue 169 | 170 | 171 | config_mnist = { 172 | 'train_size': 3_000, 173 | 'val_size': 10_000, 174 | 'model_type': 'simple_conv', 175 | 'batch_size': 256, 176 | 'patience': 3, 177 | 'epochs': 100, 178 | 'start_lr': 1e-3, 179 | 'weight_decay': 0.1, 180 | 'reload': False, 181 | 'nn_runs': 150, 182 | 'estimators': ['mc_dropout', 'decorrelating_sc', 'dpp', 'k_dpp', 'ht_dpp', 'ht_k_dpp'], 183 | 'repeats': 5, 184 | 'name': 'MNIST', 185 | 'prepare_dataset': prepare_mnist, 186 | } 187 | # config_mnist.update({ 188 | # 'epochs': 20, 189 | # # 'estimators': ['decorrelating_sc', 'mc_dropout'], 190 | # 'repeats': 5, 191 | # 'train_size': 3000 192 | # }) 193 | 194 | config_cifar = deepcopy(config_mnist) 195 | config_cifar.update({ 196 | 'train_size': 60_000, 197 | 'val_size': 10_000, 198 | 'model_type': 'resnet', 199 | 'name': 'CIFAR-10', 200 | 'prepare_dataset': prepare_cifar 201 | }) 202 | 203 | config_svhn = deepcopy(config_mnist) 204 | config_svhn.update({ 205 | 'train_size': 60_000, 206 | 'val_size': 10_000, 207 | 'model_type': 'resnet', 208 | 'name': 'SVHN', 209 | 'prepare_dataset': prepare_svhn 210 | }) 211 | 212 | # configs = [config_mnist, config_cifar, config_svhn] 213 | configs = [config_mnist] 214 | 215 | 216 | # # TODO: for debug, remove 217 | # config_mnist.update({ 218 | # 'epochs': 20, 219 | # # 'estimators': ['decorrelating_sc', 'mc_dropout'], 220 | # 'repeats': 1, 221 | # 'train_size': 3000 222 | # }) 223 | 224 | 225 | if __name__ == '__main__': 226 | for config in configs: 227 | print(config) 228 | benchmark_uncertainty(config) 229 | -------------------------------------------------------------------------------- /experiments/deprecated/classification_image.py: -------------------------------------------------------------------------------- 1 | from functools import partial 2 | 3 | import matplotlib.pyplot as plt 4 | import numpy as np 5 | import pandas as pd 6 | import seaborn as sns 7 | 8 | import torch 9 | from fastai.vision import rand_pad, flip_lr, ImageDataBunch, Learner, accuracy 10 | from fastai.callbacks import EarlyStoppingCallback 11 | 12 | 13 | from dataloader.builder import build_dataset 14 | from deprecated.utils import ImageArrayDS, Inferencer 15 | from model.cnn import AnotherConv 16 | from model.resnet import resnet_linear, resnet_masked 17 | from experiments.experiment_setup import build_estimator 18 | from uncertainty_estimator.masks import build_masks, DEFAULT_MASKS 19 | from analysis.metrics import uq_ndcg 20 | 21 | 22 | torch.backends.cudnn.benchmark = True 23 | 24 | val_size = 30_000 25 | lr = 1e-3 26 | weight_decay = 1e-3 27 | 28 | config = { 29 | 'model_runs': 3, 30 | 'repeat_runs': 3, 31 | 'nn_runs': 150, 32 | 'dropout_uq': 0.5, 33 | 'num_classes': 10 34 | } 35 | 36 | 37 | def ll(trainer, x, y): 38 | trainer.eval() 39 | logits = trainer(x).detach().cpu() 40 | probs = torch.softmax(logits, axis=-1).numpy()[np.arange(len(x)), y] 41 | return np.log(probs) 42 | 43 | 44 | def main(): 45 | data, x_train, y_train, x_val, y_val = load_data() 46 | loss_func = torch.nn.CrossEntropyLoss() 47 | 48 | models = { 49 | 'cnn': AnotherConv(), 50 | 'resnet': resnet_masked(pretrained=True), 51 | 'resnet_multiple': resnet_linear(pretrained=True) 52 | } 53 | 54 | estimation_samples = 5_000 55 | ndcgs, estimator_type, model_types = [], [], [] 56 | accuracies = [] 57 | 58 | for i in range(config['model_runs']): 59 | print('==models run==', i+1) 60 | for name, model in models.items(): 61 | callbacks = [partial(EarlyStoppingCallback, patience=3, min_delta=1e-2, monitor='valid_loss')] 62 | learner = Learner(data, model, loss_func=loss_func, metrics=[accuracy], callback_fns=callbacks) 63 | learner.fit(100, lr, wd=weight_decay) 64 | inferencer = Inferencer(model) 65 | masks = build_masks(DEFAULT_MASKS) 66 | 67 | for j in range(config['repeat_runs']): 68 | idxs = np.random.choice(len(x_val), estimation_samples, replace=False) 69 | x_current = x_val[idxs] 70 | y_current = y_val[idxs] 71 | 72 | # masks 73 | current_ll = ll(inferencer, x_current, y_current) 74 | for mask_name, mask in masks.items(): 75 | print(mask_name) 76 | estimator = build_estimator( 77 | 'bald_masked', inferencer, nn_runs=config['nn_runs'], dropout_mask=mask, 78 | dropout_rate=config['dropout_uq'], num_classes=config['num_classes']) 79 | uq = estimator.estimate(x_current) 80 | estimator.reset() 81 | ndcgs.append(uq_ndcg(-current_ll, uq)) 82 | estimator_type.append(mask_name) 83 | estimator.reset() 84 | model_types.append(name) 85 | accuracies.append(learner.recorder.metrics[-1][0].item()) 86 | try: 87 | plt.figure(figsize=(12, 8)) 88 | plt.title(f"NDCG on different train samples") 89 | 90 | df = pd.DataFrame({ 91 | 'ndcg': ndcgs, 92 | 'estimator_type': estimator_type, 93 | 'model': model_types 94 | }) 95 | sns.boxplot(data=df, x='estimator_type', y='ndcg', hue='model') 96 | plt.show() 97 | 98 | plt.figure(figsize=(12, 8)) 99 | plt.title('Accuracies') 100 | df = pd.DataFrame({ 101 | 'accuracy': accuracies, 102 | 'model': model_types 103 | }) 104 | sns.boxplot(data=df, y='accuracy', x='model') 105 | plt.show() 106 | except Exception as e: 107 | print(e) 108 | import ipdb; ipdb.set_trace() 109 | 110 | 111 | def load_data(): 112 | dataset = build_dataset('cifar_10', val_size=val_size) 113 | x_train, y_train = dataset.dataset('train') 114 | x_val, y_val = dataset.dataset('val') 115 | 116 | shape = (-1, 3, 32, 32) 117 | x_train = ((x_train - 128)/128).reshape(shape) 118 | x_val = ((x_val - 128)/128).reshape(shape) 119 | 120 | train_tfms = [*rand_pad(4, 32), flip_lr(p=0.5)] 121 | train_ds = ImageArrayDS(x_train, y_train, train_tfms) 122 | val_ds = ImageArrayDS(x_val, y_val) 123 | data = ImageDataBunch.create(train_ds, val_ds, bs=256) 124 | return data, x_train, y_train, x_val, y_val 125 | 126 | 127 | if __name__ == '__main__': 128 | main() 129 | 130 | -------------------------------------------------------------------------------- /experiments/deprecated/classification_ood_detection.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import random 3 | from copy import deepcopy 4 | from functools import partial 5 | from pathlib import Path 6 | 7 | import numpy as np 8 | import matplotlib.pyplot as plt 9 | import seaborn as sns 10 | import pandas as pd 11 | from sklearn.model_selection import train_test_split 12 | from sklearn.metrics import roc_auc_score, roc_curve 13 | 14 | import torch 15 | import torch.nn.functional as F 16 | 17 | from fastai.vision import (ImageDataBunch, Learner, accuracy) 18 | from fastai.callbacks import EarlyStoppingCallback 19 | 20 | from alpaca.uncertainty_estimator.masks import DEFAULT_MASKS 21 | from alpaca.uncertainty_estimator import build_estimator 22 | from alpaca.active_learning.simple_update import entropy 23 | 24 | from deprecated.utils import ImageArrayDS 25 | from visual_datasets import prepare_cifar, prepare_mnist, prepare_svhn, prepare_fashion_mnist 26 | from deprecated.classification_active_learning import build_model 27 | 28 | 29 | """ 30 | Experiment to detect out-of-distribution samples (OOD) by uncertainty estimation quantification 31 | Results are provided on MNIST/Fashion-MNIST and CIFAR/SVHN pairs (see config below) 32 | We report results as a boxplot ROC-AUC figure for multiple runs 33 | """ 34 | 35 | label = 'ratio' 36 | 37 | SEED = 42 38 | torch.manual_seed(SEED) 39 | np.random.seed(SEED) 40 | random.seed(SEED) 41 | 42 | if torch.cuda.is_available(): 43 | torch.cuda.set_device(0) 44 | torch.backends.cudnn.deterministic = True 45 | torch.backends.cudnn.benchmark = False 46 | device = 'cuda' 47 | else: 48 | device = 'cpu' 49 | 50 | 51 | def benchmark_ood(config): 52 | results = [] 53 | plt.figure(figsize=(10, 8)) 54 | for i in range(config['repeats']): 55 | # Load data 56 | x_set, y_set, x_val, y_val, train_tfms = config['prepare_dataset'](config) 57 | 58 | if len(x_set) > config['train_size']: 59 | _, x_train, _, y_train = train_test_split( 60 | x_set, y_set, test_size=config['train_size'], stratify=y_set) 61 | else: 62 | x_train, y_train = x_set, y_set 63 | 64 | _, _, x_alt, y_alt, _ = config['alternative_dataset'](config) 65 | 66 | train_ds = ImageArrayDS(x_train, y_train, train_tfms) 67 | val_ds = ImageArrayDS(x_val, y_val) 68 | data = ImageDataBunch.create(train_ds, val_ds, bs=config['batch_size']) 69 | 70 | # Train model 71 | loss_func = torch.nn.CrossEntropyLoss() 72 | np.set_printoptions(threshold=sys.maxsize, suppress=True) 73 | 74 | model = build_model(config['model_type']) 75 | callbacks = [partial(EarlyStoppingCallback, min_delta=1e-3, patience=config['patience'])] 76 | learner = Learner(data, model, metrics=accuracy, loss_func=loss_func, callback_fns=callbacks) 77 | learner.fit(config['epochs'], config['start_lr'], wd=config['weight_decay']) 78 | 79 | # Get data for binary classification of OOD detector 80 | original_images = torch.FloatTensor(x_val) 81 | alt_images = torch.FloatTensor(x_alt) 82 | images = torch.cat((original_images, alt_images)).to(device) 83 | y = np.array([0]*len(original_images) + [1]*len(alt_images)) 84 | 85 | probabilities = F.softmax(model(images), dim=1).detach().cpu().numpy() 86 | 87 | # Calculate uncertainty and generate ROC data for OOD detection 88 | for name in config['estimators']: 89 | ue = calc_ue(model, images, probabilities, name, config['nn_runs']) 90 | 91 | roc_auc = roc_auc_score(y, ue) 92 | print(name, roc_auc) 93 | results.append((name, roc_auc)) 94 | 95 | if i == config['repeats'] - 1: 96 | fpr, tpr, thresholds = roc_curve(y, ue, pos_label=1) 97 | plt.plot(fpr, tpr, label=name, alpha=0.8) 98 | plt.xlabel('FPR') 99 | plt.ylabel('TPR') 100 | 101 | # Plot the results and save figures 102 | dir = Path(__file__).parent.absolute() / 'data' / 'ood' 103 | plt.title(f"{config['name']} ood detection ROC") 104 | plt.legend() 105 | plt.savefig(dir / f"var_{label}_ood_roc_{config['name']}_{config['train_size']}_{config['nn_runs']}") 106 | # plt.show() 107 | 108 | plt.figure(figsize=(10, 8)) 109 | plt.title(f"{config['name']} ood detection ROC-AUC") 110 | df = pd.DataFrame(results, columns=['Estimator type', 'ROC-AUC score']) 111 | sns.boxplot('Estimator type', 'ROC-AUC score', data=df) 112 | plt.savefig(dir / f"var_{label}_ood_boxplot_{config['name']}_{config['train_size']}_{config['nn_runs']}") 113 | # plt.show() 114 | 115 | 116 | def calc_ue(model, images, probabilities, estimator_type='max_prob', nn_runs=100): 117 | if estimator_type == 'max_prob': 118 | ue = 1 - probabilities[np.arange(len(probabilities)), np.argmax(probabilities, axis=-1)] 119 | elif estimator_type == 'max_entropy': 120 | ue = entropy(probabilities) 121 | else: 122 | estimator = build_estimator( 123 | 'bald_masked', model, dropout_mask=estimator_type, num_classes=10, nn_runs=nn_runs) 124 | ue = estimator.estimate(images) 125 | print(ue[:10]) 126 | 127 | return ue 128 | 129 | 130 | config_mnist = { 131 | 'train_size': 50_000, 132 | 'val_size': 5_000, 133 | 'model_type': 'simple_conv', 134 | 'batch_size': 256, 135 | 'patience': 3, 136 | 'epochs': 50, 137 | 'start_lr': 1e-3, 138 | 'weight_decay': 0.1, 139 | 'reload': False, 140 | 'nn_runs': 150, 141 | 'estimators': DEFAULT_MASKS, 142 | 'repeats': 5, 143 | 'name': 'MNIST', 144 | 'prepare_dataset': prepare_mnist, 145 | 'alternative_dataset': prepare_fashion_mnist, 146 | } 147 | 148 | 149 | # # TODO: for debug, remove 150 | # config_mnist.update({ 151 | # 'epochs': 2, 152 | # 'estimators': ['decorrelating_sc', 'mc_dropout'], 153 | # 'repeats': 1 154 | # }) 155 | 156 | config_cifar = deepcopy(config_mnist) 157 | config_cifar.update({ 158 | 'train_size': 50_000, 159 | 'val_size': 5_000, 160 | 'model_type': 'resnet', 161 | 'name': 'CIFAR-10', 162 | 'prepare_dataset': prepare_cifar, 163 | 'alternative_dataset': prepare_svhn, 164 | }) 165 | 166 | 167 | configs = [config_mnist, config_cifar] 168 | 169 | 170 | if __name__ == '__main__': 171 | for config in configs: 172 | print(config) 173 | benchmark_ood(config) 174 | -------------------------------------------------------------------------------- /experiments/deprecated/classification_xor.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true, 8 | "pycharm": { 9 | "is_executing": false 10 | } 11 | }, 12 | "outputs": [], 13 | "source": [ 14 | "import torch\n", 15 | "from torch.utils.data import TensorDataset\n", 16 | "import numpy as np\n", 17 | "import matplotlib.pyplot as plt\n", 18 | "\n", 19 | "from fastai.vision import Learner, accuracy\n", 20 | "from fastai.basic_data import DataBunch\n", 21 | "\n", 22 | "from alpaca.model.dense import Dense\n", 23 | "from alpaca.uncertainty_estimator import build_estimator\n", 24 | "from alpaca.uncertainty_estimator.masks import DEFAULT_MASKS\n" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": null, 30 | "outputs": [], 31 | "source": [ 32 | "border = 1.2\n", 33 | "dots = 300\n", 34 | "xx, yy = np.meshgrid(np.linspace(-border, border, dots), np.linspace(-border, border, dots))\n", 35 | "x_mesh = np.vstack((xx.ravel(), yy.ravel())).T\n", 36 | "epochs = 10 \n", 37 | "layers = (2, 100, 100, 100, 2)\n" 38 | ], 39 | "metadata": { 40 | "collapsed": false, 41 | "pycharm": { 42 | "name": "#%%\n", 43 | "is_executing": false 44 | } 45 | } 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "outputs": [], 51 | "source": [ 52 | "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n", 53 | "\n", 54 | "def xor(points, noise_level=0.):\n", 55 | " rng = np.random\n", 56 | " x = 2*rng.random((points, 2)) - 1\n", 57 | " noised_x = x + noise_level * (2*rng.random((points, 2)) - 1)\n", 58 | " y = np.logical_xor(noised_x[:, 0] > 0, noised_x[:, 1] > 0)\n", 59 | " return x, y\n", 60 | "\n", 61 | "\n", 62 | "def train(x, y, x_val, y_val):\n", 63 | " loss_func = torch.nn.CrossEntropyLoss()\n", 64 | " model = Dense(layers, dropout_rate=0.5)\n", 65 | "\n", 66 | " train_ds = TensorDataset(torch.FloatTensor(x), torch.LongTensor(y))\n", 67 | " val_ds = TensorDataset(torch.FloatTensor(x_val), torch.LongTensor(y_val))\n", 68 | " data = DataBunch.create(train_ds, val_ds, bs=10)\n", 69 | "\n", 70 | " learner = Learner(data, model, metrics=accuracy, loss_func=loss_func)\n", 71 | "\n", 72 | " learner.fit_one_cycle(epochs)\n", 73 | " return model\n", 74 | "\n", 75 | "\n", 76 | "def eval(model, x, y, method='basic_bern', noise_level=0):\n", 77 | " t_mesh = torch.FloatTensor(x_mesh).to(device)\n", 78 | " estimator = build_estimator('bald_masked', model, dropout_mask=method, num_classes=2, dropout_rate=0.5, keep_runs=True)\n", 79 | " # estimations = sigmoid(estimator.estimate(t_val))\n", 80 | " estimations = model(t_mesh)[:, 0].detach()\n", 81 | " plot(x, y, estimations, noise_level, 'prediction')\n", 82 | " estimations = estimator.estimate(t_mesh)\n", 83 | " estimations = estimations\n", 84 | " plot(x, y, estimations, noise_level, method)\n", 85 | " \n", 86 | " \n", 87 | "def plot(x, y, estimations, noise_level, method):\n", 88 | " Z = estimations.reshape(xx.shape)\n", 89 | " plt.figure(figsize=(12, 9))\n", 90 | " plt.scatter(x[:, 0], x[:, 1], s=50, c=y, cmap=plt.cm.gray, edgecolors=(0, 0, 0))\n", 91 | " plt.imshow(Z, interpolation='nearest',\n", 92 | " extent=(xx.min(), xx.max(), yy.min(), yy.max()),\n", 93 | " aspect='auto', origin='lower', cmap=plt.cm.RdBu_r)\n", 94 | " plt.title(f\"{method} with noise {noise_level}\")\n", 95 | " plt.colorbar() \n", 96 | " plt.savefig(f\"data/xor/{method}_{noise_level}.png\")\n", 97 | " plt.show()\n" 98 | ], 99 | "metadata": { 100 | "collapsed": false, 101 | "pycharm": { 102 | "name": "#%%\n", 103 | "is_executing": false 104 | } 105 | } 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "source": [ 110 | "### Plot for all masks\n" 111 | ], 112 | "metadata": { 113 | "collapsed": false, 114 | "pycharm": { 115 | "name": "#%% md\n", 116 | "is_executing": true 117 | } 118 | } 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "outputs": [], 124 | "source": [ 125 | "for noise_level in np.arange(0, 1, 0.1):\n", 126 | " noise_level = round(noise_level, 1)\n", 127 | " print(noise_level)\n", 128 | " x, y = xor(500, noise_level)\n", 129 | " x_val, y_val = xor(500, noise_level)\n", 130 | " model = train(x, y, x_val, y_val)\n", 131 | " for mask_name in DEFAULT_MASKS:\n", 132 | " eval(model, x, y, mask_name, noise_level)" 133 | ], 134 | "metadata": { 135 | "collapsed": false, 136 | "pycharm": { 137 | "name": "#%%\n", 138 | "is_executing": false 139 | } 140 | } 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": null, 145 | "outputs": [], 146 | "source": [ 147 | "\n" 148 | ], 149 | "metadata": { 150 | "collapsed": false, 151 | "pycharm": { 152 | "name": "#%%\n" 153 | } 154 | } 155 | } 156 | ], 157 | "metadata": { 158 | "kernelspec": { 159 | "display_name": "Python 3", 160 | "language": "python", 161 | "name": "python3" 162 | }, 163 | "language_info": { 164 | "codemirror_mode": { 165 | "name": "ipython", 166 | "version": 2 167 | }, 168 | "file_extension": ".py", 169 | "mimetype": "text/x-python", 170 | "name": "python", 171 | "nbconvert_exporter": "python", 172 | "pygments_lexer": "ipython2", 173 | "version": "2.7.6" 174 | }, 175 | "pycharm": { 176 | "stem_cell": { 177 | "cell_type": "raw", 178 | "source": [], 179 | "metadata": { 180 | "collapsed": false 181 | } 182 | } 183 | } 184 | }, 185 | "nbformat": 4, 186 | "nbformat_minor": 0 187 | } -------------------------------------------------------------------------------- /experiments/deprecated/dpp.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true, 8 | "pycharm": { 9 | "is_executing": false 10 | } 11 | }, 12 | "outputs": [], 13 | "source": [ 14 | "import numpy as np\n", 15 | "from dppy.finite_dpps import FiniteDPP" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 2, 21 | "outputs": [], 22 | "source": [ 23 | "correlations = np.eye(5)\n", 24 | "dpp = FiniteDPP('correlation', **{'K': correlations})\n", 25 | "\n" 26 | ], 27 | "metadata": { 28 | "collapsed": false, 29 | "pycharm": { 30 | "name": "#%%\n", 31 | "is_executing": false 32 | } 33 | } 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 3, 38 | "outputs": [ 39 | { 40 | "data": { 41 | "text/plain": "[array([0, 3, 2, 1, 4])]" 42 | }, 43 | "metadata": {}, 44 | "output_type": "execute_result", 45 | "execution_count": 3 46 | } 47 | ], 48 | "source": [ 49 | "dpp.sample_exact()\n", 50 | "dpp.list_of_samples" 51 | ], 52 | "metadata": { 53 | "collapsed": false, 54 | "pycharm": { 55 | "name": "#%%\n", 56 | "is_executing": false 57 | } 58 | } 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 3, 63 | "outputs": [], 64 | "source": [ 65 | "\n" 66 | ], 67 | "metadata": { 68 | "collapsed": false, 69 | "pycharm": { 70 | "name": "#%%\n", 71 | "is_executing": false 72 | } 73 | } 74 | } 75 | ], 76 | "metadata": { 77 | "kernelspec": { 78 | "name": "python3", 79 | "language": "python", 80 | "display_name": "Python 3" 81 | }, 82 | "language_info": { 83 | "codemirror_mode": { 84 | "name": "ipython", 85 | "version": 2 86 | }, 87 | "file_extension": ".py", 88 | "mimetype": "text/x-python", 89 | "name": "python", 90 | "nbconvert_exporter": "python", 91 | "pygments_lexer": "ipython2", 92 | "version": "2.7.6" 93 | }, 94 | "pycharm": { 95 | "stem_cell": { 96 | "cell_type": "raw", 97 | "source": [], 98 | "metadata": { 99 | "collapsed": false 100 | } 101 | } 102 | } 103 | }, 104 | "nbformat": 4, 105 | "nbformat_minor": 0 106 | } -------------------------------------------------------------------------------- /experiments/deprecated/ensemble_debug.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pickle 3 | import random 4 | from pathlib import Path 5 | 6 | import seaborn as sns 7 | import pandas as pd 8 | import torch 9 | from torch.nn.functional import elu 10 | import numpy as np 11 | import matplotlib.pyplot as plt 12 | 13 | from alpaca.uncertainty_estimator.masks import build_masks, DEFAULT_MASKS 14 | from alpaca.analysis.metrics import uq_ll 15 | from alpaca.model.ensemble import MLPEnsemble 16 | from alpaca.uncertainty_estimator import build_estimator 17 | from alpaca.analysis.metrics import get_uq_metrics 18 | 19 | plt.rcParams['figure.facecolor'] = 'white' 20 | 21 | SEED = 10 22 | torch.manual_seed(SEED) 23 | np.random.seed(SEED) 24 | random.seed(SEED) 25 | 26 | torch.cuda.set_device(0) 27 | torch.backends.cudnn.deterministic = True 28 | torch.backends.cudnn.benchmark = False 29 | 30 | folder = Path('./data/regression') 31 | files = sorted([file for file in os.listdir(folder) if file.endswith('.pickle')]) 32 | files = [file for file in files if file.startswith('bos')] 33 | 34 | 35 | def load_setup(file): 36 | print(file) 37 | with open(folder / 'log_exp.log', 'w') as f: 38 | f.write(f'{cnt} / {len(files)}') 39 | with open(folder / file, 'rb') as f: 40 | dct = pickle.load(f) 41 | config = dct['config'] 42 | config['n_ue_runs'] = 1 43 | config['acc_percentile'] = .1 44 | state_dict = dct['state_dict'] 45 | x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data'] 46 | 47 | ensemble = MLPEnsemble( 48 | config['layers'], n_models=config['n_ens'], activation = elu, 49 | reduction='mean') 50 | ensemble.load_state_dict(state_dict) 51 | 52 | model = ensemble.models[2] 53 | return model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler 54 | 55 | 56 | accumulate_ll = [] 57 | data = [] 58 | 59 | 60 | # np.random.shuffle(files) 61 | for cnt, file in enumerate(files[:1]): 62 | model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler = load_setup(file) 63 | 64 | x_val_tensor = torch.tensor(x_val) 65 | unscale = lambda y : y_scaler.inverse_transform(y) 66 | 67 | predictions = model(x_val_tensor.cuda()).cpu().detach().numpy() 68 | errors = predictions - y_val 69 | # rmse_single = np.sqrt(np.mean(np.square(errors))) 70 | # accumulate.append([file[:4], 'single', rmse_single]) 71 | 72 | # predictions = ensemble(x_val_tensor.cuda()).cpu().detach().numpy() 73 | # errors = predictions - y_val 74 | # rmse_single = np.sqrt(np.mean(np.square(errors))) 75 | # accumulate.append([file[:4], 'ensemble', rmse_single]) 76 | 77 | for mask_name in DEFAULT_MASKS: 78 | estimator = build_estimator( 79 | 'mcdue_masked', model, nn_runs=100, keep_runs=True, 80 | dropout_mask=mask_name) 81 | ue =estimator.estimate(torch.Tensor(x_val).double().cuda()) 82 | runs = estimator.last_mcd_runs() 83 | predictions = np.mean(estimator.last_mcd_runs(), axis=-1) 84 | errors = predictions - y_val[:, 0] 85 | rmse_mask = np.sqrt(np.mean(np.square(errors))) 86 | # accumulate.append([file[:4], mask_name, rmse_mask]) 87 | ll = uq_ll(errors, ue) 88 | accumulate_ll.append([file[:4], mask_name, ll]) 89 | 90 | 91 | plt.figure(figsize=(10, 6)) 92 | 93 | accumulate2 = [record for record in accumulate_ll if record[1] !='decorrelating_sc'] 94 | df = pd.DataFrame(accumulate2, columns=['dataset', 'type', 'LL']) 95 | sns.boxplot('dataset', 'LL', hue='type', data=df) 96 | plt.savefig('ll_masks.png', dpi=150) 97 | 98 | -------------------------------------------------------------------------------- /experiments/deprecated/ensembles.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "outputs": [], 7 | "source": [ 8 | "import sys\n", 9 | "sys.path.append('..')\n", 10 | "%load_ext autoreload\n", 11 | "%autoreload 2\n" 12 | ], 13 | "metadata": { 14 | "collapsed": false, 15 | "pycharm": { 16 | "name": "#%%\n" 17 | } 18 | } 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "outputs": [], 24 | "source": [ 25 | "import os\n", 26 | "from pathlib import Path\n", 27 | "import random\n", 28 | "\n", 29 | "import matplotlib.pyplot as plt\n", 30 | "import pickle\n", 31 | "import torch\n", 32 | "import numpy as np\n", 33 | "\n", 34 | "# from experiment_setup import get_model, set_random, build_estimator\n", 35 | "from alpaca.model.ensemble import MLPEnsemble\n", 36 | "from alpaca.model.mlp import MLP\n", 37 | "from alpaca.dataloader.builder import build_dataset\n", 38 | "from alpaca.uncertainty_estimator import build_estimator\n", 39 | "from alpaca.analysis.metrics import uq_ll\n", 40 | "\n", 41 | "from experiments.utils.data import scale, multiple_kfold\n", 42 | "\n", 43 | "plt.rcParams['figure.facecolor'] = 'white'" 44 | ], 45 | "metadata": { 46 | "collapsed": false, 47 | "pycharm": { 48 | "name": "#%%\n" 49 | } 50 | } 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "outputs": [], 56 | "source": [ 57 | "SEED = 10\n", 58 | "torch.manual_seed(SEED)\n", 59 | "np.random.seed(SEED)\n", 60 | "random.seed(SEED)\n", 61 | "\n", 62 | "torch.cuda.set_device(0)\n", 63 | "torch.backends.cudnn.deterministic = True\n", 64 | "torch.backends.cudnn.benchmark = False\n", 65 | "\n" 66 | ], 67 | "metadata": { 68 | "collapsed": false, 69 | "pycharm": { 70 | "name": "#%%\n" 71 | } 72 | } 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "outputs": [], 78 | "source": [ 79 | "config = {\n", 80 | " 'nn_runs': 100,\n", 81 | " 'runs': 2,\n", 82 | " 'max_runs': 20,\n", 83 | " # Our train config\n", 84 | " 'layers': [8, 128, 128, 64, 1],\n", 85 | " 'epochs': 10_000,\n", 86 | " 'validation_step': 100,\n", 87 | "\n", 88 | " 'patience': 10,\n", 89 | " 'dropout_rate': 0.5,\n", 90 | " 'dropout_uq': 0.5,\n", 91 | " 'batch_size': 256,\n", 92 | " 'l2_reg': 1e-5,\n", 93 | "\n", 94 | " 'optimizer': {'type': 'Adam', 'lr': 0.01, 'weight_decay':1e-5},\n", 95 | " 'n_split_runs': 3,\n", 96 | " 'n_model_runs': 3,\n", 97 | " 'n_ens': 20,\n", 98 | " 'n_ue_runs': 5,\n", 99 | " 'k_folds': 10,\n", 100 | " 'verbose': False,\n", 101 | "}\n" 102 | ], 103 | "metadata": { 104 | "collapsed": false, 105 | "pycharm": { 106 | "name": "#%%\n" 107 | } 108 | } 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "outputs": [], 114 | "source": [ 115 | "train_opts = ['patience', 'dropout_rate', 'epochs', 'batch_size', 'validation_step', 'verbose']\n", 116 | "config['train_opts'] = {k: config[k] for k in config if k in train_opts}\n" 117 | ], 118 | "metadata": { 119 | "collapsed": false, 120 | "pycharm": { 121 | "name": "#%%\n" 122 | } 123 | } 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": null, 128 | "outputs": [], 129 | "source": [ 130 | "# datasets = [\n", 131 | "# 'boston_housing', 'concrete', 'energy_efficiency',\n", 132 | "# 'kin8nm', 'naval_propulsion', 'ccpp', 'red_wine',\n", 133 | "# 'yacht_hydrodynamics'\n", 134 | "# ]" 135 | ], 136 | "metadata": { 137 | "collapsed": false, 138 | "pycharm": { 139 | "name": "#%%\n" 140 | } 141 | } 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "outputs": [], 147 | "source": [ 148 | "# dataset_name = 'kin8nm'\n", 149 | "dataset_name = 'ccpp'\n", 150 | "dataset = build_dataset(dataset_name, val_split=0.2)\n", 151 | "x_train, y_train = dataset.dataset('train')\n", 152 | "x_val, y_val = dataset.dataset('val')\n", 153 | "\n", 154 | "x_train, x_val, x_scaler = scale(x_train, x_val)\n", 155 | "y_train, y_val, y_scaler = scale(y_train, y_val)\n", 156 | "\n", 157 | "print(dataset_name, x_train.shape)\n", 158 | "config['layers'][0] = x_train.shape[-1]\n" 159 | ], 160 | "metadata": { 161 | "collapsed": false, 162 | "pycharm": { 163 | "name": "#%%\n" 164 | } 165 | } 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": null, 170 | "outputs": [], 171 | "source": [ 172 | "model = MLP(config['layers'])\n" 173 | ], 174 | "metadata": { 175 | "collapsed": false, 176 | "pycharm": { 177 | "name": "#%%\n" 178 | } 179 | } 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "outputs": [], 185 | "source": [ 186 | "model.fit((x_train, y_train), (x_val, y_val))\n" 187 | ], 188 | "metadata": { 189 | "collapsed": false, 190 | "pycharm": { 191 | "name": "#%%\n" 192 | } 193 | } 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "outputs": [], 199 | "source": [ 200 | "x_val_tensor = torch.tensor(x_val)\n", 201 | "predictions = model(x_val_tensor.cuda()).cpu().detach().numpy()\n" 202 | ], 203 | "metadata": { 204 | "collapsed": false, 205 | "pycharm": { 206 | "name": "#%%\n" 207 | } 208 | } 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "outputs": [], 214 | "source": [ 215 | "unscale = lambda y : y_scaler.inverse_transform(y)\n", 216 | "\n", 217 | "scaled_errors = unscale(predictions) - unscale(y_val)\n", 218 | "rmse_single = np.sqrt(np.mean(np.square(scaled_errors)))" 219 | ], 220 | "metadata": { 221 | "collapsed": false, 222 | "pycharm": { 223 | "name": "#%%\n" 224 | } 225 | } 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "outputs": [], 231 | "source": [ 232 | "train_opts = ['patience', 'dropout_rate', 'epochs', 'batch_size', 'validation_step', 'verbose']\n", 233 | "config['train_opts'] = {k: config[k] for k in config if k in train_opts}" 234 | ], 235 | "metadata": { 236 | "collapsed": false, 237 | "pycharm": { 238 | "name": "#%%\n" 239 | } 240 | } 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": null, 245 | "outputs": [], 246 | "source": [ 247 | "\n", 248 | "train_opts = config['train_opts'].copy()\n", 249 | "ensemble = MLPEnsemble(config['layers'],\n", 250 | " n_models=config['n_ens'],\n", 251 | " reduction='mean')\n", 252 | "ensemble.fit((x_train, y_train),\n", 253 | " (x_val, y_val),\n", 254 | " **train_opts)\n", 255 | "\n", 256 | "\n", 257 | "estimator_ensemble = build_estimator('emcdue', ensemble, nn_runs=100)\n", 258 | "estimator_single = build_estimator('mcdue', model, nn_runs=100)\n", 259 | "\n", 260 | "x_val_tensor = torch.tensor(x_val)\n", 261 | "unscale = lambda y : y_scaler.inverse_transform(y)\n", 262 | "\n", 263 | "predictions = model(x_val_tensor.cuda()).cpu().detach().numpy()\n", 264 | "errors = predictions - y_val\n", 265 | "ue_single = estimator_single.estimate(torch.Tensor(x_val).double().cuda())\n", 266 | "ll_single = uq_ll(errors, ue_single)\n", 267 | "\n", 268 | "\n", 269 | "predictions = ensemble(x_val_tensor.cuda()).cpu().detach().numpy()\n", 270 | "errors = predictions - y_val\n", 271 | "ue_ensemble = estimator_ensemble.estimate(torch.Tensor(x_val).double().cuda())\n", 272 | "ll_ensemble = uq_ll(errors, ue_ensemble)\n", 273 | "\n", 274 | "# accumulate.append([file[:4], 'single', ll_single])\n", 275 | "# accumulate.append([file[:4], 'ensemble', ll_ensemble])\n" 276 | ], 277 | "metadata": { 278 | "collapsed": false, 279 | "pycharm": { 280 | "name": "#%%\n" 281 | } 282 | } 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": null, 287 | "outputs": [], 288 | "source": [ 289 | "ll_single\n", 290 | "ll_ensemble\n", 291 | "\n" 292 | ], 293 | "metadata": { 294 | "collapsed": false, 295 | "pycharm": { 296 | "name": "#%%\n" 297 | } 298 | } 299 | } 300 | ], 301 | "metadata": { 302 | "kernelspec": { 303 | "name": "python3", 304 | "language": "python", 305 | "display_name": "Python 3" 306 | }, 307 | "language_info": { 308 | "codemirror_mode": { 309 | "name": "ipython", 310 | "version": 2 311 | }, 312 | "file_extension": ".py", 313 | "mimetype": "text/x-python", 314 | "name": "python", 315 | "nbconvert_exporter": "python", 316 | "pygments_lexer": "ipython2", 317 | "version": "2.7.6" 318 | } 319 | }, 320 | "nbformat": 4, 321 | "nbformat_minor": 0 322 | } -------------------------------------------------------------------------------- /experiments/deprecated/ensembles_2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "outputs": [], 7 | "source": [ 8 | "%load_ext autoreload\n", 9 | "%autoreload 2" 10 | ], 11 | "metadata": { 12 | "collapsed": false, 13 | "pycharm": { 14 | "name": "#%%\n" 15 | } 16 | } 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "outputs": [], 22 | "source": [ 23 | "import os\n", 24 | "import pickle\n", 25 | "import random\n", 26 | "from pathlib import Path\n", 27 | "\n", 28 | "import pandas as pd\n", 29 | "import torch\n", 30 | "from torch.nn.functional import elu\n", 31 | "import numpy as np\n", 32 | "import matplotlib.pyplot as plt\n", 33 | "\n", 34 | "from alpaca.uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 35 | "from alpaca.model.ensemble import MLPEnsemble\n", 36 | "from alpaca.uncertainty_estimator import build_estimator\n", 37 | "from alpaca.analysis.metrics import get_uq_metrics\n", 38 | "\n", 39 | "plt.rcParams['figure.facecolor'] = 'white'\n" 40 | ], 41 | "metadata": { 42 | "collapsed": false, 43 | "pycharm": { 44 | "name": "#%%\n" 45 | } 46 | } 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "outputs": [], 52 | "source": [ 53 | "SEED = 10\n", 54 | "torch.manual_seed(SEED)\n", 55 | "np.random.seed(SEED)\n", 56 | "random.seed(SEED)\n", 57 | "\n", 58 | "torch.cuda.set_device(0)\n", 59 | "torch.backends.cudnn.deterministic = True\n", 60 | "torch.backends.cudnn.benchmark = False\n" 61 | ], 62 | "metadata": { 63 | "collapsed": false, 64 | "pycharm": { 65 | "name": "#%%\n" 66 | } 67 | } 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "outputs": [], 73 | "source": [ 74 | "folder = Path('./data/regression')\n", 75 | "files = [file for file in os.listdir(folder) if file.endswith('.pickle')]\n", 76 | "# files = [file for file in files if file.startswith('kin8')]\n" 77 | ], 78 | "metadata": { 79 | "collapsed": false, 80 | "pycharm": { 81 | "name": "#%%\n" 82 | } 83 | } 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": null, 88 | "outputs": [], 89 | "source": [ 90 | "\n", 91 | "accumulate = []" 92 | ], 93 | "metadata": { 94 | "collapsed": false, 95 | "pycharm": { 96 | "name": "#%%\n" 97 | } 98 | } 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "outputs": [], 104 | "source": [ 105 | "data = []\n", 106 | "for cnt, file in enumerate(files):\n", 107 | " print(file)\n", 108 | " with open(folder / 'log_exp.log', 'w') as f:\n", 109 | " f.write(f'{cnt} / {len(files)}')\n", 110 | " with open(folder / file, 'rb') as f:\n", 111 | " dct = pickle.load(f)\n", 112 | " print(file)\n", 113 | " config = dct['config']\n", 114 | " config['n_ue_runs'] = 1\n", 115 | " config['acc_percentile'] = .1\n", 116 | " state_dict = dct['state_dict']\n", 117 | " x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data']\n", 118 | "\n", 119 | " ensemble = MLPEnsemble(\n", 120 | " config['layers'], n_models=config['n_ens'], activation = elu,\n", 121 | " reduction='mean')\n", 122 | " ensemble.load_state_dict(state_dict)\n", 123 | "\n", 124 | " model = ensemble.models[2]\n", 125 | "\n", 126 | " x_val_tensor = torch.tensor(x_val)\n", 127 | " predictions = model(x_val_tensor.cuda()).cpu().detach().numpy()\n", 128 | " unscale = lambda y : y_scaler.inverse_transform(y)\n", 129 | " # unscale = lambda y : y\n", 130 | "\n", 131 | " scaled_errors = unscale(predictions) - unscale(y_val)\n", 132 | " rmse_single = np.sqrt(np.mean(np.square(scaled_errors)))\n", 133 | " #%%\n", 134 | " predictions = ensemble(x_val_tensor.cuda()).cpu().detach().numpy()\n", 135 | " scaled_errors = unscale(predictions) - unscale(y_val)\n", 136 | " rmse_ensemble = np.sqrt(np.mean(np.square(scaled_errors)))\n", 137 | "\n", 138 | " #%%\n", 139 | " accumulate.append([file[:4], 'single', rmse_single])\n", 140 | " accumulate.append([file[:4], 'ensemble', rmse_ensemble])\n", 141 | "\n", 142 | " print(file)\n", 143 | " print('Single', rmse_single)\n", 144 | " print('Ensemble', rmse_ensemble)\n", 145 | "\n" 146 | ], 147 | "metadata": { 148 | "collapsed": false, 149 | "pycharm": { 150 | "name": "#%%\n" 151 | } 152 | } 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "outputs": [], 158 | "source": [ 159 | "import seaborn as sns\n", 160 | "import pandas as pd" 161 | ], 162 | "metadata": { 163 | "collapsed": false, 164 | "pycharm": { 165 | "name": "#%%\n" 166 | } 167 | } 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "outputs": [], 173 | "source": [ 174 | "df = pd.DataFrame(accumulate, columns=['dataset', 'type', 'rmse'])\n", 175 | "df" 176 | ], 177 | "metadata": { 178 | "collapsed": false, 179 | "pycharm": { 180 | "name": "#%%\n" 181 | } 182 | } 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "outputs": [], 188 | "source": [ 189 | "sns.boxplot('dataset', 'rmse', hue='type', data=df)\n", 190 | "plt.savefig('rmse.png', dpi=150)\n" 191 | ], 192 | "metadata": { 193 | "collapsed": false, 194 | "pycharm": { 195 | "name": "#%%\n" 196 | } 197 | } 198 | } 199 | ], 200 | "metadata": { 201 | "kernelspec": { 202 | "name": "python3", 203 | "language": "python", 204 | "display_name": "Python 3" 205 | }, 206 | "language_info": { 207 | "codemirror_mode": { 208 | "name": "ipython", 209 | "version": 2 210 | }, 211 | "file_extension": ".py", 212 | "mimetype": "text/x-python", 213 | "name": "python", 214 | "nbconvert_exporter": "python", 215 | "pygments_lexer": "ipython2", 216 | "version": "2.7.6" 217 | } 218 | }, 219 | "nbformat": 4, 220 | "nbformat_minor": 0 221 | } -------------------------------------------------------------------------------- /experiments/deprecated/ensembles_3.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "outputs": [], 7 | "source": [ 8 | "%load_ext autoreload\n", 9 | "%autoreload 2" 10 | ], 11 | "metadata": { 12 | "collapsed": false, 13 | "pycharm": { 14 | "name": "#%%\n" 15 | } 16 | } 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "outputs": [], 22 | "source": [ 23 | "import os\n", 24 | "import pickle\n", 25 | "import random\n", 26 | "from pathlib import Path\n", 27 | "\n", 28 | "import seaborn as sns\n", 29 | "import pandas as pd\n", 30 | "import torch\n", 31 | "from torch.nn.functional import elu\n", 32 | "import numpy as np\n", 33 | "import matplotlib.pyplot as plt\n", 34 | "\n", 35 | "from alpaca.uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 36 | "from alpaca.analysis.metrics import uq_ll\n", 37 | "from alpaca.model.ensemble import MLPEnsemble\n", 38 | "from alpaca.uncertainty_estimator import build_estimator\n", 39 | "from alpaca.analysis.metrics import get_uq_metrics\n", 40 | "\n", 41 | "plt.rcParams['figure.facecolor'] = 'white'\n" 42 | ], 43 | "metadata": { 44 | "collapsed": false, 45 | "pycharm": { 46 | "name": "#%%\n" 47 | } 48 | } 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "outputs": [], 54 | "source": [ 55 | "SEED = 10\n", 56 | "torch.manual_seed(SEED)\n", 57 | "np.random.seed(SEED)\n", 58 | "random.seed(SEED)\n", 59 | "\n", 60 | "torch.cuda.set_device(0)\n", 61 | "torch.backends.cudnn.deterministic = True\n", 62 | "torch.backends.cudnn.benchmark = False\n" 63 | ], 64 | "metadata": { 65 | "collapsed": false, 66 | "pycharm": { 67 | "name": "#%%\n" 68 | } 69 | } 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "outputs": [], 75 | "source": [ 76 | "folder = Path('./data/regression')\n", 77 | "files = sorted([file for file in os.listdir(folder) if file.endswith('.pickle')])\n", 78 | "# files = [file for file in files if file.startswith('kin8')]\n", 79 | "\n" 80 | ], 81 | "metadata": { 82 | "collapsed": false, 83 | "pycharm": { 84 | "name": "#%%\n" 85 | } 86 | } 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "outputs": [], 92 | "source": [ 93 | "def load_setup(file):\n", 94 | " print(file)\n", 95 | " with open(folder / 'log_exp.log', 'w') as f:\n", 96 | " f.write(f'{cnt} / {len(files)}')\n", 97 | " with open(folder / file, 'rb') as f:\n", 98 | " dct = pickle.load(f)\n", 99 | " print(file)\n", 100 | " config = dct['config']\n", 101 | " config['n_ue_runs'] = 1\n", 102 | " config['acc_percentile'] = .1\n", 103 | " state_dict = dct['state_dict']\n", 104 | " x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data']\n", 105 | "\n", 106 | " ensemble = MLPEnsemble(\n", 107 | " config['layers'], n_models=config['n_ens'], activation = elu,\n", 108 | " reduction='mean')\n", 109 | " ensemble.load_state_dict(state_dict)\n", 110 | "\n", 111 | " model = ensemble.models[2]\n", 112 | " return model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler\n" 113 | ], 114 | "metadata": { 115 | "collapsed": false, 116 | "pycharm": { 117 | "name": "#%%\n" 118 | } 119 | } 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "outputs": [], 125 | "source": [ 126 | "accumulate = []\n", 127 | "data = []\n", 128 | "\n", 129 | "for cnt, file in enumerate(files):\n", 130 | " model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler = load_setup(file)\n", 131 | " estimator_ensemble = build_estimator('emcdue', ensemble, nn_runs=100)\n", 132 | " estimator_single = build_estimator('mcdue', model, nn_runs=100)\n", 133 | "\n", 134 | " x_val_tensor = torch.tensor(x_val)\n", 135 | " unscale = lambda y : y_scaler.inverse_transform(y)\n", 136 | "\n", 137 | " predictions = model(x_val_tensor.cuda()).cpu().detach().numpy()\n", 138 | " errors = predictions - y_val\n", 139 | " ue_single = estimator_single.estimate(torch.Tensor(x_val).double().cuda())\n", 140 | " ll_single = uq_ll(errors, ue_single)\n", 141 | "\n", 142 | "\n", 143 | " predictions = ensemble(x_val_tensor.cuda()).cpu().detach().numpy()\n", 144 | " errors = predictions - y_val\n", 145 | " ue_ensemble = estimator_ensemble.estimate(torch.Tensor(x_val).double().cuda())\n", 146 | " ll_ensemble = uq_ll(errors, ue_ensemble)\n", 147 | "\n", 148 | " accumulate.append([file[:4], 'single', ll_single])\n", 149 | " accumulate.append([file[:4], 'ensemble', ll_ensemble])\n", 150 | " #\n", 151 | " # print(file)\n", 152 | " # print('Single', rmse_single)\n", 153 | " # print('Ensemble', rmse_ensemble)\n", 154 | "\n" 155 | ], 156 | "metadata": { 157 | "collapsed": false, 158 | "pycharm": { 159 | "name": "#%%\n" 160 | } 161 | } 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": null, 166 | "outputs": [], 167 | "source": [ 168 | "df = pd.DataFrame(accumulate, columns=['dataset', 'type', 'll'])\n", 169 | "sns.boxplot('dataset', 'll', hue='type', data=df)\n", 170 | "plt.savefig('ll_emcdue.png', dpi=150)\n", 171 | "\n", 172 | "\n", 173 | "\n" 174 | ], 175 | "metadata": { 176 | "collapsed": false, 177 | "pycharm": { 178 | "name": "#%%\n" 179 | } 180 | } 181 | } 182 | ], 183 | "metadata": { 184 | "kernelspec": { 185 | "name": "python3", 186 | "language": "python", 187 | "display_name": "Python 3" 188 | }, 189 | "language_info": { 190 | "codemirror_mode": { 191 | "name": "ipython", 192 | "version": 2 193 | }, 194 | "file_extension": ".py", 195 | "mimetype": "text/x-python", 196 | "name": "python", 197 | "nbconvert_exporter": "python", 198 | "pygments_lexer": "ipython2", 199 | "version": "2.7.6" 200 | } 201 | }, 202 | "nbformat": 4, 203 | "nbformat_minor": 0 204 | } -------------------------------------------------------------------------------- /experiments/deprecated/ensembles_4.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "outputs": [], 7 | "source": [ 8 | "%load_ext autoreload\n", 9 | "%autoreload 2" 10 | ], 11 | "metadata": { 12 | "collapsed": false, 13 | "pycharm": { 14 | "name": "#%%\n" 15 | } 16 | } 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "outputs": [], 22 | "source": [ 23 | "import os\n", 24 | "import pickle\n", 25 | "import random\n", 26 | "from pathlib import Path\n", 27 | "\n", 28 | "import seaborn as sns\n", 29 | "import pandas as pd\n", 30 | "import torch\n", 31 | "from torch.nn.functional import elu\n", 32 | "import numpy as np\n", 33 | "import matplotlib.pyplot as plt\n", 34 | "\n", 35 | "from alpaca.uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 36 | "from alpaca.analysis.metrics import uq_ll\n", 37 | "from alpaca.model.ensemble import MLPEnsemble\n", 38 | "from alpaca.uncertainty_estimator import build_estimator\n", 39 | "from alpaca.analysis.metrics import get_uq_metrics\n", 40 | "\n", 41 | "plt.rcParams['figure.facecolor'] = 'white'\n" 42 | ], 43 | "metadata": { 44 | "collapsed": false, 45 | "pycharm": { 46 | "name": "#%%\n" 47 | } 48 | } 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "outputs": [], 54 | "source": [ 55 | "SEED = 10\n", 56 | "torch.manual_seed(SEED)\n", 57 | "np.random.seed(SEED)\n", 58 | "random.seed(SEED)\n", 59 | "\n", 60 | "torch.cuda.set_device(0)\n", 61 | "torch.backends.cudnn.deterministic = True\n", 62 | "torch.backends.cudnn.benchmark = False\n" 63 | ], 64 | "metadata": { 65 | "collapsed": false, 66 | "pycharm": { 67 | "name": "#%%\n" 68 | } 69 | } 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "outputs": [], 75 | "source": [ 76 | "folder = Path('./data/regression')\n", 77 | "files = sorted([file for file in os.listdir(folder) if file.endswith('.pickle')])\n", 78 | "# files = [file for file in files if file.startswith('kin8')]\n", 79 | "\n" 80 | ], 81 | "metadata": { 82 | "collapsed": false, 83 | "pycharm": { 84 | "name": "#%%\n" 85 | } 86 | } 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "outputs": [], 92 | "source": [ 93 | "def load_setup(file):\n", 94 | " print(file)\n", 95 | " with open(folder / 'log_exp.log', 'w') as f:\n", 96 | " f.write(f'{cnt} / {len(files)}')\n", 97 | " with open(folder / file, 'rb') as f:\n", 98 | " dct = pickle.load(f)\n", 99 | " config = dct['config']\n", 100 | " config['n_ue_runs'] = 1\n", 101 | " config['acc_percentile'] = .1\n", 102 | " state_dict = dct['state_dict']\n", 103 | " x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data']\n", 104 | "\n", 105 | " ensemble = MLPEnsemble(\n", 106 | " config['layers'], n_models=config['n_ens'], activation = elu,\n", 107 | " reduction='mean')\n", 108 | " ensemble.load_state_dict(state_dict)\n", 109 | "\n", 110 | " model = ensemble.models[2]\n", 111 | " return model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler\n" 112 | ], 113 | "metadata": { 114 | "collapsed": false, 115 | "pycharm": { 116 | "name": "#%%\n" 117 | } 118 | } 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "outputs": [], 124 | "source": [ 125 | "accumulate_ll = []\n", 126 | "data = []\n", 127 | "\n", 128 | "\n", 129 | "# np.random.shuffle(files)\n", 130 | "for cnt, file in enumerate(files):\n", 131 | " model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler = load_setup(file)\n", 132 | "\n", 133 | " x_val_tensor = torch.tensor(x_val)\n", 134 | " unscale = lambda y : y_scaler.inverse_transform(y)\n", 135 | "\n", 136 | " predictions = model(x_val_tensor.cuda()).cpu().detach().numpy()\n", 137 | " errors = predictions - y_val\n", 138 | " # rmse_single = np.sqrt(np.mean(np.square(errors)))\n", 139 | " # accumulate.append([file[:4], 'single', rmse_single])\n", 140 | "\n", 141 | " # predictions = ensemble(x_val_tensor.cuda()).cpu().detach().numpy()\n", 142 | " # errors = predictions - y_val\n", 143 | " # rmse_single = np.sqrt(np.mean(np.square(errors)))\n", 144 | " # accumulate.append([file[:4], 'ensemble', rmse_single])\n", 145 | "\n", 146 | " for mask_name in DEFAULT_MASKS:\n", 147 | " estimator = build_estimator(\n", 148 | " 'mcdue_masked', model, nn_runs=100, keep_runs=True,\n", 149 | " dropout_mask=mask_name)\n", 150 | " ue =estimator.estimate(torch.Tensor(x_val).double().cuda())\n", 151 | " runs = estimator.last_mcd_runs()\n", 152 | " predictions = np.mean(estimator.last_mcd_runs(), axis=-1)\n", 153 | " errors = predictions - y_val[:, 0]\n", 154 | " rmse_mask = np.sqrt(np.mean(np.square(errors)))\n", 155 | " # accumulate.append([file[:4], mask_name, rmse_mask])\n", 156 | " ll = uq_ll(errors, ue)\n", 157 | " accumulate_ll.append([file[:4], mask_name, ll])\n", 158 | "\n", 159 | "\n", 160 | "\n" 161 | ], 162 | "metadata": { 163 | "collapsed": false, 164 | "pycharm": { 165 | "name": "#%%\n" 166 | } 167 | } 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "outputs": [], 173 | "source": [ 174 | "plt.figure(figsize=(10, 6))\n", 175 | "\n", 176 | "accumulate2 = [record for record in accumulate_ll if record[1] !='decorrelating_sc']\n", 177 | "df = pd.DataFrame(accumulate2, columns=['dataset', 'type', 'LL'])\n", 178 | "sns.boxplot('dataset', 'LL', hue='type', data=df)\n", 179 | "plt.savefig('ll_masks.png', dpi=150)\n", 180 | "\n" 181 | ], 182 | "metadata": { 183 | "collapsed": false, 184 | "pycharm": { 185 | "name": "#%%\n" 186 | } 187 | } 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "outputs": [], 193 | "source": [ 194 | "\n" 195 | ], 196 | "metadata": { 197 | "collapsed": false, 198 | "pycharm": { 199 | "name": "#%%\n" 200 | } 201 | } 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": null, 206 | "outputs": [], 207 | "source": [ 208 | "\n" 209 | ], 210 | "metadata": { 211 | "collapsed": false, 212 | "pycharm": { 213 | "name": "#%%\n" 214 | } 215 | } 216 | } 217 | ], 218 | "metadata": { 219 | "kernelspec": { 220 | "name": "python3", 221 | "language": "python", 222 | "display_name": "Python 3" 223 | }, 224 | "language_info": { 225 | "codemirror_mode": { 226 | "name": "ipython", 227 | "version": 2 228 | }, 229 | "file_extension": ".py", 230 | "mimetype": "text/x-python", 231 | "name": "python", 232 | "nbconvert_exporter": "python", 233 | "pygments_lexer": "ipython2", 234 | "version": "2.7.6" 235 | } 236 | }, 237 | "nbformat": 4, 238 | "nbformat_minor": 0 239 | } -------------------------------------------------------------------------------- /experiments/deprecated/ensembles_5.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "outputs": [], 7 | "source": [ 8 | "%load_ext autoreload\n", 9 | "%autoreload 2" 10 | ], 11 | "metadata": { 12 | "collapsed": false, 13 | "pycharm": { 14 | "name": "#%%\n" 15 | } 16 | } 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "outputs": [], 22 | "source": [ 23 | "import os\n", 24 | "import pickle\n", 25 | "import random\n", 26 | "from pathlib import Path\n", 27 | "\n", 28 | "import seaborn as sns\n", 29 | "import pandas as pd\n", 30 | "import torch\n", 31 | "from torch.nn.functional import elu\n", 32 | "import numpy as np\n", 33 | "import matplotlib.pyplot as plt\n", 34 | "\n", 35 | "from alpaca.uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 36 | "from alpaca.analysis.metrics import uq_ll\n", 37 | "from alpaca.model.ensemble import MLPEnsemble\n", 38 | "from alpaca.uncertainty_estimator import build_estimator\n", 39 | "from alpaca.analysis.metrics import get_uq_metrics\n", 40 | "\n", 41 | "plt.rcParams['figure.facecolor'] = 'white'\n" 42 | ], 43 | "metadata": { 44 | "collapsed": false, 45 | "pycharm": { 46 | "name": "#%%\n" 47 | } 48 | } 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "outputs": [], 54 | "source": [ 55 | "SEED = 10\n", 56 | "torch.manual_seed(SEED)\n", 57 | "np.random.seed(SEED)\n", 58 | "random.seed(SEED)\n", 59 | "\n", 60 | "torch.cuda.set_device(0)\n", 61 | "torch.backends.cudnn.deterministic = True\n", 62 | "torch.backends.cudnn.benchmark = False\n" 63 | ], 64 | "metadata": { 65 | "collapsed": false, 66 | "pycharm": { 67 | "name": "#%%\n" 68 | } 69 | } 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "outputs": [], 75 | "source": [ 76 | "folder = Path('./data/regression')\n", 77 | "files = sorted([file for file in os.listdir(folder) if file.endswith('.pickle')])\n", 78 | "# files = [file for file in files if file.startswith('kin8')]\n", 79 | "\n" 80 | ], 81 | "metadata": { 82 | "collapsed": false, 83 | "pycharm": { 84 | "name": "#%%\n" 85 | } 86 | } 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "outputs": [], 92 | "source": [ 93 | "def load_setup(file):\n", 94 | " print(file)\n", 95 | " with open(folder / 'log_exp.log', 'w') as f:\n", 96 | " f.write(f'{cnt} / {len(files)}')\n", 97 | " with open(folder / file, 'rb') as f:\n", 98 | " dct = pickle.load(f)\n", 99 | " config = dct['config']\n", 100 | " config['n_ue_runs'] = 1\n", 101 | " config['acc_percentile'] = .1\n", 102 | " state_dict = dct['state_dict']\n", 103 | " x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data']\n", 104 | "\n", 105 | " ensemble = MLPEnsemble(\n", 106 | " config['layers'], n_models=config['n_ens'], activation = elu,\n", 107 | " reduction='mean')\n", 108 | " ensemble.load_state_dict(state_dict)\n", 109 | "\n", 110 | " model = ensemble.models[2]\n", 111 | " return model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler\n" 112 | ], 113 | "metadata": { 114 | "collapsed": false, 115 | "pycharm": { 116 | "name": "#%%\n" 117 | } 118 | } 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "outputs": [], 124 | "source": [ 125 | "accumulate_ll = []\n", 126 | "data = []\n", 127 | "\n", 128 | "\n", 129 | "np.random.shuffle(files)\n", 130 | "for cnt, file in enumerate(files[:3]):\n", 131 | " model, ensemble, x_train, y_train, x_val, y_val, x_scaler, y_scaler = load_setup(file)\n", 132 | "\n", 133 | " x_val_tensor = torch.tensor(x_val)\n", 134 | " unscale = lambda y : y_scaler.inverse_transform(y)\n", 135 | "\n", 136 | " predictions = model(x_val_tensor.cuda()).cpu().detach().numpy()\n", 137 | " errors = predictions - y_val\n", 138 | "\n", 139 | " # mask_names = ['mc_dropout', 'ht_decorrelating', 'ht_dpp', 'cov_dpp', 'ht_k_dpp', 'cov_k_dpp']\n", 140 | " mask_names = ['k_dpp', 'ht_k_dpp', 'cov_k_dpp']\n", 141 | " for mask_name in mask_names:\n", 142 | " try:\n", 143 | " estimator = build_estimator(\n", 144 | " 'mcdue_masked', model, nn_runs=100, keep_runs=True,\n", 145 | " dropout_mask=mask_name, dropout_rate=0.5)\n", 146 | " ue =estimator.estimate(torch.Tensor(x_val).double().cuda())\n", 147 | " runs = estimator.last_mcd_runs()\n", 148 | " predictions = np.mean(estimator.last_mcd_runs(), axis=-1)\n", 149 | " errors = predictions - y_val[:, 0]\n", 150 | " rmse_mask = np.sqrt(np.mean(np.square(errors)))\n", 151 | " ll = uq_ll(errors, ue)\n", 152 | " accumulate_ll.append([file[:4], mask_name, ll])\n", 153 | " print(estimator, ll)\n", 154 | " continue\n" 155 | ], 156 | "metadata": { 157 | "collapsed": false, 158 | "pycharm": { 159 | "name": "#%%\n", 160 | "is_executing": true 161 | } 162 | } 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "outputs": [], 168 | "source": [ 169 | "plt.figure(figsize=(14, 6))\n", 170 | "\n", 171 | "accumulate2 = [record for record in accumulate_ll if record[1] !='decorrelating_sc']\n", 172 | "df = pd.DataFrame(accumulate2, columns=['dataset', 'type', 'LL'])\n", 173 | "sns.boxplot('dataset', 'LL', hue='type', data=df)\n", 174 | "plt.savefig('ll_masks.png', dpi=150)\n", 175 | "\n" 176 | ], 177 | "metadata": { 178 | "collapsed": false, 179 | "pycharm": { 180 | "name": "#%%\n" 181 | } 182 | } 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "outputs": [], 188 | "source": [ 189 | "\n" 190 | ], 191 | "metadata": { 192 | "collapsed": false, 193 | "pycharm": { 194 | "name": "#%%\n" 195 | } 196 | } 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": null, 201 | "outputs": [], 202 | "source": [ 203 | "\n" 204 | ], 205 | "metadata": { 206 | "collapsed": false, 207 | "pycharm": { 208 | "name": "#%%\n" 209 | } 210 | } 211 | } 212 | ], 213 | "metadata": { 214 | "kernelspec": { 215 | "name": "python3", 216 | "language": "python", 217 | "display_name": "Python 3" 218 | }, 219 | "language_info": { 220 | "codemirror_mode": { 221 | "name": "ipython", 222 | "version": 2 223 | }, 224 | "file_extension": ".py", 225 | "mimetype": "text/x-python", 226 | "name": "python", 227 | "nbconvert_exporter": "python", 228 | "pygments_lexer": "ipython2", 229 | "version": "2.7.6" 230 | } 231 | }, 232 | "nbformat": 4, 233 | "nbformat_minor": 0 234 | } -------------------------------------------------------------------------------- /experiments/deprecated/move.py: -------------------------------------------------------------------------------- 1 | import os 2 | from shutil import copyfile 3 | 4 | 5 | 6 | for ack in ['max_prob']: 7 | for i in range(3): 8 | file_from = f"logs/{ack}/cifar_{i}/ue.pickle" 9 | file_to = f"logs/classification/cifar_{i}/ue_{ack}.pickle" 10 | copyfile(file_from, file_to) 11 | print(file_from) 12 | print(file_to) 13 | 14 | -------------------------------------------------------------------------------- /experiments/deprecated/plot_df.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import matplotlib.pyplot as plt 3 | import seaborn as sns 4 | from experiments.experiment_setup import ROOT_DIR 5 | from pathlib import Path 6 | import matplotlib 7 | 8 | matplotlib.rcParams['savefig.dpi'] = 150 9 | 10 | # 11 | # dataset = 'MNIST' 12 | # dataset = 'CIFAR-10' 13 | # dataset = 'SVHN' 14 | 15 | file = Path(ROOT_DIR) / 'experiments'/ 'data'/ f'var_ratio_4_boxplot_{dataset}_50000_150.csv' 16 | df = pd.read_csv(file) 17 | df = df.replace('mc_dropout', 'MC dropout') 18 | df = df.replace('decorrelating_sc', 'decorrelation') 19 | df = df[df['Estimator type'] != 'k_dpp_noisereg'] 20 | print(df) 21 | fig, ax = plt.subplots(figsize=(6, 6)) 22 | plt.subplots_adjust(right=.95) 23 | 24 | with sns.axes_style('whitegrid'): 25 | sns.boxplot(data=df, y='ROC-AUC score', x='Estimator type', ax=ax) 26 | plt.title(f'{dataset} wrong prediction ROC-AUC') 27 | 28 | ax.yaxis.grid(True) 29 | ax.xaxis.grid(True) 30 | 31 | 32 | plt.savefig(f'experiments/data/results/rocauc_{dataset}_4.png') 33 | plt.show() 34 | 35 | -------------------------------------------------------------------------------- /experiments/deprecated/plot_df_al.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import matplotlib.pyplot as plt 3 | import seaborn as sns 4 | from experiments.experiment_setup import ROOT_DIR 5 | from pathlib import Path 6 | import matplotlib 7 | 8 | matplotlib.rcParams['savefig.dpi'] = 600 9 | # matplotlib.rcParams['savefig.dpi'] = 2 * matplotlib.rcParams['savefig.dpi'] 10 | 11 | # 12 | # dataset = 'MNIST' 13 | # dataset = 'CIFAR-10' 14 | dataset = 'SVHN' 15 | 16 | file = Path(ROOT_DIR) / 'experiments' / 'data' / f'mnist_beauty_simple_conv_100_20.csv' 17 | df = pd.read_csv(file) 18 | df = df.replace('mc_dropout', 'MC dropout') 19 | df = df.replace('decorrelating_sc', 'decorrelation') 20 | print(df) 21 | df = df[df['Method'] != 'k_dpp_noisereg'] 22 | df2 = df[df.Method.isin(['random', 'error_oracle', 'max_entropy'])] 23 | df3 = df[~df.Method.isin(['random', 'error_oracle', 'max_entropy'])] 24 | df4 = pd.concat((df3, df2)) 25 | 26 | fig, ax = plt.subplots(figsize=(8, 5)) 27 | plt.subplots_adjust(left=0.1, right=0.95) 28 | 29 | with sns.axes_style('whitegrid'): 30 | sns.lineplot('Step', 'Accuracy', hue='Method', data=df4) 31 | plt.title(f'Active learning on MNIST') 32 | 33 | ax.yaxis.grid(True) 34 | ax.xaxis.grid(True) 35 | plt.savefig(f'experiments/data/results/al.png') 36 | plt.show() -------------------------------------------------------------------------------- /experiments/deprecated/print_confidence_accuracy_multi_ack.py: -------------------------------------------------------------------------------- 1 | import pickle 2 | import os 3 | import argparse 4 | 5 | import numpy as np 6 | import pandas as pd 7 | import matplotlib.pyplot as plt 8 | import seaborn as sns 9 | 10 | 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('name', type=str) 13 | parser.add_argument('repeats', type=int) 14 | # parser.add_argument('--acquisition', '-a', type=str, default='bald') 15 | parser.add_argument('--covariance', dest='covariance', action='store_true') 16 | args = parser.parse_args() 17 | 18 | 19 | acc_conf = [] 20 | count_conf = [] 21 | 22 | method = 'ht_k_dpp' 23 | 24 | 25 | covariance_str = '_covar' if args.covariance else '' 26 | 27 | 28 | 29 | # method = 'ht_dpp' 30 | approaches = [ 31 | ('ensemble_max_prob', 'ensemble'), 32 | ('mc_dropout', 'bald'), 33 | ('ht_dpp', 'bald'), 34 | ('ensemble_bald', 'ensemble'), 35 | ('max_prob', '') 36 | ] 37 | 38 | # file_name = f'logs/classification/mnist_0/ue.pickle' 39 | # 40 | # 41 | # with open(file_name, 'rb') as f: 42 | # record = pickle.load(f) 43 | # 44 | # print(record) 45 | # 46 | # 47 | # import ipdb; ipdb.set_trace() 48 | 49 | # for ack in ['var_ratio', 'bald_n', 'max_prob']: 50 | for estimator_name, ack in approaches: 51 | args.acquisition = ack 52 | acquisition_str = 'bald' if args.acquisition in ['bald_n', ''] else args.acquisition 53 | 54 | for i in range(args.repeats): 55 | file_name = f'logs/classification/{args.name}_{i}/ue_{acquisition_str}{covariance_str}.pickle' 56 | print(file_name) 57 | 58 | with open(file_name, 'rb') as f: 59 | record = pickle.load(f) 60 | 61 | prediction = np.argmax(np.array(record['probabilities']), axis=-1) 62 | # import ipdb; ipdb.set_trace() 63 | is_correct = (prediction == record['y_val']).astype(np.int) 64 | 65 | # bins = np.concatenate((np.arange(0, 0.9, 0.1), np.arange(0.9, 1, 0.01))) 66 | bins = np.arange(0, 1, 0.1) 67 | 68 | print(record['uncertainties'].keys()) 69 | ue = record['uncertainties'][estimator_name] 70 | 71 | print(estimator_name) 72 | print(min(ue), max(ue)) 73 | if args.acquisition == 'bald_n': 74 | ue = ue / max(ue) 75 | 76 | for confidence_level in bins: 77 | point_confidences = 1 - ue 78 | bin_correct = is_correct[point_confidences > confidence_level] 79 | if len(bin_correct) > 0: 80 | accuracy = sum(bin_correct) / len(bin_correct) 81 | else: 82 | accuracy = None 83 | acc_conf.append((confidence_level, accuracy, estimator_name+'_'+ack)) 84 | count_conf.append((confidence_level, len(bin_correct), estimator_name+' '+ack)) 85 | 86 | 87 | plt.figure(figsize=(12, 6)) 88 | 89 | plt.subplot(1, 2, 1) 90 | plt.title(f"Confidence-accuracy {args.name} {args.acquisition} {covariance_str}") 91 | df = pd.DataFrame(acc_conf, columns=['Confidence level', 'Accuracy', 'Estimator']) 92 | sns.lineplot('Confidence level', 'Accuracy', data=df, hue='Estimator') 93 | # plt.savefig(f"data/conf_accuracy_{args.name}_{args.acquisition}", dpi=150) 94 | 95 | plt.subplot(1, 2, 2) 96 | # plt.figure(figsize=(8, 6)) 97 | plt.title(f"Confidence-count {args.name} {args.acquisition} {covariance_str}") 98 | df = pd.DataFrame(count_conf, columns=['Confidence level', 'Count', 'Estimator']) 99 | sns.lineplot('Confidence level', 'Count', data=df, hue='Estimator') 100 | plt.savefig(f"data/conf_ackquisition_{args.name}_{method}", dpi=150) 101 | # plt.savefig(f"data/conf_count_{args.name}_{args.acquisition}", dpi=150) 102 | plt.show() 103 | 104 | -------------------------------------------------------------------------------- /experiments/deprecated/print_histogram.py: -------------------------------------------------------------------------------- 1 | import pickle 2 | import os 3 | import argparse 4 | 5 | import numpy as np 6 | import pandas as pd 7 | import matplotlib.pyplot as plt 8 | import seaborn as sns 9 | 10 | 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('name', type=str) 13 | parser.add_argument('repeats', type=int) 14 | parser.add_argument('--group', type=str, default='ht') 15 | args = parser.parse_args() 16 | 17 | 18 | acc_conf = [] 19 | count_conf = [] 20 | 21 | 22 | for i in range(args.repeats): 23 | file_name = f'logs/{args.group}/{args.name}_{i}/ue.pickle' 24 | 25 | with open(file_name, 'rb') as f: 26 | record = pickle.load(f) 27 | 28 | prediction = np.argmax(record['probabilities'], axis=-1) 29 | is_correct = (prediction == record['y_val']).astype(np.int) 30 | 31 | fig = plt.figure(figsize=(12, 8)) 32 | fig.suptitle(args.group) 33 | 34 | for i, estimator in enumerate(record['estimators']): 35 | if estimator in ['max_entropy']: 36 | continue 37 | ue = record['uncertainties'][estimator] 38 | print(ue.shape) 39 | print(estimator) 40 | print(min(ue), max(ue)) 41 | ax = plt.subplot(2, 3, i) 42 | ax.set_title(estimator) 43 | plt.hist(ue) 44 | 45 | plt.show() 46 | 47 | 48 | 49 | -------------------------------------------------------------------------------- /experiments/deprecated/print_it.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import matplotlib.pyplot as plt 3 | import seaborn as sns 4 | 5 | 6 | # df = pd.read_csv('data/al/ht_mnist_200_10.csv', names=['id', 'accuracy', 'step', 'method'], index_col='id', skiprows=1) 7 | # 8 | # 9 | # plt.figure(figsize=(8, 6)) 10 | # plt.title('MNIST') 11 | # sns.lineplot('step', 'accuracy', hue='method', data=df) 12 | # plt.savefig('data/al/al_ht_mnist_200_10') 13 | # plt.show() 14 | 15 | 16 | config = { 17 | 'name': 'cifar' 18 | } 19 | # df = pd.DataFrame(rocaucs, columns=['Estimator', 'ROC-AUCs']) 20 | df = pd.read_csv(f"logs/{config['name']}_ed.csv", names=['id', 'ROC-AUCs', 'Estimator'], index_col='id', skiprows=1) 21 | plt.figure(figsize=(9, 6)) 22 | sns.boxplot('Estimator', 'ROC-AUCs', data=df) 23 | plt.title(f"Error detection for {config['name']}") 24 | plt.savefig(f"data/ed/{config['name']}.png", dpi=150) 25 | plt.show() 26 | 27 | df.to_csv(f"logs/{config['name']}_ed.csv") 28 | -------------------------------------------------------------------------------- /experiments/deprecated/print_ll.py: -------------------------------------------------------------------------------- 1 | import pickle 2 | import os 3 | import argparse 4 | 5 | import numpy as np 6 | import pandas as pd 7 | import matplotlib.pyplot as plt 8 | import seaborn as sns 9 | 10 | 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('name', type=str) 13 | parser.add_argument('repeats', type=int) 14 | parser.add_argument('--acquisition', '-a', type=str, default='bald') 15 | parser.add_argument('--covariance', dest='covariance', action='store_true') 16 | args = parser.parse_args() 17 | 18 | 19 | 20 | covariance_str = '_covar' if args.covariance else '' 21 | acquisition_str = 'bald' if args.acquisition == 'bald_n' else args.acquisition 22 | 23 | 24 | ll_records = [] 25 | 26 | for i in range(args.repeats): 27 | file_name = f'logs/classification/{args.name}_{i}/ue_{acquisition_str}{covariance_str}.pickle' 28 | print(file_name) 29 | 30 | with open(file_name, 'rb') as f: 31 | record = pickle.load(f) 32 | 33 | print(i) 34 | print(record['lls']) 35 | 36 | ll_records.extend(record['lls'].items()) 37 | 38 | len_val = 10_000 39 | 40 | ll_records = [(record[0], record[1] / len_val) for record in ll_records] 41 | print(ll_records) 42 | 43 | df = pd.DataFrame(ll_records, columns=['method', 'log_likelihood']) 44 | plt.title(args.name) 45 | sns.boxplot('method', 'log_likelihood', data=df) 46 | plt.show() 47 | 48 | 49 | # prediction = np.argmax(record['probabilities'], axis=-1) 50 | # is_correct = (prediction == record['y_val']).astype(np.int) 51 | # 52 | # # bins = np.concatenate((np.arange(0, 0.9, 0.1), np.arange(0.9, 1, 0.01))) 53 | # bins = np.arange(0, 1, 0.1) 54 | # 55 | # for i, estimator in enumerate(record['estimators']): 56 | # if estimator in ['max_entropy']: 57 | # continue 58 | # ue = record['uncertainties'][estimator] 59 | # 60 | # print(estimator) 61 | # print(min(ue), max(ue)) 62 | # if args.acquisition == 'bald_n': 63 | # ue = ue / max(ue) 64 | # 65 | # for confidence_level in bins: 66 | # point_confidences = 1 - ue 67 | # bin_correct = is_correct[point_confidences > confidence_level] 68 | # if len(bin_correct) > 0: 69 | # accuracy = sum(bin_correct) / len(bin_correct) 70 | # else: 71 | # accuracy = None 72 | # acc_conf.append((confidence_level, accuracy, estimator)) 73 | # count_conf.append((confidence_level, len(bin_correct), estimator)) 74 | 75 | 76 | # plt.figure(figsize=(12, 6)) 77 | # 78 | # plt.subplot(1, 2, 1) 79 | # plt.title(f"Confidence-accuracy {args.name} {args.acquisition} {covariance_str}") 80 | # df = pd.DataFrame(acc_conf, columns=['Confidence level', 'Accuracy', 'Estimator']) 81 | # sns.lineplot('Confidence level', 'Accuracy', data=df, hue='Estimator') 82 | # # plt.savefig(f"data/conf_accuracy_{args.name}_{args.acquisition}", dpi=150) 83 | # 84 | # plt.subplot(1, 2, 2) 85 | # # plt.figure(figsize=(8, 6)) 86 | # plt.title(f"Confidence-count {args.name} {args.acquisition} {covariance_str}") 87 | # df = pd.DataFrame(count_conf, columns=['Confidence level', 'Count', 'Estimator']) 88 | # sns.lineplot('Confidence level', 'Count', data=df, hue='Estimator') 89 | # plt.savefig(f"data/conf_accuracy_{args.name}_{args.acquisition}{covariance_str}", dpi=150) 90 | # # plt.savefig(f"data/conf_count_{args.name}_{args.acquisition}", dpi=150) 91 | # plt.show() 92 | # 93 | -------------------------------------------------------------------------------- /experiments/deprecated/regression_2_prfm2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%load_ext autoreload\n", 10 | "%autoreload 2" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import os\n", 20 | "import pickle\n", 21 | "import random\n", 22 | "from pathlib import Path\n", 23 | "\n", 24 | "import pandas as pd\n", 25 | "import torch\n", 26 | "from torch.nn.functional import elu\n", 27 | "import numpy as np\n", 28 | "import matplotlib.pyplot as plt\n", 29 | "\n", 30 | "from alpaca.uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 31 | "from alpaca.model.ensemble import MLPEnsemble\n", 32 | "from alpaca.uncertainty_estimator import build_estimator\n", 33 | "from alpaca.analysis.metrics import get_uq_metrics\n", 34 | "\n", 35 | "plt.rcParams['figure.facecolor'] = 'white'\n" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [ 44 | "SEED = 10 \n", 45 | "torch.manual_seed(SEED)\n", 46 | "np.random.seed(SEED)\n", 47 | "random.seed(SEED)\n", 48 | "\n", 49 | "torch.cuda.set_device(0)\n", 50 | "torch.backends.cudnn.deterministic = True\n", 51 | "torch.backends.cudnn.benchmark = False\n" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [ 60 | "def construct_estimator(model, model_type, name):\n", 61 | " if model_type == 'mask': \n", 62 | " mask = masks[name]\n", 63 | " msk = build_estimator(\n", 64 | " 'mcdue_masked', model, nn_runs=config['nn_runs'], dropout_mask=mask,\n", 65 | " dropout_rate=config['dropout_uq'])\n", 66 | " msk.tol_level=1e-5\n", 67 | " return msk\n", 68 | " elif model_type == 'emask': \n", 69 | " mask = emasks[name]\n", 70 | " msk = build_estimator(\n", 71 | " 'emcdue_masked', model, nn_runs=config['nn_runs'], dropout_mask=mask,\n", 72 | " dropout_rate=config['dropout_uq'])\n", 73 | " msk.tol_level=1e-5\n", 74 | " return msk\n", 75 | " else:\n", 76 | " return build_estimator(name, model)\n", 77 | "\n", 78 | "\n", 79 | "class Evaluator: \n", 80 | " def __init__(self, x_test, y_test, y_scaler, tag='standard'):\n", 81 | " self.x_test = torch.DoubleTensor(x_test).cuda()\n", 82 | " self.y_test = y_test\n", 83 | " self.unscale = lambda y : y_scaler.inverse_transform(y) \n", 84 | " self.tag = tag\n", 85 | " self.results = []\n", 86 | "\n", 87 | " def bench(self, model, name, model_type='mask'): \n", 88 | " predictions = model(self.x_test).cpu().detach().numpy()\n", 89 | " \n", 90 | " errors = np.abs(predictions - self.y_test)\n", 91 | " \n", 92 | " scaled_errors = self.unscale(predictions) - self.unscale(self.y_test)\n", 93 | " rmse = np.sqrt(np.mean(np.square(scaled_errors)))\n", 94 | "\n", 95 | " estimator = construct_estimator(model, model_type, name)\n", 96 | " if model_type == 'emask':\n", 97 | " name = 'e_' + name\n", 98 | " \n", 99 | " for run in range(config['n_ue_runs']):\n", 100 | " estimations = estimator.estimate(self.x_test)\n", 101 | " acc, ndcg, ll = get_uq_metrics(estimations, errors, \n", 102 | " config['acc_percentile'],\n", 103 | " bins = [80, 95, 99]\n", 104 | " )\n", 105 | " self.results.append([acc, ndcg, ll, rmse, name, self.tag])\n", 106 | " if hasattr(estimator, 'reset'):\n", 107 | " estimator.reset()" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "folder = Path('./data/regression')\n", 117 | "files = [file for file in os.listdir(folder) if file.endswith('.pickle')]" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "data = []\n", 127 | "for cnt, file in enumerate(files[:1]):\n", 128 | " with open(folder / 'log_exp.log', 'w') as f:\n", 129 | " f.write(f'{cnt} / {len(files)}')\n", 130 | " with open(folder / file, 'rb') as f:\n", 131 | " dct = pickle.load(f)\n", 132 | " print(file)\n", 133 | " config = dct['config']\n", 134 | " config['n_ue_runs'] = 1\n", 135 | " config['acc_percentile'] = .1\n", 136 | " state_dict = dct['state_dict']\n", 137 | " x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data']\n", 138 | "\n", 139 | " model = MLPEnsemble(\n", 140 | " config['layers'], n_models=config['n_ens'], activation = elu,\n", 141 | " reduction='mean')\n", 142 | " model.load_state_dict(state_dict)\n", 143 | "\n", 144 | " standard_evaluator = Evaluator(x_val, y_val, y_scaler, 'standard')\n", 145 | " masks = build_masks(DEFAULT_MASKS)\n", 146 | " emasks = []\n", 147 | " for i in range(config['n_ens']):\n", 148 | " msk = build_masks(DEFAULT_MASKS)\n", 149 | " emasks.append(msk)\n", 150 | " emasks = {key: [e[key] for e in emasks] for key in masks.keys()}\n", 151 | "\n", 152 | "\n", 153 | " #\n", 154 | " single_model = model.models[2]\n", 155 | " # for name in masks:\n", 156 | " # print(name, end = '|')\n", 157 | " # standard_evaluator.bench(single_model, name, 'mask')\n", 158 | " # standard_evaluator.bench(model, 'eue', 'ensemble')\n", 159 | " # for name in emasks:\n", 160 | " # print(name, end = '*|')\n", 161 | " # standard_evaluator.bench(model, name, 'emask')\n", 162 | " # mask_df = pd.DataFrame(standard_evaluator.results,\n", 163 | " # columns=['Acc', 'NDCG', 'LL',\n", 164 | " # 'RMSE', 'Mask', 'Tag'])\n", 165 | " # mask_df['fname'] = file\n", 166 | " # data.append(mask_df)\n", 167 | " # pd.concat(data).to_csv(folder/'experiment_results.csv', index = None)\n" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "'finished'" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "outputs": [], 183 | "source": [ 184 | "\n" 185 | ], 186 | "metadata": { 187 | "collapsed": false, 188 | "pycharm": { 189 | "name": "#%%\n" 190 | } 191 | } 192 | } 193 | ], 194 | "metadata": { 195 | "kernelspec": { 196 | "display_name": "Python 3", 197 | "language": "python", 198 | "name": "python3" 199 | }, 200 | "language_info": { 201 | "codemirror_mode": { 202 | "name": "ipython", 203 | "version": 3 204 | }, 205 | "file_extension": ".py", 206 | "mimetype": "text/x-python", 207 | "name": "python", 208 | "nbconvert_exporter": "python", 209 | "pygments_lexer": "ipython3", 210 | "version": "3.6.7" 211 | } 212 | }, 213 | "nbformat": 4, 214 | "nbformat_minor": 2 215 | } -------------------------------------------------------------------------------- /experiments/deprecated/regression_3_dolan-more.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "outputs": [], 7 | "source": [ 8 | "# Let's build Dolan-More curves, see:\n", 9 | "# http://abelsiqueira.github.io/blog/introduction-to-performance-profile/" 10 | ], 11 | "metadata": { 12 | "collapsed": false, 13 | "pycharm": { 14 | "name": "#%%\n", 15 | "is_executing": false 16 | } 17 | } 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": null, 22 | "metadata": { 23 | "pycharm": { 24 | "is_executing": false 25 | } 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "from pathlib import Path\n", 30 | "import random\n", 31 | "\n", 32 | "import seaborn as sns\n", 33 | "import pandas as pd\n", 34 | "import numpy as np\n", 35 | "import matplotlib.pyplot as plt\n", 36 | "import torch\n", 37 | "%matplotlib inline" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "outputs": [], 44 | "source": [ 45 | "SEED = 10\n", 46 | "torch.manual_seed(SEED)\n", 47 | "np.random.seed(SEED)\n", 48 | "random.seed(SEED)\n", 49 | "\n", 50 | "torch.cuda.set_device(0)\n", 51 | "torch.backends.cudnn.deterministic = True\n", 52 | "torch.backends.cudnn.benchmark = False\n" 53 | ], 54 | "metadata": { 55 | "collapsed": false, 56 | "pycharm": { 57 | "name": "#%%\n", 58 | "is_executing": false 59 | } 60 | } 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "pycharm": { 67 | "is_executing": false 68 | } 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "folder = Path('./data/regression')\n", 73 | "df = pd.read_csv(folder / 'experiment_results.csv').drop_duplicates()" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": { 80 | "pycharm": { 81 | "is_executing": false 82 | } 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "emasks = []\n", 87 | "masks = []\n", 88 | "for msk in df.Mask.unique():\n", 89 | " if msk[:2] == 'e_':\n", 90 | " emasks.append(msk)\n", 91 | " elif msk != 'eue':\n", 92 | " masks.append(msk)\n", 93 | "print(masks)\n", 94 | "print(emasks)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": { 101 | "pycharm": { 102 | "is_executing": false 103 | } 104 | }, 105 | "outputs": [], 106 | "source": [ 107 | "dsets = []\n", 108 | "for fname in df.fname:\n", 109 | " dsets.append(fname.split('_')[0])\n", 110 | "df['dset'] = dsets" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": { 117 | "pycharm": { 118 | "is_executing": false 119 | } 120 | }, 121 | "outputs": [], 122 | "source": [] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "outputs": [], 128 | "source": [ 129 | "# %%time\n", 130 | "large_data = []\n", 131 | "metric = 'Acc'\n", 132 | "for fname, df_temp in df.groupby('fname'):\n", 133 | " data = []\n", 134 | " assert len(df_temp[df_temp.Mask == 'eue']) == 1\n", 135 | " eue_performance = df_temp[df_temp.Mask == 'eue'][metric].values[0]\n", 136 | " dct_single = {'eue': eue_performance}\n", 137 | " for mask in masks:\n", 138 | " perf = df_temp[df_temp.Mask == mask][metric].values[0]\n", 139 | " dct_single[mask] = perf\n", 140 | " dct_single = {k: 1./v for k, v in dct_single.items()}\n", 141 | " data.append(dct_single)\n", 142 | " \n", 143 | " df_res = pd.DataFrame(data)\n", 144 | " normalized_values = df_res.values/df_res.min(axis = 1).values[:,None]\n", 145 | " df_res = pd.DataFrame(normalized_values, columns = df_res.columns)\n", 146 | " large_data.append(df_res)" 147 | ], 148 | "metadata": { 149 | "collapsed": false, 150 | "pycharm": { 151 | "name": "#%%\n", 152 | "is_executing": false 153 | } 154 | } 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "metadata": { 160 | "pycharm": { 161 | "is_executing": false 162 | } 163 | }, 164 | "outputs": [], 165 | "source": [ 166 | "df_res = pd.concat(large_data)" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": { 173 | "pycharm": { 174 | "is_executing": false 175 | } 176 | }, 177 | "outputs": [], 178 | "source": [ 179 | "def make_dm_plot(df_res, masks, emasks,\n", 180 | " metric = 'some metric', \n", 181 | " exper = 'some exper',\n", 182 | " taus = np.linspace(1, 3, 101),\n", 183 | " plotter = plt.plot,\n", 184 | " **kwargs\n", 185 | " ):\n", 186 | " npp = len(df_res)\n", 187 | " for msk in masks:\n", 188 | " if 'eue' == msk:\n", 189 | " continue\n", 190 | " pas = []\n", 191 | " for tau in taus:\n", 192 | " pas.append(sum(df_res[msk].values <= tau)/npp)\n", 193 | " print(f'{msk}, {pas[0]:.2f}')\n", 194 | " plotter(taus, pas, label = msk,**kwargs)\n", 195 | " for msk in emasks:\n", 196 | " if 'eue' == msk:\n", 197 | " continue\n", 198 | " pas = []\n", 199 | " for tau in taus:\n", 200 | " pas.append(sum(df_res[msk].values <= tau)/npp)\n", 201 | " print(f'{msk}, {pas[0]:.2f}')\n", 202 | " plotter(taus, pas, label = msk,**kwargs)\n", 203 | " pas = []\n", 204 | " for tau in taus:\n", 205 | " pas.append(sum(df_res['eue'].values <= tau)/npp)\n", 206 | " print(f'{msk}, {pas[0]:.2f}')\n", 207 | " plotter(taus, pas, 'k', label = 'eue',**kwargs)\n", 208 | "\n", 209 | " plt.legend()\n", 210 | " plt.title(metric + '|' + exper)" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": null, 216 | "metadata": { 217 | "pycharm": { 218 | "is_executing": false 219 | } 220 | }, 221 | "outputs": [], 222 | "source": [ 223 | "plt.figure(figsize=(8,6))\n", 224 | "masks = ['mc_dropout', 'decorrelating_sc', 'dpp', 'k_dpp', \n", 225 | " 'eue']\n", 226 | "make_dm_plot(df_res[masks], masks, [], metric, f' ', \n", 227 | " taus = np.linspace(1, 3, 101),lw=3, alpha=.8)\n", 228 | "plt.grid()\n", 229 | "plt.title('Accuracy for single NN UE')\n", 230 | "plt.legend(['MC dropout',\n", 231 | " 'decorrelation',\n", 232 | " 'dpp', 'k_dpp', 'ensemble'\n", 233 | " ])\n", 234 | "plt.xlabel(r'$\\tau$')\n", 235 | "plt.ylabel(r'$\\rho_a(\\tau$)')\n", 236 | "plt.xlim(1, 3)\n", 237 | "plt.savefig('../figures/dolan_acc_single.png', dpi = 600)\n" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": null, 243 | "metadata": { 244 | "pycharm": { 245 | "is_executing": false 246 | } 247 | }, 248 | "outputs": [], 249 | "source": [ 250 | "import seaborn as sns\n", 251 | "cnt = 1\n", 252 | "names_as_in_paper = [\n", 253 | " 'boston', 'concrete', 'energy', 'kin8nm',\n", 254 | " 'naval', 'ccpp', 'naval', 'red wine', 'yacht'\n", 255 | "]\n", 256 | "\n", 257 | "\n", 258 | "plt.figure(figsize=(10,20))\n", 259 | "for dset in df.dset.unique():\n", 260 | " plt.subplot(4,2,cnt)\n", 261 | " dfx = df[(df.dset == dset)]\n", 262 | " g=sns.boxplot(x = 'LL', y = 'Mask', data = dfx)#, hue = 'dset')\n", 263 | " if cnt % 2 == 0:\n", 264 | " g.set(yticklabels=[])\n", 265 | " plt.grid()\n", 266 | " plt.title(names_as_in_paper[cnt-1])\n", 267 | " cnt += 1\n", 268 | "plt.tight_layout()\n" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": { 275 | "pycharm": { 276 | "is_executing": false 277 | } 278 | }, 279 | "outputs": [], 280 | "source": [ 281 | "import seaborn as sns\n", 282 | "cnt = 1\n", 283 | "names_as_in_paper = [\n", 284 | " 'boston', 'concrete', 'energy', 'kin8nm',\n", 285 | " 'naval', 'ccpp', 'naval', 'red wine', 'yacht'\n", 286 | "]\n", 287 | "\n", 288 | "plt.figure(figsize=(10,12))\n", 289 | "for dset in df.dset.unique():\n", 290 | " plt.subplot(4,2,cnt)\n", 291 | " dfx = df[(df.dset == dset) & (df.Mask.isin(masks))]\n", 292 | " g=sns.boxplot(x = 'LL', y = 'Mask', data = dfx)\n", 293 | " print(dset)\n", 294 | " g.set(ylabel='')\n", 295 | " if cnt % 2 == 0:\n", 296 | " g.set(yticklabels=[])\n", 297 | " else:\n", 298 | " g.set(yticklabels=['MC dropout', 'decorrelation', 'dpp', 'k_dpp', 'eue'])\n", 299 | " plt.grid()\n", 300 | " plt.title(names_as_in_paper[cnt-1])\n", 301 | " cnt += 1\n", 302 | "plt.tight_layout()\n", 303 | "plt.savefig('../figures/LL_UCI_single.png', dpi = 600)" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": null, 309 | "outputs": [], 310 | "source": [ 311 | "cnt = 1\n", 312 | "names_as_in_paper = [\n", 313 | " 'boston', 'concrete', 'energy', 'kin8nm',\n", 314 | " 'naval', 'ccpp', 'naval', 'red wine', 'yacht'\n", 315 | "]\n", 316 | "\n", 317 | "plt.figure(figsize=(10,12))\n", 318 | "\n", 319 | "for dset in df.dset.unique():\n", 320 | " plt.subplot(4,2,cnt)\n", 321 | " dfx = df[(df.dset == dset) & (df.Mask.isin(emasks))]\n", 322 | " g=sns.boxplot(x = 'LL', y = 'Mask', data = dfx)\n", 323 | " g.set(ylabel='')\n", 324 | " if cnt % 2 == 0:\n", 325 | " g.set(yticklabels=[])\n", 326 | " else:\n", 327 | " g.set(yticklabels=['MC dropout', 'decorrelation', 'dpp', 'k_dpp'])\n", 328 | " plt.grid()\n", 329 | " plt.title(names_as_in_paper[cnt-1])\n", 330 | " cnt += 1\n", 331 | "plt.tight_layout()\n", 332 | "plt.savefig('../figures/LL_UCI_ens.png', dpi = 600)" 333 | ], 334 | "metadata": { 335 | "collapsed": false, 336 | "pycharm": { 337 | "name": "#%%\n", 338 | "is_executing": false 339 | } 340 | } 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": null, 345 | "outputs": [], 346 | "source": [ 347 | "\n" 348 | ], 349 | "metadata": { 350 | "collapsed": false, 351 | "pycharm": { 352 | "name": "#%%\n" 353 | } 354 | } 355 | } 356 | ], 357 | "metadata": { 358 | "kernelspec": { 359 | "display_name": "Python 3", 360 | "language": "python", 361 | "name": "python3" 362 | }, 363 | "language_info": { 364 | "codemirror_mode": { 365 | "name": "ipython", 366 | "version": 3 367 | }, 368 | "file_extension": ".py", 369 | "mimetype": "text/x-python", 370 | "name": "python", 371 | "nbconvert_exporter": "python", 372 | "pygments_lexer": "ipython3", 373 | "version": "3.6.7" 374 | }, 375 | "pycharm": { 376 | "stem_cell": { 377 | "cell_type": "raw", 378 | "source": [], 379 | "metadata": { 380 | "collapsed": false 381 | } 382 | } 383 | } 384 | }, 385 | "nbformat": 4, 386 | "nbformat_minor": 2 387 | } -------------------------------------------------------------------------------- /experiments/deprecated/regression_dm_produce_results_from_models.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import sys\n", 10 | "sys.path.append('..')\n", 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "sys.path = ['/home/etsymbalov/.local/lib/python3.6/site-packages'] + sys.path" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": 3, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "import numpy as np\n", 31 | "import matplotlib.pyplot as plt\n", 32 | "\n", 33 | "from experiment_setup import get_model, set_random, build_estimator\n", 34 | "from uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 35 | "from dataloader.toy import ToyQubicData, ToySinData\n", 36 | "from model.mlp import MLP\n", 37 | "from model.ensemble import MLPEnsemble\n", 38 | "import pickle\n", 39 | "\n", 40 | "plt.rcParams['figure.facecolor'] = 'white'" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 4, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "from dataloader.builder import build_dataset\n", 50 | "from model.mlp import MLP\n", 51 | "from uncertainty_estimator.masks import build_masks \n", 52 | "from experiment_setup import build_estimator\n", 53 | "from analysis.metrics import get_uq_metrics\n", 54 | "from experiments.utils.data import scale, split_ood, multiple_kfold" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 8, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "import os\n", 64 | "import pickle\n", 65 | "import pandas as pd" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": 15, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "def construct_estimator(model, model_type, name):\n", 75 | " if model_type == 'mask': \n", 76 | " mask = masks[name]\n", 77 | " return build_estimator(\n", 78 | " 'mcdue_masked', model, nn_runs=config['nn_runs'], dropout_mask=mask,\n", 79 | " dropout_rate=config['dropout_uq'])\n", 80 | " elif model_type == 'emask': \n", 81 | " mask = emasks[name]\n", 82 | " return build_estimator(\n", 83 | " 'emcdue_masked', model, nn_runs=config['nn_runs'], dropout_mask=mask,\n", 84 | " dropout_rate=config['dropout_uq'])\n", 85 | " else:\n", 86 | " return build_estimator(name, model)\n", 87 | "\n", 88 | "\n", 89 | "class Evaluator: \n", 90 | " def __init__(self, x_test, y_test, y_scaler, tag='standard'):\n", 91 | " self.x_test = x_test\n", 92 | " self.y_test = y_test\n", 93 | " self.unscale = lambda y : y_scaler.inverse_transform(y) \n", 94 | " self.tag = tag\n", 95 | " self.results = []\n", 96 | "\n", 97 | " def bench(self, model, name, model_type='mask'): \n", 98 | " predictions = model(self.x_test).cpu().numpy()\n", 99 | " \n", 100 | " errors = np.abs(predictions - self.y_test)\n", 101 | " \n", 102 | " scaled_errors = self.unscale(predictions) - self.unscale(self.y_test)\n", 103 | " rmse = np.sqrt(np.mean(np.square(scaled_errors)))\n", 104 | "\n", 105 | " estimator = construct_estimator(model, model_type, name)\n", 106 | " if model_type == 'emask':\n", 107 | " name = 'e_' + name\n", 108 | " \n", 109 | " for run in range(config['n_ue_runs']):\n", 110 | " estimations = estimator.estimate(self.x_test)\n", 111 | " acc, ndcg, ll = get_uq_metrics(estimations, errors, \n", 112 | " config['acc_percentile'],\n", 113 | " bins = [80, 95, 99]\n", 114 | " )\n", 115 | " self.results.append([acc, ndcg, ll, rmse, name, self.tag])\n", 116 | " if hasattr(estimator, 'reset'):\n", 117 | " estimator.reset()" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 16, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "folder = './exp1/'\n", 127 | "files = os.listdir(folder)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "data = []\n", 137 | "for cnt, file in enumerate(files):\n", 138 | " with open('log_exp.log', 'w') as f:\n", 139 | " f.write(f'{cnt} / {len(files)}')\n", 140 | " with open(folder + file, 'rb') as f:\n", 141 | " dct = pickle.load(f)\n", 142 | "# print(file)\n", 143 | " config = dct['config']\n", 144 | " config['n_ue_runs'] = 1\n", 145 | " config['acc_percentile'] = .1\n", 146 | " state_dict = dct['state_dict']\n", 147 | " x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data']\n", 148 | " \n", 149 | " model = MLPEnsemble(config['layers'], \n", 150 | " n_models=config['n_ens'], \n", 151 | " reduction='mean')\n", 152 | " model.load_state_dict(state_dict)\n", 153 | " \n", 154 | " standard_evaluator = Evaluator(x_val, y_val, y_scaler, 'standard')\n", 155 | " masks = build_masks(DEFAULT_MASKS)\n", 156 | " emasks = [build_masks(DEFAULT_MASKS) for i in range(config['n_ens'])]\n", 157 | " emasks = {key: [e[key] for e in emasks] for key in masks.keys()}\n", 158 | "\n", 159 | " single_model = model.models[2]\n", 160 | " for name in masks: \n", 161 | "# print(name, end = '|')\n", 162 | " standard_evaluator.bench(single_model, name, 'mask')\n", 163 | " standard_evaluator.bench(model, 'eue', 'ensemble') \n", 164 | " for name in emasks: \n", 165 | "# print(name, end = '*|')\n", 166 | " standard_evaluator.bench(model, name, 'emask')\n", 167 | " mask_df = pd.DataFrame(standard_evaluator.results, \n", 168 | " columns=['Acc', 'NDCG', 'LL', 'RMSE', 'Mask', 'Tag'])\n", 169 | " mask_df['fname'] = file\n", 170 | " data.append(mask_df)\n", 171 | " pd.concat(data).to_csv('exper1_results.csv', index = None)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": null, 177 | "metadata": {}, 178 | "outputs": [], 179 | "source": [] 180 | } 181 | ], 182 | "metadata": { 183 | "kernelspec": { 184 | "display_name": "Python 3", 185 | "language": "python", 186 | "name": "python3" 187 | }, 188 | "language_info": { 189 | "codemirror_mode": { 190 | "name": "ipython", 191 | "version": 3 192 | }, 193 | "file_extension": ".py", 194 | "mimetype": "text/x-python", 195 | "name": "python", 196 | "nbconvert_exporter": "python", 197 | "pygments_lexer": "ipython3", 198 | "version": "3.6.7" 199 | } 200 | }, 201 | "nbformat": 4, 202 | "nbformat_minor": 2 203 | } 204 | -------------------------------------------------------------------------------- /experiments/deprecated/regression_visual.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false, 8 | "jupyter": { 9 | "outputs_hidden": false 10 | }, 11 | "pycharm": { 12 | "is_executing": false, 13 | "name": "#%%\n" 14 | } 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "import sys\n", 19 | "sys.path.append('..')\n", 20 | "%load_ext autoreload\n", 21 | "%autoreload 2" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": null, 27 | "metadata": { 28 | "collapsed": false, 29 | "jupyter": { 30 | "outputs_hidden": false 31 | }, 32 | "pycharm": { 33 | "is_executing": false, 34 | "name": "#%%\n" 35 | } 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "from itertools import product\n", 40 | "\n", 41 | "import numpy as np\n", 42 | "import matplotlib.pyplot as plt\n", 43 | "import seaborn as sns\n", 44 | "\n", 45 | "\n", 46 | "from dataloader.builder import build_dataset\n", 47 | "from model.mlp import MLP\n", 48 | "from uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 49 | "from experiment_setup import build_estimator, get_model\n", 50 | "from experiments.utils.data import scale, split_ood, multiple_kfold\n", 51 | "import torch\n", 52 | "\n", 53 | "from analysis.autoencoder import AutoEncoder \n", 54 | "\n", 55 | "plt.rcParams['figure.facecolor'] = 'white'\n", 56 | "torch.cuda.set_device(1)\n" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": { 63 | "collapsed": false, 64 | "jupyter": { 65 | "outputs_hidden": false 66 | }, 67 | "pycharm": { 68 | "is_executing": false, 69 | "name": "#%%\n" 70 | } 71 | }, 72 | "outputs": [], 73 | "source": [ 74 | "config = {\n", 75 | " 'nn_runs': 100,\n", 76 | " 'runs': 2,\n", 77 | " 'max_runs': 20,\n", 78 | " 'k_folds': 10,\n", 79 | " 'verbose': False,\n", 80 | " 'layers': [8, 256, 256, 128, 1],\n", 81 | " 'epochs': 10_000,\n", 82 | " 'validation_step': 50,\n", 83 | " 'acc_percentile': 0.1,\n", 84 | " 'patience': 3,\n", 85 | " 'dropout_rate': 0.2,\n", 86 | " 'dropout_uq': 0.5,\n", 87 | " 'batch_size': 256,\n", 88 | " # 'dataset': 'kin8nm',\n", 89 | " 'dataset': 'naval_propulsion',\n", 90 | " 'ood_percentile': 90,\n", 91 | " 'with_ensembles': True,\n", 92 | " 'optimizer': {'type': 'Adadelta', 'weight_decay':1e-3}\n", 93 | "}" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": { 100 | "collapsed": false, 101 | "jupyter": { 102 | "outputs_hidden": false 103 | }, 104 | "pycharm": { 105 | "is_executing": false, 106 | "name": "#%%\n" 107 | } 108 | }, 109 | "outputs": [], 110 | "source": [ 111 | "# Load dataset\n", 112 | "dataset = build_dataset(config['dataset'])\n", 113 | "\n", 114 | "x_train, y_train = dataset.dataset('train')\n", 115 | "x_val, y_val = dataset.dataset('val')\n", 116 | "config['layers'][0] = x_train.shape[-1]\n" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": { 123 | "collapsed": false, 124 | "jupyter": { 125 | "outputs_hidden": false 126 | }, 127 | "pycharm": { 128 | "is_executing": false, 129 | "name": "#%%\n" 130 | } 131 | }, 132 | "outputs": [], 133 | "source": [ 134 | "x_train, x_val, x_scaler = scale(x_train, x_val)\n", 135 | "y_train, y_val, y_scaler = scale(y_train, y_val)" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "metadata": { 142 | "collapsed": false, 143 | "jupyter": { 144 | "outputs_hidden": false 145 | }, 146 | "pycharm": { 147 | "is_executing": false, 148 | "name": "#%%\n" 149 | } 150 | }, 151 | "outputs": [], 152 | "source": [ 153 | "autoencoder = AutoEncoder(config['layers'][0], 128, 2, lr=1e-2)\n", 154 | "autoencoder.train()\n", 155 | "for e in range(500):\n", 156 | " loss = autoencoder.fit(x_train)\n", 157 | " if (e+1) % 5 == 0:\n", 158 | " print(e+1, loss)\n" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": { 165 | "collapsed": false, 166 | "jupyter": { 167 | "outputs_hidden": false 168 | }, 169 | "pycharm": { 170 | "is_executing": false, 171 | "name": "#%%\n" 172 | } 173 | }, 174 | "outputs": [], 175 | "source": [ 176 | "autoencoder.eval()\n", 177 | "def encode(x):\n", 178 | " samples = torch.DoubleTensor(x).to('cuda')\n", 179 | " encoded = autoencoder.encode(samples)\n", 180 | " return encoded.cpu().detach().numpy()\n", 181 | "\n", 182 | "def decode(x):\n", 183 | " samples = torch.DoubleTensor(x).to('cuda')\n", 184 | " encoded = autoencoder.decode(samples)\n", 185 | " return encoded.cpu().detach().numpy()\n" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "metadata": { 192 | "collapsed": false, 193 | "jupyter": { 194 | "outputs_hidden": false 195 | }, 196 | "pycharm": { 197 | "is_executing": false, 198 | "name": "#%%\n" 199 | } 200 | }, 201 | "outputs": [], 202 | "source": [ 203 | "encoded_train = encode(x_train)\n", 204 | "plt.figure(figsize=(12, 10))\n", 205 | "sns.scatterplot(\n", 206 | " x=encoded_train[:, 0], y=encoded_train[:, 1], hue=y_train.squeeze(),\n", 207 | " palette=\"Accent\")\n" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "metadata": { 214 | "collapsed": false, 215 | "jupyter": { 216 | "outputs_hidden": false 217 | }, 218 | "pycharm": { 219 | "is_executing": false, 220 | "name": "#%%\n" 221 | } 222 | }, 223 | "outputs": [], 224 | "source": [ 225 | "# Train or load model\n", 226 | "model = MLP(config['layers'], optimizer=config['optimizer'])\n", 227 | "model_path = f\"experiments/data/model_{config['dataset']}.ckpt\"\n", 228 | "model = get_model(model, model_path, (x_train, y_train), (x_val, y_val))\n" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": null, 234 | "metadata": { 235 | "collapsed": false, 236 | "jupyter": { 237 | "outputs_hidden": false 238 | }, 239 | "pycharm": { 240 | "is_executing": false, 241 | "name": "#%%\n" 242 | } 243 | }, 244 | "outputs": [], 245 | "source": [ 246 | "ngridx = 150\n", 247 | "ngridy = 150\n", 248 | "x = encoded_train[:, 0]\n", 249 | "y = encoded_train[:, 1]\n", 250 | "\n", 251 | "x1, x2 = 3*min(x), 3*max(x)\n", 252 | "y1, y2 = 3*min(y), 3*max(y)\n", 253 | "\n", 254 | "xi = np.linspace(x1, x2, ngridx)\n", 255 | "yi = np.linspace(y1, y2, ngridy)\n", 256 | "\n", 257 | "# Countour coord; for some reason they are ortogonal to usual coord in pyplot\n", 258 | "points = np.array(list(product(yi, xi)))\n", 259 | "x_grid = decode(points)\n", 260 | "x_grid.shape" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": null, 266 | "metadata": { 267 | "collapsed": false, 268 | "jupyter": { 269 | "outputs_hidden": false 270 | }, 271 | "pycharm": { 272 | "is_executing": false, 273 | "name": "#%%\n" 274 | } 275 | }, 276 | "outputs": [], 277 | "source": [ 278 | "\n", 279 | "# UE\n", 280 | "masks = build_masks(DEFAULT_MASKS)\n", 281 | "for name, mask in masks.items():\n", 282 | " estimator = build_estimator('mcdue_masked', model, dropout_mask=mask)\n", 283 | " estimations = estimator.estimate(x_grid)\n", 284 | " zi = estimations.reshape((ngridx, ngridy))\n", 285 | "\n", 286 | " fig, ax1 = plt.subplots(figsize=(16, 12))\n", 287 | " ax1.contour(xi, yi, zi, levels=14, linewidths=0.5, colors='k')\n", 288 | " cntr1 = ax1.contourf(xi, yi, zi, levels=14, cmap=\"gray\")\n", 289 | " fig.colorbar(cntr1, ax=ax1)\n", 290 | "\n", 291 | " ax1.scatter(x, y, c=y_train.squeeze(), alpha=0.5)\n", 292 | " ax1.set(xlim=(x1, x2), ylim=(y1, y2))\n", 293 | " ax1.set_title('%s grid and contour (%d points, %d grid points)' %\n", 294 | " (name, len(x), ngridx * ngridy))" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": null, 300 | "metadata": { 301 | "collapsed": false, 302 | "jupyter": { 303 | "outputs_hidden": false 304 | }, 305 | "pycharm": { 306 | "is_executing": false, 307 | "name": "#%%\n" 308 | } 309 | }, 310 | "outputs": [], 311 | "source": [ 312 | "decode(np.array([[10, 0.7]]))\n" 313 | ] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "execution_count": null, 318 | "metadata": { 319 | "collapsed": false, 320 | "jupyter": { 321 | "outputs_hidden": false 322 | }, 323 | "pycharm": { 324 | "is_executing": false, 325 | "name": "#%%\n" 326 | } 327 | }, 328 | "outputs": [], 329 | "source": [ 330 | "\n" 331 | ] 332 | } 333 | ], 334 | "metadata": { 335 | "kernelspec": { 336 | "display_name": "Python 3", 337 | "language": "python", 338 | "name": "python3" 339 | }, 340 | "language_info": { 341 | "codemirror_mode": { 342 | "name": "ipython", 343 | "version": 3 344 | }, 345 | "file_extension": ".py", 346 | "mimetype": "text/x-python", 347 | "name": "python", 348 | "nbconvert_exporter": "python", 349 | "pygments_lexer": "ipython3", 350 | "version": "3.7.3" 351 | }, 352 | "pycharm": { 353 | "stem_cell": { 354 | "cell_type": "raw", 355 | "source": [], 356 | "metadata": { 357 | "collapsed": false 358 | } 359 | } 360 | } 361 | }, 362 | "nbformat": 4, 363 | "nbformat_minor": 4 364 | } -------------------------------------------------------------------------------- /experiments/deprecated/uq_proc (2).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import sys\n", 10 | "sys.path = ['/home/etsymbalov/.local/lib/python3.6/site-packages'] + sys.path\n", 11 | "import torch\n", 12 | "from torch.nn.functional import elu" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": null, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "# !python -m pip install torch==1.5.0+cu92 torchvision==0.6.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html --user" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": null, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "torch.__version__" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "# !python -m pip install alpaca-ml --user --upgrade --force\n", 40 | "# я и руками ставил" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "from torch.nn.functional import leaky_relu" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "metadata": {}, 56 | "outputs": [], 57 | "source": [] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "import os\n", 66 | "import pickle\n", 67 | "import random\n", 68 | "from pathlib import Path\n", 69 | "\n", 70 | "import pandas as pd\n", 71 | "# import torch\n", 72 | "# from torch.nn.functional import elu\n", 73 | "import numpy as np\n", 74 | "import matplotlib.pyplot as plt\n", 75 | "\n", 76 | "from alpaca.uncertainty_estimator.masks import build_masks, DEFAULT_MASKS\n", 77 | "from alpaca.model.ensemble import MLPEnsemble\n", 78 | "from alpaca.uncertainty_estimator import build_estimator\n", 79 | "from alpaca.analysis.metrics import get_uq_metrics\n", 80 | "\n", 81 | "plt.rcParams['figure.facecolor'] = 'white'" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "SEED = 10 \n", 91 | "torch.manual_seed(SEED)\n", 92 | "np.random.seed(SEED)\n", 93 | "random.seed(SEED)\n", 94 | "\n", 95 | "torch.cuda.set_device(0)\n", 96 | "torch.backends.cudnn.deterministic = True\n", 97 | "torch.backends.cudnn.benchmark = False" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "def construct_estimator(model, model_type, name):\n", 107 | " if model_type == 'mask': \n", 108 | " mask = masks[name]\n", 109 | " msk = build_estimator(\n", 110 | " 'mcdue_masked', model, nn_runs=config['nn_runs'], dropout_mask=mask,\n", 111 | " dropout_rate=config['dropout_uq'])\n", 112 | " msk.tol_level=1e-5\n", 113 | " return msk\n", 114 | " elif model_type == 'emask': \n", 115 | " mask = emasks[name]\n", 116 | " msk = build_estimator(\n", 117 | " 'emcdue_masked', model, nn_runs=config['nn_runs'], dropout_mask=mask,\n", 118 | " dropout_rate=config['dropout_uq'])\n", 119 | " msk.tol_level=1e-5\n", 120 | " return msk\n", 121 | " else:\n", 122 | " return build_estimator(name, model)\n", 123 | "\n", 124 | "\n", 125 | "class Evaluator: \n", 126 | " def __init__(self, x_test, y_test, y_scaler, tag='standard'):\n", 127 | " self.x_test = torch.DoubleTensor(x_test).cuda()\n", 128 | " self.y_test = y_test\n", 129 | " self.unscale = lambda y : y_scaler.inverse_transform(y) \n", 130 | " self.tag = tag\n", 131 | " self.results = []\n", 132 | "\n", 133 | " def bench(self, model, name, model_type='mask'): \n", 134 | " predictions = model(self.x_test).cpu().detach().numpy()\n", 135 | " \n", 136 | " errors = np.abs(predictions - self.y_test)\n", 137 | " \n", 138 | " scaled_errors = self.unscale(predictions) - self.unscale(self.y_test)\n", 139 | " rmse = np.sqrt(np.mean(np.square(scaled_errors)))\n", 140 | "\n", 141 | " estimator = construct_estimator(model, model_type, name)\n", 142 | " if model_type == 'emask':\n", 143 | " name = 'e_' + name\n", 144 | " \n", 145 | " for run in range(config['n_ue_runs']):\n", 146 | " estimations = estimator.estimate(self.x_test)\n", 147 | " acc, ndcg, ll = get_uq_metrics(estimations, errors, \n", 148 | " config['acc_percentile'],\n", 149 | " bins = [80, 95, 99]\n", 150 | " )\n", 151 | " self.results.append([acc, ndcg, ll, rmse, name, self.tag])\n", 152 | " if hasattr(estimator, 'reset'):\n", 153 | " estimator.reset()\n" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "# folder = Path('./experiments/exp5_smot')\n", 163 | "folder = Path('data/regression_2')\n", 164 | "files = [file for file in os.listdir(folder) if file.endswith('.pickle')]\n" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": null, 170 | "outputs": [], 171 | "source": [ 172 | "files\n" 173 | ], 174 | "metadata": { 175 | "collapsed": false, 176 | "pycharm": { 177 | "name": "#%%\n" 178 | } 179 | } 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "# with open(folder / file, 'rb') as f:\n", 188 | "# dct = pickle.load(f)\n", 189 | "# with open('./paper_upgrade/A_log_exp.log', 'w') as f:\n", 190 | "# f.write(f'{cnt} / {len(files)}')" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": null, 196 | "metadata": {}, 197 | "outputs": [], 198 | "source": [ 199 | "DEFAULT_MASKS2 = DEFAULT_MASKS + ['decorrelating_sc', 'dpp']#, 'k_dpp']" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": null, 205 | "metadata": {}, 206 | "outputs": [], 207 | "source": [ 208 | "DEFAULT_MASKS" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "data = []\n", 218 | "errs = []\n", 219 | "for cnt, file in enumerate(files[:3]):\n", 220 | "# if cnt < 235:\n", 221 | "# continue\n", 222 | " try:\n", 223 | " with open(folder / file, 'rb') as f:\n", 224 | " dct = pickle.load(f)\n", 225 | " with open('./A_log_exp.log', 'w') as f:\n", 226 | " f.write(f'{cnt} / {len(files)}')\n", 227 | " print(file)\n", 228 | " config = dct['config']\n", 229 | " config['n_ue_runs'] = 1\n", 230 | " config['acc_percentile'] = .1\n", 231 | " state_dict = dct['state_dict']\n", 232 | " x_train, y_train, x_val, y_val, x_scaler, y_scaler = dct['data']\n", 233 | "\n", 234 | " model = MLPEnsemble(\n", 235 | " config['layers'], n_models=config['n_ens'], activation = elu,\n", 236 | " reduction='mean')\n", 237 | " model.load_state_dict(state_dict)\n", 238 | "\n", 239 | " standard_evaluator = Evaluator(x_val, y_val, y_scaler, 'standard')\n", 240 | " masks = build_masks(DEFAULT_MASKS2)\n", 241 | " emasks = []\n", 242 | " for i in range(config['n_ens']):\n", 243 | " msk = build_masks(DEFAULT_MASKS2)\n", 244 | " emasks.append(msk)\n", 245 | " emasks = {key: [e[key] for e in emasks] for key in masks.keys()}\n", 246 | "\n", 247 | " single_model = model.models[2]\n", 248 | " for name in masks: \n", 249 | " print(name, end = '|')\n", 250 | " standard_evaluator.bench(single_model, name, 'mask')\n", 251 | " standard_evaluator.bench(model, 'eue', 'ensemble') \n", 252 | " for name in emasks: \n", 253 | " print(name, end = '*|')\n", 254 | " standard_evaluator.bench(model, name, 'emask')\n", 255 | " mask_df = pd.DataFrame(standard_evaluator.results, \n", 256 | " columns=['Acc', 'NDCG', 'LL',\n", 257 | " 'RMSE', 'Mask', 'Tag'])\n", 258 | " mask_df['fname'] = file\n", 259 | " data.append(mask_df)\n", 260 | " pd.concat(data).to_csv('./AAAA_experiment_results.csv', index = None)\n", 261 | " except Exception as e:\n", 262 | " errs.append([e,cnt,file])" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": null, 268 | "metadata": {}, 269 | "outputs": [], 270 | "source": [ 271 | "pd.concat(data).head(50)" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": null, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [] 280 | } 281 | ], 282 | "metadata": { 283 | "kernelspec": { 284 | "display_name": "Python 3", 285 | "language": "python", 286 | "name": "python3" 287 | }, 288 | "language_info": { 289 | "codemirror_mode": { 290 | "name": "ipython", 291 | "version": 3 292 | }, 293 | "file_extension": ".py", 294 | "mimetype": "text/x-python", 295 | "name": "python", 296 | "nbconvert_exporter": "python", 297 | "pygments_lexer": "ipython3", 298 | "version": "3.6.7" 299 | } 300 | }, 301 | "nbformat": 4, 302 | "nbformat_minor": 2 303 | } -------------------------------------------------------------------------------- /experiments/deprecated/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/experiments/deprecated/utils/__init__.py -------------------------------------------------------------------------------- /experiments/deprecated/utils/data.py: -------------------------------------------------------------------------------- 1 | from sklearn.preprocessing import StandardScaler 2 | import numpy as np 3 | from sklearn.model_selection import KFold 4 | 5 | 6 | def scale(train, val): 7 | scaler = StandardScaler() 8 | scaler.fit(train) 9 | train = scaler.transform(train) 10 | val = scaler.transform(val) 11 | return train, val, scaler 12 | 13 | 14 | def split_ood(x_all, y_all, percentile=10): 15 | threshold = np.percentile(y_all, percentile) 16 | ood_idx = np.argwhere(y_all > threshold)[:, 0] 17 | x_ood, y_ood = x_all[ood_idx], y_all[ood_idx] 18 | train_idx = np.argwhere(y_all <= threshold)[:, 0] 19 | x_train, y_train = x_all[train_idx], y_all[train_idx] 20 | 21 | return x_train, y_train, x_ood, y_ood 22 | 23 | 24 | def multiple_kfold(k, data_size, max_iterations): 25 | kfold = KFold(k) 26 | for i in range(max_iterations): 27 | if i % k == 0: 28 | data_idx = np.random.permutation(data_size) 29 | idx_generator = kfold.split(data_idx) 30 | train_idx, val_idx = next(idx_generator) 31 | yield data_idx[train_idx], data_idx[val_idx] 32 | -------------------------------------------------------------------------------- /experiments/deprecated/utils/fastai.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | from torch.utils.data import Dataset, TensorDataset, DataLoader 4 | import fastai 5 | from fastai.vision import Image 6 | 7 | # For fastai pbar work in notebooks in vscode and pycharm 8 | from fastprogress.fastprogress import force_console_behavior 9 | master_bar, progress_bar = force_console_behavior() 10 | fastai.basic_train.master_bar, fastai.basic_train.progress_bar = master_bar, progress_bar 11 | 12 | 13 | class ImageArrayDS(Dataset): 14 | def __init__(self, images, labels, tfms=None): 15 | self.images = torch.FloatTensor(images) 16 | self.labels = torch.LongTensor(labels) 17 | self.tfms = tfms 18 | 19 | def __getitem__(self, idx): 20 | image = Image(self.images[idx]) 21 | if self.tfms is not None: 22 | image = image.apply_tfms(self.tfms) 23 | return image, self.labels[idx] 24 | 25 | def __len__(self): 26 | return len(self.images) 27 | 28 | 29 | class Inferencer: 30 | def __init__(self, model, batch_size=8192): 31 | self.model = model 32 | self.batch_size = batch_size 33 | 34 | def __call__(self, x, dropout_rate=0.5, dropout_mask=None): 35 | predictions = [] 36 | self.model.eval() 37 | 38 | if isinstance(x, np.ndarray): 39 | x = torch.Tensor(x) 40 | 41 | for batch in DataLoader(TensorDataset(x), batch_size=self.batch_size): 42 | batch = batch[0].cuda() 43 | prediction = self.model(batch, dropout_rate=dropout_rate, dropout_mask=dropout_mask).detach().cpu() #.numpy() 44 | 45 | predictions.append(prediction) 46 | 47 | if predictions: 48 | return torch.cat(predictions) 49 | else: 50 | return torch.Tensor([]) 51 | 52 | def train(self): 53 | self.model.train() 54 | 55 | def eval(self): 56 | self.model.eval() 57 | 58 | -------------------------------------------------------------------------------- /experiments/logs/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore -------------------------------------------------------------------------------- /experiments/models.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch import nn 3 | import torch.nn.functional as F 4 | from torchvision.models.resnet import ResNet, BasicBlock, conv3x3, conv1x1, Bottleneck 5 | from torch.hub import load_state_dict_from_url 6 | from alpaca.model.cnn import SimpleConv 7 | 8 | model_urls = { 9 | 'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth', 10 | } 11 | 12 | class StrongConv(nn.Module): 13 | def __init__(self, dropout_rate=0.5): 14 | super().__init__() 15 | base = 16 16 | self.conv = nn.Sequential( 17 | nn.Conv2d(3, base, 3, padding=1, bias=False), 18 | nn.BatchNorm2d(base), 19 | nn.CELU(), 20 | nn.Conv2d(base, base, 3, padding=1, bias=False), 21 | nn.CELU(), 22 | nn.MaxPool2d(2, 2), 23 | nn.Dropout2d(0.2), 24 | nn.Conv2d(base, 2*base, 3, padding=1, bias=False), 25 | nn.BatchNorm2d(2*base), 26 | nn.CELU(), 27 | nn.Conv2d(2 * base, 2 * base, 3, padding=1, bias=False), 28 | nn.CELU(), 29 | nn.Dropout2d(0.3), 30 | nn.MaxPool2d(2, 2), 31 | nn.Conv2d(2*base, 4*base, 3, padding=1, bias=False), 32 | nn.BatchNorm2d(4*base), 33 | nn.CELU(), 34 | nn.Conv2d(4*base, 4*base, 3, padding=1, bias=False), 35 | nn.CELU(), 36 | nn.Dropout2d(0.4), 37 | nn.MaxPool2d(2, 2), 38 | ) 39 | self.linear_size = 8 * 8 * base 40 | self.linear = nn.Sequential( 41 | nn.Linear(self.linear_size, 8*base), 42 | nn.CELU(), 43 | ) 44 | self.dropout = nn.Dropout(0.5) 45 | self.fc = nn.Linear(8*base, 10) 46 | 47 | def forward(self, x, dropout_rate=0.5, dropout_mask=None): 48 | x = self.conv(x) 49 | x = x.reshape(-1, self.linear_size) 50 | x = self.linear(x) 51 | if dropout_mask is None: 52 | x = self.dropout(x) 53 | else: 54 | x = x * dropout_mask(x, dropout_rate, 0) 55 | return self.fc(x) 56 | 57 | 58 | class ResNetMasked(ResNet): 59 | def forward(self, x, dropout_rate=0.3, dropout_mask=None, make_conv=True): 60 | if make_conv: 61 | x = self.conv1(x) 62 | x = self.bn1(x) 63 | x = self.relu(x) 64 | x = self.maxpool(x) 65 | 66 | x = self.layer1(x) 67 | x = self.layer2(x) 68 | x = self.layer3(x) 69 | x = self.layer4(x) 70 | 71 | x = self.avgpool(x) 72 | x = torch.flatten(x, 1) 73 | self.middle_x = x.detach().cpu().clone() 74 | if dropout_mask is not None: 75 | x = x * dropout_mask(x, dropout_rate, 0) 76 | else: 77 | x = self.dropout(x) 78 | x = self.fc(x) 79 | 80 | return x 81 | 82 | 83 | def resnet_masked(pretrained=True, dropout_rate=0.3): 84 | base = resnet18(pretrained=pretrained) 85 | base.dropout = nn.Dropout(dropout_rate) 86 | base.fc = nn.Linear(512, 10) 87 | 88 | return base 89 | 90 | 91 | def resnet_dropout(pretrained=True, dropout_rate=0.3): 92 | base = resnet18(pretrained=pretrained) 93 | base.dropout = nn.Dropout(dropout_rate) 94 | # base.fc = nn.Linear(512, 10) 95 | 96 | return base 97 | 98 | 99 | 100 | def _resnet(arch, block, layers, pretrained, progress, **kwargs): 101 | model = ResNetMasked(block, layers, **kwargs) 102 | 103 | if pretrained: 104 | state_dict = load_state_dict_from_url(model_urls[arch], 105 | progress=progress) 106 | model.load_state_dict(state_dict) 107 | return model 108 | 109 | 110 | def resnet18(pretrained=False, progress=True, **kwargs): 111 | r"""ResNet-18 model from 112 | `"Deep Residual Learning for Image Recognition" `_ 113 | 114 | Args: 115 | pretrained (bool): If True, returns a model pre-trained on ImageNet 116 | progress (bool): If True, displays a progress bar of the download to stderr 117 | """ 118 | return _resnet('resnet18', BasicBlock, [2, 2, 2, 2], pretrained, progress, 119 | **kwargs) 120 | -------------------------------------------------------------------------------- /experiments/move_chest.py: -------------------------------------------------------------------------------- 1 | import os 2 | from pathlib import Path 3 | from shutil import copy 4 | 5 | from PIL import Image 6 | 7 | ## The script to make Chexpert directory style similar to the ImageNet 8 | root_dir = Path('data/chest') 9 | source = root_dir / 'CheXpert-v1.0-small' / 'train' 10 | target = root_dir / 'train' 11 | 12 | 13 | folder_list = sorted(os.listdir(source)) 14 | 15 | for folder in folder_list: 16 | with_images = source/folder/'study1' 17 | 18 | if os.path.exists(with_images / 'view1_frontal.jpg'): 19 | file_path = with_images / 'view1_frontal.jpg' 20 | else: 21 | file_path = with_images / 'view1_lateral.jpg' 22 | image = Image.open(file_path) 23 | 24 | copy(file_path, target/f"{folder}.jpg") 25 | 26 | print(image.size, folder) 27 | 28 | 29 | 30 | -------------------------------------------------------------------------------- /experiments/print_confidence_accuracy.py: -------------------------------------------------------------------------------- 1 | import pickle 2 | import os 3 | import argparse 4 | 5 | import numpy as np 6 | import pandas as pd 7 | import matplotlib.pyplot as plt 8 | import seaborn as sns 9 | 10 | 11 | def estimator_name(estimator): 12 | return { 13 | 'max_prob': 'Max probability', 14 | 'mc_dropout': 'MC dropout', 15 | 'ht_decorrelating': 'decorrelation', 16 | 'ht_dpp': 'DPP', 17 | 'ht_k_dpp': 'k-DPP', 18 | 'cov_dpp': 'DPP (cov)', 19 | 'cov_k_dpp': 'k-DPP (cov)', 20 | 'ensemble_max_prob': 'Ensemble (max prob)', 21 | 'ensemble_bald': 'Ensemble (bald)', 22 | 'ensemble_var_ratio': 'Ensemble (var ratio)', 23 | }[estimator] 24 | 25 | 26 | def print_confidence(): 27 | parser = argparse.ArgumentParser() 28 | parser.add_argument('name', type=str) 29 | parser.add_argument('--acquisition', '-a', type=str, default='bald') 30 | parser.add_argument('--covariance', dest='covariance', action='store_true') 31 | args = parser.parse_args() 32 | 33 | args.repeats = {'mnist': 3, 'cifar': 3, 'imagenet': 1}[args.name] 34 | 35 | acc_conf = [] 36 | count_conf = [] 37 | 38 | covariance_str = '_covar' if args.covariance else '' 39 | acquisition_str = args.acquisition 40 | 41 | for i in range(args.repeats): 42 | file_name = f'logs/classification/{args.name}_{i}/ue_{acquisition_str}.pickle' 43 | if os.path.exists(file_name): 44 | process_file(file_name, args.acquisition, acc_conf, count_conf, ['mc_dropout', 'ht_dpp', 'cov_k_dpp', 'cov_dpp', 'ht_k_dpp', 'max_prob']) 45 | 46 | if args.acquisition == 'max_prob': 47 | ensemble_method = f"ensemble_{args.acquisition}" 48 | ensemble_file = f'logs/classification/{args.name}_{i}/ue_ensemble.pickle' 49 | if os.path.exists(ensemble_file): 50 | process_file(ensemble_file, args.acquisition, acc_conf, count_conf, [ensemble_method]) 51 | 52 | 53 | plt.rcParams.update({'font.size': 13}) 54 | plt.rc('grid', linestyle="--") 55 | plt.figure(figsize=(7, 5)) 56 | 57 | if args.acquisition == 'bald': 58 | metric = 'UE' 59 | else: 60 | metric = 'Confidence' 61 | 62 | plt.title(f"{metric}-accuracy {args.name} {args.acquisition} {covariance_str}") 63 | df = pd.DataFrame(acc_conf, columns=[f'{metric} level', 'Accuracy', 'Estimator']) 64 | sns.lineplot(f'{metric} level', 'Accuracy', data=df, hue='Estimator') 65 | # plt.subplots_adjust(right=0.7) 66 | # plt.legend(loc='center left', bbox_to_anchor=(1, 0.5)) 67 | sign = '<' if args.acquisition == 'bald' else '<' 68 | plt.ylabel(fr"Accuracy for samples, {metric} {sign} $\tau$") 69 | plt.xlabel(r"$\tau$") 70 | plt.grid() 71 | 72 | plt.savefig(f"data/conf_accuracy_{args.name}_{args.acquisition}{covariance_str}", dpi=150) 73 | plt.show() 74 | 75 | 76 | def process_file(file_name, acquisition, acc_conf, count_conf, methods): 77 | if acquisition == 'bald': 78 | process_file_bald(file_name, acquisition, acc_conf, count_conf, methods) 79 | return 80 | 81 | with open(file_name, 'rb') as f: 82 | record = pickle.load(f) 83 | 84 | prediction = np.argmax(np.array(record['probabilities']), axis=-1) 85 | is_correct = (prediction == record['y_val']).astype(np.int) 86 | 87 | bins = np.concatenate((np.arange(0, 1, 0.1), [0.98, 0.99, 0.999])) 88 | 89 | for estimator in methods: 90 | if estimator not in record['uncertainties'].keys(): 91 | continue 92 | 93 | ue = record['uncertainties'][estimator] 94 | print(estimator) 95 | ue = ue / max(ue) 96 | 97 | for confidence_level in bins: 98 | point_confidences = 1 - ue 99 | bin_correct = is_correct[point_confidences > confidence_level] 100 | if len(bin_correct) > 0: 101 | accuracy = sum(bin_correct) / len(bin_correct) 102 | else: 103 | accuracy = None 104 | 105 | acc_conf.append((confidence_level, accuracy, estimator_name(estimator))) 106 | count_conf.append((confidence_level, len(bin_correct), estimator_name(estimator))) 107 | 108 | 109 | def process_file_bald(file_name, acquisition, acc_conf, count_conf, methods): 110 | with open(file_name, 'rb') as f: 111 | record = pickle.load(f) 112 | 113 | prediction = np.argmax(np.array(record['probabilities']), axis=-1) 114 | is_correct = (prediction == record['y_val']).astype(np.int) 115 | 116 | if 'mnist' in file_name: 117 | bins = np.concatenate(([0, 0.01, 0.02, 0.03, 0.04, 0.05], np.arange(0.1, 1.4, 0.1))) 118 | elif 'cifar' in file_name: 119 | bins = np.concatenate(([0, 0.01, 0.02, 0.03, 0.04, 0.05], np.arange(0.1, 1, 0.1))) 120 | else: 121 | bins = np.concatenate(([0.05], np.arange(0.3, 3, 0.3))) 122 | 123 | for estimator in methods: 124 | if estimator not in record['uncertainties'].keys(): 125 | continue 126 | ue = record['uncertainties'][estimator] 127 | 128 | for ue_level in bins: 129 | bin_correct = is_correct[ue < ue_level] 130 | 131 | if len(bin_correct) > 0: 132 | accuracy = sum(bin_correct) / len(bin_correct) 133 | else: 134 | accuracy = None 135 | 136 | acc_conf.append((ue_level, accuracy, estimator_name(estimator))) 137 | count_conf.append((ue_level, len(bin_correct), estimator_name(estimator))) 138 | 139 | 140 | if __name__ == '__main__': 141 | print_confidence() 142 | -------------------------------------------------------------------------------- /experiments/print_ood.py: -------------------------------------------------------------------------------- 1 | import pickle 2 | import os 3 | import argparse 4 | 5 | import numpy as np 6 | import pandas as pd 7 | import matplotlib.pyplot as plt 8 | import seaborn as sns 9 | from print_confidence_accuracy import estimator_name 10 | 11 | 12 | parser = argparse.ArgumentParser() 13 | parser.add_argument('name', type=str) 14 | parser.add_argument('--acquisition', '-a', type=str, default='bald') 15 | args = parser.parse_args() 16 | args.repeats = {'mnist': 3, 'cifar': 3, 'imagenet': 1}[args.name] 17 | 18 | 19 | def process_file(file_name, count_conf, args, methods): 20 | 21 | with open(file_name, 'rb') as f: 22 | record = pickle.load(f) 23 | 24 | if args.acquisition == 'bald': 25 | process_file_bald(record, count_conf, args, methods) 26 | return 27 | 28 | bins = np.concatenate((np.arange(0, 1, 0.1), [0.98, 0.99, 0.999])) 29 | for estimator in methods: 30 | if estimator not in record['uncertainties'].keys(): 31 | continue 32 | ue = record['uncertainties'][estimator] 33 | 34 | print(estimator) 35 | print(min(ue), max(ue)) 36 | if args.acquisition == 'bald': 37 | ue = ue / max(ue) 38 | 39 | for confidence_level in bins: 40 | point_confidences = 1 - ue 41 | level_count = np.sum(point_confidences > confidence_level) 42 | count_conf.append((confidence_level, level_count, estimator_name(estimator))) 43 | 44 | 45 | def process_file_bald(record, count_conf, args, methods): 46 | # bins = np.concatenate((np.arange(0, 1, 0.1), [0.98, 0.99, 0.999])) 47 | 48 | max_ue = { 49 | 'mnist': 1.4, 'cifar': 1, 'imagenet': 1.1 50 | }[args.name] 51 | 52 | bins = np.concatenate(([0, 0.02, 0.04, 0.06], np.arange(0.1, max_ue, 0.02))) 53 | 54 | for estimator in methods: 55 | if estimator not in record['uncertainties'].keys(): 56 | continue 57 | ue = record['uncertainties'][estimator] 58 | 59 | print(estimator) 60 | if args.acquisition == 'bald': 61 | ue = ue / max(ue) 62 | 63 | for ue_level in bins: 64 | level_count = np.sum(ue < ue_level) 65 | count_conf.append((ue_level, level_count, estimator_name(estimator))) 66 | 67 | 68 | count_conf = [] 69 | 70 | for i in range(args.repeats): 71 | file_name = f'logs/classification/{args.name}_{i}/ue_ood_{args.acquisition}.pickle' 72 | if args.acquisition == 'max_prob': 73 | methods = ['mc_dropout', 'ht_dpp', 'cov_k_dpp', 'cov_dpp', 'ht_k_dpp', 'max_prob'] 74 | else: 75 | methods = ['mc_dropout', 'ht_dpp', 'cov_k_dpp', 'cov_dpp', 'ht_k_dpp'] 76 | 77 | 78 | process_file(file_name, count_conf, args, methods) 79 | 80 | if args.acquisition == 'max_prob': 81 | ensemble_method = f"ensemble_{args.acquisition}" 82 | file_name = f'logs/classification/{args.name}_{i}/ue_ensemble_ood.pickle' 83 | if os.path.exists(file_name): 84 | process_file(file_name, count_conf, args, [ensemble_method]) 85 | 86 | 87 | 88 | if args.acquisition == 'bald': 89 | metric = 'UE' 90 | else: 91 | metric = 'Confidence' 92 | 93 | plt.rcParams.update({'font.size': 13}) 94 | plt.rc('grid', linestyle="--") 95 | plt.figure(figsize=(7, 5)) 96 | plt.title(f"{metric}-count for OOD {args.name} {args.acquisition}") 97 | df = pd.DataFrame(count_conf, columns=[f'{metric} level', 'Count', 'Estimator']) 98 | sns.lineplot(f'{metric} level', 'Count', data=df, hue='Estimator') 99 | plt.subplots_adjust(left=0.15) 100 | # plt.legend(loc='center left', bbox_to_anchor=(1, 0.5)) 101 | 102 | sign = '<' if args.acquisition == 'bald' else '>' 103 | plt.ylabel(rf"Number of samples, {metric} {sign} $\tau$") 104 | plt.xlabel(rf"$\tau$") 105 | plt.grid() 106 | plt.savefig(f"data/conf_ood_{args.name}_{args.acquisition}", dpi=150) 107 | plt.show() 108 | 109 | -------------------------------------------------------------------------------- /experiments/regression_1_big_exper_train-clean.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "pycharm": { 8 | "is_executing": false 9 | } 10 | }, 11 | "outputs": [], 12 | "source": [ 13 | "# 1) pick dataset\n", 14 | "# 2) make n_split_runs=3 runs of 5-kfold\n", 15 | "# 3) for n_model_runs=3 \n", 16 | "# 4) train an ensemble of n_ens=5 models\n", 17 | "# 5) for each single-nn model make n_ue_runs=5 UE runs\n", 18 | "# 6) for EUE make a UE run\n", 19 | "# 7) for ensemble-UE make n_ue_runs=5 UE runs\n", 20 | "# 8) save data for a future" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": { 27 | "pycharm": { 28 | "is_executing": false 29 | } 30 | }, 31 | "outputs": [], 32 | "source": [ 33 | "import sys\n", 34 | "sys.path.append('..')\n", 35 | "%load_ext autoreload\n", 36 | "%autoreload 2\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "pycharm": { 44 | "is_executing": false 45 | } 46 | }, 47 | "outputs": [], 48 | "source": [ 49 | "import os\n", 50 | "from pathlib import Path\n", 51 | "import random\n", 52 | "\n", 53 | "import matplotlib.pyplot as plt\n", 54 | "import pickle\n", 55 | "import torch\n", 56 | "import numpy as np\n", 57 | "\n", 58 | "# from experiment_setup import get_model, set_random, build_estimator\n", 59 | "from alpaca.model.ensemble import MLPEnsemble\n", 60 | "from alpaca.dataloader.builder import build_dataset\n", 61 | "from experiments.utils.data import scale, multiple_kfold\n", 62 | "\n", 63 | "plt.rcParams['figure.facecolor'] = 'white'" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": { 70 | "pycharm": { 71 | "is_executing": false 72 | } 73 | }, 74 | "outputs": [], 75 | "source": [ 76 | "SEED = 10 \n", 77 | "torch.manual_seed(SEED)\n", 78 | "np.random.seed(SEED)\n", 79 | "random.seed(SEED)\n", 80 | "\n", 81 | "torch.cuda.set_device(0)\n", 82 | "torch.backends.cudnn.deterministic = True\n", 83 | "torch.backends.cudnn.benchmark = False\n", 84 | "\n" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": { 91 | "pycharm": { 92 | "is_executing": false 93 | } 94 | }, 95 | "outputs": [], 96 | "source": [ 97 | "config = {\n", 98 | " 'nn_runs': 100,\n", 99 | " 'runs': 2,\n", 100 | " 'max_runs': 20,\n", 101 | " # Our train config\n", 102 | " 'layers': [8, 128, 128, 64, 1],\n", 103 | "# 'layers': [8, 256, 256, 128, 1],\n", 104 | " #'layers': [8, 2048, 2048, 1024, 1],\n", 105 | " 'epochs': 10_000,\n", 106 | " 'validation_step': 100,\n", 107 | " \n", 108 | " # Our train config\n", 109 | " 'nll_layers': [8, 256, 256, 128, 2],\n", 110 | " 'nll_epochs': 400,\n", 111 | " 'nll_validation_step': 50,\n", 112 | " \n", 113 | " 'acc_percentile': 0.1,\n", 114 | " 'patience': 10,\n", 115 | " 'dropout_rate': 0.2,\n", 116 | " 'dropout_uq': 0.5,\n", 117 | " 'batch_size': 256,\n", 118 | " 'dataset': 'kin8nm',\n", 119 | " 'l2_reg': 1e-5,\n", 120 | " 'ood_percentile': 90,\n", 121 | " \n", 122 | " 'optimizer': {'type': 'Adam', 'lr': 0.01, 'weight_decay':1e-5},\n", 123 | " 'n_split_runs': 3,\n", 124 | " 'n_model_runs': 3,\n", 125 | " 'n_ens': 5,\n", 126 | " 'n_ue_runs': 5,\n", 127 | " 'k_folds': 10,\n", 128 | " 'verbose': False,\n", 129 | "}\n" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": { 136 | "pycharm": { 137 | "is_executing": false 138 | } 139 | }, 140 | "outputs": [], 141 | "source": [ 142 | "train_opts = ['patience', 'dropout_rate', 'epochs', 'batch_size', 'validation_step', 'verbose']\n", 143 | "config['train_opts'] = {k: config[k] for k in config if k in train_opts}\n" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "metadata": { 150 | "pycharm": { 151 | "is_executing": false 152 | } 153 | }, 154 | "outputs": [], 155 | "source": [ 156 | "datasets = [\n", 157 | " 'boston_housing', 'concrete', 'energy_efficiency',\n", 158 | " 'kin8nm', 'naval_propulsion', 'ccpp', 'red_wine', \n", 159 | " 'yacht_hydrodynamics'\n", 160 | "]" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": null, 166 | "metadata": { 167 | "pycharm": { 168 | "is_executing": false 169 | } 170 | }, 171 | "outputs": [], 172 | "source": [ 173 | "for dataset_name in datasets:\n", 174 | " dataset = build_dataset(dataset_name, val_split=0.0) \n", 175 | " x_set, y_set = dataset.dataset('train')\n", 176 | " print(dataset_name, x_set.shape)\n", 177 | " config['layers'][0] = x_set.shape[-1]" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": null, 183 | "metadata": { 184 | "pycharm": { 185 | "is_executing": true 186 | } 187 | }, 188 | "outputs": [ 189 | { 190 | "name": "stdout", 191 | "text": [ 192 | "Fit [3/5] model:\n", 193 | "Fit [4/5] model:\n", 194 | "Fit [5/5] model:\n", 195 | "Fit [1/5] model:\n", 196 | "Fit [2/5] model:\n", 197 | "Fit [3/5] model:\n", 198 | "Fit [4/5] model:\n" 199 | ], 200 | "output_type": "stream" 201 | } 202 | ], 203 | "source": [ 204 | "# over datasets\n", 205 | "for dataset_name in datasets:\n", 206 | " print(f'=={dataset_name}==')\n", 207 | " dataset = build_dataset(dataset_name, val_split=0.0) \n", 208 | " x_set, y_set = dataset.dataset('train')\n", 209 | " config['layers'][0] = x_set.shape[-1]\n", 210 | " # over different splits\n", 211 | " for split_cnt in range(config['n_split_runs']):\n", 212 | " kfold_iterator = multiple_kfold(config['k_folds'], \n", 213 | " len(x_set), \n", 214 | " config['k_folds'])\n", 215 | " # within one split\n", 216 | " for kfold_cnt, (val_idx, train_idx) in enumerate(kfold_iterator): \n", 217 | " # MIND THE ORDER\n", 218 | " x_train, y_train = x_set[train_idx], y_set[train_idx]\n", 219 | " x_val, y_val = x_set[val_idx], y_set[val_idx]\n", 220 | " x_train, x_val, x_scaler = scale(x_train, x_val)\n", 221 | " y_train, y_val, y_scaler = scale(y_train, y_val)\n", 222 | " \n", 223 | " train_opts = config['train_opts'].copy()\n", 224 | " # over model runs\n", 225 | " for model_cnt in range(config['n_split_runs']):\n", 226 | " model = MLPEnsemble(config['layers'], \n", 227 | " n_models=config['n_ens'], \n", 228 | " reduction='mean')\n", 229 | " model.fit((x_train, y_train),\n", 230 | " (x_val, y_val),\n", 231 | " **train_opts)\n", 232 | " \n", 233 | " fname = f'{dataset_name[:4]}_split={split_cnt}_kfold={kfold_cnt}_model={model_cnt}'\n", 234 | " dct = {\n", 235 | " 'config': config,\n", 236 | " 'state_dict': model.state_dict(),\n", 237 | " 'data': (x_train, y_train, x_val, y_val, x_scaler, y_scaler)\n", 238 | " }\n", 239 | " dir = Path(os.getcwd()) / 'data' / 'regression'\n", 240 | " with open(dir / f'{fname}.pickle', 'wb') as f:\n", 241 | " pickle.dump(dct, f)" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": null, 247 | "metadata": { 248 | "pycharm": { 249 | "is_executing": true 250 | } 251 | }, 252 | "outputs": [], 253 | "source": [ 254 | "x_train.shape, x_val.shape" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": null, 260 | "outputs": [], 261 | "source": [ 262 | "\n" 263 | ], 264 | "metadata": { 265 | "collapsed": false, 266 | "pycharm": { 267 | "name": "#%%\n", 268 | "is_executing": true 269 | } 270 | } 271 | } 272 | ], 273 | "metadata": { 274 | "kernelspec": { 275 | "display_name": "Python 3", 276 | "language": "python", 277 | "name": "python3" 278 | }, 279 | "language_info": { 280 | "codemirror_mode": { 281 | "name": "ipython", 282 | "version": 3 283 | }, 284 | "file_extension": ".py", 285 | "mimetype": "text/x-python", 286 | "name": "python", 287 | "nbconvert_exporter": "python", 288 | "pygments_lexer": "ipython3", 289 | "version": "3.6.7" 290 | }, 291 | "pycharm": { 292 | "stem_cell": { 293 | "cell_type": "raw", 294 | "source": [], 295 | "metadata": { 296 | "collapsed": false 297 | } 298 | } 299 | } 300 | }, 301 | "nbformat": 4, 302 | "nbformat_minor": 2 303 | } -------------------------------------------------------------------------------- /experiments/visual_datasets.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from torch.utils.data import Dataset 3 | from torch.utils.data import DataLoader 4 | from albumentations import ( 5 | HorizontalFlip, CLAHE, ShiftScaleRotate, Blur, HueSaturationValue, 6 | RandomBrightnessContrast, Compose, Normalize, ToFloat 7 | ) 8 | from albumentations.pytorch.transforms import ToTensorV2 9 | import matplotlib.pyplot as plt 10 | 11 | from alpaca.dataloader.builder import build_dataset 12 | 13 | def loader(x, y, batch_size=128, tfms=None, train=False): 14 | # ds = TensorDataset(torch.DoubleTensor(x), torch.LongTensor(y)) 15 | ds = ImageDataset(x, y, train=train, tfms=tfms) 16 | _loader = DataLoader(ds, batch_size=batch_size, num_workers=4, shuffle=train) 17 | return _loader 18 | 19 | 20 | augmentations = Compose([]) 21 | 22 | post_aug = Compose([ 23 | Normalize(), 24 | ToTensorV2(), 25 | ]) 26 | 27 | 28 | def prepare_cifar(config): 29 | dataset = build_dataset('cifar_10', val_size=config['val_size']) 30 | 31 | x_set, y_set = dataset.dataset('train') 32 | x_val, y_val = dataset.dataset('val') 33 | 34 | shape = (-1, 3, 32, 32) 35 | x_set = np.moveaxis(x_set.reshape(shape), 1, 3).astype(np.uint8) 36 | x_val = np.moveaxis(x_val.reshape(shape), 1, 3).astype(np.uint8) 37 | 38 | # x_set = ((x_set - 128) / 128).reshape(shape) 39 | # x_val = ((x_val - 128) / 128).reshape(shape) 40 | 41 | train_tfms = augmentations 42 | 43 | return x_set, y_set, x_val, y_val, train_tfms 44 | 45 | 46 | def prepare_mnist(config): 47 | dataset = build_dataset('mnist', val_size=config['val_size']) 48 | x_set, y_set = dataset.dataset('train') 49 | x_val, y_val = dataset.dataset('val') 50 | 51 | shape = (-1, 1, 28, 28) 52 | x_set = ((x_set - 128) / 128).reshape(shape) 53 | x_val = ((x_val - 128) / 128).reshape(shape) 54 | 55 | train_tfms = [] 56 | 57 | return x_set, y_set, x_val, y_val, train_tfms 58 | 59 | 60 | def prepare_svhn(config): 61 | dataset = build_dataset('svhn', val_size=config['val_size']) 62 | 63 | x_set, y_set = dataset.dataset('train') 64 | x_val, y_val = dataset.dataset('val') 65 | y_set[y_set == 10] = 0 66 | y_val[y_val == 10] = 0 67 | 68 | shape = (-1, 32, 32, 3) 69 | x_set = x_set.reshape(shape).astype(np.uint8) 70 | x_val = x_val.reshape(shape).astype(np.uint8) 71 | train_tfms = augmentations 72 | 73 | return x_set, y_set, x_val, y_val, train_tfms 74 | 75 | 76 | def prepare_fashion_mnist(config): 77 | dataset = build_dataset('fashion_mnist', val_size=config['val_size']) 78 | x_set, y_set = dataset.dataset('train') 79 | x_val, y_val = dataset.dataset('val') 80 | print(x_set.shape) 81 | 82 | shape = (-1, 1, 28, 28) 83 | x_set = ((x_set - 128) / 128).reshape(shape) 84 | x_val = ((x_val - 128) / 128).reshape(shape) 85 | 86 | train_tfms = [] 87 | 88 | return x_set, y_set, x_val, y_val, train_tfms 89 | 90 | 91 | class ImageDataset(Dataset): 92 | def __init__(self, x, y, train=False, tfms=None): 93 | self.x = x 94 | self.y = y 95 | self.train = train 96 | self.tfms = tfms 97 | 98 | def __len__(self): 99 | return len(self.y) 100 | 101 | def __getitem__(self, idx): 102 | return self.load_img(idx), self.y[idx] 103 | 104 | def load_img(self, idx): 105 | image = self.x[idx] 106 | if self.tfms: 107 | image = self.tfms(image=image)['image'] 108 | if image.shape[2] == 3: 109 | image = post_aug(image=image)['image'].double() 110 | return image 111 | -------------------------------------------------------------------------------- /figures/2d_toy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/2d_toy.png -------------------------------------------------------------------------------- /figures/active_learning_mnist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/active_learning_mnist.png -------------------------------------------------------------------------------- /figures/benchmark.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/benchmark.png -------------------------------------------------------------------------------- /figures/convergence.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/convergence.png -------------------------------------------------------------------------------- /figures/dolan_acc_ens.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/dolan_acc_ens.png -------------------------------------------------------------------------------- /figures/dolan_acc_single.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/dolan_acc_single.png -------------------------------------------------------------------------------- /figures/dpp_ring_contour.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/dpp_ring_contour.png -------------------------------------------------------------------------------- /figures/error_detector_cifar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/error_detector_cifar.png -------------------------------------------------------------------------------- /figures/error_detector_mnist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/error_detector_mnist.png -------------------------------------------------------------------------------- /figures/error_detector_svhn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/error_detector_svhn.png -------------------------------------------------------------------------------- /figures/ood_mnist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/ood_mnist.png -------------------------------------------------------------------------------- /figures/ring_results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stat-ml/dpp-dropout-uncertainty/3e0ac89c6717be0ce1dc322ccd3cfb1f06e0e952/figures/ring_results.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pandas>=0.25.1 2 | seaborn>=0.9.0 3 | alpaca-ml==0.8.0 4 | torch>=1.2.0 5 | torchvision= 6 | catalyst==20.3 7 | matplotlib>=3.0.0 8 | albumentations==0.4.3 9 | 10 | --------------------------------------------------------------------------------