├── README.md ├── __init__.py ├── __pycache__ ├── augment.cpython-36.pyc ├── complexity.cpython-36.pyc ├── complexitymeasures.cpython-36.pyc └── computecomplexityfinal.cpython-36.pyc ├── augment.py ├── complexity.py ├── complexitymeasures.py ├── computecomplexityfinal.py └── metadata /README.md: -------------------------------------------------------------------------------- 1 | # Winning Solution of the NeurIPS 2020 Competition on Predicting Generalization in Deep Learning 2 | 3 | We present various complexity measures that may be predictive of generalization in deep learning. The intention is to create intuitive and simple measures 4 | that can be applied post-hoc on any trained model to get a relative measure of its generalization ability. 5 | 6 | Our solutions based on consistency, robustness, and separability of representations achieved the highest (22.92) and second-highest (13.93) scores 7 | on the final phase of the [NeurIPS Competitition on Predicting Generalization in Deep Learning](https://sites.google.com/view/pgdl2020/home?authuser=0). We are **Team Interpex** on the [leaderboard](https://sites.google.com/view/pgdl2020/leaderboard?authuser=0). 8 | 9 | Detailed descriptions of our solution can be found in our paper: 10 | 11 | ``` 12 | @misc{natekar2020representation, 13 | title={Representation Based Complexity Measures for Predicting Generalization in Deep Learning}, 14 | author={Parth Natekar and Manik Sharma}, 15 | year={2020}, 16 | eprint={2012.02775}, 17 | archivePrefix={arXiv}, 18 | primaryClass={cs.LG} 19 | } 20 | ``` 21 | 22 | Our solution is based on the quality of internal representations of deep neural networks, inspired by neuroscientific theories on how the human visual system creates invariant object representations. 23 | 24 | To run our solution on the public task of the PGDL Competition, clone this repository to the ```sample_code_submission``` folder of the PGDL directory, then run 25 | ```python3 ../ingestion_program/ingestion.py``` from this folder. 26 | 27 | # Available Measures 28 | 29 | The following complexity measures are currently available: 30 | 31 | 1. Davies Bouldin Index 32 | 2. Mixup Performance 33 | 3. Perturbed Margin 34 | 4. Manifold Mixup Performance 35 | 5. Frobenius/Spectral Norm 36 | 37 | The following measures are in the pipeline: FisherRegret, Silhouette Coefficient, Ablation Performance, Pac Bayes, Noise Attenutation. 38 | 39 | # Results 40 | 41 | Scores of our final measures on various tasks of PGDL are as follows: 42 | 43 | | MEASURE | | CIFAR-10 | SVHN | CINIC-10 | CINIC-10 (No BatchNorm) | Oxford Flowers | Oxford Pets | Fashion MNIST | CIFAR 10 (With Augmentations) | 44 | |:---------------------------------:|:-:|:--------:|:-----:|:--------:|:-----------------------:|:--------------:|:-----------:|:-------------:|:-----------------------------:| 45 | | Davies Bouldin * Label-Wise Mixup | | 25.22 | 22.19 | 31.79 | 15.92 | 43.99 | 12.59 | 9.24 | 25.86 | 46 | | Mixup Margin | | 1.11 | 47.33 | 43.22 | 34.57 | 11.46 | 21.98 | 1.48 | 20.78 | 47 | | Augment Margin | | 15.66 | 48.34 | 47.22 | 22.82 | 8.67 | 11.97 | 1.28 | 15.25 | 48 | 49 | 50 | Currently the code only works in the PGDL starting-kit framework available at . You will need to manually choose the required measure in the ```complexity.py``` file and then run ```python3 ../ingestion_program/ingestion.py``` as mentioned above. 51 | 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/parthnatekar/pgdl/fadc4705c3dbf18546703c5d196e4cca661a2cfd/__init__.py -------------------------------------------------------------------------------- /__pycache__/augment.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/parthnatekar/pgdl/fadc4705c3dbf18546703c5d196e4cca661a2cfd/__pycache__/augment.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/complexity.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/parthnatekar/pgdl/fadc4705c3dbf18546703c5d196e4cca661a2cfd/__pycache__/complexity.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/complexitymeasures.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/parthnatekar/pgdl/fadc4705c3dbf18546703c5d196e4cca661a2cfd/__pycache__/complexitymeasures.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/computecomplexityfinal.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/parthnatekar/pgdl/fadc4705c3dbf18546703c5d196e4cca661a2cfd/__pycache__/computecomplexityfinal.cpython-36.pyc -------------------------------------------------------------------------------- /augment.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | 5 | class DataAugmentor: 6 | 7 | """ 8 | A class used for data augmentation (partially taken from : https://www.wouterbulten.nl/blog/tech/data-augmentation-using-tensorflow-data-dataset/) 9 | 10 | Attributes 11 | ---------- 12 | batch : tf.Tensor, optional 13 | The batch to augment 14 | batchSize: int 15 | The batch size 16 | seed: int, optional 17 | Random seed 18 | 19 | Methods 20 | ------- 21 | flip 22 | Flip Augmentation 23 | color 24 | Color Augmentation 25 | gaussian 26 | Gaussian Noise 27 | brightness 28 | Custom Brightness Augmentation 29 | zoom 30 | Crop Augmentation 31 | kerasAug 32 | Inbuilt Keras Augmentations 33 | augment 34 | Wrapper Augmentation Function 35 | """ 36 | 37 | def __init__(self, batch=None, batchSize=50, seed=0): 38 | if batch is not None: 39 | self.dataset = batch 40 | self.seed = seed 41 | tf.random.set_seed(self.seed) 42 | np.random.seed(self.seed) 43 | self.batchSize = batchSize 44 | 45 | def flip(self, x: tf.Tensor) -> tf.Tensor: 46 | """Flip augmentation 47 | 48 | Args: 49 | x: Image to flip 50 | 51 | Returns: 52 | Augmented image 53 | """ 54 | x = tf.image.random_flip_left_right(x, seed=self.seed) 55 | 56 | return x 57 | 58 | def color(self, x: tf.Tensor) -> tf.Tensor: 59 | """Color augmentation 60 | 61 | Args: 62 | x: Image 63 | 64 | Returns: 65 | Augmented image 66 | # """ 67 | x = tf.image.random_hue(x, 0.05, seed=self.seed) 68 | x = tf.image.random_saturation(x, 0.6, 1.2, seed=self.seed) 69 | x = tf.image.random_brightness(x, 0.05, seed=self.seed) 70 | x = tf.image.random_contrast(x, 0.7, 1.0, seed=self.seed) 71 | 72 | return x 73 | 74 | def gaussian(self, x: tf.Tensor) -> tf.Tensor: 75 | 76 | mean = tf.keras.backend.mean(x) 77 | std = tf.keras.backend.std(x) 78 | max_ = tf.keras.backend.max(x) 79 | min_ = tf.keras.backend.min(x) 80 | ptp = max_ - min_ 81 | noise = tf.random.normal(shape=tf.shape(x), mean=0, stddev=0.3*self.var, dtype=tf.float32, seed=self.seed) 82 | # noise_img = tf.clip_by_value(((x - mean)/std + noise)*std + mean, 83 | # clip_value_min = min_, clip_value_max=max_) 84 | noise_img = x+noise 85 | 86 | return noise_img 87 | 88 | def brightness(self, x: tf.Tensor) -> tf.Tensor: 89 | 90 | max_ = tf.keras.backend.max(x) 91 | min_ = tf.keras.backend.min(x) 92 | brightness_val = 0.1*np.random.random_sample() - 0.05 93 | noise = tf.constant(brightness_val, shape=x.shape) 94 | noise_img = x+noise 95 | noise_img = tf.clip_by_value(x, 96 | clip_value_min = min_, clip_value_max=max_) 97 | return noise_img 98 | 99 | def zoom(self, x: tf.Tensor) -> tf.Tensor: 100 | """Zoom augmentation 101 | 102 | Args: 103 | x: Image 104 | 105 | Returns: 106 | Augmented image 107 | """ 108 | 109 | # Generate 20 crop settings, ranging from a 1% to 20% crop. 110 | scales = list(np.arange(0.85, 1.0, 0.01)) 111 | boxes = np.zeros((len(scales), 4)) 112 | 113 | for i, scale in enumerate(scales): 114 | x1 = y1 = 0.5 - (0.5 * scale) 115 | x2 = y2 = 0.5 + (0.5 * scale) 116 | boxes[i] = [x1, y1, x2, y2] 117 | 118 | def random_crop(img): 119 | # Create different crops for an image 120 | crops = tf.image.crop_and_resize([img], boxes=boxes, box_indices=np.zeros(len(scales)), crop_size=(x.shape[0], x.shape[1])) 121 | # Return a random crop 122 | return crops[tf.random.uniform(shape=[], minval=0, maxval=len(scales), dtype=tf.int32, seed=self.seed)] 123 | 124 | 125 | choice = tf.random.uniform(shape=[], minval=0., maxval=1., dtype=tf.float32, seed=self.seed) 126 | 127 | # Only apply cropping 50% of the time 128 | return tf.cond(choice < 0.5, lambda: x, lambda: random_crop(x)) 129 | 130 | def kerasAug(self, x: tf.Tensor) -> tf.Tensor: 131 | 132 | datagen = tf.keras.preprocessing.image.ImageDataGenerator( 133 | rotation_range=2, 134 | width_shift_range=0, 135 | height_shift_range=0, 136 | horizontal_flip=False, 137 | shear_range = 0, 138 | fill_mode='nearest', 139 | dtype = tf.float32) 140 | 141 | return datagen.flow(x, batch_size=self.batchSize, shuffle=False, seed=self.seed).next() 142 | 143 | 144 | def augment(self, batch=None): 145 | if batch is not None: 146 | self.dataset = batch 147 | self.dataset = tf.data.Dataset.from_tensor_slices(self.dataset.numpy()) 148 | 149 | # Add augmentations 150 | augmentations = [self.flip, self.color, self.zoom] 151 | 152 | # Add the augmentations to the dataset 153 | for f in augmentations: 154 | # Apply the augmentation, run 4 jobs in parallel. 155 | self.dataset = self.dataset.map(f) 156 | 157 | self.dataset = next(iter(self.dataset.batch(self.batchSize))) 158 | 159 | return(self.dataset) 160 | 161 | 162 | 163 | 164 | -------------------------------------------------------------------------------- /complexity.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import tensorflow as tf 4 | from tensorflow import keras 5 | from collections import defaultdict 6 | import json 7 | import pickle 8 | import os 9 | import time 10 | import sys 11 | import random 12 | from computecomplexityfinal import * 13 | from complexitymeasures import * 14 | from tensorflow.keras.models import load_model 15 | import matplotlib.pyplot as plt 16 | from scipy.stats import * 17 | from augment import * 18 | import argparse 19 | 20 | def complexity(model, dataset, program_dir, measure = 'DBI, Mixup', augment=None): 21 | ''' 22 | Wrapper Complexity Function to combine various complexity measures 23 | 24 | Parameters 25 | ---------- 26 | model : tf.keras.Model() 27 | The Keras model for which the complexity measure is to be computed 28 | dataset : tf.data.Dataset 29 | Dataset object from PGDL data loader 30 | program_dir : str, optional 31 | The program directory to store and retrieve additional data 32 | measure : str, optional 33 | The complexity measure to compute, defaults to our winning solution of PGDL 34 | augment : str, optional 35 | Augmentation method to use, only relevant for some measures 36 | 37 | Returns 38 | ------- 39 | float 40 | complexity measure 41 | ''' 42 | 43 | if measure == 'DBI': 44 | complexityScore = complexityDB(model, dataset, program_dir=program_dir, pool=True, layer='initial', computeOver=400, batchSize=40) 45 | elif measure == 'Mixup': 46 | complexityScore = complexityMixup(model, dataset, program_dir=program_dir) 47 | elif measure == 'Margin': 48 | complexityScore = complexityMargin(model, dataset, augment = augment, program_dir=program_dir) 49 | elif measure == 'DBI, Mixup': 50 | complexityScore = complexityDB(model, dataset, program_dir=program_dir, pool=True, computeOver=400, batchSize=40) * (1 - complexityMixup(model, dataset, program_dir=program_dir)) 51 | elif measure == 'ManifoldMixup': 52 | complexityScore = complexityManifoldMixup(model, dataset, program_dir=program_dir) 53 | else: 54 | complexityScore = complexityDB(model, dataset, program_dir=program_dir, pool=True, computeOver=400, batchSize=40) * (1 - complexityMixup(model, dataset, program_dir=program_dir)) 55 | 56 | print('-------Final Scores---------', complexityScore) 57 | return complexityScore -------------------------------------------------------------------------------- /complexitymeasures.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | from tensorflow import keras 4 | from collections import defaultdict 5 | import json 6 | import pickle 7 | import os 8 | import time 9 | import sys 10 | import random 11 | from tensorflow.keras.models import load_model 12 | from scipy.stats import * 13 | sys.path.append('..') 14 | from sklearn.preprocessing import StandardScaler, Normalizer, LabelEncoder 15 | from sklearn.decomposition import PCA 16 | from sklearn.metrics import silhouette_score, davies_bouldin_score, pairwise_distances, calinski_harabasz_score 17 | from sklearn.utils import check_X_y, _safe_indexing 18 | from scipy.stats import wasserstein_distance, moment 19 | from math import log, exp 20 | import matplotlib.pyplot as plt 21 | from augment import * 22 | from computecomplexityfinal import * 23 | import copy 24 | import math 25 | 26 | def complexityMargin(model, dataset, augment='standard', program_dir=None): 27 | 28 | ''' 29 | Fuction to calculate margin summary measure on augmented data 30 | 31 | Parameters 32 | ---------- 33 | model : tf.keras.Model() 34 | The Keras model for which the complexity measure is to be computed 35 | dataset : tf.data.Dataset 36 | Dataset object from PGDL data loader 37 | augment : str, optional 38 | The type of augmentation to use ('standard', 'mixup', 'adverserial', 'adverserial+standard', 'mixup+standard') 39 | program_dir : str, optional 40 | The program directory to store and retrieve additional data 41 | 42 | Returns 43 | ------- 44 | float 45 | complexity measure 46 | ''' 47 | 48 | keras.backend.clear_session() 49 | 50 | np.random.seed(0) 51 | 52 | marginDistribution = {} 53 | C = CustomComplexityFinal(model, dataset, augment=augment, input_margin=False) 54 | for label in range(2, 3): 55 | marginDistribution[label], normwidth = C.computeMargins(top = label) 56 | 57 | score = 0 58 | 59 | def computeQuantiles(d): 60 | 61 | q1 = np.percentile(d, 25) 62 | q2 = np.percentile(d, 50) 63 | q3 = np.percentile(d, 75) 64 | 65 | iqr = q3 - q1 66 | 67 | if d[d < q1 - 1.5*iqr].size == 0: 68 | f_l = np.min(d) 69 | else: 70 | f_l = np.max(d[d < q1 - 1.5*iqr]) 71 | 72 | if d[d > q3 + 1.5*iqr].size == 0: 73 | f_u = np.max(d) 74 | else: 75 | f_u = np.min(d[d > q3 + 1.5*iqr]) 76 | 77 | ret = [f_l, q1, q2, q3, f_u] 78 | 79 | return np.array(ret) 80 | 81 | def computePercentiles(d): 82 | return np.array([np.percentile(d, p) for p in list(range(5, 95, 10))]) 83 | 84 | def computeMoments(d): 85 | 86 | moments = [stats.moment(d, moment=ord)**(1/ord) for ord in range(1, 6)] 87 | 88 | moments[0] = np.mean(d) 89 | moments = np.nan_to_num(moments, nan=np.mean(moments)) 90 | print(moments) 91 | return np.array(moments) 92 | 93 | for label in range(2, 3): 94 | for i, index in enumerate(list(marginDistribution[label].keys())): 95 | quantiles = np.mean(computeQuantiles(marginDistribution[label][index])) 96 | mean = np.nanmean(marginDistribution[label][index]) 97 | score += mean/len(list(marginDistribution[label].keys())) 98 | 99 | return -score 100 | 101 | 102 | def complexityNorm(model, dataset, program_dir=None): 103 | 104 | ''' 105 | Function to calculate norm based complexity measures 106 | 107 | Parameters 108 | ---------- 109 | model : tf.keras.Model() 110 | The Keras model for which the complexity measure is to be computed 111 | dataset : tf.data.Dataset 112 | Dataset object from PGDL data loader 113 | program_dir : str, optional 114 | The program directory to store and retrieve additional data 115 | 116 | Returns 117 | ------- 118 | float 119 | complexity measure 120 | ''' 121 | 122 | C = CustomComplexity(model, dataset, metric='batch_variance', augment='standard') 123 | Norm = C.getNormComplexity(norm='fro') 124 | score = 0 125 | 126 | score += Norm 127 | params = np.sum([tf.keras.backend.count_params(p) for p in model.trainable_weights]) 128 | print('Params:', params, model.layers[0].get_weights()[0].shape[-1]) 129 | print('Final Score:', score, len(model.layers)) 130 | 131 | return score 132 | 133 | def complexityDB(model, dataset, program_dir=None, pool=True, use_pca = False, layer='initial', computeOver=400, batchSize=40): 134 | 135 | ''' 136 | Function to calculate feature clustering based measures. Based on the sklearn implementation of DB Index. 137 | 138 | Parameters 139 | ---------- 140 | model : tf.keras.Model() 141 | The Keras model for which the complexity measure is to be computed 142 | dataset : tf.data.Dataset 143 | Dataset object from PGDL data loader 144 | program_dir : str, optional 145 | The program directory to store and retrieve additional data 146 | pool : bool, optional 147 | Whether to use max-pooling for dimensionality reduction, default True 148 | use_pca : bool, optional 149 | Whether to use PCA for dimensionality reduction, default False 150 | layer : str or int, optional 151 | Which layer to compute DB on. Either 'initial', for the first conv/pooling layer in the 152 | model, 'pre-penultimate' for the 3rd-from-last conv/pool layer, or an int indicating the 153 | layer. Defaults to 'initial'. 154 | 155 | Returns 156 | ------- 157 | float 158 | complexity measure 159 | ''' 160 | 161 | def check_number_of_labels(n_labels, n_samples): 162 | """Check that number of labels are valid. 163 | Parameters 164 | ---------- 165 | n_labels : int 166 | Number of labels 167 | n_samples : int 168 | Number of samples 169 | """ 170 | if not 1 < n_labels < n_samples: 171 | raise ValueError("Number of labels is %d. Valid values are 2 " 172 | "to n_samples - 1 (inclusive)" % n_labels) 173 | 174 | def db(X, labels): 175 | X, labels = check_X_y(X, labels) 176 | le = LabelEncoder() 177 | labels = le.fit_transform(labels) 178 | n_samples, _ = X.shape 179 | n_labels = len(le.classes_) 180 | check_number_of_labels(n_labels, n_samples) 181 | 182 | intra_dists = np.zeros(n_labels) 183 | centroids = np.zeros((n_labels, len(X[0])), dtype=float) 184 | for k in range(n_labels): 185 | cluster_k = _safe_indexing(X, labels == k) 186 | centroid = cluster_k.mean(axis=0) 187 | centroids[k] = centroid 188 | intra_dists[k] = np.average(pairwise_distances( 189 | cluster_k, [centroid], metric='euclidean')) 190 | 191 | centroid_distances = pairwise_distances(centroids, metric='euclidean') 192 | 193 | if np.allclose(intra_dists, 0) or np.allclose(centroid_distances, 0): 194 | return 0.0 195 | 196 | centroid_distances[centroid_distances == 0] = np.inf 197 | combined_intra_dists = intra_dists[:, None] + intra_dists 198 | scores = np.max(combined_intra_dists / centroid_distances, axis=1) 199 | return np.mean(scores) 200 | 201 | tf.keras.backend.clear_session() 202 | db_score = {} 203 | C = CustomComplexityFinal(model, dataset, augment='mixup', computeOver=computeOver, batchSize=batchSize) 204 | it = iter(dataset.repeat(-1).batch(C.batchSize)) 205 | batch=next(it) 206 | extractor = C.intermediateOutputs(batch=batch) 207 | if pool == True: 208 | max_pool = tf.keras.layers.MaxPooling2D(pool_size=(4, 4), strides=None, padding="valid", data_format=None) 209 | else: 210 | max_pool = tf.keras.layers.Lambda(lambda x: x + 0) 211 | layers = [] 212 | 213 | layer_dict = {'initial': [0, 1, 2], 'pre-penultimate': [-3, -4, -5]} 214 | 215 | if isinstance(layer, str): 216 | for l in layer_dict[layer]: 217 | c = list(model.get_layer(index = l).get_config().keys()) 218 | if 'strides' in c: 219 | layers.append(l) 220 | if len(layers) == 1: 221 | break 222 | else: 223 | for l in [layer]: 224 | c = list(model.get_layer(index = l).get_config().keys()) 225 | if 'strides' in c: 226 | layers.append(l) 227 | if len(layers) == 1: 228 | break 229 | 230 | D = DataAugmentor(batchSize = C.batchSize) 231 | for l in layers: 232 | it = iter(dataset.repeat(-1).shuffle(5000, seed=1).batch(C.batchSize)) 233 | for i in range(C.computeOver//C.batchSize): 234 | tf.keras.backend.clear_session() 235 | batch1 = next(it) 236 | # batch1 = (D.augment(batch1[0]), batch1[1]) 237 | batch2 = next(it) 238 | # batch2 = (D.augment(batch2[0]), batch2[1]) 239 | batch3 = next(it) 240 | # batch3 = (D.augment(batch3[0]), batch3[1]) 241 | feature = np.concatenate((max_pool(extractor(batch1[0].numpy())[l]).numpy().reshape(batch1[0].shape[0], -1), 242 | max_pool(extractor(batch2[0].numpy())[l]).numpy().reshape(batch2[0].shape[0], -1), 243 | max_pool(extractor(batch3[0].numpy())[l]).numpy().reshape(batch3[0].shape[0], -1)), axis = 0) 244 | target = np.concatenate((batch1[1], batch2[1], batch3[1]), axis = 0) 245 | if use_pca == True: 246 | pca = PCA(n_components=25) 247 | feature = pca.fit_transform(feature) 248 | try: 249 | db_score[l] += db(feature, target)/(C.computeOver//C.batchSize) 250 | except Exception as e: 251 | db_score[l] = db(feature, target)/(C.computeOver//C.batchSize) 252 | 253 | score = np.mean(list(db_score.values())) 254 | 255 | return(score) 256 | 257 | def complexityMixup(model, dataset, program_dir=None, 258 | computeOver=500, batchSize=50): 259 | 260 | ''' 261 | Function to calculate label-wise Mixup based measure 262 | 263 | Parameters 264 | ---------- 265 | model : tf.keras.Model() 266 | The Keras model for which the complexity measure is to be computed 267 | dataset : tf.data.Dataset 268 | Dataset object from PGDL data loader 269 | program_dir : str, optional 270 | The program directory to store and retrieve additional data 271 | computeOver : int 272 | The number of samples over which to compute the complexity measure 273 | batchSize: 274 | The batch size 275 | 276 | Returns 277 | ------- 278 | float 279 | complexity measure 280 | ''' 281 | it = iter(dataset.repeat(-1).shuffle(5000, seed=1).batch(batchSize)) 282 | batch = next(it) 283 | n_classes = 1+np.max(batch[1].numpy()) 284 | batchSize = n_classes*10 285 | computeOver = batchSize*10 286 | tf.keras.backend.clear_session() 287 | it = iter(dataset.repeat(-1).batch(batchSize)) 288 | N = computeOver//batchSize 289 | batches = [next(it) for i in range(N)] 290 | vr = [] 291 | 292 | def intrapolateImages(img, alpha=0.5): 293 | temp = np.stack([img]*img.shape[0]) 294 | tempT = np.transpose(temp, axes = (1,0,2,3,4)) 295 | ret = alpha*temp + (1-alpha)*tempT 296 | mask = np.triu_indices(img.shape[0], 1) 297 | return ret[mask] 298 | 299 | def choose(n, k): 300 | """ 301 | A fast way to calculate binomial coefficients by Andrew Dalke (contrib). 302 | """ 303 | if 0 <= k <= n: 304 | ntok = 1 305 | ktok = 1 306 | for t in range(1, min(k, n - k) + 1): 307 | ntok *= n 308 | ktok *= t 309 | n -= 1 310 | return ntok // ktok 311 | else: 312 | return 313 | 314 | def veracityRatio(model, batches, label, version_loss=None, label_smoothing=0.1): 315 | ret = [] 316 | lossObject = tf.keras.losses.SparseCategoricalCrossentropy(from_logits = True) 317 | for b in batches: 318 | img = b[0][b[1]==label] 319 | lbl = b[1][b[1]==label] 320 | int_img = intrapolateImages(img) 321 | int_lbl = np.stack([label]*int_img.shape[0]) 322 | int_logits = model(int_img) 323 | if version_loss == 'log': 324 | logLikelihood = lossObject(int_lbl, int_logits) 325 | ret.append(logLikelihood) 326 | elif version_loss == 'cosine': 327 | int_preds = tf.nn.softmax(int_logits, axis = 1) 328 | target = tf.one_hot(int_lbl, int_preds.shape[-1]) * (1 - label_smoothing) + label_smoothing/2 329 | ret.append((tf.keras.losses.CosineSimilarity()(target, int_preds)+1)/2) 330 | elif version_loss == 'mse': 331 | int_preds = tf.nn.softmax(int_logits, axis = 1) 332 | target = tf.one_hot(int_lbl, int_preds.shape[-1]) #* (1 - label_smoothing) + label_smoothing/2 333 | ret.append(tf.keras.losses.MeanSquaredError()(target, int_preds)) 334 | else: 335 | int_preds = tf.argmax(int_logits, axis=1) 336 | ret.append(np.sum(int_preds==label)/np.size(int_preds)) 337 | return np.mean(ret) 338 | 339 | for l in range(n_classes): 340 | try: 341 | vr.append(veracityRatio(model, batches, l)) 342 | except: 343 | pass 344 | 345 | return np.mean(vr) 346 | 347 | 348 | def complexityManifoldMixup(model, dataset, program_dir=None, 349 | computeOver=1000, batchSize=50): 350 | 351 | ''' 352 | Function to calculate Manifold Mixup based measure 353 | 354 | Parameters 355 | ---------- 356 | model : tf.keras.Model() 357 | The Keras model for which the complexity measure is to be computed 358 | dataset : tf.data.Dataset 359 | Dataset object from PGDL data loader 360 | program_dir : str, optional 361 | The program directory to store and retrieve additional data 362 | computeOver : int 363 | The number of samples over which to compute the complexity measure 364 | batchSize: int 365 | The batch size 366 | 367 | Returns 368 | ------- 369 | float 370 | complexity measure 371 | ''' 372 | 373 | it = iter(dataset.repeat(-1).batch(batchSize)) 374 | N = computeOver//batchSize 375 | batches = [next(it) for i in range(N)] 376 | digress = [] 377 | 378 | def intrapolateImages(img, alpha=0.5): 379 | temp = np.stack([img]*img.shape[0]) 380 | tempT = np.transpose(temp, axes = (1,0,2,3,4)) 381 | ret = alpha*temp + (1-alpha)*tempT 382 | mask = np.triu_indices(img.shape[0], 1) 383 | return ret[mask] 384 | 385 | def multiplicativeNoise(img, std=2): 386 | return img*(tf.random.normal(mean=1., stddev=std, shape=img.shape, seed=1)) 387 | 388 | def veracityRatio(model, batches, label, layer=0, version_loss=False): 389 | 390 | cloned_model = keras.models.clone_model(model, input_tensors=keras.Input(shape=(batches[0][0][0].shape))) 391 | cloned_model.set_weights(model.get_weights()) 392 | if layer != 0: 393 | cloned_model.layers[layer-1].activation = keras.activations.linear 394 | for b in batches: 395 | ret = [] 396 | img = b[0][b[1]==label] 397 | orig_logits = cloned_model(img) 398 | representation = cloned_model.layers[layer]._last_seen_input 399 | int_repr = intrapolateImages(representation) 400 | int_lbl = np.stack([label]*int_repr.shape[0]) 401 | x = int_repr 402 | for i in range(layer, len(model.layers)): 403 | if i == layer and i != 0: 404 | x = keras.activations.relu(x) 405 | x = model.layers[i](x) 406 | else: 407 | x = model.layers[i](x) 408 | if version_loss: 409 | logLikelihood = lossObject(int_lbl, x) 410 | ret.append(logLikelihood/np.size(int_preds)) 411 | else: 412 | int_preds = tf.argmax(x, axis=1) 413 | ret.append(np.sum(int_preds==label)/np.size(int_preds)) 414 | return np.mean(ret) 415 | 416 | for l in range(10): 417 | try: 418 | digress.append(veracityRatio(model, batches, l, layer = 1)) 419 | except: 420 | pass 421 | return np.mean(digress) 422 | 423 | def complexityMixupSoft(model, dataset, program_dir=None, 424 | computeOver=1000, batchSize=100): 425 | 426 | ''' 427 | Function to calculate Mixup based measure 428 | 429 | Parameters 430 | ---------- 431 | model : tf.keras.Model() 432 | The Keras model for which the complexity measure is to be computed 433 | dataset : tf.data.Dataset 434 | Dataset object from PGDL data loader 435 | program_dir : str, optional 436 | The program directory to store and retrieve additional data 437 | computeOver : int 438 | The number of samples over which to compute the complexity measure 439 | batchSize: int 440 | The batch size 441 | 442 | Returns 443 | ------- 444 | float 445 | complexity measure 446 | ''' 447 | 448 | it = iter(dataset.repeat(-1).shuffle(5000, seed=1).batch(batchSize)) 449 | batch = next(it) 450 | n_classes = 1+np.max(batch[1].numpy()) 451 | batchSize = n_classes*10 452 | computeOver = batchSize*10 453 | tf.keras.backend.clear_session() 454 | it = iter(dataset.repeat(-1).batch(batchSize)) 455 | N = computeOver//batchSize 456 | batches = [next(it) for i in range(N)] 457 | np.random.seed(0) 458 | vr = [] 459 | 460 | def intrapolateImages(img, lbl, alpha=0.5): 461 | indices = np.random.randint(0, img.shape[0], size = img.shape[0]) 462 | img = img*alpha + img[indices]*(1-alpha) 463 | lbl = lbl*alpha + lbl[indices]*(1-alpha) 464 | return (tf.convert_to_tensor(img), tf.convert_to_tensor(lbl)) 465 | 466 | def choose(n, k): 467 | """ 468 | A fast way to calculate binomial coefficients by Andrew Dalke (contrib). 469 | """ 470 | if 0 <= k <= n: 471 | ntok = 1 472 | ktok = 1 473 | for t in range(1, min(k, n - k) + 1): 474 | ntok *= n 475 | ktok *= t 476 | n -= 1 477 | return ntok // ktok 478 | else: 479 | return 480 | 481 | def veracityRatio(model, batches, version_loss=None, label_smoothing=0.1): 482 | ret = [] 483 | lossObject = tf.keras.losses.SparseCategoricalCrossentropy(from_logits = True) 484 | for b in batches: 485 | int_img, int_lbl = intrapolateImages(b[0].numpy(), b[1].numpy()) 486 | int_logits = model(int_img) 487 | if version_loss == 'log': 488 | logLikelihood = lossObject(int_lbl, int_logits) 489 | ret.append(logLikelihood) 490 | elif version_loss == 'cosine': 491 | int_preds = tf.nn.softmax(int_logits, axis = 1) 492 | target = tf.one_hot(int_lbl, int_preds.shape[-1]) * (1 - label_smoothing) + label_smoothing/2 493 | ret.append((tf.keras.losses.CosineSimilarity()(target, int_preds)+1)/2) 494 | elif version_loss == 'mse': 495 | int_preds = tf.nn.softmax(int_logits, axis = 1) 496 | target = tf.one_hot(int_lbl, int_preds.shape[-1]) #* (1 - label_smoothing) + label_smoothing/2 497 | ret.append(tf.keras.losses.MeanSquaredError()(target, int_preds)) 498 | else: 499 | int_preds = tf.argmax(int_logits, axis=1) 500 | ret.append(np.sum(int_preds==label)/np.size(int_preds)) 501 | return np.mean(ret) 502 | 503 | vr.append(veracityRatio(model, batches, version_loss='log')) 504 | 505 | return np.mean(vr) -------------------------------------------------------------------------------- /computecomplexityfinal.py: -------------------------------------------------------------------------------- 1 | """Custom Complexity Measure based on Margin Distribution""" 2 | 3 | import numpy as np 4 | import tensorflow as tf 5 | from tensorflow import keras 6 | from collections import defaultdict 7 | import json 8 | import pickle 9 | import os 10 | import time 11 | import sys 12 | import random 13 | import math 14 | from tensorflow.keras.models import load_model 15 | import tensorflow.keras.backend as K 16 | from scipy.stats import * 17 | sys.path.append('..') 18 | from sklearn.preprocessing import StandardScaler, Normalizer 19 | from sklearn.metrics import silhouette_score 20 | from math import log 21 | import matplotlib.pyplot as plt 22 | from augment import * 23 | import gc 24 | import time 25 | 26 | class CustomComplexityFinal: 27 | 28 | """ 29 | A class used to create margin based complexity measures 30 | 31 | Attributes 32 | ---------- 33 | model : tf.keras.Model() 34 | The Keras model for which the complexity measure is to be computed 35 | dataset : tf.data.Dataset 36 | Dataset object from PGDL data loader 37 | rootpath : str, optional 38 | Path to root directory 39 | computeOver : int 40 | The number of samples over which to compute the complexity measure 41 | batchSize: int 42 | The batch size 43 | basename: str, optional 44 | basename argument of PGDL directory structure 45 | metric: str, optional 46 | Metric to use to scale margin Distribution 47 | augment : str, optional 48 | The type of augmentation to use ('standard', 'mixup', 'adverserial', 'adverserial+standard', 'mixup+standard') 49 | penalize : bool, optional 50 | Whether to penalize misclassified samples 51 | input_margin : bool, optional 52 | Whether to compute margin on input data instead of intermediate representations 53 | network_scale: bool, optional 54 | Only used for auxiliary experiments involving regularizing for network topology 55 | seed: int, optional 56 | Random seed 57 | 58 | """ 59 | 60 | def __init__(self, model, ds, rootpath=None, mid=None, computeOver = 500, batchSize = 50, basename=None, metric='batch_variance', augment='standard', penalize=True, input_margin=False, network_scale = False, seed=1): 61 | self.model = model 62 | self.dataset = ds 63 | self.computeOver = computeOver 64 | if rootpath is not None: 65 | self.rootPath = os.path.join(rootpath, 'computed_data/pickles/{}'.format(mid)) 66 | if not os.path.exists(self.rootPath): 67 | os.makedirs(self.rootPath) 68 | self.mid = mid 69 | self.basename = basename 70 | self.batchSize = batchSize 71 | self.metric = metric 72 | self.verbose = False 73 | self.norm = 'l2' 74 | self.penalize = penalize 75 | self.augment = augment 76 | self.input_margin = input_margin 77 | self.network_scale = network_scale 78 | self.seed=seed 79 | 80 | # ====================================== Functions for Margin Based Solution ===================================== 81 | 82 | def computeMargins(self, top = 2): 83 | 84 | ''' 85 | Fuction to compute margin distribution 86 | 87 | Returns 88 | ------- 89 | marginDistribution : dict 90 | Dictionary containing lists of margin distribution for each layer 91 | 92 | ''' 93 | 94 | it = iter(self.dataset.repeat(-1).shuffle(5000, seed=self.seed).batch(self.batchSize)) 95 | marginDistribution = {} 96 | totalVariance = {} 97 | totalVarianceTensor = {} 98 | totalNorm = {} 99 | totalNormTensor = {} 100 | self.layers = [] 101 | ratio_list = {} 102 | 103 | for l in range(len(self.model.layers)): 104 | c = list(self.model.get_layer(index = l).get_config().keys()) 105 | if 'filters' in c or 'units' in c: 106 | self.layers.append(l) 107 | if len(self.layers) == 1: 108 | break 109 | 110 | if self.input_margin == True: 111 | self.layers = [-1] 112 | 113 | if self.verbose == True: 114 | for l, layer in enumerate(self.model.layers): 115 | print(self.model.get_layer(index = l).get_config()) 116 | 117 | for i in range(self.computeOver//self.batchSize): 118 | batch = next(it) 119 | if self.augment == 'standard': 120 | D = DataAugmentor(batch[0], batchSize = self.batchSize) 121 | batch_ = (D.augment(), batch[1]) 122 | elif self.augment == 'adverserial': 123 | batch_ = (self.getAdverserialBatch(batch), batch[1]) 124 | elif self.augment == 'adverserial+standard': 125 | D = DataAugmentor(batch[0], batchSize = self.batchSize) 126 | batch_ = (D.augment(), batch[1]) 127 | batch_ = (self.getAdverserialBatch(batch_), batch_[1]) 128 | elif self.augment == 'mixup': 129 | batch_ = self.batchMixupLabelwiseLinear(batch) 130 | elif self.augment == 'mixup+standard': 131 | D = DataAugmentor(batch[0], batchSize = self.batchSize) 132 | batch_ = (D.augment(), batch[1]) 133 | batch_ = self.batchMixupLabelwise(batch_) 134 | else: 135 | batch_ = batch 136 | 137 | for layer in self.layers: 138 | if self.augment is not None: 139 | grads, inter = self.distancefromMargin(batch_, layer+1, top) 140 | else: 141 | grads, inter = self.distancefromMargin(batch_, layer+1, top) 142 | try: 143 | marginDistribution[layer] += grads 144 | totalVarianceTensor[layer] = np.vstack((totalVarianceTensor[layer], np.array(inter).reshape(inter.shape[0], -1))) 145 | except Exception as e: 146 | marginDistribution[layer] = grads 147 | totalVarianceTensor[layer] = np.array(inter).reshape(inter.shape[0], -1) 148 | 149 | if self.network_scale == True: 150 | return marginDistribution, {} 151 | 152 | normWidth = {} 153 | for layer in self.layers: 154 | totalVariance[layer] = (trim_mean(np.var(totalVarianceTensor[layer].reshape(totalVarianceTensor[layer].shape[0], -1), axis = 0), proportiontocut=0.05))**(1/2) 155 | normWidth[layer] = math.sqrt(np.prod(totalVarianceTensor[layer].shape[1:])) 156 | marginDistribution[layer] = np.array(marginDistribution[layer])/(np.array(totalVariance[layer])+1e-7) #/np.sqrt(m_factor) #/totalVarianceTensor[layer].shape[1:][0] 157 | 158 | return marginDistribution, normWidth 159 | 160 | def distancefromMargin(self, batch, layer, top = 2): 161 | 162 | ''' 163 | Fuction to calculate margin distance for a given layer 164 | 165 | Parameters 166 | ---------- 167 | batch : tf.data.Dataset() 168 | The batch over which to compute the margin distance. A tuple of tf.Tensor of the form (input data, labels) 169 | layer : int 170 | The layer for which to compute margin distance 171 | top : int, optional 172 | Index for which to compute margin. For example, top = 2 will compute the margins between the class with the highes and second-highest softmax scores 173 | 174 | Returns 175 | ------- 176 | grads : list 177 | A list containing the scaled margin distances 178 | np_out : np.array 179 | An array containing the flattened intermediate feature vector 180 | ''' 181 | 182 | if self.network_scale == True: 183 | batch_ = tf.ones(shape = batch[0].shape) 184 | else: 185 | batch_ = batch[0] 186 | with tf.GradientTape(persistent=True) as tape: 187 | intermediateVal = [batch_] 188 | tape.watch(intermediateVal) 189 | for l, layer_ in enumerate(self.model.layers): 190 | intermediateVal.append(layer_(intermediateVal[-1])) 191 | 192 | out_hard = tf.math.top_k(tf.nn.softmax(intermediateVal[-1], axis = 1), k = top)[1] 193 | top_1 = out_hard[:,top-2] 194 | misclassified = np.where(top_1 != batch[1]) 195 | 196 | if self.penalize: 197 | top_1_og = out_hard[:,top-2] 198 | top_2_og = out_hard[:,top-1] 199 | mask = np.array(top_1_og == batch[1]).astype(int) 200 | top_1 = top_1_og*mask + batch[1]*(1.- mask) 201 | top_2 = top_2_og*mask + top_1_og*(1.- mask) 202 | else: 203 | top_1 = out_hard[:,top-2] 204 | top_2 = out_hard[:,top-1] 205 | mask = np.array(top_1 == batch[1]).astype(int) 206 | top_2 = top_2*mask + batch[1]*(1.- mask) 207 | 208 | index = list(range(batch[0].shape[0])) 209 | index1 = np.array(list(zip(index, top_1))) 210 | index2 = np.array(list(zip(index, top_2))) 211 | preds = intermediateVal[-1] 212 | logit1 = tf.gather_nd(preds, tf.constant(index1, tf.int32)) 213 | logit2 = tf.gather_nd(preds, tf.constant(index2, tf.int32)) 214 | if self.network_scale == True: 215 | grad_i = tape.gradient(intermediateVal[-1], intermediateVal[layer]) 216 | grad_diff = (np.reshape(grad_i.numpy(), (self.batchSize, -1))) 217 | denominator = np.linalg.norm(grad_diff, axis = 1, ord=2) 218 | np_out = np.array(intermediateVal[layer]) 219 | print(denominator, np.mean(grad_i**2)) 220 | return denominator, np_out, grad_diff 221 | else: 222 | grad_i = tape.gradient(logit1, intermediateVal[layer]) 223 | grad_j = tape.gradient(logit2, intermediateVal[layer]) 224 | numerator = tf.gather_nd(preds, tf.constant(index1, tf.int32)) - tf.gather_nd(preds, tf.constant(index2, tf.int32)).numpy() 225 | grad_diff = (np.reshape(grad_i.numpy(), (grad_i.numpy().shape[0], -1)) - np.reshape(grad_j.numpy(), (grad_j.numpy().shape[0], -1))) 226 | denominator = np.linalg.norm(grad_diff, axis = 1, ord=2) 227 | inf = np.linalg.norm(grad_diff, axis = 1, ord=np.inf) 228 | 229 | if self.penalize == False: 230 | numerator = np.delete(numerator, misclassified, axis = 0) 231 | denominator = np.delete(denominator, misclassified, axis = 0) 232 | 233 | if self.metric == 'spectral': 234 | grads = numerator 235 | else: 236 | grads = numerator/(denominator+1e-7) 237 | 238 | np_out = np.array(intermediateVal[layer]) 239 | 240 | gc.collect() 241 | 242 | return list(grads), np_out 243 | 244 | 245 | # ====================================== Functions for Mixup Based Solution ====================================== 246 | 247 | def batchMixup(self, batch, seed=1): 248 | 249 | ''' 250 | Fuction to perform mixup on a batch of data 251 | 252 | Parameters 253 | ---------- 254 | batch : tf.data.Dataset() 255 | The batch over which to compute the margin distance. A tuple of tf.Tensor of the form (input data, labels) 256 | seed : int, optional 257 | Random seed 258 | 259 | Returns 260 | ------- 261 | tf.tensor 262 | The mixed-up batch 263 | ''' 264 | 265 | np.random.seed(seed) 266 | x = batch[0] 267 | mix_x = batch[0].numpy() 268 | for i in range(np.max(batch[1])): 269 | lam = (np.random.randint(0, 3, size=x[batch[1] == i].shape[0])/10)[..., None, None, None] 270 | mix_x[(batch[1] == i)] = x[batch[1] == i]*lam + tf.random.shuffle(x[batch[1] == i], seed=seed)*(1-lam) 271 | 272 | if self.augment == 'mixup': 273 | mix_x = mix_x[np.random.randint(0, mix_x.shape[0], size=self.batchSize//3)] 274 | target = batch[1].numpy()[np.random.randint(0, mix_x.shape[0], size=self.batchSize//3)] 275 | return (tf.convert_to_tensor(mix_x), tf.convert_to_tensor(target)) 276 | else: 277 | return tf.convert_to_tensor(mix_x) 278 | 279 | def batchMixupLabelwiseLinear(self, batch, seed=2): 280 | 281 | 282 | np.random.seed(seed) 283 | labels = batch[1] 284 | sorted_indices = np.argsort(batch[1]) 285 | sorted_labels = batch[1].numpy()[sorted_indices] 286 | sorted_images = batch[0].numpy()[sorted_indices] 287 | edges = np.array([len(sorted_labels[sorted_labels == i]) for i in range(max(sorted_labels)+1)]) 288 | edges = [0] + list(np.cumsum(edges)) 289 | shuffled_indices = [] 290 | for i in range(len(edges)-1): 291 | # print(sorted_indices[edges[i]:edges[i+1]], sorted_labels[edges[i]:edges[i+1]]) 292 | shuffled_indices += list(np.random.choice(list(range(edges[i], edges[i+1])), replace=False, size=edges[i+1] - edges[i])) 293 | # print(shuffled_indices[edges[i]:edges[i+1]]) 294 | intrapolateImages = (sorted_images + sorted_images[shuffled_indices])/2 295 | return (tf.convert_to_tensor(intrapolateImages), tf.convert_to_tensor(sorted_labels)) 296 | 297 | def batchMixupLabelwise(self, batch, seed=1): 298 | 299 | ''' 300 | Fuction to perform label-wise mixup on a batch of data 301 | 302 | Parameters 303 | ---------- 304 | batch : tf.data.Dataset() 305 | The batch over which to compute the margin distance. A tuple of tf.Tensor of the form (input data, labels) 306 | seed : int, optional 307 | Random seed 308 | 309 | Returns 310 | ------- 311 | tf.tensor 312 | The label-wise mixed-up batch 313 | ''' 314 | 315 | np.random.seed(seed) 316 | 317 | def intrapolateImages(img, alpha=0.5): 318 | temp = np.stack([img]*img.shape[0]) 319 | try: 320 | tempT = np.transpose(temp, axes = (1,0,2,3,4)) 321 | except: 322 | tempT = np.transpose(temp, axes = (1,0,2,3)) 323 | ret = alpha*temp + (1-alpha)*tempT 324 | mask = np.triu_indices(img.shape[0], 1) 325 | return ret[mask] 326 | 327 | def randomSample(batch, size): 328 | indices = np.random.randint(0, batch.shape[0], size=size) 329 | return batch[indices] 330 | 331 | for label in range(1+np.max(batch[1].numpy())): 332 | try: 333 | img = batch[0][batch[1]==label] 334 | lbl = batch[1][batch[1]==label] 335 | try: 336 | mixedBatch = np.vstack((mixedBatch, randomSample(intrapolateImages(img), img.shape[0]))) 337 | labels = np.concatenate((labels, lbl)) 338 | except Exception as e: 339 | mixedBatch = randomSample(intrapolateImages(img), img.shape[0]) 340 | labels = lbl 341 | except: 342 | img = batch[0][batch[1]==label] 343 | lbl = batch[1][batch[1]==label] 344 | try: 345 | mixedBatch = np.vstack((mixedBatch, img)) 346 | labels = np.concatenate((labels, lbl)) 347 | except: 348 | mixedBatch = img 349 | labels = lbl 350 | 351 | return (tf.convert_to_tensor(mixedBatch), tf.convert_to_tensor(labels)) 352 | 353 | # ====================================== Utility Functions ====================================== 354 | 355 | def intermediateOutputs(self, batch, layer=None, mode=None): 356 | 357 | ''' 358 | Fuction to get intermadiate feature vectors 359 | 360 | Parameters 361 | ---------- 362 | batch : tf.Tensor 363 | A batch of data 364 | layer : int, optional 365 | The layer for which to get intermediate features 366 | mode : str, optional 367 | 'pre' to create a pre-model which takes in the input data and gives out intermediate activations, 368 | 'post' to take in intermediate activations and give out the model predictions 369 | 370 | Returns 371 | ------- 372 | tf.keras.Model() 373 | An extractor model 374 | ''' 375 | 376 | model_ = keras.Sequential() 377 | model_.add(keras.Input(shape=(batch[0][0].shape))) 378 | for layer_ in self.model.layers: 379 | model_.add(layer_) 380 | 381 | if layer is not None and mode=='pre': 382 | if layer >= 0: 383 | extractor = keras.Model(inputs=self.model.layers[0].input, 384 | outputs=self.model.layers[layer].output) 385 | else: 386 | 387 | extractor = keras.Model(inputs=self.model.layers[0].input, 388 | outputs=self.model.layers[0].input) 389 | elif layer is not None and mode=='post': 390 | input_ = keras.Input(shape = (self.model.layers[layer].input.shape[1:])) 391 | next_layer = input_ 392 | for layer in self.model.layers[layer:layer+2]: 393 | next_layer = layer(next_layer) 394 | extractor = keras.Model(input_, next_layer) 395 | else: 396 | extractor = keras.Model(inputs=self.model.layers[0].input, 397 | outputs=[layer.output for layer in self.model.layers]) 398 | 399 | return extractor -------------------------------------------------------------------------------- /metadata: -------------------------------------------------------------------------------- 1 | description: Provides prediction model to be executed by the ingestion program --------------------------------------------------------------------------------