├── .gitignore ├── README.md ├── caffe_utils.py ├── cnn_occlusion.py ├── dataset_utils.py ├── occlusion_output.png ├── synset_words.txt ├── synsets.txt ├── t-sne.gif ├── tsne.py └── tsneCNN.py /.gitignore: -------------------------------------------------------------------------------- 1 | Images 2 | Networks 3 | *.pyc 4 | mnist* 5 | .* 6 | !/.gitignore 7 | image_dir 8 | images_dir 9 | 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CNN_Visualization 2 | Implementation of visualization techniques for CNN in Caffe (t-SNE, DeconvNet, Image occlusions) 3 | 4 | #### Requirements: 5 | 6 | - **Caffe** and **pyCaffe**, [Installation guide](http://caffe.berkeleyvision.org/installation.html). 7 | 8 | 9 | - numpy, scikit-image, sklearn, skdata 10 | 11 | 12 | ## t-sne: 13 | The file tsne.py contains an implementation of t-Stochastic Neighbor Embedding as described in the following paper: [Visualizing Data using t-SNE](https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf). 14 | 15 | It's possible to test this implementation stand-alone on the well known mnist dataset. It's necessary to download data and labels from [here](https://github.com/azinik/java-deeplearning/tree/master/deeplearning4j-core/src/main/resources), put them in the same folder of the source code and run 16 | 17 | 18 | $ python tsne.py 19 | 20 | To test how the implementation works with features extracted from a CNN, we will use the file tsneCNN.py. 21 | 22 | 23 | Parameter | Description 24 | ------------ | ------------- 25 | --input_im, -i PATH | Path to the folder containing the synset image folders 26 | --weights, -w PATH | The model path (default: /path/to/caffenet_model) 27 | --prototxt, -p PATH | The prototxt path (default: /path/to/caffenet_prototxt) 28 | --gpu, -g | If this flag is used, the code will run in gpu mode 29 | --net_type, -n STRING | The type of the CNN, options are resnet, googlenet, caffenet (default: caffenet) 30 | 31 | The simplest way to test the code consists in downloading the standard CaffeNet network using 32 | 33 | $ wget -O path/to/caffe/models/bvls_reference_caffenet/bvlc_reference_caffenet.caffemodel http://dl.caffe.berkeleyvision.org/bvlc_reference_caffenet.caffemodel 34 | 35 | 36 | Get the mapping from network output to synsets using 37 | 38 | $ ./path/to/caffe/data/ilsvrc12/data/get_ilsvrc_aux.sh 39 | 40 | Download some synsets of images from ImageNet and place them in a folder called "image_dir". 41 | The script will take up to 150 images from each subfolder of "image_dir". It's assumed that each of these subfolder is a synset directory as downloaded from ImageNet. 42 | 43 | Launch the script exploiting default parameters. 44 | 45 | $ python tsneCNN.py -i image_dir -g 46 | 47 | 48 | It's possible to test different networks simply by downloading their prototxt and weights and providing the paths to these file when launching the script. Do not forget to set also the network type! This argument is used to choose from which layer features have to be extracted. 49 | 50 | The expected output consists in: 51 | 52 | - The 2-dimensional embedding of the extracted features (each image is represented as a "dot" coloured accoring to its real or predicted class). This allows to see how this unsupervised method performs really well regardless of the CNN prediction. 53 | 54 | - A video showing how the embedding changes through the t-SNE iterations. 55 | 56 | - The real images organized according to the embedding. This allows to see how t-SNE preserves local differences within the same class of images (N.B. Zoom on an area of the image to see something meaningful) 57 | 58 | ![alt text](https://github.com/albioTQ/CNN_Visualization/blob/master/t-sne.gif) 59 | 60 | 61 | 62 | 63 | ## OCCLUSION: 64 | The file cnn_occlusion.py is an implementation of the occlusion technics described in the section **4.2** of the following paper: [Visualizing and Understanding Convolutional Networks](https://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf). 65 | 66 | ### How to use 67 | In the following, the procedure to use **caffenet** will be described. It is possible to use different models by changing the default parameters. The **caffenet** prototxt is already included in the **caffe** installation. 68 | 69 | To download the caffemodel file in the proper place run: 70 | 71 | $ wget -O path/to/caffe/models/bvls_reference_caffenet/bvlc_reference_caffenet.caffemodel http://dl.caffe.berkeleyvision.org/bvlc_reference_caffenet.caffemodel 72 | 73 | Choose a class from the file **synset_words.txt** and download at least one image from [Image-Net](www.image-net.org). 74 | Suppose the name of the image is *n01440764_18.JPEG*. It's important to don't change the name of the image since it contains the synset reference. 75 | 76 | To test the code: 77 | 78 | $ python cnn_occlusion.py /path/to/n01440764_18.JPEG 79 | 80 | 81 | You can expect something like this: 82 | 83 | 84 | ![alt text](https://github.com/albioTQ/CNN_Visualization/blob/master/occlusion_output.png) 85 | 86 | 87 | ### Parameters: 88 | 89 | 90 | Parameter | Description 91 | ------------ | ------------- 92 | image_path, PATH | The image path 93 | --weights, -w PATH | The model path (default: /path/to/caffenet_model) 94 | --prototxt, -p PATH | The prototxt path (default: /path/to/caffenet_prototxt) 95 | --layer, -l PATH | Extraction layer (default: pool5) 96 | --gpu, -g INT | GPU number to use (default: -1, aka cpu_mode) 97 | --batch_size INT| The batch size (default: 1) 98 | --stride INT | The stride of the applied mask (default: 100) 99 | --mask_size INT | The length of the side of the square mask (default: 100) 100 | 101 | 102 | 103 | 104 | -------------------------------------------------------------------------------- /caffe_utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | os.environ['GLOG_minloglevel'] = '3' 3 | 4 | import caffe 5 | import numpy as np 6 | import sys 7 | import itertools 8 | 9 | import matplotlib.pyplot as plt 10 | 11 | from sklearn.metrics import confusion_matrix 12 | 13 | def backspace(n): 14 | sys.stdout.write('\r'+n) 15 | sys.stdout.flush() 16 | 17 | class CaffeNet(): 18 | """The class initializes a caffenet and brings some usefull methods. 19 | 20 | Parameters 21 | --------- 22 | model_path: the path to the prototxt 23 | weights_path: the path to the weights 24 | mean_path: the path to the mean of the dataset, default: None 25 | image_scale: the color scale accepted by the net, default: 255.0 26 | batch_size: batch size for the forward pass, default: 1 27 | input_shape: (rows, cols) the input shape of the net, default: (227, 227)""" 28 | 29 | def __init__(self, model_path, 30 | weights_path, 31 | mean_path=None, 32 | image_scale=255.0, 33 | batch_size=1, 34 | input_shape=(227, 227)): 35 | 36 | self.net = caffe.Net(model_path, # defines the structure of the model 37 | weights_path, # contains the trained weights 38 | caffe.TEST) # use test mode (e.g., don't perform dropout) 39 | 40 | self.net.blobs['data'].reshape(batch_size, 3, input_shape[0], input_shape[1]) 41 | self.net.blobs['prob'].reshape(batch_size, ) 42 | 43 | self.mean_path = mean_path 44 | self.image_scale = image_scale 45 | self.batch_size = batch_size 46 | 47 | self.transformer = self.set_transformer() 48 | 49 | def set_transformer(self): 50 | transformer = caffe.io.Transformer({'data': self.net.blobs['data'].data.shape}) 51 | transformer.set_transpose('data', (2, 0, 1)) #move image channels to outermost dimension 52 | transformer.set_channel_swap('data', (2, 1, 0)) # if using RGB instead of BGR 53 | transformer.set_raw_scale('data', self.image_scale) 54 | 55 | if self.mean_path: 56 | transformer.set_mean('data', np.load(self.mean_path).mean(1).mean(1)) 57 | 58 | return transformer 59 | 60 | def preprocess_images(self, image_set): 61 | transformed_images = [] 62 | 63 | for image in image_set: 64 | transformed_images.append(self.transformer.preprocess('data', image)) 65 | 66 | return transformed_images 67 | 68 | def deprocess_images(self, image_set): 69 | transformed_images = [] 70 | 71 | for image in image_set: 72 | transformed_images.append(self.transformer.deprocess('data', image)) 73 | 74 | return transformed_images 75 | 76 | def get_features(self, batch, extraction_layer, most_active_filter=None): 77 | 78 | features_vector = [] 79 | 80 | for image in batch: 81 | 82 | self.net.blobs['data'].data[...] = image 83 | self.net.forward() 84 | 85 | features = self.net.blobs[extraction_layer].data[0] 86 | if most_active_filter is int: 87 | features_vector.append(features[most_active_filter].copy()) 88 | else: 89 | features_vector.append(features.copy()) 90 | 91 | return features_vector 92 | 93 | @staticmethod 94 | def get_most_active_filters(images_features, n=10): 95 | 96 | best_filters = [] 97 | 98 | for filters in images_features: 99 | 100 | mean_filters = [np.mean(filter_) for filter_ in filters] 101 | 102 | # reversing the order 103 | filters_sorted = np.argsort(mean_filters)[::-1] 104 | best_filters.append(filters_sorted[:n]) 105 | 106 | return best_filters 107 | 108 | def get_probs(self, batch): 109 | 110 | batch_probabilities = [] 111 | for img in batch: 112 | self.net.blobs['data'].data[...] = img 113 | batch_output = self.net.forward() 114 | 115 | batch_probabilities.append(batch_output['prob'][0]) 116 | 117 | return batch_probabilities 118 | 119 | def get_probs_and_features(self, batch, extraction_layer, most_active_filter=None): 120 | 121 | batch_probabilities = [] 122 | features_vector = [] 123 | 124 | for img in batch: 125 | 126 | self.net.blobs['data'].data[...] = img 127 | batch_output = self.net.forward() 128 | batch_probabilities.append(batch_output['prob'][0].copy()) 129 | 130 | features = self.net.blobs[extraction_layer].data[0] 131 | if most_active_filter is int: 132 | features_vector.append(features[most_active_filter].copy()) 133 | else: 134 | features_vector.append(features.copy()) 135 | 136 | return batch_probabilities, features_vector 137 | 138 | @staticmethod 139 | def batch_iterator(images, batch_size): 140 | batch = [] 141 | for idx, image in enumerate(images): 142 | batch.append(image) 143 | 144 | idx+=1 145 | 146 | if idx % batch_size == 0 and idx != 0: 147 | yield batch 148 | batch = [] 149 | 150 | 151 | @staticmethod 152 | def get_precision(trueLabels, predictedLabels): 153 | 154 | countCorrect1 = 0 155 | countCorrect5 = 0 156 | 157 | if len(trueLabels) != len(predictedLabels): 158 | print 'True and Predicted lists have different size.' 159 | print len(trueLabels), " True labels" 160 | print len(predictedLabels), " Predicted labels" 161 | 162 | return 0 163 | 164 | for index, item in enumerate(trueLabels): 165 | 166 | prediction = predictedLabels[index] 167 | if prediction[0] == item: 168 | countCorrect1 += 1 169 | 170 | if item in prediction: 171 | countCorrect5 += 1 172 | 173 | percentage1 = 100.0 * countCorrect1/len(trueLabels) 174 | percentage5 = 100.0 * countCorrect5/len(trueLabels) 175 | 176 | print percentage1, ' % Top1 Correct predictions' 177 | print percentage5, ' % Top5 Correct predictions' 178 | 179 | 180 | return percentage1 181 | 182 | 183 | @staticmethod 184 | def plot_confusion_matrix(truePredicted, inlierPredicted, classes, 185 | title='Confusion matrix'): 186 | """ 187 | This function prints and plots the confusion matrix. 188 | Normalization can be applied by setting `normalize=True`. 189 | """ 190 | cm = confusion_matrix(truePredicted, inlierPredicted) 191 | 192 | cmap=plt.cm.Blues 193 | 194 | cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] 195 | plt.imshow(cm, interpolation='nearest', cmap=cmap) 196 | plt.title(title) 197 | plt.colorbar() 198 | tick_marks = np.arange(len(classes)) 199 | plt.xticks(tick_marks, classes, rotation=45) 200 | plt.yticks(tick_marks, classes) 201 | np.set_printoptions(precision=2) 202 | 203 | cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] 204 | thresh = cm.max() / 2. 205 | for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])): 206 | plt.text(j, i, np.around(cm[i, j], decimals=2), 207 | horizontalalignment="center", 208 | color="white" if cm[i, j] > thresh else "black") 209 | 210 | plt.tight_layout() 211 | plt.ylabel('True labels') 212 | plt.xlabel('Predicted labels') 213 | plt.savefig(title + ".png") 214 | 215 | return 216 | 217 | @staticmethod 218 | def outputs_to_synsets(output, word_synsets): 219 | synsets = [] 220 | 221 | for value in output: 222 | synset = word_synsets[value] 223 | synsets.append(synset) 224 | 225 | return synsets 226 | 227 | @staticmethod 228 | def synsets_to_words(synsets): 229 | new_labels = [] 230 | 231 | for synset in synsets: 232 | 233 | word = synset.split(',')[0] 234 | new_labels.append(word) 235 | 236 | return new_labels 237 | -------------------------------------------------------------------------------- /cnn_occlusion.py: -------------------------------------------------------------------------------- 1 | import os 2 | os.environ['GLOG_minloglevel'] = '3' 3 | 4 | import warnings 5 | warnings.filterwarnings("ignore") 6 | 7 | import sys 8 | sys.path.append("/home/ale/libs/caffe/python") 9 | 10 | import caffe 11 | import argparse 12 | import itertools 13 | import caffe_utils 14 | 15 | import numpy as np 16 | import matplotlib.pyplot as plt 17 | 18 | from matplotlib.patches import Rectangle 19 | from matplotlib.colors import Normalize 20 | 21 | 22 | def backspace(n): 23 | sys.stdout.write('\r'+n) 24 | sys.stdout.flush() 25 | 26 | 27 | def apply_mask_iterator(img, mask_size=20, stride=1, batch_size=100): 28 | half_mask_size = int(mask_size/2) 29 | 30 | x_max_img = img.shape[0] 31 | y_max_img = img.shape[1] 32 | x_min_img = 0 33 | y_min_img = 0 34 | 35 | batch = [] 36 | positions = [] 37 | 38 | x_range = range(x_min_img, x_max_img, stride) 39 | y_range = range(y_min_img, y_max_img, stride) 40 | 41 | total_samples = len(x_range)*len(y_range) 42 | 43 | for idx, (x, y) in enumerate(itertools.product(x_range, y_range)): 44 | 45 | x_min = max(x - half_mask_size, x_min_img) 46 | x_max = min(x + half_mask_size, x_max_img) 47 | 48 | y_min = max(y - half_mask_size, y_min_img) 49 | y_max = min(y + half_mask_size, y_max_img) 50 | 51 | new_img = img.copy() 52 | new_img[x_min:x_max, y_min:y_max] = [0, 0, 0] 53 | batch.append(new_img) 54 | positions.append([x,y]) 55 | 56 | if len(batch) % batch_size == 0 or idx == total_samples - 1: 57 | yield batch, positions, total_samples 58 | batch = [] 59 | positions = [] 60 | 61 | 62 | def check_positive(value): 63 | try: 64 | ivalue = int(value) 65 | except ValueError: 66 | raise argparse.ArgumentTypeError("%s is not an int value" % value) 67 | if ivalue <= 0: 68 | raise argparse.ArgumentTypeError("%s is not a positive int value" % value) 69 | return ivalue 70 | 71 | 72 | def main(): 73 | 74 | # retrieving files path 75 | pycaffe_path = os.path.dirname(caffe.__file__) 76 | caffe_path = os.path.normpath(os.path.join(pycaffe_path, '../../')) 77 | mean_path = os.path.join(pycaffe_path, 'imagenet/ilsvrc_2012_mean.npy') 78 | synsets_num_path = os.path.join(os.getcwd(), 'synsets.txt') 79 | synsets_to_class_path = os.path.join(os.getcwd(), 'synset_words.txt') 80 | 81 | # building dictionaries and inverse ones 82 | idx_to_synset = {} 83 | synset_to_idx = {} 84 | 85 | with open(synsets_num_path, 'r') as fp: 86 | for idx, synset in enumerate(fp): 87 | synset = synset.strip() 88 | idx_to_synset[idx] = synset 89 | synset_to_idx[synset] = idx 90 | 91 | synset_to_class = {} 92 | class_to_synset = {} 93 | 94 | with open(synsets_to_class_path, 'r') as fp: 95 | for line in fp: 96 | synset_class = line.strip().split() 97 | synset = synset_class[0] 98 | class_ = " ".join(synset_class[1:]) 99 | synset_to_class[synset] = class_ 100 | class_to_synset[class_] = synset 101 | 102 | parser = argparse.ArgumentParser() 103 | 104 | parser.add_argument('image_path', metavar='PATH', type=str, 105 | help="Input image path, an ImageNet one is required.") 106 | parser.add_argument("-w", "--weights", metavar='PATH', type=str, 107 | help="the model file, (default: %(default)s).", 108 | default=os.path.join(caffe_path, 'models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel')) 109 | parser.add_argument("-p", "--prototxt", metavar='PATH', type=str, 110 | help="prototxt file, (default: %(default)s).", 111 | default=os.path.join(caffe_path, 'models/bvlc_reference_caffenet/deploy.prototxt')) 112 | parser.add_argument("-l", "--layer", default='pool5', metavar='layer_name', type=str, 113 | help="Extraction layer, (default: %(default)s)") 114 | parser.add_argument("-g", "--gpu", default=-1, metavar='INT', type=int, 115 | help="GPU number, (default: %(default)s aka disabled)") 116 | parser.add_argument("--batch_size", type=int, default=1, metavar='INT', 117 | help="Batch size, (default: %(default)s).") 118 | parser.add_argument("--stride", type=check_positive, default=20, metavar='INT', 119 | help="The stride of the applied mask, (default: %(default)s).") 120 | parser.add_argument("--mask_size", type=check_positive, default=50, metavar='INT', 121 | help="The length of the side of the square mask, (default: %(default)s).") 122 | args = parser.parse_args() 123 | 124 | model_filename = args.prototxt 125 | weight_filename = args.weights 126 | image_path = args.image_path 127 | batch_size = args.batch_size 128 | extraction_layer = args.layer 129 | stride = args.stride 130 | mask_size = args.mask_size 131 | 132 | # setting the mode, the default is cpu mode 133 | caffe.set_mode_cpu() 134 | 135 | if args.gpu > -1: 136 | caffe.set_mode_gpu() 137 | caffe.set_device(args.gpu) 138 | 139 | if os.path.isfile(model_filename): 140 | print 'Caffe model found.' 141 | else: 142 | print 'Caffe model NOT found...' 143 | sys.exit() 144 | 145 | # Loading net and utilities 146 | net = caffe_utils.CaffeNet(model_filename, weight_filename, mean_path, batch_size=batch_size) 147 | 148 | # Loading image to process 149 | img = caffe.io.load_image(image_path) 150 | synset = os.path.basename(image_path).split('_')[0] 151 | 152 | # preprocessing and extracting most active filter 153 | preprocessed_img = net.preprocess_images([img]) 154 | img_features = net.get_features(preprocessed_img, extraction_layer) 155 | 156 | # getting the most active filter index for the image 157 | most_active_filter = net.get_most_active_filters(img_features)[0][0] 158 | 159 | images_features = [] 160 | synset_probabilities = [] 161 | predicted_idxs = [] 162 | all_positions = [] 163 | true_synset_idx = synset_to_idx[synset] 164 | 165 | print '####################################' 166 | print 'True synset: ', synset 167 | print 'True class: ', synset_to_class[synset] 168 | print '####################################' 169 | 170 | # the mask is applied before the image preprocessing 171 | iterator_ = apply_mask_iterator(img, mask_size=mask_size, stride=stride, batch_size=batch_size) 172 | 173 | for masked_images, positions, total_samples in iterator_: 174 | 175 | # storing the central position of the applied mask 176 | all_positions.extend(positions) 177 | 178 | # preprocessing the image according to the network input 179 | preprocessed_images = net.preprocess_images(masked_images) 180 | 181 | # extracting the softmax output and the most active convolutional features from the extraction layer 182 | probs, features = net.get_probs_and_features(preprocessed_images, extraction_layer, most_active_filter) 183 | images_features.extend(features) 184 | 185 | synset_probabilities.extend([x[true_synset_idx] for x in probs]) 186 | best_synsets_idxs = [np.argsort(x)[-1] for x in probs] 187 | predicted_idxs.extend(best_synsets_idxs) 188 | 189 | to_print = '{} of {}'.format(len(images_features), total_samples) 190 | 191 | backspace(to_print) 192 | 193 | print 194 | 195 | # heat_map of probability of the true class, 196 | heat_map_size = img.shape[:2] 197 | 198 | # initializing heatmaps 199 | heat_map_probs = np.zeros(heat_map_size) 200 | heat_map_features = np.zeros(heat_map_size) 201 | heat_map_synsets = np.zeros(heat_map_size) 202 | heat_map_num = np.zeros(heat_map_size) 203 | 204 | # filling heatmaps 205 | for [x, y], prob, feature, predicted_idx in zip(all_positions, synset_probabilities, images_features, predicted_idxs): 206 | heat_map_probs[x, y] = prob 207 | heat_map_features[x, y] = np.mean(feature) 208 | heat_map_synsets[x, y] = predicted_idx 209 | heat_map_num[x, y] = 1 210 | 211 | # deleting empty rows and columns 212 | heat_map_probs = np.nan_to_num(np.divide(heat_map_probs, heat_map_num)) 213 | means_0 = np.mean(heat_map_num, axis=1) 214 | heat_map_num = np.delete(heat_map_num, np.where(means_0 == 0)[0], axis=0) 215 | means_1 = np.mean(heat_map_num, axis=0) 216 | heat_map_num = np.delete(heat_map_num, np.where(means_1 == 0)[0], axis=1) 217 | 218 | heat_map_probs = np.delete(heat_map_probs, np.where(means_0 == 0)[0], axis=0) 219 | heat_map_probs = np.delete(heat_map_probs, np.where(means_1 == 0)[0], axis=1) 220 | 221 | heat_map_features = np.delete(heat_map_features, np.where(means_0 == 0)[0], axis=0) 222 | heat_map_features = np.delete(heat_map_features, np.where(means_1 == 0)[0], axis=1) 223 | max_feature = np.max(heat_map_features) 224 | min_feature = np.min(heat_map_features) 225 | 226 | heat_map_features = (heat_map_features - min_feature)/(max_feature - min_feature) 227 | heat_map_synsets = np.delete(heat_map_synsets, np.where(means_0 == 0)[0], axis=0) 228 | heat_map_synsets = np.delete(heat_map_synsets, np.where(means_1 == 0)[0], axis=1) 229 | 230 | f, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2) 231 | f.subplots_adjust() 232 | 233 | ax1.imshow(img) 234 | ax1.set_title('Original image') 235 | 236 | cmap = plt.get_cmap('YlOrRd') 237 | img = ax2.imshow(heat_map_probs, cmap=cmap, vmin=0, vmax=1, interpolation='none') 238 | ax2.axis('off') 239 | ax2.set_title('Classifier, probability of correct class') 240 | plt.colorbar(img, ax=ax2, fraction=0.046, pad=0.04) 241 | 242 | img = ax3.imshow(heat_map_features, cmap=cmap, interpolation='none') 243 | ax3.axis('off') 244 | ax3.set_title('Strongest feature map') 245 | plt.colorbar(img, ax=ax3, fraction=0.046, pad=0.04) 246 | 247 | norm = Normalize(vmin=0, vmax=len(idx_to_synset)) 248 | 249 | cmap = plt.get_cmap('rainbow') 250 | ax4.imshow(heat_map_synsets, cmap=cmap, interpolation='none', norm=norm) 251 | ax4.axis('off') 252 | ax4.set_title('Classifier, most probable class') 253 | synsets_set = list(set(heat_map_synsets.flatten().tolist())) 254 | class_set = [synset_to_class[idx_to_synset[x]].split(',')[0] for x in synsets_set] 255 | #synsets_set = [x for x in xrange(0,999,1000/len(synsets_set))] 256 | colors = [cmap(norm(x)) for x in synsets_set] 257 | handles = [] 258 | for synset_id, color in zip(synsets_set, colors): 259 | handles.append(Rectangle((0,0),1,1, color=list(color[:3]))) 260 | box = ax4.get_position() 261 | ax4.set_position([box.x0, box.y0 + box.height * 0.1, 262 | box.width, box.height * 0.9]) 263 | ax4.legend(handles, class_set, loc='upper center', bbox_to_anchor=(0.5, -0.05), 264 | fancybox=True, shadow=True, ncol=3) 265 | 266 | plt.show() 267 | 268 | 269 | if __name__=='__main__': 270 | main() 271 | -------------------------------------------------------------------------------- /dataset_utils.py: -------------------------------------------------------------------------------- 1 | import xml.etree.ElementTree as ET 2 | from os import walk, mkdir, remove, stat, listdir 3 | import os 4 | import numpy as np 5 | import caffe 6 | 7 | 8 | def createSamplesDatastructures(images_dir, annotations_dir, interesting_labels): 9 | samplesNames = [] 10 | samplesImages = [] 11 | samplesLabels = [] 12 | 13 | for root, dirs, files in walk(images_dir): 14 | for image_name in files: 15 | name, extension = image_name.split(".") 16 | 17 | samplesNames.append(name) 18 | 19 | imageCompletePath = images_dir + '/' + image_name 20 | image = caffe.io.load_image(imageCompletePath) 21 | samplesImages.append(image) 22 | 23 | annotationCompletePath = annotations_dir + '/' + name + '.xml' 24 | label = readLabelFromAnnotation(annotationCompletePath, interesting_labels) 25 | samplesLabels.append(label) 26 | 27 | return [samplesNames, samplesImages, samplesLabels] 28 | 29 | 30 | def normalizeData(featuresVector): 31 | 32 | featureVectorsNormalized = [] 33 | 34 | for vec in featuresVector: 35 | vecNormalized = vec / np.linalg.norm(vec) 36 | featureVectorsNormalized.append(vecNormalized) 37 | 38 | mean = np.mean(featureVectorsNormalized, axis=0) 39 | 40 | featureVectorsNormalizedCentered = [] 41 | 42 | for vec in featureVectorsNormalized: 43 | vecCentered = vec - mean 44 | featureVectorsNormalizedCentered.append(vecCentered) 45 | 46 | return featureVectorsNormalizedCentered 47 | 48 | 49 | def readLabelFromAnnotation(annotationFileName, interesting_labels): 50 | # Parse the given annotation file and read the label 51 | 52 | tree = ET.parse(annotationFileName) 53 | root = tree.getroot() 54 | for obj in root.findall('object'): 55 | label = obj.find("name").text 56 | if label in interesting_labels: 57 | return label 58 | else: 59 | return 'unknown' 60 | 61 | 62 | def loadImageNetFiles(folder, numImages = 100, classes=None): 63 | 64 | samples_names = [] 65 | samples_images = [] 66 | samples_labels = [] 67 | 68 | for root, dirs, files in walk(folder): 69 | 70 | label = root.split(os.path.sep)[-1] 71 | 72 | if classes and label not in classes: 73 | continue 74 | 75 | for count, image_name in enumerate(files): 76 | 77 | if count == numImages: 78 | break 79 | 80 | name, extension = image_name.split(".") 81 | 82 | samples_names.append(name) 83 | 84 | image_complete_path = os.path.join(root, image_name) 85 | image = caffe.io.load_image(image_complete_path) 86 | samples_images.append(image) 87 | samples_labels.append(label) 88 | 89 | return [samples_names, samples_images, samples_labels] 90 | 91 | 92 | 93 | -------------------------------------------------------------------------------- /occlusion_output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alsora/CNN_Visualization/782071bd912c42342dc28b0edc71a9431fa7c173/occlusion_output.png -------------------------------------------------------------------------------- /synset_words.txt: -------------------------------------------------------------------------------- 1 | n01440764 tench, Tinca tinca 2 | n01443537 goldfish, Carassius auratus 3 | n01484850 great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias 4 | n01491361 tiger shark, Galeocerdo cuvieri 5 | n01494475 hammerhead, hammerhead shark 6 | n01496331 electric ray, crampfish, numbfish, torpedo 7 | n01498041 stingray 8 | n01514668 cock 9 | n01514859 hen 10 | n01518878 ostrich, Struthio camelus 11 | n01530575 brambling, Fringilla montifringilla 12 | n01531178 goldfinch, Carduelis carduelis 13 | n01532829 house finch, linnet, Carpodacus mexicanus 14 | n01534433 junco, snowbird 15 | n01537544 indigo bunting, indigo finch, indigo bird, Passerina cyanea 16 | n01558993 robin, American robin, Turdus migratorius 17 | n01560419 bulbul 18 | n01580077 jay 19 | n01582220 magpie 20 | n01592084 chickadee 21 | n01601694 water ouzel, dipper 22 | n01608432 kite 23 | n01614925 bald eagle, American eagle, Haliaeetus leucocephalus 24 | n01616318 vulture 25 | n01622779 great grey owl, great gray owl, Strix nebulosa 26 | n01629819 European fire salamander, Salamandra salamandra 27 | n01630670 common newt, Triturus vulgaris 28 | n01631663 eft 29 | n01632458 spotted salamander, Ambystoma maculatum 30 | n01632777 axolotl, mud puppy, Ambystoma mexicanum 31 | n01641577 bullfrog, Rana catesbeiana 32 | n01644373 tree frog, tree-frog 33 | n01644900 tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui 34 | n01664065 loggerhead, loggerhead turtle, Caretta caretta 35 | n01665541 leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea 36 | n01667114 mud turtle 37 | n01667778 terrapin 38 | n01669191 box turtle, box tortoise 39 | n01675722 banded gecko 40 | n01677366 common iguana, iguana, Iguana iguana 41 | n01682714 American chameleon, anole, Anolis carolinensis 42 | n01685808 whiptail, whiptail lizard 43 | n01687978 agama 44 | n01688243 frilled lizard, Chlamydosaurus kingi 45 | n01689811 alligator lizard 46 | n01692333 Gila monster, Heloderma suspectum 47 | n01693334 green lizard, Lacerta viridis 48 | n01694178 African chameleon, Chamaeleo chamaeleon 49 | n01695060 Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis 50 | n01697457 African crocodile, Nile crocodile, Crocodylus niloticus 51 | n01698640 American alligator, Alligator mississipiensis 52 | n01704323 triceratops 53 | n01728572 thunder snake, worm snake, Carphophis amoenus 54 | n01728920 ringneck snake, ring-necked snake, ring snake 55 | n01729322 hognose snake, puff adder, sand viper 56 | n01729977 green snake, grass snake 57 | n01734418 king snake, kingsnake 58 | n01735189 garter snake, grass snake 59 | n01737021 water snake 60 | n01739381 vine snake 61 | n01740131 night snake, Hypsiglena torquata 62 | n01742172 boa constrictor, Constrictor constrictor 63 | n01744401 rock python, rock snake, Python sebae 64 | n01748264 Indian cobra, Naja naja 65 | n01749939 green mamba 66 | n01751748 sea snake 67 | n01753488 horned viper, cerastes, sand viper, horned asp, Cerastes cornutus 68 | n01755581 diamondback, diamondback rattlesnake, Crotalus adamanteus 69 | n01756291 sidewinder, horned rattlesnake, Crotalus cerastes 70 | n01768244 trilobite 71 | n01770081 harvestman, daddy longlegs, Phalangium opilio 72 | n01770393 scorpion 73 | n01773157 black and gold garden spider, Argiope aurantia 74 | n01773549 barn spider, Araneus cavaticus 75 | n01773797 garden spider, Aranea diademata 76 | n01774384 black widow, Latrodectus mactans 77 | n01774750 tarantula 78 | n01775062 wolf spider, hunting spider 79 | n01776313 tick 80 | n01784675 centipede 81 | n01795545 black grouse 82 | n01796340 ptarmigan 83 | n01797886 ruffed grouse, partridge, Bonasa umbellus 84 | n01798484 prairie chicken, prairie grouse, prairie fowl 85 | n01806143 peacock 86 | n01806567 quail 87 | n01807496 partridge 88 | n01817953 African grey, African gray, Psittacus erithacus 89 | n01818515 macaw 90 | n01819313 sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita 91 | n01820546 lorikeet 92 | n01824575 coucal 93 | n01828970 bee eater 94 | n01829413 hornbill 95 | n01833805 hummingbird 96 | n01843065 jacamar 97 | n01843383 toucan 98 | n01847000 drake 99 | n01855032 red-breasted merganser, Mergus serrator 100 | n01855672 goose 101 | n01860187 black swan, Cygnus atratus 102 | n01871265 tusker 103 | n01872401 echidna, spiny anteater, anteater 104 | n01873310 platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus 105 | n01877812 wallaby, brush kangaroo 106 | n01882714 koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus 107 | n01883070 wombat 108 | n01910747 jellyfish 109 | n01914609 sea anemone, anemone 110 | n01917289 brain coral 111 | n01924916 flatworm, platyhelminth 112 | n01930112 nematode, nematode worm, roundworm 113 | n01943899 conch 114 | n01944390 snail 115 | n01945685 slug 116 | n01950731 sea slug, nudibranch 117 | n01955084 chiton, coat-of-mail shell, sea cradle, polyplacophore 118 | n01968897 chambered nautilus, pearly nautilus, nautilus 119 | n01978287 Dungeness crab, Cancer magister 120 | n01978455 rock crab, Cancer irroratus 121 | n01980166 fiddler crab 122 | n01981276 king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica 123 | n01983481 American lobster, Northern lobster, Maine lobster, Homarus americanus 124 | n01984695 spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish 125 | n01985128 crayfish, crawfish, crawdad, crawdaddy 126 | n01986214 hermit crab 127 | n01990800 isopod 128 | n02002556 white stork, Ciconia ciconia 129 | n02002724 black stork, Ciconia nigra 130 | n02006656 spoonbill 131 | n02007558 flamingo 132 | n02009229 little blue heron, Egretta caerulea 133 | n02009912 American egret, great white heron, Egretta albus 134 | n02011460 bittern 135 | n02012849 crane 136 | n02013706 limpkin, Aramus pictus 137 | n02017213 European gallinule, Porphyrio porphyrio 138 | n02018207 American coot, marsh hen, mud hen, water hen, Fulica americana 139 | n02018795 bustard 140 | n02025239 ruddy turnstone, Arenaria interpres 141 | n02027492 red-backed sandpiper, dunlin, Erolia alpina 142 | n02028035 redshank, Tringa totanus 143 | n02033041 dowitcher 144 | n02037110 oystercatcher, oyster catcher 145 | n02051845 pelican 146 | n02056570 king penguin, Aptenodytes patagonica 147 | n02058221 albatross, mollymawk 148 | n02066245 grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus 149 | n02071294 killer whale, killer, orca, grampus, sea wolf, Orcinus orca 150 | n02074367 dugong, Dugong dugon 151 | n02077923 sea lion 152 | n02085620 Chihuahua 153 | n02085782 Japanese spaniel 154 | n02085936 Maltese dog, Maltese terrier, Maltese 155 | n02086079 Pekinese, Pekingese, Peke 156 | n02086240 Shih-Tzu 157 | n02086646 Blenheim spaniel 158 | n02086910 papillon 159 | n02087046 toy terrier 160 | n02087394 Rhodesian ridgeback 161 | n02088094 Afghan hound, Afghan 162 | n02088238 basset, basset hound 163 | n02088364 beagle 164 | n02088466 bloodhound, sleuthhound 165 | n02088632 bluetick 166 | n02089078 black-and-tan coonhound 167 | n02089867 Walker hound, Walker foxhound 168 | n02089973 English foxhound 169 | n02090379 redbone 170 | n02090622 borzoi, Russian wolfhound 171 | n02090721 Irish wolfhound 172 | n02091032 Italian greyhound 173 | n02091134 whippet 174 | n02091244 Ibizan hound, Ibizan Podenco 175 | n02091467 Norwegian elkhound, elkhound 176 | n02091635 otterhound, otter hound 177 | n02091831 Saluki, gazelle hound 178 | n02092002 Scottish deerhound, deerhound 179 | n02092339 Weimaraner 180 | n02093256 Staffordshire bullterrier, Staffordshire bull terrier 181 | n02093428 American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier 182 | n02093647 Bedlington terrier 183 | n02093754 Border terrier 184 | n02093859 Kerry blue terrier 185 | n02093991 Irish terrier 186 | n02094114 Norfolk terrier 187 | n02094258 Norwich terrier 188 | n02094433 Yorkshire terrier 189 | n02095314 wire-haired fox terrier 190 | n02095570 Lakeland terrier 191 | n02095889 Sealyham terrier, Sealyham 192 | n02096051 Airedale, Airedale terrier 193 | n02096177 cairn, cairn terrier 194 | n02096294 Australian terrier 195 | n02096437 Dandie Dinmont, Dandie Dinmont terrier 196 | n02096585 Boston bull, Boston terrier 197 | n02097047 miniature schnauzer 198 | n02097130 giant schnauzer 199 | n02097209 standard schnauzer 200 | n02097298 Scotch terrier, Scottish terrier, Scottie 201 | n02097474 Tibetan terrier, chrysanthemum dog 202 | n02097658 silky terrier, Sydney silky 203 | n02098105 soft-coated wheaten terrier 204 | n02098286 West Highland white terrier 205 | n02098413 Lhasa, Lhasa apso 206 | n02099267 flat-coated retriever 207 | n02099429 curly-coated retriever 208 | n02099601 golden retriever 209 | n02099712 Labrador retriever 210 | n02099849 Chesapeake Bay retriever 211 | n02100236 German short-haired pointer 212 | n02100583 vizsla, Hungarian pointer 213 | n02100735 English setter 214 | n02100877 Irish setter, red setter 215 | n02101006 Gordon setter 216 | n02101388 Brittany spaniel 217 | n02101556 clumber, clumber spaniel 218 | n02102040 English springer, English springer spaniel 219 | n02102177 Welsh springer spaniel 220 | n02102318 cocker spaniel, English cocker spaniel, cocker 221 | n02102480 Sussex spaniel 222 | n02102973 Irish water spaniel 223 | n02104029 kuvasz 224 | n02104365 schipperke 225 | n02105056 groenendael 226 | n02105162 malinois 227 | n02105251 briard 228 | n02105412 kelpie 229 | n02105505 komondor 230 | n02105641 Old English sheepdog, bobtail 231 | n02105855 Shetland sheepdog, Shetland sheep dog, Shetland 232 | n02106030 collie 233 | n02106166 Border collie 234 | n02106382 Bouvier des Flandres, Bouviers des Flandres 235 | n02106550 Rottweiler 236 | n02106662 German shepherd, German shepherd dog, German police dog, alsatian 237 | n02107142 Doberman, Doberman pinscher 238 | n02107312 miniature pinscher 239 | n02107574 Greater Swiss Mountain dog 240 | n02107683 Bernese mountain dog 241 | n02107908 Appenzeller 242 | n02108000 EntleBucher 243 | n02108089 boxer 244 | n02108422 bull mastiff 245 | n02108551 Tibetan mastiff 246 | n02108915 French bulldog 247 | n02109047 Great Dane 248 | n02109525 Saint Bernard, St Bernard 249 | n02109961 Eskimo dog, husky 250 | n02110063 malamute, malemute, Alaskan malamute 251 | n02110185 Siberian husky 252 | n02110341 dalmatian, coach dog, carriage dog 253 | n02110627 affenpinscher, monkey pinscher, monkey dog 254 | n02110806 basenji 255 | n02110958 pug, pug-dog 256 | n02111129 Leonberg 257 | n02111277 Newfoundland, Newfoundland dog 258 | n02111500 Great Pyrenees 259 | n02111889 Samoyed, Samoyede 260 | n02112018 Pomeranian 261 | n02112137 chow, chow chow 262 | n02112350 keeshond 263 | n02112706 Brabancon griffon 264 | n02113023 Pembroke, Pembroke Welsh corgi 265 | n02113186 Cardigan, Cardigan Welsh corgi 266 | n02113624 toy poodle 267 | n02113712 miniature poodle 268 | n02113799 standard poodle 269 | n02113978 Mexican hairless 270 | n02114367 timber wolf, grey wolf, gray wolf, Canis lupus 271 | n02114548 white wolf, Arctic wolf, Canis lupus tundrarum 272 | n02114712 red wolf, maned wolf, Canis rufus, Canis niger 273 | n02114855 coyote, prairie wolf, brush wolf, Canis latrans 274 | n02115641 dingo, warrigal, warragal, Canis dingo 275 | n02115913 dhole, Cuon alpinus 276 | n02116738 African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus 277 | n02117135 hyena, hyaena 278 | n02119022 red fox, Vulpes vulpes 279 | n02119789 kit fox, Vulpes macrotis 280 | n02120079 Arctic fox, white fox, Alopex lagopus 281 | n02120505 grey fox, gray fox, Urocyon cinereoargenteus 282 | n02123045 tabby, tabby cat 283 | n02123159 tiger cat 284 | n02123394 Persian cat 285 | n02123597 Siamese cat, Siamese 286 | n02124075 Egyptian cat 287 | n02125311 cougar, puma, catamount, mountain lion, painter, panther, Felis concolor 288 | n02127052 lynx, catamount 289 | n02128385 leopard, Panthera pardus 290 | n02128757 snow leopard, ounce, Panthera uncia 291 | n02128925 jaguar, panther, Panthera onca, Felis onca 292 | n02129165 lion, king of beasts, Panthera leo 293 | n02129604 tiger, Panthera tigris 294 | n02130308 cheetah, chetah, Acinonyx jubatus 295 | n02132136 brown bear, bruin, Ursus arctos 296 | n02133161 American black bear, black bear, Ursus americanus, Euarctos americanus 297 | n02134084 ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus 298 | n02134418 sloth bear, Melursus ursinus, Ursus ursinus 299 | n02137549 mongoose 300 | n02138441 meerkat, mierkat 301 | n02165105 tiger beetle 302 | n02165456 ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle 303 | n02167151 ground beetle, carabid beetle 304 | n02168699 long-horned beetle, longicorn, longicorn beetle 305 | n02169497 leaf beetle, chrysomelid 306 | n02172182 dung beetle 307 | n02174001 rhinoceros beetle 308 | n02177972 weevil 309 | n02190166 fly 310 | n02206856 bee 311 | n02219486 ant, emmet, pismire 312 | n02226429 grasshopper, hopper 313 | n02229544 cricket 314 | n02231487 walking stick, walkingstick, stick insect 315 | n02233338 cockroach, roach 316 | n02236044 mantis, mantid 317 | n02256656 cicada, cicala 318 | n02259212 leafhopper 319 | n02264363 lacewing, lacewing fly 320 | n02268443 dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk 321 | n02268853 damselfly 322 | n02276258 admiral 323 | n02277742 ringlet, ringlet butterfly 324 | n02279972 monarch, monarch butterfly, milkweed butterfly, Danaus plexippus 325 | n02280649 cabbage butterfly 326 | n02281406 sulphur butterfly, sulfur butterfly 327 | n02281787 lycaenid, lycaenid butterfly 328 | n02317335 starfish, sea star 329 | n02319095 sea urchin 330 | n02321529 sea cucumber, holothurian 331 | n02325366 wood rabbit, cottontail, cottontail rabbit 332 | n02326432 hare 333 | n02328150 Angora, Angora rabbit 334 | n02342885 hamster 335 | n02346627 porcupine, hedgehog 336 | n02356798 fox squirrel, eastern fox squirrel, Sciurus niger 337 | n02361337 marmot 338 | n02363005 beaver 339 | n02364673 guinea pig, Cavia cobaya 340 | n02389026 sorrel 341 | n02391049 zebra 342 | n02395406 hog, pig, grunter, squealer, Sus scrofa 343 | n02396427 wild boar, boar, Sus scrofa 344 | n02397096 warthog 345 | n02398521 hippopotamus, hippo, river horse, Hippopotamus amphibius 346 | n02403003 ox 347 | n02408429 water buffalo, water ox, Asiatic buffalo, Bubalus bubalis 348 | n02410509 bison 349 | n02412080 ram, tup 350 | n02415577 bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis 351 | n02417914 ibex, Capra ibex 352 | n02422106 hartebeest 353 | n02422699 impala, Aepyceros melampus 354 | n02423022 gazelle 355 | n02437312 Arabian camel, dromedary, Camelus dromedarius 356 | n02437616 llama 357 | n02441942 weasel 358 | n02442845 mink 359 | n02443114 polecat, fitch, foulmart, foumart, Mustela putorius 360 | n02443484 black-footed ferret, ferret, Mustela nigripes 361 | n02444819 otter 362 | n02445715 skunk, polecat, wood pussy 363 | n02447366 badger 364 | n02454379 armadillo 365 | n02457408 three-toed sloth, ai, Bradypus tridactylus 366 | n02480495 orangutan, orang, orangutang, Pongo pygmaeus 367 | n02480855 gorilla, Gorilla gorilla 368 | n02481823 chimpanzee, chimp, Pan troglodytes 369 | n02483362 gibbon, Hylobates lar 370 | n02483708 siamang, Hylobates syndactylus, Symphalangus syndactylus 371 | n02484975 guenon, guenon monkey 372 | n02486261 patas, hussar monkey, Erythrocebus patas 373 | n02486410 baboon 374 | n02487347 macaque 375 | n02488291 langur 376 | n02488702 colobus, colobus monkey 377 | n02489166 proboscis monkey, Nasalis larvatus 378 | n02490219 marmoset 379 | n02492035 capuchin, ringtail, Cebus capucinus 380 | n02492660 howler monkey, howler 381 | n02493509 titi, titi monkey 382 | n02493793 spider monkey, Ateles geoffroyi 383 | n02494079 squirrel monkey, Saimiri sciureus 384 | n02497673 Madagascar cat, ring-tailed lemur, Lemur catta 385 | n02500267 indri, indris, Indri indri, Indri brevicaudatus 386 | n02504013 Indian elephant, Elephas maximus 387 | n02504458 African elephant, Loxodonta africana 388 | n02509815 lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens 389 | n02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca 390 | n02514041 barracouta, snoek 391 | n02526121 eel 392 | n02536864 coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch 393 | n02606052 rock beauty, Holocanthus tricolor 394 | n02607072 anemone fish 395 | n02640242 sturgeon 396 | n02641379 gar, garfish, garpike, billfish, Lepisosteus osseus 397 | n02643566 lionfish 398 | n02655020 puffer, pufferfish, blowfish, globefish 399 | n02666196 abacus 400 | n02667093 abaya 401 | n02669723 academic gown, academic robe, judge's robe 402 | n02672831 accordion, piano accordion, squeeze box 403 | n02676566 acoustic guitar 404 | n02687172 aircraft carrier, carrier, flattop, attack aircraft carrier 405 | n02690373 airliner 406 | n02692877 airship, dirigible 407 | n02699494 altar 408 | n02701002 ambulance 409 | n02704792 amphibian, amphibious vehicle 410 | n02708093 analog clock 411 | n02727426 apiary, bee house 412 | n02730930 apron 413 | n02747177 ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin 414 | n02749479 assault rifle, assault gun 415 | n02769748 backpack, back pack, knapsack, packsack, rucksack, haversack 416 | n02776631 bakery, bakeshop, bakehouse 417 | n02777292 balance beam, beam 418 | n02782093 balloon 419 | n02783161 ballpoint, ballpoint pen, ballpen, Biro 420 | n02786058 Band Aid 421 | n02787622 banjo 422 | n02788148 bannister, banister, balustrade, balusters, handrail 423 | n02790996 barbell 424 | n02791124 barber chair 425 | n02791270 barbershop 426 | n02793495 barn 427 | n02794156 barometer 428 | n02795169 barrel, cask 429 | n02797295 barrow, garden cart, lawn cart, wheelbarrow 430 | n02799071 baseball 431 | n02802426 basketball 432 | n02804414 bassinet 433 | n02804610 bassoon 434 | n02807133 bathing cap, swimming cap 435 | n02808304 bath towel 436 | n02808440 bathtub, bathing tub, bath, tub 437 | n02814533 beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon 438 | n02814860 beacon, lighthouse, beacon light, pharos 439 | n02815834 beaker 440 | n02817516 bearskin, busby, shako 441 | n02823428 beer bottle 442 | n02823750 beer glass 443 | n02825657 bell cote, bell cot 444 | n02834397 bib 445 | n02835271 bicycle-built-for-two, tandem bicycle, tandem 446 | n02837789 bikini, two-piece 447 | n02840245 binder, ring-binder 448 | n02841315 binoculars, field glasses, opera glasses 449 | n02843684 birdhouse 450 | n02859443 boathouse 451 | n02860847 bobsled, bobsleigh, bob 452 | n02865351 bolo tie, bolo, bola tie, bola 453 | n02869837 bonnet, poke bonnet 454 | n02870880 bookcase 455 | n02871525 bookshop, bookstore, bookstall 456 | n02877765 bottlecap 457 | n02879718 bow 458 | n02883205 bow tie, bow-tie, bowtie 459 | n02892201 brass, memorial tablet, plaque 460 | n02892767 brassiere, bra, bandeau 461 | n02894605 breakwater, groin, groyne, mole, bulwark, seawall, jetty 462 | n02895154 breastplate, aegis, egis 463 | n02906734 broom 464 | n02909870 bucket, pail 465 | n02910353 buckle 466 | n02916936 bulletproof vest 467 | n02917067 bullet train, bullet 468 | n02927161 butcher shop, meat market 469 | n02930766 cab, hack, taxi, taxicab 470 | n02939185 caldron, cauldron 471 | n02948072 candle, taper, wax light 472 | n02950826 cannon 473 | n02951358 canoe 474 | n02951585 can opener, tin opener 475 | n02963159 cardigan 476 | n02965783 car mirror 477 | n02966193 carousel, carrousel, merry-go-round, roundabout, whirligig 478 | n02966687 carpenter's kit, tool kit 479 | n02971356 carton 480 | n02974003 car wheel 481 | n02977058 cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM 482 | n02978881 cassette 483 | n02979186 cassette player 484 | n02980441 castle 485 | n02981792 catamaran 486 | n02988304 CD player 487 | n02992211 cello, violoncello 488 | n02992529 cellular telephone, cellular phone, cellphone, cell, mobile phone 489 | n02999410 chain 490 | n03000134 chainlink fence 491 | n03000247 chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour 492 | n03000684 chain saw, chainsaw 493 | n03014705 chest 494 | n03016953 chiffonier, commode 495 | n03017168 chime, bell, gong 496 | n03018349 china cabinet, china closet 497 | n03026506 Christmas stocking 498 | n03028079 church, church building 499 | n03032252 cinema, movie theater, movie theatre, movie house, picture palace 500 | n03041632 cleaver, meat cleaver, chopper 501 | n03042490 cliff dwelling 502 | n03045698 cloak 503 | n03047690 clog, geta, patten, sabot 504 | n03062245 cocktail shaker 505 | n03063599 coffee mug 506 | n03063689 coffeepot 507 | n03065424 coil, spiral, volute, whorl, helix 508 | n03075370 combination lock 509 | n03085013 computer keyboard, keypad 510 | n03089624 confectionery, confectionary, candy store 511 | n03095699 container ship, containership, container vessel 512 | n03100240 convertible 513 | n03109150 corkscrew, bottle screw 514 | n03110669 cornet, horn, trumpet, trump 515 | n03124043 cowboy boot 516 | n03124170 cowboy hat, ten-gallon hat 517 | n03125729 cradle 518 | n03126707 crane 519 | n03127747 crash helmet 520 | n03127925 crate 521 | n03131574 crib, cot 522 | n03133878 Crock Pot 523 | n03134739 croquet ball 524 | n03141823 crutch 525 | n03146219 cuirass 526 | n03160309 dam, dike, dyke 527 | n03179701 desk 528 | n03180011 desktop computer 529 | n03187595 dial telephone, dial phone 530 | n03188531 diaper, nappy, napkin 531 | n03196217 digital clock 532 | n03197337 digital watch 533 | n03201208 dining table, board 534 | n03207743 dishrag, dishcloth 535 | n03207941 dishwasher, dish washer, dishwashing machine 536 | n03208938 disk brake, disc brake 537 | n03216828 dock, dockage, docking facility 538 | n03218198 dogsled, dog sled, dog sleigh 539 | n03220513 dome 540 | n03223299 doormat, welcome mat 541 | n03240683 drilling platform, offshore rig 542 | n03249569 drum, membranophone, tympan 543 | n03250847 drumstick 544 | n03255030 dumbbell 545 | n03259280 Dutch oven 546 | n03271574 electric fan, blower 547 | n03272010 electric guitar 548 | n03272562 electric locomotive 549 | n03290653 entertainment center 550 | n03291819 envelope 551 | n03297495 espresso maker 552 | n03314780 face powder 553 | n03325584 feather boa, boa 554 | n03337140 file, file cabinet, filing cabinet 555 | n03344393 fireboat 556 | n03345487 fire engine, fire truck 557 | n03347037 fire screen, fireguard 558 | n03355925 flagpole, flagstaff 559 | n03372029 flute, transverse flute 560 | n03376595 folding chair 561 | n03379051 football helmet 562 | n03384352 forklift 563 | n03388043 fountain 564 | n03388183 fountain pen 565 | n03388549 four-poster 566 | n03393912 freight car 567 | n03394916 French horn, horn 568 | n03400231 frying pan, frypan, skillet 569 | n03404251 fur coat 570 | n03417042 garbage truck, dustcart 571 | n03424325 gasmask, respirator, gas helmet 572 | n03425413 gas pump, gasoline pump, petrol pump, island dispenser 573 | n03443371 goblet 574 | n03444034 go-kart 575 | n03445777 golf ball 576 | n03445924 golfcart, golf cart 577 | n03447447 gondola 578 | n03447721 gong, tam-tam 579 | n03450230 gown 580 | n03452741 grand piano, grand 581 | n03457902 greenhouse, nursery, glasshouse 582 | n03459775 grille, radiator grille 583 | n03461385 grocery store, grocery, food market, market 584 | n03467068 guillotine 585 | n03476684 hair slide 586 | n03476991 hair spray 587 | n03478589 half track 588 | n03481172 hammer 589 | n03482405 hamper 590 | n03483316 hand blower, blow dryer, blow drier, hair dryer, hair drier 591 | n03485407 hand-held computer, hand-held microcomputer 592 | n03485794 handkerchief, hankie, hanky, hankey 593 | n03492542 hard disc, hard disk, fixed disk 594 | n03494278 harmonica, mouth organ, harp, mouth harp 595 | n03495258 harp 596 | n03496892 harvester, reaper 597 | n03498962 hatchet 598 | n03527444 holster 599 | n03529860 home theater, home theatre 600 | n03530642 honeycomb 601 | n03532672 hook, claw 602 | n03534580 hoopskirt, crinoline 603 | n03535780 horizontal bar, high bar 604 | n03538406 horse cart, horse-cart 605 | n03544143 hourglass 606 | n03584254 iPod 607 | n03584829 iron, smoothing iron 608 | n03590841 jack-o'-lantern 609 | n03594734 jean, blue jean, denim 610 | n03594945 jeep, landrover 611 | n03595614 jersey, T-shirt, tee shirt 612 | n03598930 jigsaw puzzle 613 | n03599486 jinrikisha, ricksha, rickshaw 614 | n03602883 joystick 615 | n03617480 kimono 616 | n03623198 knee pad 617 | n03627232 knot 618 | n03630383 lab coat, laboratory coat 619 | n03633091 ladle 620 | n03637318 lampshade, lamp shade 621 | n03642806 laptop, laptop computer 622 | n03649909 lawn mower, mower 623 | n03657121 lens cap, lens cover 624 | n03658185 letter opener, paper knife, paperknife 625 | n03661043 library 626 | n03662601 lifeboat 627 | n03666591 lighter, light, igniter, ignitor 628 | n03670208 limousine, limo 629 | n03673027 liner, ocean liner 630 | n03676483 lipstick, lip rouge 631 | n03680355 Loafer 632 | n03690938 lotion 633 | n03691459 loudspeaker, speaker, speaker unit, loudspeaker system, speaker system 634 | n03692522 loupe, jeweler's loupe 635 | n03697007 lumbermill, sawmill 636 | n03706229 magnetic compass 637 | n03709823 mailbag, postbag 638 | n03710193 mailbox, letter box 639 | n03710637 maillot 640 | n03710721 maillot, tank suit 641 | n03717622 manhole cover 642 | n03720891 maraca 643 | n03721384 marimba, xylophone 644 | n03724870 mask 645 | n03729826 matchstick 646 | n03733131 maypole 647 | n03733281 maze, labyrinth 648 | n03733805 measuring cup 649 | n03742115 medicine chest, medicine cabinet 650 | n03743016 megalith, megalithic structure 651 | n03759954 microphone, mike 652 | n03761084 microwave, microwave oven 653 | n03763968 military uniform 654 | n03764736 milk can 655 | n03769881 minibus 656 | n03770439 miniskirt, mini 657 | n03770679 minivan 658 | n03773504 missile 659 | n03775071 mitten 660 | n03775546 mixing bowl 661 | n03776460 mobile home, manufactured home 662 | n03777568 Model T 663 | n03777754 modem 664 | n03781244 monastery 665 | n03782006 monitor 666 | n03785016 moped 667 | n03786901 mortar 668 | n03787032 mortarboard 669 | n03788195 mosque 670 | n03788365 mosquito net 671 | n03791053 motor scooter, scooter 672 | n03792782 mountain bike, all-terrain bike, off-roader 673 | n03792972 mountain tent 674 | n03793489 mouse, computer mouse 675 | n03794056 mousetrap 676 | n03796401 moving van 677 | n03803284 muzzle 678 | n03804744 nail 679 | n03814639 neck brace 680 | n03814906 necklace 681 | n03825788 nipple 682 | n03832673 notebook, notebook computer 683 | n03837869 obelisk 684 | n03838899 oboe, hautboy, hautbois 685 | n03840681 ocarina, sweet potato 686 | n03841143 odometer, hodometer, mileometer, milometer 687 | n03843555 oil filter 688 | n03854065 organ, pipe organ 689 | n03857828 oscilloscope, scope, cathode-ray oscilloscope, CRO 690 | n03866082 overskirt 691 | n03868242 oxcart 692 | n03868863 oxygen mask 693 | n03871628 packet 694 | n03873416 paddle, boat paddle 695 | n03874293 paddlewheel, paddle wheel 696 | n03874599 padlock 697 | n03876231 paintbrush 698 | n03877472 pajama, pyjama, pj's, jammies 699 | n03877845 palace 700 | n03884397 panpipe, pandean pipe, syrinx 701 | n03887697 paper towel 702 | n03888257 parachute, chute 703 | n03888605 parallel bars, bars 704 | n03891251 park bench 705 | n03891332 parking meter 706 | n03895866 passenger car, coach, carriage 707 | n03899768 patio, terrace 708 | n03902125 pay-phone, pay-station 709 | n03903868 pedestal, plinth, footstall 710 | n03908618 pencil box, pencil case 711 | n03908714 pencil sharpener 712 | n03916031 perfume, essence 713 | n03920288 Petri dish 714 | n03924679 photocopier 715 | n03929660 pick, plectrum, plectron 716 | n03929855 pickelhaube 717 | n03930313 picket fence, paling 718 | n03930630 pickup, pickup truck 719 | n03933933 pier 720 | n03935335 piggy bank, penny bank 721 | n03937543 pill bottle 722 | n03938244 pillow 723 | n03942813 ping-pong ball 724 | n03944341 pinwheel 725 | n03947888 pirate, pirate ship 726 | n03950228 pitcher, ewer 727 | n03954731 plane, carpenter's plane, woodworking plane 728 | n03956157 planetarium 729 | n03958227 plastic bag 730 | n03961711 plate rack 731 | n03967562 plow, plough 732 | n03970156 plunger, plumber's helper 733 | n03976467 Polaroid camera, Polaroid Land camera 734 | n03976657 pole 735 | n03977966 police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria 736 | n03980874 poncho 737 | n03982430 pool table, billiard table, snooker table 738 | n03983396 pop bottle, soda bottle 739 | n03991062 pot, flowerpot 740 | n03992509 potter's wheel 741 | n03995372 power drill 742 | n03998194 prayer rug, prayer mat 743 | n04004767 printer 744 | n04005630 prison, prison house 745 | n04008634 projectile, missile 746 | n04009552 projector 747 | n04019541 puck, hockey puck 748 | n04023962 punching bag, punch bag, punching ball, punchball 749 | n04026417 purse 750 | n04033901 quill, quill pen 751 | n04033995 quilt, comforter, comfort, puff 752 | n04037443 racer, race car, racing car 753 | n04039381 racket, racquet 754 | n04040759 radiator 755 | n04041544 radio, wireless 756 | n04044716 radio telescope, radio reflector 757 | n04049303 rain barrel 758 | n04065272 recreational vehicle, RV, R.V. 759 | n04067472 reel 760 | n04069434 reflex camera 761 | n04070727 refrigerator, icebox 762 | n04074963 remote control, remote 763 | n04081281 restaurant, eating house, eating place, eatery 764 | n04086273 revolver, six-gun, six-shooter 765 | n04090263 rifle 766 | n04099969 rocking chair, rocker 767 | n04111531 rotisserie 768 | n04116512 rubber eraser, rubber, pencil eraser 769 | n04118538 rugby ball 770 | n04118776 rule, ruler 771 | n04120489 running shoe 772 | n04125021 safe 773 | n04127249 safety pin 774 | n04131690 saltshaker, salt shaker 775 | n04133789 sandal 776 | n04136333 sarong 777 | n04141076 sax, saxophone 778 | n04141327 scabbard 779 | n04141975 scale, weighing machine 780 | n04146614 school bus 781 | n04147183 schooner 782 | n04149813 scoreboard 783 | n04152593 screen, CRT screen 784 | n04153751 screw 785 | n04154565 screwdriver 786 | n04162706 seat belt, seatbelt 787 | n04179913 sewing machine 788 | n04192698 shield, buckler 789 | n04200800 shoe shop, shoe-shop, shoe store 790 | n04201297 shoji 791 | n04204238 shopping basket 792 | n04204347 shopping cart 793 | n04208210 shovel 794 | n04209133 shower cap 795 | n04209239 shower curtain 796 | n04228054 ski 797 | n04229816 ski mask 798 | n04235860 sleeping bag 799 | n04238763 slide rule, slipstick 800 | n04239074 sliding door 801 | n04243546 slot, one-armed bandit 802 | n04251144 snorkel 803 | n04252077 snowmobile 804 | n04252225 snowplow, snowplough 805 | n04254120 soap dispenser 806 | n04254680 soccer ball 807 | n04254777 sock 808 | n04258138 solar dish, solar collector, solar furnace 809 | n04259630 sombrero 810 | n04263257 soup bowl 811 | n04264628 space bar 812 | n04265275 space heater 813 | n04266014 space shuttle 814 | n04270147 spatula 815 | n04273569 speedboat 816 | n04275548 spider web, spider's web 817 | n04277352 spindle 818 | n04285008 sports car, sport car 819 | n04286575 spotlight, spot 820 | n04296562 stage 821 | n04310018 steam locomotive 822 | n04311004 steel arch bridge 823 | n04311174 steel drum 824 | n04317175 stethoscope 825 | n04325704 stole 826 | n04326547 stone wall 827 | n04328186 stopwatch, stop watch 828 | n04330267 stove 829 | n04332243 strainer 830 | n04335435 streetcar, tram, tramcar, trolley, trolley car 831 | n04336792 stretcher 832 | n04344873 studio couch, day bed 833 | n04346328 stupa, tope 834 | n04347754 submarine, pigboat, sub, U-boat 835 | n04350905 suit, suit of clothes 836 | n04355338 sundial 837 | n04355933 sunglass 838 | n04356056 sunglasses, dark glasses, shades 839 | n04357314 sunscreen, sunblock, sun blocker 840 | n04366367 suspension bridge 841 | n04367480 swab, swob, mop 842 | n04370456 sweatshirt 843 | n04371430 swimming trunks, bathing trunks 844 | n04371774 swing 845 | n04372370 switch, electric switch, electrical switch 846 | n04376876 syringe 847 | n04380533 table lamp 848 | n04389033 tank, army tank, armored combat vehicle, armoured combat vehicle 849 | n04392985 tape player 850 | n04398044 teapot 851 | n04399382 teddy, teddy bear 852 | n04404412 television, television system 853 | n04409515 tennis ball 854 | n04417672 thatch, thatched roof 855 | n04418357 theater curtain, theatre curtain 856 | n04423845 thimble 857 | n04428191 thresher, thrasher, threshing machine 858 | n04429376 throne 859 | n04435653 tile roof 860 | n04442312 toaster 861 | n04443257 tobacco shop, tobacconist shop, tobacconist 862 | n04447861 toilet seat 863 | n04456115 torch 864 | n04458633 totem pole 865 | n04461696 tow truck, tow car, wrecker 866 | n04462240 toyshop 867 | n04465501 tractor 868 | n04467665 trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi 869 | n04476259 tray 870 | n04479046 trench coat 871 | n04482393 tricycle, trike, velocipede 872 | n04483307 trimaran 873 | n04485082 tripod 874 | n04486054 triumphal arch 875 | n04487081 trolleybus, trolley coach, trackless trolley 876 | n04487394 trombone 877 | n04493381 tub, vat 878 | n04501370 turnstile 879 | n04505470 typewriter keyboard 880 | n04507155 umbrella 881 | n04509417 unicycle, monocycle 882 | n04515003 upright, upright piano 883 | n04517823 vacuum, vacuum cleaner 884 | n04522168 vase 885 | n04523525 vault 886 | n04525038 velvet 887 | n04525305 vending machine 888 | n04532106 vestment 889 | n04532670 viaduct 890 | n04536866 violin, fiddle 891 | n04540053 volleyball 892 | n04542943 waffle iron 893 | n04548280 wall clock 894 | n04548362 wallet, billfold, notecase, pocketbook 895 | n04550184 wardrobe, closet, press 896 | n04552348 warplane, military plane 897 | n04553703 washbasin, handbasin, washbowl, lavabo, wash-hand basin 898 | n04554684 washer, automatic washer, washing machine 899 | n04557648 water bottle 900 | n04560804 water jug 901 | n04562935 water tower 902 | n04579145 whiskey jug 903 | n04579432 whistle 904 | n04584207 wig 905 | n04589890 window screen 906 | n04590129 window shade 907 | n04591157 Windsor tie 908 | n04591713 wine bottle 909 | n04592741 wing 910 | n04596742 wok 911 | n04597913 wooden spoon 912 | n04599235 wool, woolen, woollen 913 | n04604644 worm fence, snake fence, snake-rail fence, Virginia fence 914 | n04606251 wreck 915 | n04612504 yawl 916 | n04613696 yurt 917 | n06359193 web site, website, internet site, site 918 | n06596364 comic book 919 | n06785654 crossword puzzle, crossword 920 | n06794110 street sign 921 | n06874185 traffic light, traffic signal, stoplight 922 | n07248320 book jacket, dust cover, dust jacket, dust wrapper 923 | n07565083 menu 924 | n07579787 plate 925 | n07583066 guacamole 926 | n07584110 consomme 927 | n07590611 hot pot, hotpot 928 | n07613480 trifle 929 | n07614500 ice cream, icecream 930 | n07615774 ice lolly, lolly, lollipop, popsicle 931 | n07684084 French loaf 932 | n07693725 bagel, beigel 933 | n07695742 pretzel 934 | n07697313 cheeseburger 935 | n07697537 hotdog, hot dog, red hot 936 | n07711569 mashed potato 937 | n07714571 head cabbage 938 | n07714990 broccoli 939 | n07715103 cauliflower 940 | n07716358 zucchini, courgette 941 | n07716906 spaghetti squash 942 | n07717410 acorn squash 943 | n07717556 butternut squash 944 | n07718472 cucumber, cuke 945 | n07718747 artichoke, globe artichoke 946 | n07720875 bell pepper 947 | n07730033 cardoon 948 | n07734744 mushroom 949 | n07742313 Granny Smith 950 | n07745940 strawberry 951 | n07747607 orange 952 | n07749582 lemon 953 | n07753113 fig 954 | n07753275 pineapple, ananas 955 | n07753592 banana 956 | n07754684 jackfruit, jak, jack 957 | n07760859 custard apple 958 | n07768694 pomegranate 959 | n07802026 hay 960 | n07831146 carbonara 961 | n07836838 chocolate sauce, chocolate syrup 962 | n07860988 dough 963 | n07871810 meat loaf, meatloaf 964 | n07873807 pizza, pizza pie 965 | n07875152 potpie 966 | n07880968 burrito 967 | n07892512 red wine 968 | n07920052 espresso 969 | n07930864 cup 970 | n07932039 eggnog 971 | n09193705 alp 972 | n09229709 bubble 973 | n09246464 cliff, drop, drop-off 974 | n09256479 coral reef 975 | n09288635 geyser 976 | n09332890 lakeside, lakeshore 977 | n09399592 promontory, headland, head, foreland 978 | n09421951 sandbar, sand bar 979 | n09428293 seashore, coast, seacoast, sea-coast 980 | n09468604 valley, vale 981 | n09472597 volcano 982 | n09835506 ballplayer, baseball player 983 | n10148035 groom, bridegroom 984 | n10565667 scuba diver 985 | n11879895 rapeseed 986 | n11939491 daisy 987 | n12057211 yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum 988 | n12144580 corn 989 | n12267677 acorn 990 | n12620546 hip, rose hip, rosehip 991 | n12768682 buckeye, horse chestnut, conker 992 | n12985857 coral fungus 993 | n12998815 agaric 994 | n13037406 gyromitra 995 | n13040303 stinkhorn, carrion fungus 996 | n13044778 earthstar 997 | n13052670 hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa 998 | n13054560 bolete 999 | n13133613 ear, spike, capitulum 1000 | n15075141 toilet tissue, toilet paper, bathroom tissue 1001 | -------------------------------------------------------------------------------- /synsets.txt: -------------------------------------------------------------------------------- 1 | n01440764 2 | n01443537 3 | n01484850 4 | n01491361 5 | n01494475 6 | n01496331 7 | n01498041 8 | n01514668 9 | n01514859 10 | n01518878 11 | n01530575 12 | n01531178 13 | n01532829 14 | n01534433 15 | n01537544 16 | n01558993 17 | n01560419 18 | n01580077 19 | n01582220 20 | n01592084 21 | n01601694 22 | n01608432 23 | n01614925 24 | n01616318 25 | n01622779 26 | n01629819 27 | n01630670 28 | n01631663 29 | n01632458 30 | n01632777 31 | n01641577 32 | n01644373 33 | n01644900 34 | n01664065 35 | n01665541 36 | n01667114 37 | n01667778 38 | n01669191 39 | n01675722 40 | n01677366 41 | n01682714 42 | n01685808 43 | n01687978 44 | n01688243 45 | n01689811 46 | n01692333 47 | n01693334 48 | n01694178 49 | n01695060 50 | n01697457 51 | n01698640 52 | n01704323 53 | n01728572 54 | n01728920 55 | n01729322 56 | n01729977 57 | n01734418 58 | n01735189 59 | n01737021 60 | n01739381 61 | n01740131 62 | n01742172 63 | n01744401 64 | n01748264 65 | n01749939 66 | n01751748 67 | n01753488 68 | n01755581 69 | n01756291 70 | n01768244 71 | n01770081 72 | n01770393 73 | n01773157 74 | n01773549 75 | n01773797 76 | n01774384 77 | n01774750 78 | n01775062 79 | n01776313 80 | n01784675 81 | n01795545 82 | n01796340 83 | n01797886 84 | n01798484 85 | n01806143 86 | n01806567 87 | n01807496 88 | n01817953 89 | n01818515 90 | n01819313 91 | n01820546 92 | n01824575 93 | n01828970 94 | n01829413 95 | n01833805 96 | n01843065 97 | n01843383 98 | n01847000 99 | n01855032 100 | n01855672 101 | n01860187 102 | n01871265 103 | n01872401 104 | n01873310 105 | n01877812 106 | n01882714 107 | n01883070 108 | n01910747 109 | n01914609 110 | n01917289 111 | n01924916 112 | n01930112 113 | n01943899 114 | n01944390 115 | n01945685 116 | n01950731 117 | n01955084 118 | n01968897 119 | n01978287 120 | n01978455 121 | n01980166 122 | n01981276 123 | n01983481 124 | n01984695 125 | n01985128 126 | n01986214 127 | n01990800 128 | n02002556 129 | n02002724 130 | n02006656 131 | n02007558 132 | n02009229 133 | n02009912 134 | n02011460 135 | n02012849 136 | n02013706 137 | n02017213 138 | n02018207 139 | n02018795 140 | n02025239 141 | n02027492 142 | n02028035 143 | n02033041 144 | n02037110 145 | n02051845 146 | n02056570 147 | n02058221 148 | n02066245 149 | n02071294 150 | n02074367 151 | n02077923 152 | n02085620 153 | n02085782 154 | n02085936 155 | n02086079 156 | n02086240 157 | n02086646 158 | n02086910 159 | n02087046 160 | n02087394 161 | n02088094 162 | n02088238 163 | n02088364 164 | n02088466 165 | n02088632 166 | n02089078 167 | n02089867 168 | n02089973 169 | n02090379 170 | n02090622 171 | n02090721 172 | n02091032 173 | n02091134 174 | n02091244 175 | n02091467 176 | n02091635 177 | n02091831 178 | n02092002 179 | n02092339 180 | n02093256 181 | n02093428 182 | n02093647 183 | n02093754 184 | n02093859 185 | n02093991 186 | n02094114 187 | n02094258 188 | n02094433 189 | n02095314 190 | n02095570 191 | n02095889 192 | n02096051 193 | n02096177 194 | n02096294 195 | n02096437 196 | n02096585 197 | n02097047 198 | n02097130 199 | n02097209 200 | n02097298 201 | n02097474 202 | n02097658 203 | n02098105 204 | n02098286 205 | n02098413 206 | n02099267 207 | n02099429 208 | n02099601 209 | n02099712 210 | n02099849 211 | n02100236 212 | n02100583 213 | n02100735 214 | n02100877 215 | n02101006 216 | n02101388 217 | n02101556 218 | n02102040 219 | n02102177 220 | n02102318 221 | n02102480 222 | n02102973 223 | n02104029 224 | n02104365 225 | n02105056 226 | n02105162 227 | n02105251 228 | n02105412 229 | n02105505 230 | n02105641 231 | n02105855 232 | n02106030 233 | n02106166 234 | n02106382 235 | n02106550 236 | n02106662 237 | n02107142 238 | n02107312 239 | n02107574 240 | n02107683 241 | n02107908 242 | n02108000 243 | n02108089 244 | n02108422 245 | n02108551 246 | n02108915 247 | n02109047 248 | n02109525 249 | n02109961 250 | n02110063 251 | n02110185 252 | n02110341 253 | n02110627 254 | n02110806 255 | n02110958 256 | n02111129 257 | n02111277 258 | n02111500 259 | n02111889 260 | n02112018 261 | n02112137 262 | n02112350 263 | n02112706 264 | n02113023 265 | n02113186 266 | n02113624 267 | n02113712 268 | n02113799 269 | n02113978 270 | n02114367 271 | n02114548 272 | n02114712 273 | n02114855 274 | n02115641 275 | n02115913 276 | n02116738 277 | n02117135 278 | n02119022 279 | n02119789 280 | n02120079 281 | n02120505 282 | n02123045 283 | n02123159 284 | n02123394 285 | n02123597 286 | n02124075 287 | n02125311 288 | n02127052 289 | n02128385 290 | n02128757 291 | n02128925 292 | n02129165 293 | n02129604 294 | n02130308 295 | n02132136 296 | n02133161 297 | n02134084 298 | n02134418 299 | n02137549 300 | n02138441 301 | n02165105 302 | n02165456 303 | n02167151 304 | n02168699 305 | n02169497 306 | n02172182 307 | n02174001 308 | n02177972 309 | n02190166 310 | n02206856 311 | n02219486 312 | n02226429 313 | n02229544 314 | n02231487 315 | n02233338 316 | n02236044 317 | n02256656 318 | n02259212 319 | n02264363 320 | n02268443 321 | n02268853 322 | n02276258 323 | n02277742 324 | n02279972 325 | n02280649 326 | n02281406 327 | n02281787 328 | n02317335 329 | n02319095 330 | n02321529 331 | n02325366 332 | n02326432 333 | n02328150 334 | n02342885 335 | n02346627 336 | n02356798 337 | n02361337 338 | n02363005 339 | n02364673 340 | n02389026 341 | n02391049 342 | n02395406 343 | n02396427 344 | n02397096 345 | n02398521 346 | n02403003 347 | n02408429 348 | n02410509 349 | n02412080 350 | n02415577 351 | n02417914 352 | n02422106 353 | n02422699 354 | n02423022 355 | n02437312 356 | n02437616 357 | n02441942 358 | n02442845 359 | n02443114 360 | n02443484 361 | n02444819 362 | n02445715 363 | n02447366 364 | n02454379 365 | n02457408 366 | n02480495 367 | n02480855 368 | n02481823 369 | n02483362 370 | n02483708 371 | n02484975 372 | n02486261 373 | n02486410 374 | n02487347 375 | n02488291 376 | n02488702 377 | n02489166 378 | n02490219 379 | n02492035 380 | n02492660 381 | n02493509 382 | n02493793 383 | n02494079 384 | n02497673 385 | n02500267 386 | n02504013 387 | n02504458 388 | n02509815 389 | n02510455 390 | n02514041 391 | n02526121 392 | n02536864 393 | n02606052 394 | n02607072 395 | n02640242 396 | n02641379 397 | n02643566 398 | n02655020 399 | n02666196 400 | n02667093 401 | n02669723 402 | n02672831 403 | n02676566 404 | n02687172 405 | n02690373 406 | n02692877 407 | n02699494 408 | n02701002 409 | n02704792 410 | n02708093 411 | n02727426 412 | n02730930 413 | n02747177 414 | n02749479 415 | n02769748 416 | n02776631 417 | n02777292 418 | n02782093 419 | n02783161 420 | n02786058 421 | n02787622 422 | n02788148 423 | n02790996 424 | n02791124 425 | n02791270 426 | n02793495 427 | n02794156 428 | n02795169 429 | n02797295 430 | n02799071 431 | n02802426 432 | n02804414 433 | n02804610 434 | n02807133 435 | n02808304 436 | n02808440 437 | n02814533 438 | n02814860 439 | n02815834 440 | n02817516 441 | n02823428 442 | n02823750 443 | n02825657 444 | n02834397 445 | n02835271 446 | n02837789 447 | n02840245 448 | n02841315 449 | n02843684 450 | n02859443 451 | n02860847 452 | n02865351 453 | n02869837 454 | n02870880 455 | n02871525 456 | n02877765 457 | n02879718 458 | n02883205 459 | n02892201 460 | n02892767 461 | n02894605 462 | n02895154 463 | n02906734 464 | n02909870 465 | n02910353 466 | n02916936 467 | n02917067 468 | n02927161 469 | n02930766 470 | n02939185 471 | n02948072 472 | n02950826 473 | n02951358 474 | n02951585 475 | n02963159 476 | n02965783 477 | n02966193 478 | n02966687 479 | n02971356 480 | n02974003 481 | n02977058 482 | n02978881 483 | n02979186 484 | n02980441 485 | n02981792 486 | n02988304 487 | n02992211 488 | n02992529 489 | n02999410 490 | n03000134 491 | n03000247 492 | n03000684 493 | n03014705 494 | n03016953 495 | n03017168 496 | n03018349 497 | n03026506 498 | n03028079 499 | n03032252 500 | n03041632 501 | n03042490 502 | n03045698 503 | n03047690 504 | n03062245 505 | n03063599 506 | n03063689 507 | n03065424 508 | n03075370 509 | n03085013 510 | n03089624 511 | n03095699 512 | n03100240 513 | n03109150 514 | n03110669 515 | n03124043 516 | n03124170 517 | n03125729 518 | n03126707 519 | n03127747 520 | n03127925 521 | n03131574 522 | n03133878 523 | n03134739 524 | n03141823 525 | n03146219 526 | n03160309 527 | n03179701 528 | n03180011 529 | n03187595 530 | n03188531 531 | n03196217 532 | n03197337 533 | n03201208 534 | n03207743 535 | n03207941 536 | n03208938 537 | n03216828 538 | n03218198 539 | n03220513 540 | n03223299 541 | n03240683 542 | n03249569 543 | n03250847 544 | n03255030 545 | n03259280 546 | n03271574 547 | n03272010 548 | n03272562 549 | n03290653 550 | n03291819 551 | n03297495 552 | n03314780 553 | n03325584 554 | n03337140 555 | n03344393 556 | n03345487 557 | n03347037 558 | n03355925 559 | n03372029 560 | n03376595 561 | n03379051 562 | n03384352 563 | n03388043 564 | n03388183 565 | n03388549 566 | n03393912 567 | n03394916 568 | n03400231 569 | n03404251 570 | n03417042 571 | n03424325 572 | n03425413 573 | n03443371 574 | n03444034 575 | n03445777 576 | n03445924 577 | n03447447 578 | n03447721 579 | n03450230 580 | n03452741 581 | n03457902 582 | n03459775 583 | n03461385 584 | n03467068 585 | n03476684 586 | n03476991 587 | n03478589 588 | n03481172 589 | n03482405 590 | n03483316 591 | n03485407 592 | n03485794 593 | n03492542 594 | n03494278 595 | n03495258 596 | n03496892 597 | n03498962 598 | n03527444 599 | n03529860 600 | n03530642 601 | n03532672 602 | n03534580 603 | n03535780 604 | n03538406 605 | n03544143 606 | n03584254 607 | n03584829 608 | n03590841 609 | n03594734 610 | n03594945 611 | n03595614 612 | n03598930 613 | n03599486 614 | n03602883 615 | n03617480 616 | n03623198 617 | n03627232 618 | n03630383 619 | n03633091 620 | n03637318 621 | n03642806 622 | n03649909 623 | n03657121 624 | n03658185 625 | n03661043 626 | n03662601 627 | n03666591 628 | n03670208 629 | n03673027 630 | n03676483 631 | n03680355 632 | n03690938 633 | n03691459 634 | n03692522 635 | n03697007 636 | n03706229 637 | n03709823 638 | n03710193 639 | n03710637 640 | n03710721 641 | n03717622 642 | n03720891 643 | n03721384 644 | n03724870 645 | n03729826 646 | n03733131 647 | n03733281 648 | n03733805 649 | n03742115 650 | n03743016 651 | n03759954 652 | n03761084 653 | n03763968 654 | n03764736 655 | n03769881 656 | n03770439 657 | n03770679 658 | n03773504 659 | n03775071 660 | n03775546 661 | n03776460 662 | n03777568 663 | n03777754 664 | n03781244 665 | n03782006 666 | n03785016 667 | n03786901 668 | n03787032 669 | n03788195 670 | n03788365 671 | n03791053 672 | n03792782 673 | n03792972 674 | n03793489 675 | n03794056 676 | n03796401 677 | n03803284 678 | n03804744 679 | n03814639 680 | n03814906 681 | n03825788 682 | n03832673 683 | n03837869 684 | n03838899 685 | n03840681 686 | n03841143 687 | n03843555 688 | n03854065 689 | n03857828 690 | n03866082 691 | n03868242 692 | n03868863 693 | n03871628 694 | n03873416 695 | n03874293 696 | n03874599 697 | n03876231 698 | n03877472 699 | n03877845 700 | n03884397 701 | n03887697 702 | n03888257 703 | n03888605 704 | n03891251 705 | n03891332 706 | n03895866 707 | n03899768 708 | n03902125 709 | n03903868 710 | n03908618 711 | n03908714 712 | n03916031 713 | n03920288 714 | n03924679 715 | n03929660 716 | n03929855 717 | n03930313 718 | n03930630 719 | n03933933 720 | n03935335 721 | n03937543 722 | n03938244 723 | n03942813 724 | n03944341 725 | n03947888 726 | n03950228 727 | n03954731 728 | n03956157 729 | n03958227 730 | n03961711 731 | n03967562 732 | n03970156 733 | n03976467 734 | n03976657 735 | n03977966 736 | n03980874 737 | n03982430 738 | n03983396 739 | n03991062 740 | n03992509 741 | n03995372 742 | n03998194 743 | n04004767 744 | n04005630 745 | n04008634 746 | n04009552 747 | n04019541 748 | n04023962 749 | n04026417 750 | n04033901 751 | n04033995 752 | n04037443 753 | n04039381 754 | n04040759 755 | n04041544 756 | n04044716 757 | n04049303 758 | n04065272 759 | n04067472 760 | n04069434 761 | n04070727 762 | n04074963 763 | n04081281 764 | n04086273 765 | n04090263 766 | n04099969 767 | n04111531 768 | n04116512 769 | n04118538 770 | n04118776 771 | n04120489 772 | n04125021 773 | n04127249 774 | n04131690 775 | n04133789 776 | n04136333 777 | n04141076 778 | n04141327 779 | n04141975 780 | n04146614 781 | n04147183 782 | n04149813 783 | n04152593 784 | n04153751 785 | n04154565 786 | n04162706 787 | n04179913 788 | n04192698 789 | n04200800 790 | n04201297 791 | n04204238 792 | n04204347 793 | n04208210 794 | n04209133 795 | n04209239 796 | n04228054 797 | n04229816 798 | n04235860 799 | n04238763 800 | n04239074 801 | n04243546 802 | n04251144 803 | n04252077 804 | n04252225 805 | n04254120 806 | n04254680 807 | n04254777 808 | n04258138 809 | n04259630 810 | n04263257 811 | n04264628 812 | n04265275 813 | n04266014 814 | n04270147 815 | n04273569 816 | n04275548 817 | n04277352 818 | n04285008 819 | n04286575 820 | n04296562 821 | n04310018 822 | n04311004 823 | n04311174 824 | n04317175 825 | n04325704 826 | n04326547 827 | n04328186 828 | n04330267 829 | n04332243 830 | n04335435 831 | n04336792 832 | n04344873 833 | n04346328 834 | n04347754 835 | n04350905 836 | n04355338 837 | n04355933 838 | n04356056 839 | n04357314 840 | n04366367 841 | n04367480 842 | n04370456 843 | n04371430 844 | n04371774 845 | n04372370 846 | n04376876 847 | n04380533 848 | n04389033 849 | n04392985 850 | n04398044 851 | n04399382 852 | n04404412 853 | n04409515 854 | n04417672 855 | n04418357 856 | n04423845 857 | n04428191 858 | n04429376 859 | n04435653 860 | n04442312 861 | n04443257 862 | n04447861 863 | n04456115 864 | n04458633 865 | n04461696 866 | n04462240 867 | n04465501 868 | n04467665 869 | n04476259 870 | n04479046 871 | n04482393 872 | n04483307 873 | n04485082 874 | n04486054 875 | n04487081 876 | n04487394 877 | n04493381 878 | n04501370 879 | n04505470 880 | n04507155 881 | n04509417 882 | n04515003 883 | n04517823 884 | n04522168 885 | n04523525 886 | n04525038 887 | n04525305 888 | n04532106 889 | n04532670 890 | n04536866 891 | n04540053 892 | n04542943 893 | n04548280 894 | n04548362 895 | n04550184 896 | n04552348 897 | n04553703 898 | n04554684 899 | n04557648 900 | n04560804 901 | n04562935 902 | n04579145 903 | n04579432 904 | n04584207 905 | n04589890 906 | n04590129 907 | n04591157 908 | n04591713 909 | n04592741 910 | n04596742 911 | n04597913 912 | n04599235 913 | n04604644 914 | n04606251 915 | n04612504 916 | n04613696 917 | n06359193 918 | n06596364 919 | n06785654 920 | n06794110 921 | n06874185 922 | n07248320 923 | n07565083 924 | n07579787 925 | n07583066 926 | n07584110 927 | n07590611 928 | n07613480 929 | n07614500 930 | n07615774 931 | n07684084 932 | n07693725 933 | n07695742 934 | n07697313 935 | n07697537 936 | n07711569 937 | n07714571 938 | n07714990 939 | n07715103 940 | n07716358 941 | n07716906 942 | n07717410 943 | n07717556 944 | n07718472 945 | n07718747 946 | n07720875 947 | n07730033 948 | n07734744 949 | n07742313 950 | n07745940 951 | n07747607 952 | n07749582 953 | n07753113 954 | n07753275 955 | n07753592 956 | n07754684 957 | n07760859 958 | n07768694 959 | n07802026 960 | n07831146 961 | n07836838 962 | n07860988 963 | n07871810 964 | n07873807 965 | n07875152 966 | n07880968 967 | n07892512 968 | n07920052 969 | n07930864 970 | n07932039 971 | n09193705 972 | n09229709 973 | n09246464 974 | n09256479 975 | n09288635 976 | n09332890 977 | n09399592 978 | n09421951 979 | n09428293 980 | n09468604 981 | n09472597 982 | n09835506 983 | n10148035 984 | n10565667 985 | n11879895 986 | n11939491 987 | n12057211 988 | n12144580 989 | n12267677 990 | n12620546 991 | n12768682 992 | n12985857 993 | n12998815 994 | n13037406 995 | n13040303 996 | n13044778 997 | n13052670 998 | n13054560 999 | n13133613 1000 | n15075141 -------------------------------------------------------------------------------- /t-sne.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alsora/CNN_Visualization/782071bd912c42342dc28b0edc71a9431fa7c173/t-sne.gif -------------------------------------------------------------------------------- /tsne.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import sys 3 | 4 | from matplotlib import pyplot as plt 5 | from matplotlib.animation import FuncAnimation 6 | from sklearn.decomposition import PCA 7 | from sklearn.metrics.pairwise import pairwise_distances 8 | from skdata.mnist.views import OfficialImageClassification 9 | from skimage.transform import resize 10 | from matplotlib.offsetbox import OffsetImage, AnnotationBbox 11 | 12 | def backspace(n): 13 | sys.stdout.write('\r'+n) 14 | sys.stdout.flush() 15 | 16 | 17 | def binarySearch(comparisonMethod, terms, tolerance=1e-5, objective=30.0): 18 | valuemin = 0.1 19 | 20 | valuemax = 10000 21 | 22 | value = (valuemin + valuemax) / 2.0 23 | 24 | PP = comparisonMethod(terms, value) 25 | 26 | Pdiff = PP - objective 27 | binaryTries = 0 28 | 29 | while binaryTries < 100 and np.abs(Pdiff) > tolerance: 30 | #Try new values until the perplexity difference is under the threshold 31 | if Pdiff < 0: 32 | valuemin = value 33 | else: 34 | valuemax = value 35 | 36 | value = (valuemin + valuemax) / 2.0 37 | 38 | #Recompute the perplexity 39 | PP = comparisonMethod(terms, value) 40 | 41 | Pdiff = PP - objective 42 | binaryTries += 1 43 | 44 | return value 45 | 46 | 47 | def computePerplexity(D=np.array([]), sigma=1.0): 48 | #Compute perplexity as function of the gaussian distribution variance sigma 49 | 50 | precision = 1.0 / sigma 51 | 52 | P = np.exp(-D * precision) 53 | P = P / sum(P) 54 | 55 | log2P = np.log2(P) 56 | Plog2P = P * log2P 57 | H = - sum(Plog2P) 58 | 59 | PP = 2 ** H 60 | 61 | return PP 62 | 63 | 64 | def computeMapPoints(P, numIter=200, numOutputDimensions=2): 65 | numPoints = P.shape[0] 66 | initial_momentum = 0.5 67 | final_momentum = 0.8 68 | eta = 100 69 | min_gain = 0.01 70 | #initialize containers with the correct dimension 71 | gradients = np.zeros((numPoints, numOutputDimensions)) 72 | increment = np.zeros((numPoints, numOutputDimensions)) 73 | gains = np.ones((numPoints, numOutputDimensions)) 74 | 75 | mapPointsStorage = [] 76 | 77 | #Compute symmetric conditional probabilities (high dimensional space) 78 | numeratorPsymmetric = P + np.transpose(P) 79 | denominatorPsymmetric = np.sum(numeratorPsymmetric) 80 | Psymmetric = numeratorPsymmetric / denominatorPsymmetric 81 | 82 | Psymmetric *= 4 #Early exaggeration 83 | #Lower bound on minimum value of high dimensional probabilities 84 | Psymmetric = np.maximum(Psymmetric, 1e-12) 85 | 86 | #Initial random low dimensional embedding 87 | lowDimPoints = np.random.normal(0, 1e-4, (numPoints, numOutputDimensions)) 88 | 89 | momentum = initial_momentum 90 | #T-sne iterations 91 | for iter in range(numIter): 92 | 93 | mapPointsStorage.append(lowDimPoints) 94 | 95 | #Stop early exaggeration 96 | if iter == 100: 97 | Psymmetric /= 4 98 | 99 | #Switch to the higher momentum after some iterations 100 | if iter == 250: 101 | momentum = final_momentum 102 | 103 | #Compute low dimensional pairwise distances 104 | D = pairwise_distances(lowDimPoints, squared=True) 105 | 106 | #Compute joint probabilities (low dimensional space) Q 107 | numeratorQ = 1 / (1 + D) 108 | #Set values on the diagonal to 0 109 | numeratorQ[range(numPoints), range(numPoints)] = 0 110 | denominatorQ = np.sum(numeratorQ) 111 | Q = numeratorQ/ denominatorQ 112 | #Lower bound on the minimum value for low dim probabilities 113 | Q = np.maximum(Q, 1e-12) 114 | 115 | #Differences between high and low dimensional probabilities 116 | P_Q = Psymmetric - Q 117 | 118 | #Compute Kullback-Leibler divergence gradient 119 | for i in range(numPoints): 120 | gradients[i, :] = np.sum(np.tile(P_Q[:, i] * numeratorQ[:, i], (numOutputDimensions, 1)).T * (lowDimPoints[i, :] - lowDimPoints), 0) 121 | 122 | #Select which values have to be increased or decreased 123 | toBeIncreased = increment * gradients < 0 124 | toBeDecreased = increment * gradients >= 0 125 | #Set the corresponding gains 126 | gains[toBeIncreased] += 0.2 127 | gains[toBeDecreased] *= 0.8 128 | 129 | gains[gains < min_gain] = min_gain 130 | 131 | learningRate = eta * gains 132 | 133 | increment = momentum * increment - learningRate * gradients 134 | 135 | lowDimPoints = lowDimPoints + increment 136 | lowDimPoints = lowDimPoints - np.tile(np.mean(lowDimPoints, 0), (numPoints, 1)) 137 | 138 | #Compute and display the current cost every 10 iterations 139 | if (iter + 1) % 10 == 0: 140 | cost = np.sum(Psymmetric * np.log(Psymmetric / Q)) 141 | to_print = "Iteration {} : error {}".format(iter + 1, cost) 142 | backspace(to_print) 143 | 144 | 145 | #Insert last iteration points 146 | mapPointsStorage.append(lowDimPoints) 147 | return mapPointsStorage 148 | 149 | 150 | def computeProbabilities(X, perplexity=30.0, tolerance=1e-5): 151 | #Perform an initial dimensionality reduction 152 | pca = PCA(n_components=50) 153 | 154 | X = pca.fit_transform(X) 155 | 156 | numSamples = X.shape[0] 157 | 158 | P = np.zeros((numSamples, numSamples)) 159 | 160 | D = pairwise_distances(X, squared=True) 161 | 162 | for i in range(numSamples): 163 | indices = np.concatenate((np.arange(i), np.arange(i + 1, numSamples))) 164 | 165 | distancesFromI = D[i, indices] 166 | 167 | sigma = binarySearch(computePerplexity, distancesFromI, tolerance, perplexity) 168 | 169 | precision = 1.0 / sigma 170 | #Compute a "row" of matrix P: the probabilities wrt point I 171 | PwrtI = np.exp(- distancesFromI * precision) 172 | PwrtI /= sum(PwrtI) 173 | #Insert an element corresponding to I wrt I 174 | PwrtI = np.concatenate((PwrtI[0:i], [0.0], PwrtI[i:numSamples])) 175 | #Insert the row 176 | P[i, :] = PwrtI 177 | 178 | return P 179 | 180 | 181 | def showPoints(position, labels): 182 | classes = list(set(labels)) 183 | 184 | numClasses = len(classes) 185 | 186 | perClassPositions_t = [[] for x in range(numClasses)] 187 | 188 | for ind, point in enumerate(position): 189 | subListID = classes.index(labels[ind]) 190 | 191 | perClassPositions_t[subListID].append(point) 192 | 193 | finalData = perClassPositions_t 194 | plotStorage = [] 195 | colors = [] 196 | 197 | cmap = plt.cm.get_cmap('hsv', numClasses + 1) 198 | 199 | for index, lab in enumerate(finalData): 200 | lab = np.asarray(lab) 201 | 202 | x = plt.scatter(lab[:, 0], lab[:, 1], 20, c=cmap(index)) 203 | 204 | plotStorage.append(x) 205 | colors.append(cmap(index)) 206 | 207 | plt.legend(plotStorage, 208 | classes, 209 | scatterpoints=1, 210 | loc='lower left', 211 | ncol=3, 212 | fontsize=8) 213 | 214 | plt.show() 215 | 216 | 217 | def showMovie(positions, labels): 218 | positions = np.asarray(positions) 219 | 220 | classes = list(set(labels)) 221 | 222 | numClasses = len(classes) 223 | 224 | fig = plt.figure(figsize=(10, 10)) 225 | ax = fig.add_axes([0, 0, 1, 1], frameon=False) 226 | 227 | maxX = np.amax(positions[:, :, 0]) 228 | minX = np.amin(positions[:, :, 0]) 229 | maxY = np.amax(positions[:, :, 1]) 230 | minY = np.amin(positions[:, :, 1]) 231 | 232 | limit = max(maxX, maxY, minX, minY, key=abs) * 1.2 233 | 234 | ax.set_xlim(-limit, limit), ax.set_xticks([]) 235 | ax.set_ylim(-limit, limit), ax.set_yticks([]) 236 | rect = fig.patch 237 | rect.set_facecolor('white') 238 | 239 | currentPositions = positions[0] 240 | 241 | colors = [] 242 | cmap = plt.cm.get_cmap('hsv', numClasses + 1) 243 | 244 | for ind in range(numClasses): 245 | colors.append(cmap(ind)) 246 | 247 | coloredLabels = [colors[classes.index(label)] for label in labels] 248 | 249 | scat = ax.scatter(currentPositions[:, 0], currentPositions[:, 1], 20, coloredLabels) 250 | 251 | def increment(frame_number): 252 | num = frame_number * 5 253 | 254 | currentPositions = positions[num] 255 | 256 | scat.set_offsets(currentPositions) 257 | 258 | numFrames = len(positions) / 5 259 | 260 | animation = FuncAnimation(fig, increment, interval=100, frames=numFrames, repeat=False) 261 | plt.show() 262 | animation.save('movie.mp4') 263 | 264 | 265 | 266 | def imagesPlot(images, positions, zoom=0.25): 267 | fig, ax = plt.subplots() 268 | 269 | for num in range(len(images)): 270 | 271 | x = positions[num, 0] 272 | y = positions[num, 1] 273 | image = images[num] 274 | 275 | im = OffsetImage(image, zoom=zoom) 276 | x, y = np.atleast_1d(x, y) 277 | 278 | for x0, y0 in zip(x, y): 279 | ab = AnnotationBbox(im, (x0, y0), xycoords='data', frameon=False) 280 | ax.add_artist(ab) 281 | 282 | ax.update_datalim(np.column_stack([x, y])) 283 | ax.autoscale() 284 | 285 | plt.show() 286 | 287 | def test(): 288 | X = np.loadtxt("mnist2500_X.txt") 289 | labels = np.loadtxt("mnist2500_labels.txt") 290 | 291 | perplexity = 20.0 292 | tolerance = 1e-5 293 | 294 | iterations = 800 295 | 296 | P = computeProbabilities(X, perplexity, tolerance) 297 | 298 | positions = computeMapPoints(P, iterations) 299 | 300 | showPoints(positions[-1], labels) 301 | 302 | showMovie(positions, labels) 303 | 304 | 305 | if __name__ == "__main__": 306 | print 'Running t-sne test example' 307 | test() 308 | -------------------------------------------------------------------------------- /tsneCNN.py: -------------------------------------------------------------------------------- 1 | import os 2 | os.environ['GLOG_minloglevel'] = '3' 3 | 4 | import warnings 5 | warnings.filterwarnings("ignore") 6 | 7 | 8 | import caffe 9 | import sys 10 | import argparse 11 | import numpy as np 12 | 13 | import tsne 14 | #import caffeCNN_utils as CNN 15 | import caffe_utils 16 | 17 | from dataset_utils import loadImageNetFiles 18 | 19 | netLayers = { 20 | 'caffenet': 'fc7', 21 | 22 | #'googlenet': 'inception_5b/output' 23 | 'googlenet': 'loss3/classifier', 24 | 'resnet': 'fc1000' 25 | } 26 | 27 | 28 | def main(argv): 29 | 30 | pycaffe_path = os.path.dirname(caffe.__file__) 31 | caffe_path = os.path.normpath(os.path.join(pycaffe_path, '../../')) 32 | mean_path = os.path.join(pycaffe_path, 'imagenet/ilsvrc_2012_mean.npy') 33 | synsets_num_path = os.path.join(caffe_path, 'data/ilsvrc12/synsets.txt') 34 | synsets_to_class_path = os.path.join(os.getcwd(), 'synset_words.txt') 35 | 36 | model_filename = os.path.join(caffe_path, 'models/bvlc_reference_caffenet/deploy.prototxt') 37 | weight_filename = os.path.join(caffe_path, 'models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel') 38 | cnn_type = 'caffenet' 39 | images_dir = 'Images' 40 | caffe.set_mode_cpu() 41 | 42 | parser = argparse.ArgumentParser() 43 | parser.add_argument("-w", "--weights", help="caffemodel file") 44 | parser.add_argument("-p", "--prototxt", help="prototxt file") 45 | parser.add_argument("-i", "--input_im", help="input images dir") 46 | parser.add_argument("-n", "--net_type", help="cnn type (resnet/googlenet/vggnet") 47 | parser.add_argument("-g", "--gpu", help="enable gpu mode", action='store_true') 48 | args = parser.parse_args() 49 | 50 | if args.prototxt: 51 | model_filename = args.prototxt 52 | if args.weights: 53 | weight_filename = args.weights 54 | if args.input_im: 55 | images_dir = args.input_im 56 | if args.net_type: 57 | cnn_type = args.net_type 58 | if args.gpu: 59 | caffe.set_mode_gpu() 60 | caffe.set_device(0) 61 | 62 | if os.path.isfile(model_filename): 63 | print 'Caffe model found.' 64 | else: 65 | print 'Caffe model NOT found...' 66 | sys.exit(2) 67 | 68 | extractionLayerName = netLayers[cnn_type] 69 | 70 | # building dictionaries and inverse ones 71 | idx_to_synset = {} 72 | synset_to_idx = {} 73 | 74 | with open(synsets_num_path, 'r') as fp: 75 | for idx, synset in enumerate(fp): 76 | synset = synset.strip() 77 | idx_to_synset[idx] = synset 78 | synset_to_idx[synset] = idx 79 | 80 | synset_to_class = {} 81 | class_to_synset = {} 82 | 83 | with open(synsets_to_class_path, 'r') as fp: 84 | for line in fp: 85 | [synset, class_] = line.strip().split(' ', 1) 86 | synset_to_class[synset] = class_ 87 | class_to_synset[class_] = synset 88 | 89 | #Loading net and utilities 90 | net = caffe_utils.CaffeNet(model_filename, weight_filename, mean_path) 91 | 92 | #Create Images and labels 93 | numberImagesPerClass = 150 94 | 95 | #classes = ['n02119789', 'n03773504', 'n04254680', 'n04429376', 'n04507155'] 96 | #If you don't provide synsets, it will take images from all the sub-folders in images_dir 97 | print "Loading images from provided directories... " 98 | names, images, labels = loadImageNetFiles(images_dir, numberImagesPerClass)#, classes) 99 | 100 | #Preprocess images 101 | preprocessedImages = net.preprocess_images(images) 102 | 103 | #Forward pass to extract features and predict labels 104 | print "Performing a forward pass on all the images.... " 105 | probs, features = net.get_probs_and_features(preprocessedImages, extractionLayerName) 106 | 107 | predictedProbsTop5 =[np.argsort(x)[-1:-5:-1] for x in probs] 108 | 109 | predictedLabelsTop5 = [] 110 | for predictions in predictedProbsTop5: 111 | predictions = [idx_to_synset[x] for x in predictions] 112 | predictedLabelsTop5.append(predictions) 113 | 114 | 115 | predictedLabelsTop1 = [predictions[0] for predictions in predictedLabelsTop5] 116 | 117 | net.get_precision(labels, predictedLabelsTop5) 118 | 119 | #Apply tsne 120 | 121 | perplexity = 10.0 122 | tolerance = 1e-5 123 | iterations = 800 124 | 125 | features = np.array(features) 126 | 127 | P = tsne.computeProbabilities(features, perplexity, tolerance) 128 | positions = tsne.computeMapPoints(P, iterations) 129 | 130 | mappedLabels = net.synsets_to_words([synset_to_class[x] for x in labels]) 131 | mappedPredictedLabels = net.synsets_to_words([synset_to_class[x] for x in predictedLabelsTop1]) 132 | 133 | tsne.showPoints(positions[-1], mappedLabels) 134 | tsne.showPoints(positions[-1], mappedPredictedLabels) 135 | 136 | tsne.showMovie(positions, mappedLabels) 137 | 138 | tsne.imagesPlot(images, positions[-1]) 139 | 140 | 141 | if __name__=='__main__': 142 | main(sys.argv[1:]) 143 | --------------------------------------------------------------------------------