├── .gitignore ├── LICENSE.md ├── README.md ├── dataset └── README.md ├── dissection └── README.md ├── plot ├── ResNet-152-model_imagenet.csv ├── ResNet-152-model_places365.csv ├── caffe_reference_imagenet.csv ├── caffe_reference_places205.csv ├── caffe_reference_places365.csv ├── extract_csv.m ├── getPrintName.m ├── network2layer.m ├── plot_histogram.m ├── plot_semantics.m ├── printLabels.m ├── result.txt ├── return_annotation.m ├── semantics_allnetwork.pdf ├── semantics_cvpr_release.mat └── semantics_samples.mat ├── script ├── dlbroden.sh ├── dlbroden_227.sh ├── dlzoo.sh ├── dlzoo_example.sh ├── makebroden.sh ├── makelinks.sh ├── rundissect.sh ├── rundissect_pytorch.sh └── rundissect_pytorch_external.sh ├── src ├── ade20k.py ├── adeseg.py ├── bargraph.py ├── brian.ipynb ├── colorname.py ├── computealpha.py ├── consolidatelineardiscprobe.py ├── consolidatelinearprobe.py ├── customoptimizer_pytorch.py ├── distance_correlation.py ├── dtdseg.py ├── expdir.py ├── extract_concept_data.py ├── fieldmap.py ├── figures.py ├── figures_playground.py ├── follow_up.ipynb ├── graphprobe.py ├── indexdata.py ├── indexdataset.py ├── intersect.py ├── invert.py ├── joinseg.py ├── labelprobe.py ├── labelprobe_pytorch.py ├── linearmaxprobe.py ├── linearprobe_disc_pytorch.py ├── linearprobe_pytorch.py ├── loadseg.py ├── makeresult.py ├── maxprobe.py ├── multi_filter_multi_concept.py ├── netprobe.py ├── netprobe_pytorch.py ├── netprobe_pytorch_imagenet.py ├── neural_debugging.ipynb ├── osseg.py ├── pascalseg.py ├── playground.ipynb ├── printmean.py ├── probelinear_pytorch.py ├── quantile.py ├── quantprobe.py ├── rebuttal.ipynb ├── report.py ├── reverse_prediction.py ├── rotate.py ├── saveneuralstatistics.py ├── supp_figures.ipynb ├── synonym.py ├── synset_words.txt ├── unicsv.py ├── upsample.py ├── upsample_blob_data.py ├── vecquantile.py ├── viewprobe.py ├── voclabels.py └── w2color.npy └── zoo └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .* 2 | *.pyc 3 | dataset 4 | dissection 5 | zoo 6 | !/dataset/README.md 7 | !/dissection/README.md 8 | !/zoo/README.md 9 | !/.gitignore 10 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT LICENSE 2 | =========== 3 | 4 | Copyright (c) 2017 MIT, Computer Science and Artificial Intelligence Laboratory, David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 7 | 8 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 11 | 12 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Net2Vec 2 | 3 | ## Introduction 4 | This repository contains the code for our [arxiv'18 paper](https://arxiv.org/abs/1801.03454) Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks. It is [forked code-wise](https://github.com/CSAILVision/NetDissect) and builds on the work from [Bau et al's CVPR'17 paper](http://netdissect.csail.mit.edu/final-network-dissection.pdf) Network Dissection: 5 | Quantifying Interpretability of Deep Visual Representations. 6 | 7 | Pardon the current appearance of the repo: this code is still being developed and will be cleaned up (with more user-friendly README instructions) shortly. 8 | 9 | ## Probing a Caffe/Pytorch Network 10 | 11 | First, collect network activations using either [src/netprobe.py](src/netprobe.py) to prove a Caffe network or [src/netprobe_pytorch.py](src/netprobe_pytorch.py) to probe a PyTorch network. For PyTorch networks, blobs are based on the path to the module of interest, with "." denoting entering into a nn.Module/nn.Sequential (i.e., to probe conv5 in the pytorch implementation of pytorch, pass in "features.11" as the blob name). 12 | 13 | Second, collect activation quantiles as done in NetDissect using [src/quantprobe.py](src/quantprobe.py). 14 | 15 | ### Segmentation 16 | 17 | #### Single-Filter 18 | Run [src/labelprobe_pytorch.py](src/labelprobe_pytorch.py) to probe network activations using single filters (use this regardless of if the network was from Caffe or PyTorch; this function approximately does the same thing as the original [src/labelprobe.py](src/labelprobe.py) but we save results in a different format and have a few minor implementation differences (i.e., 1., upsampling after thresholding for consistently with the multi-filter approach, 2., upsampling bilinearly without respect to the receptive field, 3., using the BRODEN train/val split to choose the best filter, see our [paper](https://arxiv.org/abs/1801.03454) for more details). 19 | 20 | #### Multi-Filter 21 | Run [src/linearprobe_pytorch.py](src/linearprobe_pytorch.py) first and then [src/probelinear_pytorch.py](src/probelinear_pytorch.py). 22 | 23 | To be continued... 24 | 25 | ### Classification 26 | 27 | To be continued... 28 | 29 | # README from NetDissect 30 | 31 | You can use this code with naive [Caffe](https://github.com/BVLC/caffe), with matcaffe and pycaffe compiled. We also provide a [PyTorch wrapper](script/rundissect_pytorch.sh) to apply NetDissect to probe networks in PyTorch format. There are dissection results for several networks at the [project page](http://netdissect.csail.mit.edu/). 32 | 33 | This code includes 34 | 35 | * Code to run network dissection on an arbitrary deep convolutional 36 | neural network provided as a Caffe deploy.prototxt and .caffemodel. 37 | The script `rundissect.sh` runs all the needed phases. 38 | 39 | * Code to create the merged Broden dataset from constituent datasets 40 | ADE, PASCAL, PASCAL Parts, PASCAL Context, OpenSurfaces, and DTD. 41 | The script `makebroden.sh` runs all the needed steps. 42 | 43 | 44 | ## Download 45 | * Clone the code of Network Dissection from github 46 | ``` 47 | https://github.com/CSAILVision/NetDissect.git 48 | cd NetDissect 49 | ``` 50 | * Download the Broden dataset (~1GB space) and the example pretrained models. 51 | ``` 52 | script/dlbroden_227.sh 53 | script/dlzoo_example.sh 54 | ``` 55 | 56 | Note that you can run ```script/dlbroden.sh``` to download Broden dataset with images in all three resolution (227x227,224x224,384x384), or run ```script/dlzoo.sh``` to download more CNN models. AlexNet models work with 227x227 image input, while VGG, ResNet, GoogLeNet works with 224x224 image input. 57 | 58 | ## Run in Caffe 59 | * Run Network Dissection in Caffe to probe the conv5 layer of the AlexNet trained on Places365. Results will be saved to ```dissection/caffe_reference_model_places365/```, in which ```html``` contains the visualization of all the units in a html page and ```conv5-result.csv``` contains the raw predicted labels for each unit. The code takes about 40 mintues to run, and it will generate about 1.5GB intermediate results (mmap) for one layer, which you could delete after the code finishes running. 60 | 61 | ``` 62 | script/rundissect.sh --model caffe_reference_places365 --layers "conv5" --dataset dataset/broden1_227 --resolution 227 63 | ``` 64 | 65 | * Run Network Dissection to compare three layers of AlexNet trained on ImageNet. Results will be saved to ```dissection/caffe_reference_model_imagenet/```. 66 | 67 | ``` 68 | script/rundissect.sh --model caffe_reference_imagenet --layers "conv3 conv4 conv5" --dataset dataset/broden1_227 --resolution 227 69 | ``` 70 | 71 | * If you need to regenerate the Broden dataset from scratch, you can run ```script/makebroden.sh```. The script will download the pieces and merge them. 72 | 73 | * Network dissection depends on scipy as well as pycaffe. [Details on installing pycaffe can be found here](http://caffe.berkeleyvision.org/tutorial/interfaces.html#python). 74 | 75 | ## Run in PyTorch 76 | 77 | * Run Network Dissection in PyTorch. Please install [PyTorch](http://pytorch.org/) and [Torchvision](https://github.com/pytorch/vision) first. We provide a [feature extract wrapper](src/netprobe_pytorch.py) for PyTorch. So you could run ```script/rundissect_pytorch.sh``` to probe the existing networks trained on ImageNet in [Torchvision](https://github.com/pytorch/vision/tree/master/torchvision/models). 78 | 79 | ``` 80 | script/rundissect_pytorch.sh 81 | ``` 82 | 83 | * Or try ```script/rundissect_pytorch_external.sh``` on a resnet18 trained on [Places365](https://github.com/CSAILVision/places365). 84 | 85 | ``` 86 | script/rundissect_pytorch_external.sh 87 | ``` 88 | 89 | ## Report 90 | * At the end of the dissection script, a report will be generated that summarizes the semantics of the networks. For example, after you have tested the conv5 layer of caffe_reference_places365, you will have: 91 | 92 | ``` 93 | dissection/caffe_reference_places365/html/conv5.html 94 | dissection/caffe_reference_places365/html/image/conv5-bargraph.svg 95 | dissection/caffe_reference_places365/html/image/conv5-0[###].png 96 | dissection/caffe_reference_places365/conv5-result.csv 97 | ``` 98 | 99 | These are, respectively, the HTML-formatted report, the semantics of the units of the layer summarized as a bar graph, visualizations of all the units of the layer (using zero-indexed unit numbers), and a CSV file containing raw scores of the top matching semantic concepts in each category for each unit of the layer. 100 | 101 | 104 | 105 | 106 | ## Reference 107 | If you find the code useful, please cite the following papers 108 | ``` 109 | @article{fong2018, 110 | title={Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks}, 111 | author={Fong, Ruth and Vedaldi, Andrea}, 112 | journal={arXiv preprint arXiv:1801.03454}, 113 | year={2018} 114 | } 115 | ``` 116 | 117 | ``` 118 | @inproceedings{net 119 | 120 | 2017, 121 | title={Network Dissection: Quantifying Interpretability of Deep Visual Representations}, 122 | author={Bau, David and Zhou, Bolei and Khosla, Aditya and Oliva, Aude and Torralba, Antonio}, 123 | booktitle={Computer Vision and Pattern Recognition}, 124 | year={2017} 125 | } 126 | ``` 127 | 128 | -------------------------------------------------------------------------------- /dataset/README.md: -------------------------------------------------------------------------------- 1 | Dataset directory 2 | ================= 3 | 4 | Install or link datasets at this directory. 5 | 6 | The script dlbroden.sh will download the Broden dataset here. 7 | -------------------------------------------------------------------------------- /dissection/README.md: -------------------------------------------------------------------------------- 1 | Dissection directory 2 | ==================== 3 | 4 | Indermediate and final results of network dissection are written 5 | into this directory, with final output files of the form: 6 | 7 | [modelname]/[layername].csv 8 | [modelname]/html/[layername].html 9 | 10 | -------------------------------------------------------------------------------- /plot/extract_csv.m: -------------------------------------------------------------------------------- 1 | % loading the csv files generate by NetDissect 2 | disp('extract semantics from raw data') 3 | params = {}; 4 | params.networks_name = {'ResNet-152-model_places365','ResNet-152-model_imagenet','caffe_reference_places365','caffe_reference_places205','caffe_reference_imagenet'}; 5 | params.csv_files = {'ResNet-152-model_places365.csv','ResNet-152-model_imagenet.csv','caffe_reference_places365.csv','caffe_reference_places205.csv','caffe_reference_imagenet.csv'}; 6 | 7 | params.layers_name = network2layer(params.networks_name, 'networkprobe'); 8 | 9 | 10 | [unit_semantics] = return_annotation( params ); 11 | 12 | 13 | thresh = 0.04; % IoU threshold to decide if an unit is a interpretable detector. 14 | concepts = {'object','part','scene','material','texture', 'color'}; 15 | indices_concepts = [1,6,4,3,5,2]; % (objet,part,scene,material,texture,color) as the column index inside the csv file 16 | stat = {}; 17 | stat.networks_name = params.networks_name; 18 | stat.layers_name = params.layers_name; 19 | stat.ratio_detectors = zeros(numel(params.networks_name), numel(concepts)+1); 20 | stat.num_uniquedetectors = zeros(numel(params.networks_name), numel(concepts)+1); 21 | stat.num_detectors = zeros(numel(params.networks_name), numel(concepts)+1); 22 | stat.concepts = ['all' concepts]; 23 | stat.thresh = thresh; 24 | stat.indices_concepts = indices_concepts; 25 | 26 | 27 | for netID = 1:numel(params.networks_name) 28 | semantics_network = unit_semantics{netID,3}; 29 | 30 | scores_allconcept = str2double(semantics_network(:,2:2:end)); 31 | [max_scores, max_idx] = max(scores_allconcept,[],2); 32 | num_unit = size(max_scores,1); 33 | 34 | num_detector_concepts = zeros(1,6); 35 | num_uniquedetector_concepts = zeros(1,6); 36 | for conceptID = 1:numel(indices_concepts) 37 | num_detector_overall = sum(scores_allconcept(:, indices_concepts(conceptID))>thresh); 38 | num_uniquedetector_overall = numel(unique(semantics_network(scores_allconcept(:,indices_concepts(conceptID))>thresh,indices_concepts(conceptID)*2-1))); 39 | num_detector_concepts(conceptID) = num_detector_overall; 40 | num_uniquedetector_concepts(conceptID) = num_uniquedetector_overall; 41 | end 42 | 43 | num_detector_overall = sum(max_scores>thresh); 44 | num_uniquedetector_overall = sum(num_detector_concepts);%num_detector_object_unique+num_detector_objectpart_unique+num_detector_texture_unique+num_detector_material_unique+num_detector_color_unique+num_detector_scene_unique; 45 | stat.num_detectors(netID,:) = [num_detector_overall num_detector_concepts];%[num_detector, num_detector_object, num_detector_objectpart, num_detector_scene, num_detector_material, num_detector_texture, num_detector_color]; 46 | stat.ratio_detectors(netID,:) = stat.num_detectors(netID,:)./num_unit; %[num_detector/num_unit num_detector_object/num_unit num_detector_objectpart/num_unit num_detector_scene/num_unit num_detector_material/num_unit num_detector_texture/num_unit num_detector_color/num_unit]; 47 | stat.num_uniquedetectors(netID,:) = [num_uniquedetector_overall num_uniquedetector_concepts];%[num_uniquedetector num_detector_object_unique num_detector_objectpart_unique num_detector_scene_unique num_detector_material_unique num_detector_texture_unique num_detector_color_unique]; 48 | end 49 | save('semantics_samples.mat','stat','unit_semantics'); 50 | -------------------------------------------------------------------------------- /plot/getPrintName.m: -------------------------------------------------------------------------------- 1 | function list_output = getPrintName( list_featurename, type_print) 2 | if strcmp(type_print, 'feature') 3 | printMap = construct_dictionary_feature(); 4 | elseif strcmp(type_print, 'semantics') 5 | printMap = construct_dictionary_semantics(); 6 | elseif strcmp(type_print,'width') 7 | printMap = construct_dictionary_width(); 8 | end 9 | list_output = cell(numel(list_featurename),1); 10 | 11 | for i = 1:numel(list_featurename) 12 | try 13 | list_output{i} = printMap(list_featurename{i}); 14 | catch exception 15 | error(list_featurename{i}) 16 | end 17 | end 18 | 19 | 20 | end 21 | 22 | function printMap = construct_dictionary_width() 23 | printMap = containers.Map(); 24 | printMap('caffe_reference_places365_GAP_gap') = 'AlexNet-Places365-GAP'; 25 | 26 | end 27 | 28 | function printMap = construct_dictionary_feature() 29 | 30 | printMap = containers.Map(); 31 | printMap('ResNet-152-model_imagenet-pool5') = 'ResNet152-ImageNet'; 32 | printMap('ResNet-50-model_imagenet-pool5') = 'ResNet50-ImageNet'; 33 | printMap('googlenet_imagenet-pool5/7x7_s1') = 'GoogLeNet-ImageNet'; 34 | printMap('ResNet-152-model_places365-pool5') = 'ResNet152-Places365'; 35 | printMap('vgg16_hybrid1365-fc7') = 'VGG-Hybrid'; 36 | printMap('vgg16_imagenet-fc7') = 'VGG-ImageNet'; 37 | printMap('vgg16_places205-fc7') = 'VGG-Places205'; 38 | printMap('vgg16_places365-fc7') = 'VGG-Places365'; 39 | printMap('caffe_reference_imagenetplaces205-pool5') = 'AlexNet-Hybrid'; 40 | printMap('googlenet_places205-pool5/7x7_s1') = 'GoogLeNet-Places205'; 41 | printMap('googlenet_places365-pool5/7x7_s1') = 'GoogLeNet-Places365'; 42 | printMap('caffe_reference_imagenet-pool5') = 'AlexNet-ImageNet'; 43 | printMap('caffe_reference_places205_batchnorm-pool5') = 'AlexNet-Places205-BN'; 44 | printMap('caffe_reference_places365_GAP-pool5_gap') = 'AlexNet-Places365-GAP'; 45 | printMap('caffe_reference_places365-pool5') = 'AlexNet-Places365'; 46 | printMap('caffe_reference_places205-pool5') = 'AlexNet-Places205'; 47 | printMap('weakly_deepcontext-pool5') = 'context'; 48 | printMap('weakly_colorization-pool5') = 'colorization'; 49 | printMap('weakly_audio-pool5') = 'audio'; 50 | printMap('weakly_splitbrain-pool5') = 'crosschannel'; 51 | printMap('weakly_videotracking-pool5') = 'tracking'; 52 | printMap('weakly_solvingpuzzle-pool5') = 'puzzle'; 53 | printMap('weakly_objectcentric-pool5') = 'objectcentric'; 54 | printMap('weakly_egomotion-cls_pool5') = 'egomotion'; 55 | printMap('weakly_learningbymoving-pool5') = 'moving'; 56 | printMap('caffenet_random-pool5') = 'AlexNet-random'; 57 | printMap('weakly_videoorder-pool5') = 'frameorder'; 58 | 59 | end 60 | 61 | function printMap = construct_dictionary_semantics() 62 | 63 | printMap = containers.Map(); 64 | printMap('ResNet-152-model_imagenet') = 'ResNet152-ImageNet'; 65 | printMap('ResNet-152-model_places365') = 'ResNet152-Places365'; 66 | printMap('ResNet-50-model_imagenet') = 'ResNet50-ImageNet'; 67 | printMap('resnet-152-torch-places365') = 'ResNet152-Places365'; 68 | printMap('resnet-152-torch-imagenet') = 'ResNet152-ImageNet'; 69 | printMap('vgg16_imagenet') = 'VGG-ImageNet'; 70 | printMap('vgg16_places205') = 'VGG-Places205'; 71 | printMap('vgg16_places365') = 'VGG-Places365'; 72 | printMap('vgg16_hybrid1365') = 'VGG-Hybrid'; 73 | printMap('googlenet_imagenet') = 'GoogLeNet-ImageNet'; 74 | printMap('googlenet_places205') = 'GoogLeNet-Places205'; 75 | printMap('googlenet_places365') = 'GoogLeNet-Places365'; 76 | printMap('caffenet_random') = 'AlexNet-random'; 77 | printMap('caffe_reference_imagenet') = 'AlexNet-ImageNet'; 78 | printMap('caffe_reference_places205') = 'AlexNet-Places205'; 79 | printMap('caffe_reference_imagenetplaces205') = 'AlexNet-Hybrid'; 80 | printMap('caffe_reference_places205_batchnorm') = 'AlexNet-Places205-BatchNorm'; 81 | printMap('caffe_reference_places365') = 'AlexNet-Places365'; 82 | printMap('caffe_reference_places365_GAP') = 'AlexNet-Places365-GAP'; 83 | printMap('caffe_reference_places365_GAP1024') = 'AlexNet-Places365-GAP1024'; 84 | printMap('caffe_reference_places205_nodropout') = 'AlexNet-Places205-NoDropout'; 85 | printMap('caffe_reference_places205_repeat1') = 'AlexNet-Places205-repeat1'; 86 | printMap('caffe_reference_places205_repeat2') = 'AlexNet-Places205-repeat2'; 87 | printMap('caffe_reference_places205_repeat3') = 'AlexNet-Places205-repeat3'; 88 | printMap('caffe_reference_places205_repeat4') = 'AlexNet-Places205-repeat4'; 89 | 90 | printMap('weakly_deepcontext') = 'context'; 91 | printMap('weakly_audio') = 'audio'; 92 | printMap('weakly_videoorder') = 'frameorder'; 93 | printMap('weakly_videotracking') = 'tracking'; 94 | printMap('weakly_egomotion') = 'egomotion'; 95 | printMap('weakly_learningbymoving') = 'moving'; 96 | printMap('weakly_objectcentric') = 'objectcentric'; 97 | printMap('weakly_solvingpuzzle') = 'puzzle'; 98 | printMap('weakly_colorization') = 'colorization'; 99 | printMap('weakly_splitbrain') = 'crosschannel'; 100 | 101 | end -------------------------------------------------------------------------------- /plot/network2layer.m: -------------------------------------------------------------------------------- 1 | function [layers] = network2layer( networks, type_networks) 2 | %NETWORK2FEATURE Summary of this function goes here 3 | % Detailed explanation goes here 4 | 5 | layers = cell(numel(networks),1); 6 | if strcmp(type_networks,'genericfeature') 7 | layerMap = get_layerMap_genericfeature(); 8 | elseif strcmp(type_networks, 'networkprobe') 9 | layerMap = get_layerMap_networkprobe(); 10 | end 11 | 12 | try 13 | for netID = 1:numel(networks) 14 | if strcmp(networks{netID}(1:11),'places_iter') 15 | layers{netID} = 'conv5'; 16 | else 17 | layers{netID} = layerMap(networks{netID}); 18 | end 19 | end 20 | catch exception 21 | 22 | error(['no info for ' networks{netID}]); 23 | end 24 | 25 | end 26 | 27 | function layerMap = get_layerMap_networkprobe() 28 | layerMap = containers.Map(); 29 | layerMap('caffenet_random') = 'conv5'; 30 | layerMap('caffe_reference_imagenet') = 'conv5'; 31 | layerMap('caffe_reference_places205_bn') = 'conv5'; 32 | layerMap('caffe_reference_places205') = 'conv5'; 33 | layerMap('caffe_reference_places365') = 'conv5'; 34 | layerMap('caffe_reference_places205_batchnorm') = 'conv5'; 35 | layerMap('caffe_reference_imagenetplaces205') = 'conv5'; 36 | layerMap('caffe_reference_places365_GAP') = 'conv5'; 37 | layerMap('caffe_reference_places365_GAPplus') = 'conv5'; 38 | layerMap('caffe_reference_places365_GAP1024') = 'conv5'; 39 | layerMap('caffe_reference_places205_repeat1') = 'conv5'; 40 | layerMap('caffe_reference_places205_repeat2') = 'conv5'; 41 | layerMap('caffe_reference_places205_repeat3') = 'conv5'; 42 | layerMap('caffe_reference_places205_repeat4') = 'conv5'; 43 | layerMap('caffe_reference_places205_nodropout') = 'conv5'; 44 | 45 | layerMap('vgg16_hybrid1365') = 'conv5_3'; 46 | layerMap('vgg16_imagenet') = 'conv5_3'; 47 | layerMap('vgg16_places205') = 'conv5_3'; 48 | layerMap('vgg16_places365') = 'conv5_3'; 49 | 50 | layerMap('weakly_audio') = 'conv5'; 51 | layerMap('weakly_deepcontext') = 'conv5'; 52 | layerMap('weakly_videoorder') = 'conv5'; 53 | layerMap('weakly_videotracking') = 'conv5'; 54 | layerMap('weakly_egomotion') = 'cls_conv5'; 55 | layerMap('weakly_learningbymoving') = 'conv5'; 56 | layerMap('weakly_objectcentric') = 'conv5'; 57 | layerMap('weakly_solvingpuzzle') = 'conv5_s1'; 58 | layerMap('weakly_colorization') = 'conv5'; 59 | layerMap('weakly_splitbrain') = 'conv5'; 60 | 61 | layerMap('googlenet_imagenet') = 'inception_5b-output'; 62 | layerMap('googlenet_places205') = 'inception_5b-output'; 63 | layerMap('googlenet_places365') = 'inception_5b-output'; 64 | 65 | layerMap('ResNet-152-model_imagenet') = 'res5c'; 66 | layerMap('ResNet-152-model_places365') = 'res5c'; 67 | layerMap('ResNet-50-model_imagenet') = 'res5c'; 68 | layerMap('resnet-152-torch-places365') = 'caffe.Eltwise_510'; 69 | layerMap('resnet-152-torch-imagenet') = 'caffe.Eltwise_510'; 70 | layerMap('rotation_020') = 'conv5'; 71 | layerMap('rotation_040') = 'conv5'; 72 | layerMap('rotation_060') = 'conv5'; 73 | layerMap('rotation_080') = 'conv5'; 74 | layerMap('rotation_100') = 'conv5'; 75 | end 76 | 77 | function layerMap = get_layerMap_genericfeature() 78 | layerMap = containers.Map(); 79 | layerMap('caffenet_random') = 'pool5'; 80 | layerMap('caffe_reference_imagenet') = 'pool5'; 81 | layerMap('caffe_reference_places205_bn') = 'pool5'; 82 | layerMap('caffe_reference_places205') = 'pool5'; 83 | layerMap('caffe_reference_places365') = 'pool5'; 84 | layerMap('caffe_reference_places365_GAP') = 'pool5_gap'; 85 | layerMap('caffe_reference_places205_batchnorm') = 'pool5'; 86 | 87 | layerMap('caffe_reference_imagenetplaces205') = 'pool5'; 88 | layerMap('vgg16_hybrid1365') = 'fc7'; 89 | layerMap('vgg16_imagenet') = 'fc7'; 90 | layerMap('vgg16_places205') = 'fc7'; 91 | layerMap('vgg16_places365') = 'fc7'; 92 | 93 | layerMap('weakly_audio') = 'pool5'; 94 | layerMap('weakly_deepcontext') = 'pool5'; 95 | layerMap('weakly_videoorder') = 'pool5'; 96 | layerMap('weakly_videotracking') = 'pool5'; 97 | layerMap('weakly_egomotion') = 'cls_pool5'; 98 | layerMap('weakly_learningbymoving') = 'pool5'; 99 | layerMap('weakly_objectcentric') = 'pool5'; 100 | layerMap('weakly_solvingpuzzle') = 'pool5'; 101 | layerMap('colorization_berkeley') = 'pool5'; 102 | layerMap('googlenet_imagenet') = 'pool5/7x7_s1'; 103 | layerMap('googlenet_places205') = 'pool5/7x7_s1'; 104 | layerMap('googlenet_places365') = 'pool5/7x7_s1'; 105 | layerMap('ResNet-152-model_imagenet') = 'pool5'; 106 | layerMap('ResNet-152-model_places365') = 'pool5'; 107 | layerMap('ResNet-50-model_imagenet') = 'pool5'; 108 | layerMap('weakly_colorization') = 'pool5'; 109 | layerMap('weakly_colorization_nors') = 'pool5'; 110 | layerMap('weakly_splitbrain') = 'pool5'; 111 | layerMap('weakly_splitbrain_nors') = 'pool5'; 112 | end 113 | -------------------------------------------------------------------------------- /plot/plot_histogram.m: -------------------------------------------------------------------------------- 1 | % plot the histogram of all the detectors the network dissection identifies 2 | % for the given networks. 3 | clear 4 | load('semantics_samples.mat'); 5 | thresh = stat.thresh; 6 | concepts = stat.concepts(2:end);% the first one is the all the detector 7 | indices_concepts = stat.indices_concepts; 8 | 9 | concepts_select = {'object','texture','scene'}; % select the concepts to plot 10 | [concepts_select, ia, ib] = intersect(concepts_select, concepts); 11 | indices_concepts_select = indices_concepts(ib); 12 | 13 | for netID = 1:numel(stat.networks_name) 14 | figure 15 | semantics_network = unit_semantics{netID,3}; 16 | num_unit = size(semantics_network,1); 17 | scores_allconcept = str2double(semantics_network(:,2:2:end)); 18 | 19 | for conceptID = 1:numel(concepts_select) 20 | detector_concept = semantics_network(scores_allconcept(:,indices_concepts_select(conceptID))>thresh, indices_concepts_select(conceptID)*2-1); 21 | 22 | [unique_data,junk,ind] = unique(detector_concept); 23 | freq_unique_data = histc(ind,1:numel(unique_data)); 24 | [value_sort, idx_sort] = sort(freq_unique_data,'descend'); 25 | uniquedetectors = unique_data(idx_sort); 26 | freq_detectors = value_sort; 27 | 28 | subplot(numel(concepts_select),1,conceptID); 29 | set(gcf,'Color',[1 1 1]); 30 | 31 | bar( freq_detectors,'stacked'), 32 | title(sprintf('Histogram of %s detectors', concepts_select{conceptID})); 33 | xticks([1:numel(uniquedetectors)]) 34 | xticklabels(printLabels(uniquedetectors)), xtickangle(45) 35 | 36 | set(gca,'FontSize',20); 37 | xlim(gca,[0 numel(uniquedetectors)+1]) 38 | end 39 | xlabel(strrep(stat.networks_name{netID},'_','-')); 40 | end -------------------------------------------------------------------------------- /plot/plot_semantics.m: -------------------------------------------------------------------------------- 1 | % sample script to plot the summary of detector numbers for all the 2 | % networks 3 | % run extract_csv.m first to extract the semantics from the raw csv if you 4 | % jush finish running network dissection 5 | 6 | clear 7 | load('semantics_cvpr_release.mat'); 8 | 9 | selectIDX_networks = [1:numel(stat.networks_name)]; 10 | 11 | stat.networks_name = stat.networks_name(selectIDX_networks); 12 | stat.layers_name = stat.layers_name(selectIDX_networks); 13 | stat.ratio_detectors = stat.ratio_detectors(selectIDX_networks,:); 14 | stat.num_uniquedetectors = stat.num_uniquedetectors(selectIDX_networks,:); 15 | stat.num_detectors = stat.num_detectors(selectIDX_networks,:); 16 | 17 | sum_uniquedetectors = sum(stat.num_uniquedetectors(:,2:end),2); 18 | [value_sort, idx_sort] = sort(sum_uniquedetectors,'descend'); 19 | stat.networks_name = stat.networks_name(idx_sort); 20 | stat.layers_name = stat.layers_name(idx_sort); 21 | stat.ratio_detectors = stat.ratio_detectors(idx_sort,:); 22 | stat.num_uniquedetectors = stat.num_uniquedetectors(idx_sort,:); 23 | stat.num_detectors = stat.num_detectors(idx_sort,:); 24 | 25 | disp(stat); 26 | networks_print = getPrintName(stat.networks_name,'semantics'); 27 | 28 | % figure, 29 | % %plot([1:size(stat.ratio_detectors,1)], stat.ratio_detectors', '--o'), 30 | % bar(stat.ratio_detectors, 'stacked'), 31 | % legend(stat.concepts),title('Ratio of detectors'); 32 | % xticks([1:size(stat.ratio_detectors,1)]) 33 | % xticklabels(networks_print),xtickangle(45) 34 | 35 | figure, 36 | subplot(1,2,1); 37 | bar( stat.num_uniquedetectors(:,2:end),'stacked'), 38 | legend(stat.concepts(2:end)),title('Number of unique detectors'); 39 | xticks([1:size(stat.ratio_detectors,1)]) 40 | xticklabels(networks_print), xtickangle(45) 41 | 42 | subplot(1,2,2); 43 | bar( stat.num_detectors(:,2:end),'stacked'), 44 | legend(stat.concepts(2:end)),title('Number of detectors'); 45 | xticks([1:size(stat.num_detectors,1)]) 46 | xticklabels(networks_print), xtickangle(45) -------------------------------------------------------------------------------- /plot/printLabels.m: -------------------------------------------------------------------------------- 1 | function [list_output] = printLabels( list_input) 2 | % remove some legacy signs in the names of concepts 3 | list_output = cell(numel(list_input),1); 4 | for i = 1:numel(list_input) 5 | tmp = list_input{i}; 6 | tmp(1:end-2) = strrep(tmp(1:end-2),'_',' '); 7 | tmp(1:end-2) = strrep(tmp(1:end-2),'-',' '); 8 | tmp = strrep(tmp,'-s',''); 9 | tmp = strrep(tmp,'-c',''); 10 | list_output{i} = tmp; 11 | end 12 | 13 | 14 | end 15 | 16 | -------------------------------------------------------------------------------- /plot/result.txt: -------------------------------------------------------------------------------- 1 | ./vgg16_places205/backup/conv5_3-result.csv 2 | ./vgg16_places205/conv3_3-result.csv 3 | ./vgg16_places205/conv5_3-result.csv 4 | ./vgg16_places205/conv4_3-result.csv 5 | ./places_iter_492/conv5-result.csv 6 | ./places_iter_492/conv3-result.csv 7 | ./places_iter_492/conv4-result.csv 8 | ./googlenet_imagenet/inception_4e-output-result.csv 9 | ./googlenet_imagenet/inception_5b-output-backup.csv 10 | ./googlenet_imagenet/inception_3b-output-result.csv 11 | ./googlenet_imagenet/inception_5b-output-result.csv 12 | ./googlenet_imagenet/conv2-norm2-result.csv 13 | ./places_iter_27396/conv5-result.csv 14 | ./places_iter_27396/conv3-result.csv 15 | ./places_iter_27396/conv4-result.csv 16 | ./caffe_reference_places205_batchnorm/conv5-result.csv 17 | ./caffe_reference_places205_batchnorm/conv2-result.csv 18 | ./caffe_reference_places205_batchnorm/conv3-result.csv 19 | ./caffe_reference_places205_batchnorm/conv4-result.csv 20 | ./caffe_reference_places205_batchnorm/conv1-result.csv 21 | ./places_iter_1/conv5-result.csv 22 | ./places_iter_1/conv3-result.csv 23 | ./places_iter_1/conv4-result.csv 24 | ./weakly_objectcentric/conv5-result.csv 25 | ./places_iter_600818/conv5-result.csv 26 | ./places_iter_600818/conv3-result.csv 27 | ./places_iter_600818/conv4-result.csv 28 | ./colorization_berkeley/conv5-result.csv 29 | ./places_iter_60491/conv3-result.csv 30 | ./places_iter_60491/conv4-result.csv 31 | ./places_iter_60491/conv5-result.csv 32 | ./places_iter_12164/conv5-result.csv 33 | ./places_iter_12164/conv4-result.csv 34 | ./places_iter_12164/conv3-result.csv 35 | ./rotation_060/conv5-result.csv 36 | ./rotation_060/conv2-result.csv 37 | ./rotation_060/conv1-result.csv 38 | ./rotation_060/conv3-result.csv 39 | ./rotation_060/conv4-result.csv 40 | ./vgg16_imagenet/conv4_3-result.csv 41 | ./vgg16_imagenet/conv5_3-result.csv 42 | ./vgg16_imagenet/conv3_3-result.csv 43 | ./places_iter_601603/conv5-result.csv 44 | ./places_iter_601603/conv3-result.csv 45 | ./places_iter_601603/conv4-result.csv 46 | ./caffe_reference_places205_repeat2/conv2-result.csv 47 | ./caffe_reference_places205_repeat2/conv5-result.csv 48 | ./caffe_reference_places205_repeat2/conv4-result.csv 49 | ./caffe_reference_places205_repeat2/conv3-result.csv 50 | ./caffe_reference_places205_repeat2/conv1-result.csv 51 | ./weakly_splitbrain/conv5-result.csv 52 | ./vgg16_hybrid1365/conv3_3-result.csv 53 | ./vgg16_hybrid1365/conv5_3-result.csv 54 | ./vgg16_hybrid1365/conv4_3-result.csv 55 | ./weakly_audio/conv5-result.csv 56 | ./places_iter_1200818/conv5-result.csv 57 | ./places_iter_1200818/conv3-result.csv 58 | ./places_iter_1200818/conv4-result.csv 59 | ./rotation_080/conv3-result.csv 60 | ./rotation_080/conv4-result.csv 61 | ./rotation_080/conv1-result.csv 62 | ./rotation_080/conv5-result.csv 63 | ./rotation_080/conv2-result.csv 64 | ./places_iter_300818/conv5-result.csv 65 | ./places_iter_300818/conv3-result.csv 66 | ./places_iter_300818/conv4-result.csv 67 | ./vgg16_places365/conv4_3-result.csv 68 | ./vgg16_places365/conv5_3-result.csv 69 | ./vgg16_places365/conv3_3-result.csv 70 | ./places_iter_99/conv4-result.csv 71 | ./places_iter_99/conv3-result.csv 72 | ./places_iter_99/conv5-result.csv 73 | ./places_iter_2/conv3-result.csv 74 | ./places_iter_2/conv4-result.csv 75 | ./places_iter_2/conv5-result.csv 76 | ./rotation_100/conv1-result.csv 77 | ./rotation_100/conv3-result.csv 78 | ./rotation_100/conv4-result.csv 79 | ./rotation_100/conv5-result.csv 80 | ./rotation_100/conv2-result.csv 81 | ./caffe_reference_imagenetplaces205/conv2-result.csv 82 | ./caffe_reference_imagenetplaces205/conv5-result.csv 83 | ./caffe_reference_imagenetplaces205/conv1-result.csv 84 | ./caffe_reference_imagenetplaces205/conv4-result.csv 85 | ./caffe_reference_imagenetplaces205/conv3-result.csv 86 | ./caffe_reference_places365_GAP/conv5-result.csv 87 | ./caffe_reference_places205_repeat1/conv5-result.csv 88 | ./caffe_reference_places205_repeat1/conv2-result.csv 89 | ./caffe_reference_places205_repeat1/conv1-result.csv 90 | ./caffe_reference_places205_repeat1/conv3-result.csv 91 | ./caffe_reference_places205_repeat1/conv4-result.csv 92 | ./caffe_reference_places205_repeat3/conv2-result.csv 93 | ./caffe_reference_places205_repeat3/conv5-result.csv 94 | ./caffe_reference_places205_repeat3/conv1-result.csv 95 | ./caffe_reference_places205_repeat3/conv4-result.csv 96 | ./caffe_reference_places205_repeat3/conv3-result.csv 97 | ./places_iter_136238/conv5-result.csv 98 | ./places_iter_136238/conv3-result.csv 99 | ./places_iter_136238/conv4-result.csv 100 | ./places_iter_1108/conv5-result.csv 101 | ./places_iter_1108/conv3-result.csv 102 | ./places_iter_1108/conv4-result.csv 103 | ./resnet-152-torch-imagenet/caffe.Eltwise_34-result.csv 104 | ./resnet-152-torch-imagenet/caffe.Eltwise_510-result.csv 105 | ./resnet-152-torch-imagenet/caffe.Eltwise_238-result.csv 106 | ./resnet-152-torch-imagenet/caffe.BN_1-result.csv 107 | ./resnet-152-torch-imagenet/caffe.Eltwise_116-result.csv 108 | ./resnet-152-torch-imagenet/caffe.Eltwise_478-result.csv 109 | ./resnet-152-torch-imagenet/caffe.Eltwise_358-result.csv 110 | ./rotation_040/conv4-result.csv 111 | ./rotation_040/conv3-result.csv 112 | ./rotation_040/conv1-result.csv 113 | ./rotation_040/conv2-result.csv 114 | ./rotation_040/conv5-result.csv 115 | ./weakly_videoorder/conv5-result.csv 116 | ./caffe_reference_places365_GAPplus/conv5-result.csv 117 | ./caffe_reference_places365_GAPplus/conv6-result.csv 118 | ./caffenet_random/conv5-result.csv 119 | ./ResNet-152-model_imagenet/res4b24-result.csv 120 | ./ResNet-152-model_imagenet/res5c-result.csv 121 | ./ResNet-152-model_imagenet/res4b12-result.csv 122 | ./ResNet-152-model_imagenet/res4b35-result.csv 123 | ./ResNet-152-model_imagenet/res3b7-result.csv 124 | ./weakly_colorization/conv5-result.csv 125 | ./googlenet_places205/inception_5b-output-result.csv 126 | ./googlenet_places205/inception_4e-output-result.csv 127 | ./googlenet_places205/conv2-norm2-result.csv 128 | ./googlenet_places205/inception_3b-output-result.csv 129 | ./caffe_reference_places205_nodropout/conv1-result.csv 130 | ./caffe_reference_places205_nodropout/conv4-result.csv 131 | ./caffe_reference_places205_nodropout/conv3-result.csv 132 | ./caffe_reference_places205_nodropout/conv2-result.csv 133 | ./caffe_reference_places205_nodropout/conv5-result.csv 134 | ./places_iter_5509/conv5-result.csv 135 | ./places_iter_5509/conv4-result.csv 136 | ./places_iter_5509/conv3-result.csv 137 | ./caffe_reference_places205_repeat4/conv3-result.csv 138 | ./caffe_reference_places205_repeat4/conv4-result.csv 139 | ./caffe_reference_places205_repeat4/conv1-result.csv 140 | ./caffe_reference_places205_repeat4/conv5-result.csv 141 | ./caffe_reference_places205_repeat4/conv2-result.csv 142 | ./weakly_egomotion/cls_conv5-result.csv 143 | ./caffe_reference_places205/conv1-result.csv 144 | ./caffe_reference_places205/conv3-result.csv 145 | ./caffe_reference_places205/conv4-result.csv 146 | ./caffe_reference_places205/conv5-result.csv 147 | ./caffe_reference_places205/conv2-result.csv 148 | ./ResNet-152-model_places365/res3b7-result.csv 149 | ./ResNet-152-model_places365/res4b24-result.csv 150 | ./ResNet-152-model_places365/res4b12-result.csv 151 | ./ResNet-152-model_places365/res4b35-result.csv 152 | ./ResNet-152-model_places365/res5c-result.csv 153 | ./weakly_deepcontext/conv5-result.csv 154 | ./places_iter_223/conv5-result.csv 155 | ./places_iter_223/conv3-result.csv 156 | ./places_iter_223/conv4-result.csv 157 | ./places_iter_2446/conv4-result.csv 158 | ./places_iter_2446/conv3-result.csv 159 | ./places_iter_2446/conv5-result.csv 160 | ./resnet-152-torch-places365/caffe.Eltwise_238-result.csv 161 | ./resnet-152-torch-places365/caffe.Eltwise_34-result.csv 162 | ./resnet-152-torch-places365/caffe.BN_1-result.csv 163 | ./resnet-152-torch-places365/caffe.Eltwise_478-result.csv 164 | ./resnet-152-torch-places365/caffe.Eltwise_358-result.csv 165 | ./resnet-152-torch-places365/caffe.Eltwise_116-result.csv 166 | ./resnet-152-torch-places365/caffe.Eltwise_510-result.csv 167 | ./places_iter_4/conv5-result.csv 168 | ./places_iter_4/conv3-result.csv 169 | ./places_iter_4/conv4-result.csv 170 | ./weakly_videotracking/conv5-result.csv 171 | ./weakly_solvingpuzzle/conv5_s1-result.csv 172 | ./ResNet-50-model_imagenet/res4f-result.csv 173 | ./ResNet-50-model_imagenet/res3d-result.csv 174 | ./ResNet-50-model_imagenet/res5c-result.csv 175 | ./caffe_reference_places365/conv5-result.csv 176 | ./places_iter_44/conv4-result.csv 177 | ./places_iter_44/conv3-result.csv 178 | ./places_iter_44/conv5-result.csv 179 | ./places_iter_9/conv3-result.csv 180 | ./places_iter_9/conv4-result.csv 181 | ./places_iter_9/conv5-result.csv 182 | ./weakly_learningbymoving/conv5-result.csv 183 | ./places_iter_2400818/conv4-result.csv 184 | ./places_iter_2400818/conv3-result.csv 185 | ./places_iter_2400818/conv5-result.csv 186 | ./rotation_020/conv1-result.csv 187 | ./rotation_020/conv4-result.csv 188 | ./rotation_020/conv3-result.csv 189 | ./rotation_020/conv2-result.csv 190 | ./rotation_020/conv5-result.csv 191 | ./caffe_reference_imagenet/conv3-result.csv 192 | ./caffe_reference_imagenet/conv4-result.csv 193 | ./caffe_reference_imagenet/conv1-result.csv 194 | ./caffe_reference_imagenet/conv5-result.csv 195 | ./caffe_reference_imagenet/conv2-result.csv 196 | ./places_iter_20/conv5-result.csv 197 | ./places_iter_20/conv4-result.csv 198 | ./places_iter_20/conv3-result.csv 199 | ./googlenet_places365/inception_5b-output-result.csv 200 | ./googlenet_places365/inception_3b-output-result.csv 201 | ./googlenet_places365/inception_4e-output-result.csv 202 | -------------------------------------------------------------------------------- /plot/return_annotation.m: -------------------------------------------------------------------------------- 1 | function [unit_activations] = return_annotation(params) 2 | % extract semantics from the raw csv file. 3 | 4 | unit_activations = cell(numel(params.networks_name),4); 5 | 6 | for i = 1:numel(params.csv_files) 7 | 8 | semantics_file = fullfile(params.csv_files{i}); 9 | fprintf('processing %s\n', semantics_file) 10 | 11 | data = fopen(semantics_file); 12 | csv_data = textscan(data,'%s','Delimiter','\n'); 13 | semantics_headers = textscan(csv_data{1}{1},'%s','Delimiter',','); 14 | semantics_headers = semantics_headers{1}; 15 | num_units = numel(csv_data{1})-1; 16 | semantics_values = cell(num_units, numel(semantics_headers)); 17 | for unitID = 1:num_units 18 | C = textscan(csv_data{1}{unitID+1},'%s','Delimiter',','); 19 | D = C{1}; 20 | semantics_values(str2double(D{1}),:) = D; 21 | end 22 | unit_activations{i, 1} = params.networks_name{i}; 23 | unit_activations{i, 2} = params.layers_name{i}; 24 | unit_activations{i, 3} = select_semantics(semantics_values); 25 | unit_activations{i, 4} = semantics_headers; 26 | fclose(data); 27 | end 28 | 29 | end 30 | 31 | function semantics_new = select_semantics(semantics) 32 | tmp_1 = [5:5:size(semantics,2)]; 33 | tmp_2 = [9:5:size(semantics,2)]; 34 | index_concepts = [tmp_1, tmp_2]; 35 | index_concepts = sort(index_concepts,'ascend'); 36 | semantics_new = semantics(:, index_concepts); 37 | end 38 | -------------------------------------------------------------------------------- /plot/semantics_allnetwork.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ruthcfong/net2vec/4d314fe2975762532afde7f7cf1c7f4a71ce1045/plot/semantics_allnetwork.pdf -------------------------------------------------------------------------------- /plot/semantics_cvpr_release.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ruthcfong/net2vec/4d314fe2975762532afde7f7cf1c7f4a71ce1045/plot/semantics_cvpr_release.mat -------------------------------------------------------------------------------- /plot/semantics_samples.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ruthcfong/net2vec/4d314fe2975762532afde7f7cf1c7f4a71ce1045/plot/semantics_samples.mat -------------------------------------------------------------------------------- /script/dlbroden.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | # Download broden1_224 8 | if [ ! -f dataset/broden1_224/index.csv ] 9 | then 10 | 11 | echo "Downloading broden1_224" 12 | mkdir -p dataset 13 | pushd dataset 14 | wget --progress=bar \ 15 | http://netdissect.csail.mit.edu/data/broden1_224.zip \ 16 | -O broden1_224.zip 17 | unzip broden1_224.zip 18 | rm broden1_224.zip 19 | popd 20 | 21 | fi 22 | 23 | # Download broden1_227 24 | if [ ! -f dataset/broden1_227/index.csv ] 25 | then 26 | 27 | echo "Downloading broden1_227" 28 | mkdir -p dataset 29 | pushd dataset 30 | wget --progress=bar \ 31 | http://netdissect.csail.mit.edu/data/broden1_227.zip \ 32 | -O broden1_227.zip 33 | unzip broden1_227.zip 34 | rm broden1_227.zip 35 | popd 36 | 37 | fi 38 | 39 | # Download broden1_384 40 | if [ ! -f dataset/broden1_384/index.csv ] 41 | then 42 | 43 | echo "Downloading broden1_384" 44 | mkdir -p dataset 45 | pushd dataset 46 | wget --progress=bar \ 47 | http://netdissect.csail.mit.edu/data/broden1_384.zip \ 48 | -O broden1_384.zip 49 | unzip broden1_384.zip 50 | rm broden1_384.zip 51 | popd 52 | 53 | fi 54 | 55 | -------------------------------------------------------------------------------- /script/dlbroden_227.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | # Download broden1_227 8 | if [ ! -f dataset/broden1_227/index.csv ] 9 | then 10 | 11 | echo "Downloading broden1_227" 12 | mkdir -p dataset 13 | pushd dataset 14 | wget --progress=bar \ 15 | http://netdissect.csail.mit.edu/data/broden1_227.zip \ 16 | -O broden1_227.zip 17 | unzip broden1_227.zip 18 | rm broden1_227.zip 19 | popd 20 | 21 | fi 22 | 23 | 24 | -------------------------------------------------------------------------------- /script/dlzoo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | declare -a MODELS=( 8 | "weakly_objectcentric" 9 | "weakly_videotracking" 10 | "weakly_audio" 11 | "weakly_colorization" 12 | "weakly_deepcontext" 13 | "weakly_splitbrain" 14 | "weakly_egomotion" 15 | "weakly_videoorder" 16 | "weakly_learningbymoving" 17 | "weakly_solvingpuzzle" 18 | "caffe_reference_imagenet" # alexnet-imagenet 19 | #"caffe_reference_places205" # alexnet-places205 20 | "caffe_reference_places365" # alexnet-places365 21 | "vgg16_imagenet" # vgg-imagenet 22 | #"vgg16_places205" # vgg-places205 23 | "vgg16_places365" # vgg-places365 24 | "googlenet_imagenet" # googlenet-imagenet 25 | #"googlenet_places205" # googlenet-places205 26 | "googlenet_places365" # googlenet-places365 27 | "resnet-152-torch-imagenet" # resnet-imagenet 28 | "resnet-152-torch-places365" # resnet-places365 29 | #"densenet161_imagenet" 30 | #"densenet161_places365" 31 | ) 32 | 33 | for MODEL in "${MODELS[@]}" 34 | do 35 | 36 | if [ ! -f zoo/${MODEL}.prototxt ] || [ ! -f zoo/${MODEL}.caffemodel ] 37 | then 38 | 39 | echo "Downloading $MODEL" 40 | mkdir -p zoo 41 | pushd zoo 42 | wget --progress=bar \ 43 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.prototxt 44 | wget --progress=bar \ 45 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.caffemodel 46 | popd 47 | 48 | fi 49 | 50 | done 51 | -------------------------------------------------------------------------------- /script/dlzoo_example.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | echo "Download AlexNet trained on Places365" 8 | declare -a MODELS=( 9 | "caffe_reference_places365" # alexnet-places365 10 | ) 11 | 12 | for MODEL in "${MODELS[@]}" 13 | do 14 | 15 | if [ ! -f zoo/${MODEL}.prototxt ] || [ ! -f zoo/${MODEL}.caffemodel ] 16 | then 17 | 18 | echo "Downloading $MODEL" 19 | mkdir -p zoo 20 | pushd zoo 21 | wget --progress=bar \ 22 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.prototxt 23 | wget --progress=bar \ 24 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.caffemodel 25 | popd 26 | 27 | fi 28 | 29 | done 30 | -------------------------------------------------------------------------------- /script/makebroden.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | # PASCAL 2010 Images 8 | if [ ! -f dataset/pascal/VOC2010/ImageSets/Segmentation/train.txt ] 9 | then 10 | 11 | echo "Downloading Pascal VOC2010 images" 12 | mkdir -p dataset/pascal 13 | pushd dataset/pascal 14 | wget --progress=bar \ 15 | http://host.robots.ox.ac.uk/pascal/VOC/voc2010/VOCtrainval_03-May-2010.tar \ 16 | -O VOCtrainval_03-May-2010.tar 17 | tar xvf VOCtrainval_03-May-2010.tar 18 | rm VOCtrainval_03-May-2010.tar 19 | mv VOCdevkit/* . 20 | rmdir VOCdevkit 21 | popd 22 | 23 | fi 24 | 25 | 26 | # PASCAL Part dataset 27 | if [ ! -f dataset/pascal/part/part2ind.m ] 28 | then 29 | 30 | echo "Downloading Pascal Part Dataset" 31 | mkdir -p dataset/pascal/part 32 | pushd dataset/pascal/part 33 | wget --progress=bar \ 34 | http://www.stat.ucla.edu/~xianjie.chen/pascal_part_dataset/trainval.tar.gz \ 35 | -O trainval.tar.gz 36 | tar xvfz trainval.tar.gz 37 | rm trainval.tar.gz 38 | popd 39 | 40 | fi 41 | 42 | 43 | # PASCAL Context dataset 44 | if [ ! -f dataset/pascal/context/labels.txt ] 45 | then 46 | 47 | echo "Downloading Pascal Context Dataset" 48 | mkdir -p dataset/pascal/context 49 | pushd dataset/pascal/context 50 | wget --progress=bar \ 51 | http://www.cs.stanford.edu/~roozbeh/pascal-context/trainval.tar.gz \ 52 | -O trainval.tar.gz 53 | tar xvfz trainval.tar.gz 54 | rm trainval.tar.gz 55 | popd 56 | 57 | fi 58 | 59 | 60 | # DTD 61 | if [ ! -f dataset/dtd/dtd-r1.0.1/imdb/imdb.mat ] 62 | then 63 | 64 | echo "Downloading Describable Textures Dataset" 65 | mkdir -p dataset/dtd 66 | pushd dataset/dtd 67 | wget --progress=bar \ 68 | https://www.robots.ox.ac.uk/~vgg/data/dtd/download/dtd-r1.0.1.tar.gz \ 69 | -O dtd-r1.0.1.tar.gz 70 | tar xvzf dtd-r1.0.1.tar.gz 71 | mv dtd dtd-r1.0.1 72 | rm dtd-r1.0.1.tar.gz 73 | popd 74 | 75 | fi 76 | 77 | 78 | # OpenSurfaces 79 | if [ ! -f dataset/opensurfaces/photos.csv ] 80 | then 81 | 82 | echo "Downloading OpenSurfaces Dataset" 83 | mkdir -p dataset/opensurfaces 84 | pushd dataset/opensurfaces 85 | wget --progress=bar \ 86 | http://labelmaterial.s3.amazonaws.com/release/opensurfaces-release-0.zip \ 87 | -O opensurfaces-release-0.zip 88 | unzip opensurfaces-release-0.zip 89 | rm opensurfaces-release-0.zip 90 | PROCESS=process_opensurfaces_release_0.py 91 | wget --progress=bar \ 92 | http://labelmaterial.s3.amazonaws.com/release/$PROCESS \ 93 | -O $PROCESS 94 | python $PROCESS 95 | popd 96 | 97 | fi 98 | 99 | 100 | # ADE20K 101 | if [ ! -f dataset/ade20k/index_ade20k.mat ] 102 | then 103 | 104 | echo "Downloading ADE20K Dataset" 105 | mkdir -p dataset/ade20k 106 | pushd dataset/ade20k 107 | wget --progress=bar \ 108 | http://groups.csail.mit.edu/vision/datasets/ADE20K/ADE20K_2016_07_26.zip \ 109 | -O ADE20K_2016_07_26.zip 110 | unzip ADE20K_2016_07_26.zip 111 | rm ADE20K_2016_07_26.zip 112 | popd 113 | 114 | fi 115 | 116 | 117 | # Now make broden in various sizes 118 | if [ ! -f dataset/broden1_224/index.csv ] 119 | then 120 | echo "Building Broden1 224" 121 | python src/joinseg.py --size=224 122 | fi 123 | 124 | if [ ! -f dataset/broden1_227/index.csv ] 125 | then 126 | echo "Building Broden1 227" 127 | python src/joinseg.py --size=227 128 | fi 129 | 130 | if [ ! -f dataset/broden1_384/index.csv ] 131 | then 132 | echo "Building Broden1 384" 133 | python src/joinseg.py --size=384 134 | fi 135 | -------------------------------------------------------------------------------- /script/makelinks.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This script replaces dummy directories with symbolic links to your 4 | # own directories ~/ndlinks/[dataset|dissection|sourcedata|zoo], which 5 | # of course may be further symbolic links to wherever you choose 6 | # to keep your datasets, models, and dissections. 7 | 8 | # Start from parent directory of script 9 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 10 | 11 | # Remove dummy directories 12 | for DUMMY in dataset dissection zoo 13 | do 14 | if [ -h ${DUMMY} ] 15 | then 16 | rm ${DUMMY} 17 | echo "Removed link ${DUMMY}" 18 | fi 19 | if [ -d ${DUMMY} ] 20 | then 21 | rm -f ${DUMMY}/README.md 22 | rmdir ${DUMMY} 23 | echo "Removed dummy directory ${DUMMY}" 24 | fi 25 | ln -s --target-directory=. ~/ndlinks/${DUMMY} 26 | echo "Created link ${DUMMY}" 27 | done 28 | 29 | # Remove dummy directories from git using sparse-checkout 30 | if [ -e .git/info ] 31 | then 32 | git config core.sparsecheckout true 33 | cat << EOF >> .git/info/sparse-checkout 34 | !dataset/* 35 | !dissection/* 36 | !zoo/* 37 | /* 38 | EOF 39 | git read-tree -mu HEAD 40 | # git checkout dataset dissection zoo 41 | fi 42 | -------------------------------------------------------------------------------- /script/rundissect.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # To use this, put the caffe model to be tested in the "zoo" directory 4 | # the following naming convention for a model called "vgg16_places365": 5 | # 6 | # zoo/caffe_reference_places365.caffemodel 7 | # zoo/caffe_reference_places365.prototxt 8 | # 9 | # and then, with scipy and pycaffe available in your python, run: 10 | # 11 | # ./rundissect.sh --model caffe_reference_places365 --layers "conv4 conv5" 12 | # 13 | # the output will be placed in a directory dissection/caffe_reference_places365/ 14 | # 15 | # More options are listed below. 16 | 17 | # Defaults 18 | THRESHOLD="0.04" 19 | #WORKDIR="dissection" 20 | WORKDIR="/scratch/shared/nfs1/ruthfong" 21 | TALLYDEPTH=2048 22 | PARALLEL=4 23 | TALLYBATCH=16 24 | PROBEBATCH=64 25 | QUANTILE="0.005" 26 | COLORDEPTH="3" 27 | CENTERED="c" 28 | MEAN="0 0 0" 29 | FORCE="none" 30 | ENDAFTER="none" 31 | MODELDIR="zoo" 32 | ROTATION_SEED="" 33 | ROTATION_POWER="1" 34 | 35 | # Start from parent directory of script 36 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 37 | 38 | # Parse command-line arguments. http://stackoverflow.com/questions/192249 39 | 40 | while [[ $# -gt 1 ]] 41 | do 42 | key="$1" 43 | 44 | case $key in 45 | -d|--model) 46 | DIR="$2" 47 | shift 48 | ;; 49 | -w|--weights) 50 | WEIGHTS="$2" 51 | shift 52 | ;; 53 | -p|--proto) 54 | PROTO="$2" 55 | shift 56 | ;; 57 | -l|--layers) 58 | LAYERS="$2" 59 | shift 60 | ;; 61 | -s|--dataset) 62 | DATASET="$2" 63 | shift 64 | ;; 65 | --colordepth) 66 | COLORDEPTH="$2" 67 | shift 68 | ;; 69 | --resolution) 70 | RESOLUTION="$2" 71 | shift 72 | ;; 73 | --probebatch) 74 | PROBEBATCH="$2" 75 | shift 76 | ;; 77 | -t|--threshold) 78 | THRESHOLD="$2" 79 | shift 80 | ;; 81 | --tallydepth) 82 | TALLYDEPTH="$2" 83 | shift 84 | ;; 85 | --tallybatch) 86 | TALLYBATCH="$2" 87 | shift 88 | ;; 89 | --mean) 90 | MEAN="$2" 91 | shift 92 | ;; 93 | --rotation_seed) 94 | ROTATION_SEED="$2" 95 | shift 96 | ;; 97 | --rotation_power) 98 | ROTATION_POWER="$2" 99 | shift 100 | ;; 101 | -w|--workdir) 102 | WORKDIR="$2" 103 | shift 104 | ;; 105 | -f|--force) 106 | FORCE="$2" 107 | shift 108 | ;; 109 | --endafter) 110 | ENDAFTER="$2" 111 | shift 112 | ;; 113 | --gpu) 114 | export CUDA_VISIBLE_DEVICES="$2" 115 | shift 116 | ;; 117 | *) 118 | echo "Unknown option" $key 119 | exit 3 120 | # unknown option 121 | ;; 122 | esac 123 | shift # past argument or value 124 | done 125 | 126 | # Get rid of slashes in layer names for directory purposes 127 | LAYERA=(${LAYERS//\//-}) 128 | 129 | # For expanding globs http://stackoverflow.com/questions/2937407 130 | function globexists { 131 | set +f 132 | test -e "$1" -o -L "$1";set -f 133 | } 134 | 135 | if [ -z $DIR ]; then 136 | echo '--model directory' must be specified 137 | exit 1 138 | fi 139 | 140 | if [ -z "${LAYERS}" ]; then 141 | echo '--layers layers' must be specified 142 | exit 1 143 | fi 144 | 145 | # Set up directory to work in, and lay down pid file etc. 146 | mkdir -p $WORKDIR/$DIR 147 | if [ -z "${FORCE##*pid*}" ] || [ ! -e $WORKDIR/$DIR/job.pid ] 148 | then 149 | exec &> >(tee -a "$WORKDIR/$DIR/job.log") 150 | echo "Beginning pid $$ on host $(hostname) at $(date)" 151 | trap "rm -rf $WORKDIR/$DIR/job.pid" EXIT 152 | echo $(hostname) $$ > $WORKDIR/$DIR/job.pid 153 | else 154 | echo "Already running $DIR at $(cat $WORKDIR/$DIR/job.pid)" 155 | exit 1 156 | fi 157 | 158 | if [ "$COLORDEPTH" -le 0 ] 159 | then 160 | (( COLORDEPTH = -COLORDEPTH )) 161 | CENTERED="" 162 | fi 163 | 164 | if [ -z "${CENTERED##*c*}" ] 165 | then 166 | MEAN="109.5388 118.6897 124.6901" 167 | fi 168 | 169 | # Convention: dir, weights, and proto all have the same name 170 | if [[ -z "${WEIGHTS}" && -z "${PROTO}" ]] 171 | then 172 | WEIGHTS="zoo/$DIR.caffemodel" 173 | PROTO="zoo/$DIR.prototxt" 174 | fi 175 | 176 | echo DIR = "${DIR}" 177 | echo LAYERS = "${LAYERS}" 178 | echo DATASET = "${DATASET}" 179 | echo COLORDEPTH = "${COLORDEPTH}" 180 | echo RESOLUTION = "${RESOLUTION}" 181 | echo WORKDIR = "${WORKDIR}" 182 | echo WEIGHTS = "${WEIGHTS}" 183 | echo PROTO = "${PROTO}" 184 | echo THRESHOLD = "${THRESHOLD}" 185 | echo PROBEBATCH = "${PROBEBATCH}" 186 | echo TALLYDEPTH = "${TALLYDEPTH}" 187 | echo TALLYBATCH = "${TALLYBATCH}" 188 | echo MEAN = "${MEAN}" 189 | echo FORCE = "${FORCE}" 190 | echo ENDAFTER = "${ENDAFTER}" 191 | 192 | # Set up rotation flag if rotation is selected 193 | ROTATION_FLAG="" 194 | if [ ! -z "${ROTATION_SEED}" ] 195 | then 196 | ROTATION_FLAG=" --rotation_seed ${ROTATION_SEED} 197 | --rotation_unpermute 1 198 | --rotation_power ${ROTATION_POWER} " 199 | fi 200 | 201 | # Step 1: do a forward pass over the network specified by the model files. 202 | # The output is e.g.,: "conv5.mmap", "conv5-info.txt" 203 | if [ -z "${FORCE##*probe*}" ] || \ 204 | ! ls $(printf " $WORKDIR/$DIR/%s.mmap" "${LAYERA[@]}") 2>/dev/null 205 | then 206 | 207 | echo 'Testing activations' 208 | python src/netprobe.py \ 209 | --directory $WORKDIR/$DIR \ 210 | --blobs $LAYERS \ 211 | --weights $WEIGHTS \ 212 | --definition $PROTO \ 213 | --batch_size $PROBEBATCH \ 214 | --mean $MEAN \ 215 | --colordepth $COLORDEPTH \ 216 | ${ROTATION_FLAG} \ 217 | --dataset $DATASET 218 | 219 | [[ $? -ne 0 ]] && exit $? 220 | echo netprobe > $WORKDIR/$DIR/job.done 221 | fi 222 | 223 | if [ -z "${ENDAFTER##*probe*}" ] 224 | then 225 | exit 0 226 | fi 227 | 228 | # Step 2: compute quantiles 229 | # saves results in conv5-quant-1000.mmap; conv5-rank-1001.mmap inverts this too. 230 | #echo 'Computing quantiles' 231 | if [ -z "${FORCE##*sort*}" ] || \ 232 | ! ls $(printf " $WORKDIR/$DIR/%s-quant-*.mmap" "${LAYERA[@]}") 2>/dev/null 233 | then 234 | 235 | echo 'Collecting quantiles of activations' 236 | python src/quantprobe.py \ 237 | --directory $WORKDIR/$DIR \ 238 | --blobs $LAYERS 239 | [[ $? -ne 0 ]] && exit $? 240 | 241 | echo quantprobe > $WORKDIR/$DIR/job.done 242 | fi 243 | 244 | if [ -z "${ENDAFTER##*quant*}" ] 245 | then 246 | exit 0 247 | fi 248 | 249 | # Step 3: the output here is the ~1G file called "conv5-tally-005.mmap". 250 | # It contains all the I/O/U etc counts for every label and unit (at the 0.5% 251 | # top activtation mask) for EVERY image. I.e., for image #n, we can read 252 | # out the number of pixels that light up both unit #u and label #l for 253 | # any combination of (n, u, l). That is a a very large sparse matrix, so 254 | # we encode that matrix specially. This is a 255 | # (#images, 2048, 3) dimensional file with entries such as this: 256 | # E.g., for image 412, we will have a list of up to 2048 triples. 257 | # Each triple has (count, #unit, #label) in it. 258 | if [ -z "${FORCE##*tally*}" ] || \ 259 | ! ls $(printf " $WORKDIR/$DIR/%s-tally-*.mmap" "${LAYERA[@]}") 2>/dev/null 260 | then 261 | 262 | echo 'Tallying counts' 263 | python src/labelprobe.py \ 264 | --directory $WORKDIR/$DIR \ 265 | --quantile $QUANTILE \ 266 | --tally_depth $TALLYDEPTH \ 267 | --blobs $LAYERS \ 268 | --parallel $PARALLEL \ 269 | --batch_size $TALLYBATCH \ 270 | --ahead 4 271 | 272 | [[ $? -ne 0 ]] && exit $? 273 | 274 | echo tallyprobe > $WORKDIR/$DIR/job.done 275 | fi 276 | 277 | if [ -z "${ENDAFTER##*tally*}" ] 278 | then 279 | exit 0 280 | fi 281 | 282 | # Step 4: compute conv5-imgmax.mmap / conv5-imgmax.mat 283 | # This contains the per-imgae maximum activation for every unit. 284 | if [ -z "${FORCE##*imgmax*}" ] || \ 285 | ! ls $(printf " $WORKDIR/$DIR/%s-imgmax.mmap" "${LAYERA[@]}") 2>/dev/null 286 | then 287 | 288 | echo 'Computing imgmax' 289 | python src/maxprobe.py \ 290 | --directory $WORKDIR/$DIR \ 291 | --blobs $LAYERS 292 | 293 | [[ $? -ne 0 ]] && exit $? 294 | echo maxprobe > $WORKDIR/$DIR/job.done 295 | fi 296 | 297 | if [ -z "${ENDAFTER##*imgmax*}" ] 298 | then 299 | exit 0 300 | fi 301 | 302 | 303 | # Step 5: we just run over the tally file to extract whatever score we 304 | # want to derive. That gets summarized in a [layer]-result.csv file. 305 | if [ -z "${FORCE##*result*}" ] || \ 306 | ! ls $(printf " $WORKDIR/$DIR/%s-result.csv" "${LAYERA[@]}") 307 | then 308 | 309 | echo 'Generating result.csv' 310 | python src/makeresult.py \ 311 | --directory $WORKDIR/$DIR \ 312 | --blobs $LAYERS 313 | 314 | [[ $? -ne 0 ]] && exit $? 315 | 316 | echo makeresult > $WORKDIR/$DIR/job.done 317 | fi 318 | 319 | if [ -z "${ENDAFTER##*result*}" ] 320 | then 321 | exit 0 322 | fi 323 | 324 | # Step 6: now generate the HTML visualization and images. 325 | if [ -z "${FORCE##*report*}" ] || \ 326 | ! ls $(printf " $WORKDIR/$DIR/html/%s.html" "${LAYERA[@]}") || \ 327 | ! ls $(printf " $WORKDIR/$DIR/html/image/%s-bargraph.svg" "${LAYERA[@]}") 328 | then 329 | 330 | echo 'Generating report' 331 | python src/report.py \ 332 | --directory $WORKDIR/$DIR \ 333 | --blobs $LAYERS \ 334 | --threshold ${THRESHOLD} 335 | 336 | [[ $? -ne 0 ]] && exit $? 337 | 338 | echo viewprobe > $WORKDIR/$DIR/job.done 339 | fi 340 | 341 | if [ -z "${ENDAFTER##*view*}" ] 342 | then 343 | exit 0 344 | fi 345 | 346 | # 347 | # Step 6: graph the results. 348 | if [ -z "${FORCE##*graph*}" ] || ! globexists $WORKDIR/$DIR/html/*-graph.png 349 | then 350 | 351 | # Compute text for labeling graphs. 352 | PERCENT=$(printf '%g%%' $(echo "scale=0; $THRESHOLD * 100" | bc)) 353 | 354 | echo 'Generating graph' 355 | python src/graphprobe.py \ 356 | --directories $WORKDIR/$DIR \ 357 | --blobs $LAYERS \ 358 | --labels $LAYERS \ 359 | --threshold $THRESHOLD \ 360 | --include_total true \ 361 | --title "Interpretability By Layer ($PERCENT IOU)" \ 362 | --out $WORKDIR/$DIR/html/layer-graph.png 363 | 364 | [[ $? -ne 0 ]] && exit $? 365 | echo graphprobe > $WORKDIR/$DIR/job.done 366 | fi 367 | 368 | if [ "$WORKDIR" != "$WORKDIR" ] 369 | then 370 | rm -rf $WORKDIR/$DIR/ 371 | [[ $? -ne 0 ]] && exit $? 372 | fi 373 | 374 | echo finished > $WORKDIR/$DIR/job.done 375 | 376 | if [ -z "${ENDAFTER##*graph*}" ] 377 | then 378 | exit 0 379 | fi 380 | 381 | -------------------------------------------------------------------------------- /script/rundissect_pytorch.sh: -------------------------------------------------------------------------------- 1 | # pre-defined setting 2 | 3 | #WORKDIR=probes 4 | WORKDIR="/scratch/shared/nfs1/ruthfong" 5 | #DIR=pytorch_alexnet_imagenet 6 | #DIR="pytorch_vgg19_imagenet" 7 | DIR="pytorch_alexnet_imagenet" 8 | ARCH='alexnet' # [alexnet,squeezenet1_1,resnet18,...]. It should work for all the models in https://github.com/pytorch/vision/tree/master/torchvision/models 9 | #LAYERS="features" 10 | #LAYERS="features.1 features.4 features.7 features.9 features.11" 11 | LAYERS="classifier.2 classifier.5 classifier.6" 12 | #ARCH="vgg19" 13 | #LAYERS="features.3 features.8 features.17 features.26 features.35" 14 | NUMCLASSES=1000 15 | DATASET=dataset/broden1_227 16 | #DATASET=dataset/broden1_224 17 | INPUTSIZE=227 18 | #INPUTSIZE=224 19 | GPU="0 1" 20 | WEIGHTS="None" 21 | 22 | # default setting 23 | THRESHOLD="0.04" 24 | TALLYDEPTH=2048 25 | #PARALLEL=4 26 | PARALLEL=0 27 | TALLYBATCH=16 28 | PROBEBATCH=64 29 | QUANTILE="0.005" 30 | COLORDEPTH="3" 31 | CENTERED="c" 32 | MEAN="0 0 0" 33 | FORCE="none" 34 | ENDAFTER="none" 35 | RESOLUTION="120" 36 | 37 | 38 | # Set up directory to work in, and lay down pid file etc. 39 | mkdir -p $WORKDIR/$DIR 40 | if [ -z "${FORCE##*pid*}" ] || [ ! -e $WORKDIR/$DIR/job.pid ] 41 | then 42 | exec &> >(tee -a "$WORKDIR/$DIR/job.log") 43 | #tee -a "$WORKDIR/$DIR/job.log" 44 | echo "Beginning pid $$ on host $(hostname) at $(date)" 45 | trap "rm -rf $WORKDIR/$DIR/job.pid $WORKDIR/$DIR/job.host" EXIT 46 | echo $$ > $WORKDIR/$DIR/job.pid 47 | echo $(hostname) > $WORKDIR/$DIR/job.host 48 | else 49 | echo "Already running $DIR under pid $(cat $WORKDIR/$DIR/job.pid)" 50 | exit 1 51 | fi 52 | 53 | if [ "$COLORDEPTH" -le 0 ] 54 | then 55 | (( COLORDEPTH = -COLORDEPTH )) 56 | CENTERED="" 57 | fi 58 | 59 | if [ -z "${CENTERED##*c*}" ] 60 | then 61 | MEAN="109.5388 118.6897 124.6901" 62 | fi 63 | 64 | # Get rid of slashes in layer names for directory purposes 65 | LAYERA=(${LAYERS//\//-}) 66 | 67 | # Step 1: do a forward pass over the network specified by the model files. 68 | # The output is e.g.,: "conv5.mmap", "conv5-info.txt" 69 | if [ -z "${FORCE##*probe*}" ] || \ 70 | ! ls $(printf " $WORKDIR/$DIR/%s.mmap" "${LAYERA[@]}") 2>/dev/null 71 | then 72 | 73 | echo 'Testing activations' 74 | python src/netprobe_pytorch.py \ 75 | --directory $WORKDIR/$DIR \ 76 | --blobs $LAYERS \ 77 | --mean $MEAN \ 78 | --definition $ARCH \ 79 | --num_classes $NUMCLASSES \ 80 | --dataset $DATASET \ 81 | --input_size $INPUTSIZE \ 82 | --gpu $GPU 83 | # --weights $WEIGHTS 84 | 85 | [[ $? -ne 0 ]] && exit $? 86 | echo netprobe > $WORKDIR/$DIR/job.done 87 | fi 88 | 89 | if [ -z "${ENDAFTER##*probe*}" ] 90 | then 91 | exit 0 92 | fi 93 | 94 | # Step 2: compute quantiles 95 | # saves results in conv5-quant-1000.mmap; conv5-rank-1001.mmap inverts this too. 96 | #echo 'Computing quantiles' 97 | if [ -z "${FORCE##*sort*}" ] || \ 98 | ! ls $(printf " $WORKDIR/$DIR/%s-quant-*.mmap" "${LAYERA[@]}") 2>/dev/null 99 | then 100 | 101 | echo 'Collecting quantiles of activations' 102 | python src/quantprobe.py \ 103 | --directory $WORKDIR/$DIR \ 104 | --blobs $LAYERS 105 | [[ $? -ne 0 ]] && exit $? 106 | 107 | echo quantprobe > $WORKDIR/$DIR/job.done 108 | fi 109 | 110 | if [ -z "${ENDAFTER##*quant*}" ] 111 | then 112 | exit 0 113 | fi 114 | 115 | # Step 3: the output here is the ~1G file called "conv5-tally-005.mmap". 116 | # It contains all the I/O/U etc counts for every label and unit (at the 0.5% 117 | # top activtation mask) for EVERY image. I.e., for image #n, we can read 118 | # out the number of pixels that light up both unit #u and label #l for 119 | # any combination of (n, u, l). That is a a very large sparse matrix, so 120 | # we encode that matrix specially. This is a 121 | # (#images, 2048, 3) dimensional file with entries such as this: 122 | # E.g., for image 412, we will have a list of up to 2048 triples. 123 | # Each triple has (count, #unit, #label) in it. 124 | if [ -z "${FORCE##*tally*}" ] || \ 125 | ! ls $(printf " $WORKDIR/$DIR/%s-tally-*.mmap" "${LAYERA[@]}") 2>/dev/null 126 | then 127 | 128 | echo 'Tallying counts' 129 | python src/labelprobe.py \ 130 | --directory $WORKDIR/$DIR \ 131 | --quantile $QUANTILE \ 132 | --tally_depth $TALLYDEPTH \ 133 | --blobs $LAYERS \ 134 | --parallel $PARALLEL \ 135 | --batch_size $TALLYBATCH \ 136 | --ahead 4 137 | 138 | [[ $? -ne 0 ]] && exit $? 139 | 140 | echo tallyprobe > $WORKDIR/$DIR/job.done 141 | fi 142 | 143 | if [ -z "${ENDAFTER##*tally*}" ] 144 | then 145 | exit 0 146 | fi 147 | 148 | # Step 4: compute conv5-imgmax.mmap / conv5-imgmax.mat 149 | # This contains the per-imgae maximum activation for every unit. 150 | if [ -z "${FORCE##*imgmax*}" ] || \ 151 | ! ls $(printf " $WORKDIR/$DIR/%s-imgmax.mmap" "${LAYERA[@]}") 2>/dev/null 152 | then 153 | 154 | echo 'Computing imgmax' 155 | python src/maxprobe.py \ 156 | --directory $WORKDIR/$DIR \ 157 | --blobs $LAYERS 158 | 159 | [[ $? -ne 0 ]] && exit $? 160 | echo maxprobe > $WORKDIR/$DIR/job.done 161 | fi 162 | 163 | if [ -z "${ENDAFTER##*imgmax*}" ] 164 | then 165 | exit 0 166 | fi 167 | 168 | 169 | # Step 5: we just run over the tally file to extract whatever score we 170 | # want to derive. That's pretty easy, so we also generate HTML at the 171 | # same time. 172 | if [ -z "${FORCE##*view*}" ] || \ 173 | ! ls $(printf " $WORKDIR/$DIR/html/%s.html" "${LAYERA[@]}") || \ 174 | ! ls $(printf " $WORKDIR/$DIR/%s-result.csv" "${LAYERA[@]}") 175 | then 176 | 177 | echo 'Generating views' 178 | echo $LAYERS 179 | python src/viewprobe.py \ 180 | --directory $WORKDIR/$DIR \ 181 | --format csv,html,quantmat \ 182 | --imscale 72 \ 183 | --imsize $RESOLUTION \ 184 | --blobs $LAYERS 185 | 186 | [[ $? -ne 0 ]] && exit $? 187 | 188 | echo viewprobe > $WORKDIR/$DIR/job.done 189 | fi 190 | 191 | if [ -z "${ENDAFTER##*view*}" ] 192 | then 193 | exit 0 194 | fi 195 | 196 | # 197 | # Step 6: graph the results. 198 | if [ -z "${FORCE##*graph*}" ] || ! globexists $WORKDIR/$DIR/html/*-graph.png 199 | then 200 | 201 | # Compute text for labeling graphs. 202 | PERCENT=$(printf '%g%%' $(echo "scale=0; $THRESHOLD * 100" | bc)) 203 | 204 | echo 'Generating graph' 205 | python src/graphprobe.py \ 206 | --directories $WORKDIR/$DIR \ 207 | --blobs $LAYERS \ 208 | --labels $LAYERS \ 209 | --threshold $THRESHOLD \ 210 | --include_total true \ 211 | --title "Interpretability By Layer ($PERCENT IOU)" \ 212 | --out $WORKDIR/$DIR/html/layer-graph.png 213 | 214 | [[ $? -ne 0 ]] && exit $? 215 | echo graphprobe > $WORKDIR/$DIR/job.done 216 | fi 217 | 218 | if [ "$WORKDIR" != "$WORKDIR" ] 219 | then 220 | rm -rf $WORKDIR/$DIR/ 221 | [[ $? -ne 0 ]] && exit $? 222 | fi 223 | 224 | echo finished > $WORKDIR/$DIR/job.done 225 | 226 | if [ -z "${ENDAFTER##*graph*}" ] 227 | then 228 | exit 0 229 | fi 230 | 231 | -------------------------------------------------------------------------------- /script/rundissect_pytorch_external.sh: -------------------------------------------------------------------------------- 1 | # pre-defined setting 2 | WORKDIR=probes 3 | DIR=pytorch_alexnet_imagenet 4 | ARCH='alexnet' # [alexnet,squeezenet1_1,resnet18,...]. It should work for all the models in https://github.com/pytorch/vision/tree/master/torchvision/models 5 | LAYERS="features" 6 | DATASET=dataset/broden1_224 7 | WEIGHTS="none" 8 | NUMCLASSES=1000 9 | 10 | # default setting 11 | THRESHOLD=0.04 12 | TALLYDEPTH=2048 13 | PARALLEL=4 14 | TALLYBATCH=16 15 | PROBEBATCH=64 16 | QUANTILE="0.005" 17 | COLORDEPTH="3" 18 | CENTERED="c" 19 | MEAN="0 0 0" 20 | FORCE="none" 21 | ENDAFTER="none" 22 | RESOLUTION=100 23 | 24 | # Download the external model and reset the parameters 25 | DIR="pytorch_resnet18_places365" 26 | ARCH="resnet18" 27 | NUMCLASSES=365 28 | LAYERS="layer4" 29 | WEIGHTS=zoo/resnet18_places365.pth.tar 30 | if [ ! -e $WEIGHTS ] 31 | then 32 | echo "Download the resnet18 trained on Places365" 33 | wget http://places2.csail.mit.edu/models_places365/resnet18_places365.pth.tar -O $WEIGHTS 34 | fi 35 | 36 | # Set up directory to work in, and lay down pid file etc. 37 | mkdir -p $WORKDIR/$DIR 38 | if [ -z "${FORCE##*pid*}" ] || [ ! -e $WORKDIR/$DIR/job.pid ] 39 | then 40 | exec &> >(tee -a "$WORKDIR/$DIR/job.log") 41 | echo "Beginning pid $$ on host $(hostname) at $(date)" 42 | trap "rm -rf $WORKDIR/$DIR/job.pid $WORKDIR/$DIR/job.host" EXIT 43 | echo $$ > $WORKDIR/$DIR/job.pid 44 | echo $(hostname) > $WORKDIR/$DIR/job.host 45 | else 46 | echo "Already running $DIR under pid $(cat $WORKDIR/$DIR/job.pid)" 47 | exit 1 48 | fi 49 | 50 | if [ "$COLORDEPTH" -le 0 ] 51 | then 52 | (( COLORDEPTH = -COLORDEPTH )) 53 | CENTERED="" 54 | fi 55 | 56 | if [ -z "${CENTERED##*c*}" ] 57 | then 58 | MEAN="109.5388 118.6897 124.6901" 59 | fi 60 | 61 | # Get rid of slashes in layer names for directory purposes 62 | LAYERA=(${LAYERS//\//-}) 63 | 64 | # Step 1: do a forward pass over the network specified by the model files. 65 | # The output is e.g.,: "conv5.mmap", "conv5-info.txt" 66 | if [ -z "${FORCE##*probe*}" ] || \ 67 | ! ls $(printf " $WORKDIR/$DIR/%s.mmap" "${LAYERA[@]}") 2>/dev/null 68 | then 69 | 70 | echo 'Testing activations' 71 | python src/netprobe_pytorch.py \ 72 | --directory $WORKDIR/$DIR \ 73 | --blobs $LAYERS \ 74 | --mean $MEAN \ 75 | --definition $ARCH \ 76 | --weights $WEIGHTS \ 77 | --num_classes $NUMCLASSES \ 78 | --dataset $DATASET 79 | 80 | 81 | [[ $? -ne 0 ]] && exit $? 82 | echo netprobe > $WORKDIR/$DIR/job.done 83 | fi 84 | 85 | if [ -z "${ENDAFTER##*probe*}" ] 86 | then 87 | exit 0 88 | fi 89 | 90 | # Step 2: compute quantiles 91 | # saves results in conv5-quant-1000.mmap; conv5-rank-1001.mmap inverts this too. 92 | #echo 'Computing quantiles' 93 | if [ -z "${FORCE##*sort*}" ] || \ 94 | ! ls $(printf " $WORKDIR/$DIR/%s-quant-*.mmap" "${LAYERA[@]}") 2>/dev/null 95 | then 96 | 97 | echo 'Collecting quantiles of activations' 98 | python src/quantprobe.py \ 99 | --directory $WORKDIR/$DIR \ 100 | --blobs $LAYERS 101 | [[ $? -ne 0 ]] && exit $? 102 | 103 | echo quantprobe > $WORKDIR/$DIR/job.done 104 | fi 105 | 106 | if [ -z "${ENDAFTER##*quant*}" ] 107 | then 108 | exit 0 109 | fi 110 | 111 | # Step 3: the output here is the ~1G file called "conv5-tally-005.mmap". 112 | # It contains all the I/O/U etc counts for every label and unit (at the 0.5% 113 | # top activtation mask) for EVERY image. I.e., for image #n, we can read 114 | # out the number of pixels that light up both unit #u and label #l for 115 | # any combination of (n, u, l). That is a a very large sparse matrix, so 116 | # we encode that matrix specially. This is a 117 | # (#images, 2048, 3) dimensional file with entries such as this: 118 | # E.g., for image 412, we will have a list of up to 2048 triples. 119 | # Each triple has (count, #unit, #label) in it. 120 | if [ -z "${FORCE##*tally*}" ] || \ 121 | ! ls $(printf " $WORKDIR/$DIR/%s-tally-*.mmap" "${LAYERA[@]}") 2>/dev/null 122 | then 123 | 124 | echo 'Tallying counts' 125 | python src/labelprobe.py \ 126 | --directory $WORKDIR/$DIR \ 127 | --quantile $QUANTILE \ 128 | --tally_depth $TALLYDEPTH \ 129 | --blobs $LAYERS \ 130 | --parallel $PARALLEL \ 131 | --batch_size $TALLYBATCH \ 132 | --ahead 4 133 | 134 | [[ $? -ne 0 ]] && exit $? 135 | 136 | echo tallyprobe > $WORKDIR/$DIR/job.done 137 | fi 138 | 139 | if [ -z "${ENDAFTER##*tally*}" ] 140 | then 141 | exit 0 142 | fi 143 | 144 | # Step 4: compute conv5-imgmax.mmap / conv5-imgmax.mat 145 | # This contains the per-imgae maximum activation for every unit. 146 | if [ -z "${FORCE##*imgmax*}" ] || \ 147 | ! ls $(printf " $WORKDIR/$DIR/%s-imgmax.mmap" "${LAYERA[@]}") 2>/dev/null 148 | then 149 | 150 | echo 'Computing imgmax' 151 | python src/maxprobe.py \ 152 | --directory $WORKDIR/$DIR \ 153 | --blobs $LAYERS 154 | 155 | [[ $? -ne 0 ]] && exit $? 156 | echo maxprobe > $WORKDIR/$DIR/job.done 157 | fi 158 | 159 | if [ -z "${ENDAFTER##*imgmax*}" ] 160 | then 161 | exit 0 162 | fi 163 | 164 | 165 | # Step 5: we just run over the tally file to extract whatever score we 166 | # want to derive. That's pretty easy, so we also generate HTML at the 167 | # same time. 168 | if [ -z "${FORCE##*view*}" ] || \ 169 | ! ls $(printf " $WORKDIR/$DIR/html/%s.html" "${LAYERA[@]}") || \ 170 | ! ls $(printf " $WORKDIR/$DIR/%s-result.csv" "${LAYERA[@]}") 171 | then 172 | 173 | echo 'Generating views' 174 | python src/viewprobe.py \ 175 | --directory $WORKDIR/$DIR \ 176 | --format csv,html,quantmat \ 177 | --imscale 72 \ 178 | --imsize $RESOLUTION \ 179 | --blobs $LAYERS 180 | 181 | [[ $? -ne 0 ]] && exit $? 182 | 183 | echo viewprobe > $WORKDIR/$DIR/job.done 184 | fi 185 | 186 | if [ -z "${ENDAFTER##*view*}" ] 187 | then 188 | exit 0 189 | fi 190 | 191 | # 192 | # Step 6: graph the results. 193 | if [ -z "${FORCE##*graph*}" ] || ! globexists $WORKDIR/$DIR/html/*-graph.png 194 | then 195 | 196 | # Compute text for labeling graphs. 197 | PERCENT=$(printf '%g%%' $(echo "scale=0; $THRESHOLD * 100" | bc)) 198 | 199 | echo 'Generating graph' 200 | python src/graphprobe.py \ 201 | --directories $WORKDIR/$DIR \ 202 | --blobs $LAYERS \ 203 | --labels $LAYERS \ 204 | --threshold $THRESHOLD \ 205 | --include_total true \ 206 | --title "Interpretability By Layer ($PERCENT IOU)" \ 207 | --out $WORKDIR/$DIR/html/layer-graph.png 208 | 209 | [[ $? -ne 0 ]] && exit $? 210 | echo graphprobe > $WORKDIR/$DIR/job.done 211 | fi 212 | 213 | if [ "$WORKDIR" != "$WORKDIR" ] 214 | then 215 | rm -rf $WORKDIR/$DIR/ 216 | [[ $? -ne 0 ]] && exit $? 217 | fi 218 | 219 | echo finished > $WORKDIR/$DIR/job.done 220 | 221 | if [ -z "${ENDAFTER##*graph*}" ] 222 | then 223 | exit 0 224 | fi 225 | 226 | -------------------------------------------------------------------------------- /src/adeseg.py: -------------------------------------------------------------------------------- 1 | import colorname 2 | import glob 3 | import os 4 | import re 5 | import numpy 6 | from loadseg import AbstractSegmentation 7 | from scipy.io import loadmat 8 | from scipy.misc import imread 9 | from collections import namedtuple 10 | 11 | class AdeSegmentation(AbstractSegmentation): 12 | def __init__(self, directory=None, version=None): 13 | # Default to value of ADE20_ROOT env variable 14 | if directory is None: 15 | directory = os.environ['ADE20K_ROOT'] 16 | directory = os.path.expanduser(directory) 17 | # Default to the latest version present in the directory 18 | if version is None: 19 | contents = os.listdir(directory) 20 | if not list(c for c in contents if re.match('^index.*mat$', c)): 21 | version = sorted(c for c in contents if os.path.isdir( 22 | os.path.join(directory, c)))[-1] 23 | else: 24 | version = '' 25 | self.root = directory 26 | self.version = version 27 | mat = loadmat(self.expand(self.version, 'index*.mat'), squeeze_me=True) 28 | index = mat['index'] 29 | Ade20kIndex = namedtuple('Ade20kIndex', index.dtype.names) 30 | self.index = Ade20kIndex( 31 | **{name: index[name][()] for name in index.dtype.names}) 32 | self.scenes = ['-'] + [ 33 | norm_name(s) for s in sorted(set(self.index.scene))] 34 | self.scene_map = dict((s, i) for i, s in enumerate(self.scenes)) 35 | # Zero out special ~ scene names, which mean unlabeled. 36 | for k in self.scene_map: 37 | if k.startswith('~'): 38 | self.scene_map[k] = 0 39 | self.raw_mat = mat 40 | 41 | def all_names(self, category, j): 42 | if j == 0: 43 | return [] 44 | if category == 'color': 45 | return [colorname.color_names[j - 1] + '-c'] 46 | if category == 'scene': 47 | return [self.scenes[j] + '-s'] 48 | result = self.index.objectnames[j - 1] 49 | return re.split(',\s*', result) 50 | 51 | def size(self): 52 | '''Returns the number of images in this dataset.''' 53 | return len(self.index.filename) 54 | 55 | def filename(self, n): 56 | '''Returns the filename for the nth dataset image.''' 57 | filename = self.index.filename[n] 58 | folder = self.index.folder[n] 59 | return self.expand(folder, filename) 60 | 61 | def metadata(self, i): 62 | '''Returns an object that can be used to create all segmentations.''' 63 | return dict( 64 | filename=self.filename(i), 65 | seg_filename=self.seg_filename(i), 66 | part_filenames=self.part_filenames(i), 67 | scene=self.scene_map[norm_name(self.index.scene[i])] 68 | ) 69 | 70 | @classmethod 71 | def resolve_segmentation(cls, m, categories=None): 72 | result = {} 73 | if wants('scene', categories): 74 | result['scene'] = m['scene'] 75 | if wants('part', categories): 76 | result['part'] = load_parts(m) 77 | if wants('object', categories): 78 | result['object'] = load_segmentation(m) 79 | if wants('color', categories): 80 | result['color'] = colorname.label_major_colors(load_image(m)) + 1 81 | arrs = [a for a in result.values() if numpy.shape(a) >= 2] 82 | shape = arrs[0].shape[-2:] if arrs else (1, 1) 83 | return result, shape 84 | 85 | ### End of contract for AbstractSegmentation 86 | 87 | def seg_filename(self, n): 88 | '''Returns the segmentation filename for the nth dataset image.''' 89 | return re.sub(r'\.jpg$', '_seg.png', self.filename(n)) 90 | 91 | def part_filenames(self, n): 92 | '''Returns all the subpart images for the nth dataset image.''' 93 | filename = self.filename(n) 94 | level = 1 95 | result = [] 96 | while True: 97 | probe = re.sub(r'\.jpg$', '_parts_%d.png' % level, filename) 98 | if not os.path.isfile(probe): 99 | break 100 | result.append(probe) 101 | level += 1 102 | return result 103 | 104 | def expand(self, *path): 105 | '''Expands a filename and directories with the ADE dataset''' 106 | result = os.path.join(self.root, *path) 107 | if '*' in result or '?' in result: 108 | globbed = glob.glob(result) 109 | if len(globbed): 110 | return globbed[0] 111 | return result 112 | 113 | def norm_name(s): 114 | return s.replace(' - ', '-').replace('/', '-') 115 | 116 | def load_image(m): 117 | '''Returns the nth dataset image as a numpy array.''' 118 | return imread(m['filename']) 119 | 120 | def load_segmentation(m): 121 | '''Returns the nth dataset segmentation as a numpy array, 122 | where each entry at a pixel is an object class value. 123 | ''' 124 | data = imread(m['seg_filename']) 125 | return decodeClassMask(data) 126 | 127 | def load_parts(m): 128 | '''Returns an list of part segmentations for the nth dataset item, 129 | with one array for each level available. 130 | ''' 131 | result = [] 132 | for fn in m['part_filenames']: 133 | data = imread(fn) 134 | result.append(decodeClassMask(data)) 135 | if not result: 136 | return [] 137 | return numpy.concatenate(tuple(m[numpy.newaxis] for m in result)) 138 | 139 | def decodeClassMask(im): 140 | '''Decodes pixel-level object/part class and instance data from 141 | the given image, previously encoded into RGB channels.''' 142 | # Classes are a combination of RG channels (dividing R by 10) 143 | return (im[:,:,0] // 10) * 256 + im[:,:,1] 144 | 145 | def wants(what, option): 146 | if option is None: 147 | return True 148 | return what in option 149 | -------------------------------------------------------------------------------- /src/bargraph.py: -------------------------------------------------------------------------------- 1 | import re 2 | import expdir 3 | import itertools 4 | import operator 5 | from xml.etree import ElementTree as et 6 | 7 | default_category_order = [ 8 | 'object', 9 | 'scene', 10 | 'part', 11 | 'material', 12 | 'texture', 13 | 'color' 14 | ] 15 | 16 | palette = [ 17 | ('#4B4CBF', '#B6B6F2'), 18 | ('#55B05B', '#B6F2BA'), 19 | ('#50BDAC', '#A5E5DB'), 20 | ('#D4CF24', '#F2F1B6'), 21 | ('#F0883B', '#F2CFB6'), 22 | ('#D92E2B', '#F2B6B6') 23 | ] 24 | 25 | def most_common(L): 26 | # get an iterable of (item, iterable) pairs 27 | SL = sorted((x, i) for i, x in enumerate(L)) 28 | groups = itertools.groupby(SL, key=operator.itemgetter(0)) 29 | # auxiliary function to get "quality" for an item 30 | def _auxfun(g): 31 | item, iterable = g 32 | count = 0 33 | min_index = len(L) 34 | for _, where in iterable: 35 | count += 1 36 | min_index = min(min_index, where) 37 | return count, -min_index 38 | # pick the highest-count/earliest item 39 | return max(groups, key=_auxfun)[0] 40 | 41 | def bar_graph_svg(ed, blob, barheight, barwidth, 42 | order=None, 43 | show_labels=True, 44 | threshold=0.04, 45 | rendered_order=None, 46 | save=None): 47 | records = ed.load_csv(blob=blob, part='result') 48 | # ['unit', 'category', 'label', 'score'] 49 | # Examine each label 50 | label_cats = {} 51 | label_score = {} 52 | for record in records: 53 | if float(record['score']) < threshold: 54 | continue 55 | label = record['label'] 56 | if label not in label_cats: 57 | label_cats[label] = [] 58 | label_cats[label].append(record['category']) 59 | if (label not in label_score 60 | or label_score[label] < float(record['score'])): 61 | label_score[label] = float(record['score']) 62 | # Count each label, and note its cateogry 63 | label_counts = {} 64 | for label, cats in label_cats.items(): 65 | label_counts[label] = len(cats) 66 | label_cats[label] = most_common(cats) 67 | # Sort labels by frequency and max score 68 | sorted_labels = sorted(label_counts.keys(), 69 | key=lambda x: (-label_counts[x], -label_score[x])) 70 | category_order = order 71 | if not category_order: 72 | # Default category order: broden order plus any missing categories 73 | category_order = list(default_category_order) 74 | for label in sorted_labels: 75 | if label_cats[label] not in category_order: 76 | category_order.append(label_cats[label]) 77 | # Now make a plot 78 | heights = [] 79 | colors = [] 80 | categories = [] 81 | labels = [] 82 | for cat in category_order: 83 | filtered = [label for label in sorted_labels 84 | if label_cats[label] == cat] 85 | labels.extend(filtered) 86 | heights.extend([label_counts[label] for label in filtered]) 87 | categories.append((cat, len(filtered))) 88 | # Sort records in histogram order and output them if requested 89 | if rendered_order is not None: 90 | rendered_order.extend(sorted(records, key=lambda record: ( 91 | # Items below score threshold are sorted last, by score 92 | (len(category_order), 0, 0, -float(record['score'])) 93 | if float(record['score']) < threshold else 94 | # Others are sorted by category, label count/score, and score 95 | (category_order.index(label_cats[record['label']]), 96 | -label_counts[record['label']], -label_score[record['label']], 97 | -float(record['score']))))) 98 | filename = None 99 | if save: 100 | if save == True: 101 | filename = ed.filename('bargraph.svg', blob=blob, directory='html') 102 | else: 103 | filename = save 104 | ed.ensure_dir('html') 105 | return make_svg_bargraph(labels, heights, categories, 106 | barheight, barwidth, show_labels, filename) 107 | 108 | def make_svg_bargraph(labels, heights, categories, 109 | barheight=100, barwidth=12, show_labels=True, filename=None): 110 | unitheight = float(barheight) / max(heights) 111 | textheight = barheight if show_labels else 0 112 | labelsize = float(barwidth) 113 | gap = float(barwidth) / 4 114 | textsize = barwidth + gap 115 | rollup = max(heights) 116 | textmargin = float(labelsize) * 2 / 3 117 | leftmargin = 32 118 | rightmargin = 8 119 | svgwidth = len(heights) * (barwidth + gap) + 2 * leftmargin + rightmargin 120 | svgheight = barheight + textheight 121 | 122 | # create an SVG XML element 123 | svg = et.Element('svg', width=str(svgwidth), height=str(svgheight), 124 | version='1.1', xmlns='http://www.w3.org/2000/svg') 125 | 126 | # Draw the bar graph 127 | basey = svgheight - textheight 128 | x = leftmargin 129 | # Add units scale on left 130 | for h in [1, (max(heights) + 1) // 2, max(heights)]: 131 | et.SubElement(svg, 'text', x='0', y='0', 132 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:end;'+ 133 | 'alignment-baseline:hanging;' + 134 | 'transform:translate(%dpx, %dpx);') % 135 | (textsize, x - gap, basey - h * unitheight)).text = str(h) 136 | et.SubElement(svg, 'text', x='0', y='0', 137 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:middle;'+ 138 | 'transform:translate(%dpx, %dpx) rotate(-90deg)') % 139 | (textsize, x - gap - 1.2 * textsize, basey - h * unitheight / 2) 140 | ).text = 'units' 141 | # Draw big category background rectangles 142 | for catindex, (cat, catcount) in enumerate(categories): 143 | if not catcount: 144 | continue 145 | et.SubElement(svg, 'rect', x=str(x), y=str(basey - rollup * unitheight), 146 | width=(str((barwidth + gap) * catcount - gap)), 147 | height = str(rollup*unitheight), 148 | fill=palette[catindex % len(palette)][1]) 149 | x += (barwidth + gap) * catcount 150 | # Draw small bars as well as 45degree text labels 151 | x = leftmargin 152 | catindex = -1 153 | catcount = 0 154 | for label, height in zip(labels, heights): 155 | while not catcount and catindex <= len(categories): 156 | catindex += 1 157 | catcount = categories[catindex][1] 158 | color = palette[catindex % len(palette)][0] 159 | et.SubElement(svg, 'rect', x=str(x), y=str(basey-(height * unitheight)), 160 | width=str(barwidth), height=str(height * unitheight), 161 | fill=color) 162 | x += barwidth 163 | if show_labels: 164 | et.SubElement(svg, 'text', x='0', y='0', 165 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:end;'+ 166 | 'transform:translate(%dpx, %dpx) rotate(-45deg);') % 167 | (labelsize, x, basey + textmargin)).text = fix(label) 168 | x += gap 169 | catcount -= 1 170 | # Text lables for each category 171 | x = leftmargin 172 | for cat, catcount in categories: 173 | if not catcount: 174 | continue 175 | et.SubElement(svg, 'text', x='0', y='0', 176 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:end;'+ 177 | 'transform:translate(%dpx, %dpx) rotate(-90deg);') % 178 | (textsize, x + (barwidth + gap) * catcount - gap, 179 | basey - rollup * unitheight + gap)).text = '%d %s' % ( 180 | catcount, fix(cat + ('s' if catcount != 1 else ''))) 181 | x += (barwidth + gap) * catcount 182 | # Output - this is the bare svg. 183 | result = et.tostring(svg) 184 | if filename: 185 | f = open(filename, 'w') 186 | # When writing to a file a special header is needed. 187 | f.write('\n') 188 | f.write('\n') 190 | f.write(result) 191 | f.close() 192 | return result 193 | 194 | replacements = [(re.compile(r[0]), r[1]) for r in [ 195 | (r'-[sc]$', ''), 196 | (r'_', ' '), 197 | ]] 198 | 199 | def fix(s): 200 | for pattern, subst in replacements: 201 | s = re.sub(pattern, subst, s) 202 | return s 203 | 204 | # Usage: 205 | # bargraph.py --directory dir --blob layer --barheight h --barwidth w 206 | if __name__ == '__main__': 207 | import argparse 208 | import sys 209 | import traceback 210 | 211 | parser = argparse.ArgumentParser( 212 | description='Plot graph of unique interpretations.') 213 | parser.add_argument( 214 | '--directory', 215 | help='directory to graph') 216 | parser.add_argument( 217 | '--blob', 218 | nargs='*', 219 | help='blob to graph') 220 | parser.add_argument( 221 | '--barheight', 222 | type=int, default=100, 223 | help='graph big color bar height') 224 | parser.add_argument( 225 | '--barwidth', 226 | type=int, default=12, 227 | help='graph little color bar width') 228 | args = parser.parse_args() 229 | 230 | bar_graph_svg(args.directory, args.blob, args.barheight, args.barwidth, 231 | save=True) 232 | -------------------------------------------------------------------------------- /src/brian.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "data": { 12 | "text/plain": [ 13 | "2" 14 | ] 15 | }, 16 | "execution_count": 1, 17 | "metadata": {}, 18 | "output_type": "execute_result" 19 | } 20 | ], 21 | "source": [ 22 | "1+1" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 2, 28 | "metadata": { 29 | "collapsed": false 30 | }, 31 | "outputs": [ 32 | { 33 | "data": { 34 | "text/plain": [ 35 | "4" 36 | ] 37 | }, 38 | "execution_count": 2, 39 | "metadata": {}, 40 | "output_type": "execute_result" 41 | } 42 | ], 43 | "source": [ 44 | "2+2" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 3, 50 | "metadata": { 51 | "collapsed": false 52 | }, 53 | "outputs": [ 54 | { 55 | "name": "stdout", 56 | "output_type": "stream", 57 | "text": [ 58 | "Brian is an awesome husband\n" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "print \"Brian is an awesome husband\"" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": { 70 | "collapsed": true 71 | }, 72 | "outputs": [], 73 | "source": [] 74 | } 75 | ], 76 | "metadata": { 77 | "kernelspec": { 78 | "display_name": "Python 2", 79 | "language": "python", 80 | "name": "python2" 81 | }, 82 | "language_info": { 83 | "codemirror_mode": { 84 | "name": "ipython", 85 | "version": 2 86 | }, 87 | "file_extension": ".py", 88 | "mimetype": "text/x-python", 89 | "name": "python", 90 | "nbconvert_exporter": "python", 91 | "pygments_lexer": "ipython2", 92 | "version": "2.7.13" 93 | } 94 | }, 95 | "nbformat": 4, 96 | "nbformat_minor": 2 97 | } 98 | -------------------------------------------------------------------------------- /src/colorname.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Discretizes colors to eleven color names as described in 3 | 4 | Learning Color Names for Real-World Applications 5 | J. van de Weijer, C. Schmid, J. Verbeek, D. Larlus. 6 | IEEE Transactions in Image Processing 2009. 7 | 8 | The work above uses color names observed in the wild (on Google Images) 9 | to derive probabilities for eleven color names for every point in RGB space. 10 | 11 | The function provided here simply assigns each pixel according to its 12 | maximum probability label based on probabilities from van de Weijer. 13 | ''' 14 | 15 | from scipy.io import loadmat 16 | import numpy 17 | 18 | # The 11 fundamental named colors in English. 19 | color_names = [ 20 | 'black', 21 | 'blue', 22 | 'brown', 23 | 'grey', 24 | 'green', 25 | 'orange', 26 | 'pink', 27 | 'purple', 28 | 'red', 29 | 'white', 30 | 'yellow'] 31 | 32 | color_values = numpy.array([ 33 | [0, 0, 0], # black 34 | [64, 64, 255], # blue 35 | [127, 101, 63], # brown 36 | [127, 127, 127], # grey 37 | [0, 192, 0], # green 38 | [255, 203, 0], # orange 39 | [192, 64, 192], # pink 40 | [255, 64, 255], # purple 41 | [255, 0, 0], # red 42 | [255, 255, 255], # white 43 | [255, 255, 0]], # yellow 44 | dtype='uint8') 45 | 46 | # Flat weighted mean of RGB values, generated from van de Weijer's data via: 47 | # w2c = numpy.swapaxes(scipy.io.loadmat( 48 | # 'w2c.mat')['w2c'].reshape(32,32,32,11), 0, 2) 49 | # r = numpy.tile(numpy.arange(0,256,8), (32,32,1)) 50 | # rr = numpy.stack([numpy.swapaxes(r, 2, n) 51 | # for n in range(3)])[:,:,:,:,None] 52 | # ((w2c[None] * rr).sum(axis=(1,2,3)) / 53 | # w2c.sum(axis=(0,1,2))[None]).transpose().astype('uint8') 54 | # What we actually need is weightings by real 55 | # color ocurrences. 56 | mean_color_values = numpy.array( 57 | [[ 97, 107, 113], # black 58 | [ 64, 119, 180], # blue 59 | [140, 116, 88], # brown 60 | [114, 125, 125], # grey 61 | [ 84, 186, 90], # green 62 | [182, 106, 65], # orange 63 | [188, 90, 136], # pink 64 | [138, 65, 172], # purple 65 | [155, 84, 77], # red 66 | [130, 166, 170], 67 | [162, 173, 75]], 68 | dtype='uint8') 69 | 70 | # Load 71 | _cached_w2color = None 72 | def get_color_map(): 73 | global _cached_w2color 74 | if _cached_w2color is None: 75 | # w2c = loadmat('w2c.mat')['w2c'] 76 | # _cached_w2color = numpy.swapaxes( 77 | # w2c.argmax(axis=1).reshape(32, 32, 32), 0, 2) 78 | # numpy.save('w2color', _cached_w2color, False) 79 | # instead just load this from the saved .npy file 80 | from inspect import getsourcefile 81 | import os.path 82 | _cached_w2color = numpy.load(os.path.join( 83 | os.path.dirname(getsourcefile(lambda:0)), 'w2color.npy')) 84 | return _cached_w2color 85 | 86 | def label_colors(im): 87 | try: 88 | if len(im.shape) == 2: 89 | im = numpy.repeat(im[:,:,numpy.newaxis], 3, axis=2) 90 | return get_color_map()[tuple(im[:,:,c] // 8 for c in range(3))] 91 | except: 92 | print im.shape 93 | print (im // 8).max() 94 | raise 95 | 96 | def label_major_colors(im, threshold=0): 97 | raw = label_colors(im) 98 | if not threshold: 99 | return raw 100 | pixel_threshold = int(im.shape[0] * im.shape[1] * threshold) 101 | counts = numpy.bincount(raw.ravel(), minlength=11) 102 | mask = (counts > pixel_threshold) 103 | return (raw + 1).astype('int') * mask[raw] - 1 104 | 105 | def posterize(im, use_mean_colors=False): 106 | if use_mean_colors: 107 | return mean_color_values[label_colors(im)] 108 | else: 109 | return color_values[label_colors(im)] 110 | 111 | if __name__ == '__main__': 112 | print get_color_map() 113 | -------------------------------------------------------------------------------- /src/computealpha.py: -------------------------------------------------------------------------------- 1 | import loadseg 2 | import expdir 3 | from indexdata import load_image_to_label, has_image_to_label, create_image_to_label 4 | 5 | import numpy as np 6 | 7 | def compute_alpha(directory): 8 | ed = expdir.ExperimentDirectory(directory) 9 | info = ed.load_info() 10 | ds = loadseg.SegmentationData(info.dataset) 11 | L = ds.label_size() 12 | if not has_image_to_label(directory): 13 | create_image_to_label(directory) 14 | image_to_label = load_image_to_label(directory) 15 | 16 | label_names = np.array([ds.label[i]['name'] for i in range(L)]) 17 | 18 | alphas = np.zeros((L,)) 19 | 20 | for label_i in range(1,L): 21 | label_categories = ds.label[label_i]['category'].keys() 22 | label_idx = np.where(image_to_label[:, label_i])[0] 23 | train_loader = loadseg.SegmentationPrefetcher(ds, categories=label_categories, split='train', indexes=label_idx, 24 | once=True, batch_size=64, ahead=4, thread=True) 25 | train_idx = np.array(train_loader.indexes) 26 | #sw = 0 27 | #sh = 0 28 | perc_label = [] 29 | #train_label_categories = [] 30 | for batch in train_loader.batches(): 31 | for rec in batch: 32 | sw, sh = [rec[k] for k in ['sw', 'sh']] 33 | #sw_r, sh_r = [rec[k] for k in ['sw', 'sh']] 34 | #if sw == 0 and sh == 0: 35 | # sw = sw_r 36 | # sh = sh_r 37 | #else: 38 | # assert(sw == sw_r and sh == sh_r) 39 | for cat in label_categories: 40 | if rec[cat] != []: 41 | #train_label_categories.append(cat) 42 | if type(rec[cat]) is np.ndarray: 43 | perc_label.append(np.sum(rec[cat] == label_i) / float(sw * sh)) 44 | else: 45 | perc_label.append(1.) 46 | break 47 | assert(len(perc_label) == len(train_idx)) 48 | 49 | alphas[label_i] = float(1. - np.mean(perc_label)) 50 | print label_i, label_names[label_i], alphas[label_i] 51 | train_loader.close() 52 | 53 | alphas_mmap = ed.open_mmap(part='train_alphas', mode='w+', dtype='float32', shape=alphas.shape) 54 | alphas_mmap[:] = alphas[:] 55 | ed.finish_mmap(alphas_mmap) 56 | 57 | if __name__ == '__main__': 58 | import argparse 59 | import sys 60 | import traceback 61 | 62 | try: 63 | parser = argparse.ArgumentParser( 64 | description='Learn linear weights for maximizing segmentation loss\ 65 | for a given label') 66 | parser.add_argument( 67 | '--directory', 68 | default='.', 69 | help='output directory for the net probe') 70 | 71 | args = parser.parse_args() 72 | compute_alpha(args.directory) 73 | except: 74 | traceback.print_exc(file=sys.stdout) 75 | sys.exit(1) 76 | 77 | 78 | -------------------------------------------------------------------------------- /src/consolidatelineardiscprobe.py: -------------------------------------------------------------------------------- 1 | import expdir 2 | import loadseg 3 | import os 4 | #from labelprobe import cached_memmap 5 | 6 | def consolidate_disc_probe(directory, blob, bias=False, num_filters=None, suffix='', epochs=30, delete=False): 7 | ed = expdir.ExperimentDirectory(directory) 8 | try: 9 | info = ed.load_info() 10 | except: 11 | ed = expdir.ExperimentDirectory(os.path.join(directory, 'train')) 12 | #ed_val = expdir.ExperimentDirectory(os.path.join(directory, 'val')) 13 | info = ed.load_info() 14 | assert('imagenet' in info.dataset or 'ILSVRC' in info.dataset) 15 | 16 | blob_info = ed.load_info(blob=blob) 17 | shape = blob_info.shape 18 | if 'broden' in info.dataset: 19 | ds = loadseg.SegmentationData(info.dataset) 20 | L = ds.label_size() 21 | elif 'imagenet' in info.dataset or 'ILSVRC' in info.dataset: 22 | L = 1000 23 | N = shape[0] # total number of images in the dataset 24 | K = shape[1] # number of units for the given blob 25 | F = 1 if num_filters is None else len(num_filters) + 1 26 | 27 | if num_filters is None: 28 | suffixes = [suffix] 29 | else: 30 | suffixes = ['%s_num_filters_%d' % (suffix, n) for n in num_filters] 31 | suffixes.append(suffix) 32 | suffix = '%s_num_filters_%d' % (suffix, F) 33 | 34 | 35 | if (ed.has_mmap(blob=blob, part='linear_weights_disc%s' % suffix) 36 | and (not bias or ed.has_mmap(blob=blob, part='linear_bias_disc%s' % suffix))): 37 | print('Linear weights (and bias) have already been consolidated') 38 | print ed.mmap_filename(blob=blob, part='linear_weights_disc%s' % suffix) 39 | else: 40 | weights_mmap = ed.open_mmap(blob=blob, part='linear_weights_disc%s' % suffix, mode='w+', 41 | dtype='float32', shape=(L,F,K)) 42 | 43 | if bias: 44 | bias_mmap = ed.open_mmap(blob=blob, part='linear_bias_disc%s' % suffix, 45 | mode='w+', dtype='float32', shape=(L,F)) 46 | 47 | results_mmap = ed.open_mmap(blob=blob, part='linear_results_disc%s' % suffix, 48 | mode='w+', dtype='float32', shape=(L, F, 4, epochs)) 49 | 50 | 51 | missing_idx = [] 52 | for l in range(L): 53 | if not ed.has_mmap(blob=blob, part='label_i_%d_weights_disc%s' % (l, suffixes[0])): 54 | missing_idx.append(l) 55 | continue 56 | for i in range(len(suffixes)): 57 | suffix = suffixes[i] 58 | label_weights_mmap = ed.open_mmap(blob=blob, part='label_i_%d_weights_disc%s' 59 | % (l, suffix), mode='r', dtype='float32', shape=(K,)) 60 | 61 | weights_mmap[l,i,:] = label_weights_mmap[:] 62 | 63 | if bias: 64 | label_bias_mmap = ed.open_mmap(blob=blob, part='label_i_%d_bias_disc%s' 65 | % (l, suffix), mode='r', dtype='float32', shape=(1,)) 66 | bias_mmap[l,i] = label_bias_mmap[:] 67 | 68 | label_results_mmap = ed.open_mmap(blob=blob, part='label_i_%d_results_disc%s' 69 | % (l, suffix), mode='r', dtype='float32', 70 | shape=(4, epochs)) 71 | results_mmap[l,i,:,:] = label_results_mmap[:,:] 72 | print l 73 | 74 | ed.finish_mmap(weights_mmap) 75 | if bias: 76 | ed.finish_mmap(bias_mmap) 77 | ed.finish_mmap(results_mmap) 78 | 79 | print('Finished consolidating existing weights files ' 80 | + '(Files for %d labels were missing).' % len(missing_idx)) 81 | print(missing_idx) 82 | 83 | if delete: 84 | c_w = 0 85 | c_b = 0 86 | c_r = 0 87 | print('Deleting all unnecessary label weights (and bias) files...') 88 | for l in range(L): 89 | for i in range(F): 90 | suffix = suffixes[i] 91 | if ed.has_mmap(blob=blob, part='label_i_%d_weights_disc%s' % (l, suffix)): 92 | fn = ed.mmap_filename(blob=blob, part='label_i_%d_weights_disc%s' % (l, suffix)) 93 | os.remove(fn) 94 | c_w += 1 95 | if bias and ed.has_mmap(blob=blob, part='label_i_%d_bias_disc%s' % (l, suffix)): 96 | fn = ed.mmap_filename(blob=blob, part='label_i_%d_bias_disc%s' % (l, suffix)) 97 | os.remove(fn) 98 | c_b += 1 99 | if ed.has_mmap(blob=blob, part='label_i_%d_results_disc%s' % (l, suffix)): 100 | fn = ed.mmap_filename(blob=blob, part='label_i_%d_results_disc%s' % (l, suffix)) 101 | os.remove(fn) 102 | c_r += 1 103 | 104 | print('Removed %d weights, %d bias, and %d results files.' % (c_w, c_b, c_r)) 105 | 106 | 107 | if __name__ == '__main__': 108 | import argparse 109 | import sys 110 | import traceback 111 | 112 | try: 113 | parser = argparse.ArgumentParser() 114 | parser.add_argument('--directory', default='.') 115 | parser.add_argument('--blobs', nargs='*') 116 | parser.add_argument('--bias', action='store_true', default=False) 117 | parser.add_argument('--suffix', type=str, default='') 118 | parser.add_argument('--num_filters', type=int, nargs='*', default=None) 119 | parser.add_argument('--epochs', type=int, default=30) 120 | parser.add_argument('--delete', action='store_true', default=False) 121 | 122 | args = parser.parse_args() 123 | 124 | for blob in args.blobs: 125 | consolidate_disc_probe(args.directory, blob, bias=args.bias, 126 | num_filters=args.num_filters, epochs=args.epochs, suffix=args.suffix, delete=args.delete) 127 | 128 | except: 129 | traceback.print_exc(file=sys.stdout) 130 | sys.exit(1) 131 | -------------------------------------------------------------------------------- /src/consolidatelinearprobe.py: -------------------------------------------------------------------------------- 1 | import expdir 2 | import loadseg 3 | import os 4 | #from labelprobe import cached_memmap 5 | 6 | def consolidate_probe(directory, blob, bias=False, num_filters=None, suffix='', delete=False): 7 | ed = expdir.ExperimentDirectory(directory) 8 | info = ed.load_info() 9 | blob_info = ed.load_info(blob=blob) 10 | shape = blob_info.shape 11 | ds = loadseg.SegmentationData(info.dataset) 12 | 13 | L = ds.label_size() # number of labels (including background at index 0) 14 | N = ds.size() # total number of images in the dataset 15 | K = shape[1] # number of units for the given blob 16 | F = 1 if num_filters is None else len(num_filters) + 1 17 | 18 | if num_filters is None: 19 | suffixes = [suffix] 20 | else: 21 | suffixes = ['%s_num_filters_%d' % (suffix, n) for n in num_filters] 22 | suffixes.append(suffix) 23 | suffix = '%s_num_filters_%d' % (suffix, F) 24 | 25 | if (ed.has_mmap(blob=blob, part='linear_weights%s' % suffix) 26 | and (not bias 27 | or ed.has_mmap(blob=blob, part='linear_bias%s' % suffix))): 28 | print('Linear weights (and bias) have already been consolidated') 29 | else: 30 | weights_mmap = ed.open_mmap(blob=blob, part='linear_weights%s' % suffix, mode='w+', 31 | dtype='float32', shape=(L,F,K)) 32 | 33 | if bias: 34 | bias_mmap = ed.open_mmap(blob=blob, part='linear_bias%s' % suffix, 35 | mode='w+', dtype='float32', shape=(L,F)) 36 | 37 | missing_idx = [] 38 | for l in range(L): 39 | if not ed.has_mmap(blob=blob, part='label_i_%d_weights%s' % (l, suffixes[0])): 40 | missing_idx.append(l) 41 | continue 42 | for i in range(F): 43 | suffix = suffixes[i] 44 | if ed.has_mmap(blob=blob, part='label_i_%d_weights%s' % (l, suffix)): 45 | try: 46 | label_weights_mmap = ed.open_mmap(blob=blob, part='label_i_%d_weights%s' 47 | % (l, suffix), mode='r', dtype='float32', shape=(K,)) 48 | except ValueError: 49 | print('here') 50 | # SUPPORT LEGACY CODE, TODO: remove eventually 51 | label_weights_mmap = ed.open_mmap(blob=blob, part='label_i_%d_weights%s' 52 | % (l, suffix), mode='r', dtype=float, shape=(K,)) 53 | else: 54 | all_weights_mmap = ed.open_mmap(blob=blob, part='linear_weights%s' % suffix, 55 | mode='r', dtype='float32', shape=(L,K)) 56 | label_weights_mmap = all_weights_mmap[l] 57 | 58 | 59 | weights_mmap[l,i,:] = label_weights_mmap[:] 60 | 61 | if bias: 62 | if ed.has_mmap(blob=blob, part='label_i_%d_bias' % (l, suffix)): 63 | label_bias_mmap = ed.open_mmap(blob=blob, part='label_i_%d_bias%s' 64 | % (l, suffix), mode='r', dtype='float32', shape=(1,)) 65 | else: 66 | all_bias_mmap = ed.open_mmap(blob=blob, part='linear_bias%s' % suffix, 67 | mode='r', dtype='float32', shape=(K,)) 68 | label_bias_mmap = all_bias_mmap[l] 69 | bias_mmap[l,i] = label_bias_mmap[:] 70 | 71 | ed.finish_mmap(weights_mmap) 72 | if bias: 73 | ed.finish_mmap(bias_mmap) 74 | 75 | print('Finished consolidating existing weights files ' 76 | + '(Files for %d labels were missing).' % len(missing_idx)) 77 | print(missing_idx) 78 | 79 | if delete: 80 | c_w = 0 81 | c_b = 0 82 | print('Deleting all unnecessary label weights (and bias) files...') 83 | for l in range(L): 84 | for i in range(F): 85 | suffix = suffixes[i] 86 | if ed.has_mmap(blob=blob, part='label_i_%d_weights%s' % (l, suffix)): 87 | fn = ed.mmap_filename(blob=blob, part='label_i_%d_weights%s' % (l, suffix)) 88 | os.remove(fn) 89 | c_w += 1 90 | if bias and ed.has_mmap(blob=blob, part='label_i_%d_bias%s' % (l, suffix)): 91 | fn = ed.mmap_filename(blob=blob, part='label_i_%d_bias%s' % (l, suffix)) 92 | os.remove(fn) 93 | c_b += 1 94 | 95 | print('Removed %d weights and %d bias files.' % (c_w, c_b)) 96 | 97 | 98 | if __name__ == '__main__': 99 | import argparse 100 | import sys 101 | import traceback 102 | 103 | try: 104 | parser = argparse.ArgumentParser() 105 | parser.add_argument('--directory', default='.') 106 | parser.add_argument('--blobs', nargs='*') 107 | parser.add_argument('--bias', action='store_true', default=False) 108 | parser.add_argument('--suffix', type=str, default='') 109 | parser.add_argument('--delete', action='store_true', default=False) 110 | parser.add_argument('--num_filters', type=int, nargs='*', default=None) 111 | 112 | args = parser.parse_args() 113 | 114 | for blob in args.blobs: 115 | consolidate_probe(args.directory, blob, bias=args.bias, 116 | suffix=args.suffix, num_filters=args.num_filters, delete=args.delete) 117 | 118 | except: 119 | traceback.print_exc(file=sys.stdout) 120 | sys.exit(1) 121 | -------------------------------------------------------------------------------- /src/customoptimizer_pytorch.py: -------------------------------------------------------------------------------- 1 | import math 2 | import torch 3 | from torch.optim import Optimizer 4 | 5 | required = object() 6 | 7 | class Custom_Adam(Optimizer): 8 | def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, 9 | l1_weight_decay=0, l2_weight_decay=0, lower_bound=None, upper_bound=None): 10 | defaults = dict(lr=lr, betas=betas, eps=eps, 11 | l1_weight_decay=l1_weight_decay, 12 | l2_weight_decay=l2_weight_decay, 13 | lower_bound=lower_bound, 14 | upper_bound=upper_bound) 15 | super(Custom_Adam, self).__init__(params, defaults) 16 | 17 | def step(self, closure=None): 18 | """Performs a single optimization step. 19 | Arguments: 20 | closure (callable, optional): A closure that reevaluates the model 21 | and returns the loss. 22 | """ 23 | loss = None 24 | if closure is not None: 25 | loss = closure() 26 | 27 | for group in self.param_groups: 28 | for p in group['params']: 29 | if p.grad is None: 30 | continue 31 | grad = p.grad.data 32 | state = self.state[p] 33 | 34 | # State initialization 35 | if len(state) == 0: 36 | state['step'] = 0 37 | # Exponential moving average of gradient values 38 | state['exp_avg'] = grad.new().resize_as_(grad).zero_() 39 | # Exponential moving average of squared gradient values 40 | state['exp_avg_sq'] = grad.new().resize_as_(grad).zero_() 41 | 42 | exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq'] 43 | beta1, beta2 = group['betas'] 44 | 45 | state['step'] += 1 46 | 47 | if group['l1_weight_decay'] != 0: 48 | grad = grad.add(group['l1_weight_decay'], torch.sign(p.data)) 49 | if group['l2_weight_decay'] != 0: 50 | grad = grad.add(group['l2_weight_decay'], p.data) 51 | 52 | # Decay the first and second moment running average coefficient 53 | exp_avg.mul_(beta1).add_(1 - beta1, grad) 54 | exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad) 55 | 56 | denom = exp_avg_sq.sqrt().add_(group['eps']) 57 | 58 | bias_correction1 = 1 - beta1 ** state['step'] 59 | bias_correction2 = 1 - beta2 ** state['step'] 60 | step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1 61 | 62 | p.data.addcdiv_(-step_size, exp_avg, denom) 63 | 64 | # custom clamping 65 | if group['lower_bound'] is not None: 66 | p.data.clamp_(min=group['lower_bound']) 67 | if group['upper_bound'] is not None: 68 | p.data.clamp_(max=group['upper_bound']) 69 | 70 | return loss 71 | 72 | class Custom_SGD(Optimizer): 73 | r"""Implements stochastic gradient descent (optionally with momentum). 74 | Nesterov momentum is based on the formula from 75 | `On the importance of initialization and momentum in deep learning`__. 76 | Args: 77 | params (iterable): iterable of parameters to optimize or dicts defining 78 | parameter groups 79 | lr (float): learning rate 80 | momentum (float, optional): momentum factor (default: 0) 81 | weight_decay (float, optional): weight decay (L2 penalty) (default: 0) 82 | dampening (float, optional): dampening for momentum (default: 0) 83 | nesterov (bool, optional): enables Nesterov momentum (default: False) 84 | Example: 85 | >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) 86 | >>> optimizer.zero_grad() 87 | >>> loss_fn(model(input), target).backward() 88 | >>> optimizer.step() 89 | __ http://www.cs.toronto.edu/%7Ehinton/absps/momentum.pdf 90 | .. note:: 91 | The implementation of SGD with Momentum/Nesterov subtly differs from 92 | Sutskever et. al. and implementations in some other frameworks. 93 | Considering the specific case of Momentum, the update can be written as 94 | .. math:: 95 | v = \rho * v + g \\ 96 | p = p - lr * v 97 | where p, g, v and :math:`\rho` denote the parameters, gradient, velocity, and 98 | momentum respectively. 99 | This is in constrast to Sutskever et. al. and 100 | other frameworks which employ an update of the form 101 | .. math:: 102 | v = \rho * v + lr * g \\ 103 | p = p - v 104 | The Nesterov version is analogously modified. 105 | """ 106 | 107 | def __init__(self, params, lr=required, momentum=0, dampening=0, 108 | l1_weight_decay=0, l2_weight_decay=0, nesterov=False, 109 | lower_bound=None, upper_bound=None): 110 | defaults = dict(lr=lr, momentum=momentum, dampening=dampening, 111 | l1_weight_decay=l1_weight_decay, l2_weight_decay=l2_weight_decay, 112 | nesterov=nesterov, lower_bound=lower_bound, upper_bound=upper_bound) 113 | if nesterov and (momentum <= 0 or dampening != 0): 114 | raise ValueError("Nesterov momentum requires a momentum and zero dampening") 115 | super(Custom_SGD, self).__init__(params, defaults) 116 | 117 | def __setstate__(self, state): 118 | super(Custom_SGD, self).__setstate__(state) 119 | for group in self.param_groups: 120 | group.setdefault('nesterov', False) 121 | 122 | def step(self, closure=None): 123 | """Performs a single optimization step. 124 | Arguments: 125 | closure (callable, optional): A closure that reevaluates the model 126 | and returns the loss. 127 | """ 128 | loss = None 129 | if closure is not None: 130 | loss = closure() 131 | 132 | for group in self.param_groups: 133 | l1_weight_decay = group['l1_weight_decay'] 134 | l2_weight_decay = group['l2_weight_decay'] 135 | momentum = group['momentum'] 136 | dampening = group['dampening'] 137 | nesterov = group['nesterov'] 138 | 139 | for p in group['params']: 140 | if p.grad is None: 141 | continue 142 | d_p = p.grad.data 143 | # L1 weight decay 144 | if l1_weight_decay != 0: 145 | d_p.add_(l1_weight_decay, torch.sign(p.data)) 146 | if l2_weight_decay != 0: 147 | d_p.add_(l2_weight_decay, p.data) 148 | if momentum != 0: 149 | param_state = self.state[p] 150 | if 'momentum_buffer' not in param_state: 151 | buf = param_state['momentum_buffer'] = d_p.clone() 152 | else: 153 | buf = param_state['momentum_buffer'] 154 | buf.mul_(momentum).add_(1 - dampening, d_p) 155 | if nesterov: 156 | d_p = d_p.add(momentum, buf) 157 | else: 158 | d_p = buf 159 | 160 | p.data.add_(-group['lr'], d_p) 161 | 162 | # custom clamping 163 | if group['lower_bound'] is not None: 164 | p.data.clamp_(min=group['lower_bound']) 165 | if group['upper_bound'] is not None: 166 | p.data.clamp_(max=group['upper_bound']) 167 | 168 | return loss 169 | 170 | -------------------------------------------------------------------------------- /src/dtdseg.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | import colorname 4 | from loadseg import AbstractSegmentation 5 | from scipy.misc import imread 6 | 7 | class DtdSegmentation(AbstractSegmentation): 8 | def __init__(self, directory=None): 9 | directory = os.path.expanduser(directory) 10 | self.directory = directory 11 | with open(os.path.join(directory, 'labels', 12 | 'labels_joint_anno.txt')) as f: 13 | self.dtd_meta = [line.split(None, 1) for line in f.readlines()] 14 | self.textures = ['-'] + sorted(list(set(sum( 15 | [c.split() for f, c in self.dtd_meta], [])))) 16 | self.texture_map = dict((t, i) for i, t in enumerate(self.textures)) 17 | 18 | def all_names(self, category, j): 19 | if j == 0: 20 | return [] 21 | if category == 'color': 22 | return [colorname.color_names[j - 1] + '-c'] 23 | if category == 'texture': 24 | return [self.textures[j]] 25 | return [] 26 | 27 | def size(self): 28 | '''Returns the number of images in this dataset.''' 29 | return len(self.dtd_meta) 30 | 31 | def filename(self, i): 32 | '''Returns the filename for the nth dataset image.''' 33 | return os.path.join(self.directory, 'images', self.dtd_meta[i][0]) 34 | 35 | def metadata(self, i): 36 | '''Returns an object that can be used to create all segmentations.''' 37 | fn, tnames = self.dtd_meta[i] 38 | tnumbers = [self.texture_map[n] for n in tnames.split()] 39 | return os.path.join(self.directory, 'images', fn), tnumbers 40 | 41 | @classmethod 42 | def resolve_segmentation(cls, m, categories=None): 43 | filename, tnumbers = m 44 | result = {} 45 | if wants('texture', categories): 46 | result['texture'] = tnumbers 47 | if wants('color', categories): 48 | result['color'] = colorname.label_major_colors(imread(filename)) + 1 49 | arrs = [a for a in result.values() if numpy.shape(a) >= 2] 50 | shape = arrs[0].shape[-2:] if arrs else (1, 1) 51 | return result, shape 52 | 53 | def wants(what, option): 54 | if option is None: 55 | return True 56 | return what in option 57 | -------------------------------------------------------------------------------- /src/extract_concept_data.py: -------------------------------------------------------------------------------- 1 | import expdir 2 | import loadseg 3 | import time 4 | 5 | import numpy as np 6 | 7 | 8 | def get_seg_size(input_dim): 9 | if input_dim == [227, 227]: 10 | seg_size = (113, 113) 11 | elif input_dim == [224, 224]: 12 | seg_size = (112, 112) 13 | elif input_dim == [384, 384]: 14 | seg_size = (192, 192) 15 | else: 16 | print input_dim 17 | assert(False) 18 | return seg_size 19 | 20 | 21 | def extract_concept_data(directory, batch_size=64, ahead=16, verbose=True): 22 | ed = expdir.ExperimentDirectory(directory) 23 | if ed.has_mmap(part='concept_data'): 24 | print('%s already has %s, so skipping' % (directory, 25 | ed.mmap_filename(part='concept_data'))) 26 | return 27 | info = ed.load_info() 28 | (sh, sw) = get_seg_size(info.input_dim) 29 | ds = loadseg.SegmentationData(info.dataset) 30 | categories = np.array(ds.category_names()) 31 | L = ds.label_size() 32 | N = ds.size() 33 | pf = loadseg.SegmentationPrefetcher(ds, categories=categories, once=True, batch_size=batch_size, 34 | ahead=ahead, thread=True) 35 | 36 | if verbose: 37 | print 'Creating new mmap at %s' % ed.mmap_filename(part='concept_data') 38 | data = ed.open_mmap(part='concept_data', mode='w+', shape=(N,L,sh,sw)) 39 | 40 | start_time = time.time() 41 | last_batch_time = start_time 42 | index = 0 43 | for batch in pf.batches(): 44 | batch_time = time.time() 45 | rate = index / (batch_time - start_time + 1e-15) 46 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 47 | last_batch_time = batch_time 48 | if verbose: 49 | print 'extract_concept_data index %d/%d (%.2f)\titems per sec %.2f\t%.2f' % (index, 50 | N, 51 | index/float(N), 52 | batch_rate, 53 | rate) 54 | for rec in batch: 55 | for cat in categories: 56 | if len(rec[cat]) == 0: 57 | continue 58 | if cat == 'texture' or cat == 'scene': 59 | for i in range(len(rec[cat])): 60 | data[index][rec[cat][i]-1,:,:] = 1 61 | else: 62 | for i in range(len(rec[cat])): 63 | ys, xs = np.where(rec[cat][i]) 64 | for j in range(len(xs)): 65 | data[index][rec[cat][i][ys[j]][xs[j]]-1][ys[j]][xs[j]] = 1 66 | index += 1 67 | 68 | assert index == N, ("Data source should return every item once %d %d." % 69 | (index, N)) 70 | if verbose: 71 | print 'Renaming mmap.' 72 | ed.finish_mmap(data) 73 | 74 | 75 | if __name__ == '__main__': 76 | import sys 77 | import traceback 78 | import argparse 79 | 80 | try: 81 | parser = argparse.ArgumentParser(description='TODO') 82 | parser.add_argument( 83 | '--directory', 84 | default='.', 85 | help='output directory for the net probe') 86 | parser.add_argument( 87 | '--batch_size', 88 | type=int, 89 | default=64, 90 | help='batch size') 91 | parser.add_argument( 92 | '--ahead', 93 | type=int, 94 | default=16, 95 | help='TODO') 96 | args = parser.parse_args() 97 | extract_concept_data(args.directory, batch_size=args.batch_size, ahead=args.ahead, 98 | verbose=True) 99 | except: 100 | traceback.print_exc(file=sys.stdout) 101 | sys.exit(1) 102 | 103 | 104 | -------------------------------------------------------------------------------- /src/fieldmap.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import os 4 | 5 | os.environ['GLOG_minloglevel'] = '2' 6 | import caffe 7 | from caffe.proto import caffe_pb2 8 | from google.protobuf import text_format 9 | import upsample 10 | import rotate 11 | 12 | def print_fieldmap(definition, blob): 13 | ''' 14 | definition: the filename for the caffe prototxt 15 | blobs: 'conv3' to probe 16 | ''' 17 | np = caffe_pb2.NetParameter() 18 | with open(definition, 'r') as dfn_file: 19 | text_format.Merge(dfn_file.read(), np) 20 | # Compute the input-to-layer fieldmap for each blob 21 | # ind = max(i for i, lay in enumerate(np.layer) if blob in lay.top) 22 | # path = upsample.shortest_layer_path(np.input, blob, np.layer) 23 | net = caffe.Net(definition, caffe.TEST) 24 | fieldmap, path = upsample.composed_fieldmap(np.layer, blob) 25 | for b, layer in path: 26 | f, _ = upsample.composed_fieldmap(np.layer, b) 27 | a = net.blobs[b].data.shape[-2:] 28 | s = upsample.upsampled_shape(f, a) 29 | print b, f, a, '->', s 30 | print blob, fieldmap 31 | 32 | if __name__ == '__main__': 33 | import argparse 34 | import loadseg 35 | 36 | parser = argparse.ArgumentParser( 37 | description='Calculate fieldmap for a blob in a caffe network.') 38 | parser.add_argument( 39 | '--blob', 40 | help='network blob name to probe') 41 | parser.add_argument( 42 | '--definition', 43 | help='the deploy prototext defining the net') 44 | args = parser.parse_args() 45 | 46 | print_fieldmap(args.definition, args.blob) 47 | -------------------------------------------------------------------------------- /src/figures.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import os 3 | import sys 4 | import numpy as np 5 | import re 6 | #import upsample 7 | import time 8 | import loadseg 9 | from scipy.misc import imread, imresize, imsave 10 | from loadseg import normalize_label 11 | import expdir 12 | import matplotlib 13 | import matplotlib.pyplot as plt 14 | 15 | import torch 16 | import torch.nn as nn 17 | from torch.autograd import Variable 18 | 19 | from linearprobe_pytorch import CustomLayer 20 | 21 | def to_percent(y, position): 22 | # Ignore the passed in position. This has the effect of scaling the default 23 | # tick locations. 24 | s = str(int(100 * y)) 25 | 26 | # The percent symbol needs escaping in latex 27 | if matplotlib.rcParams['text.usetex'] is True: 28 | return s + r'$\%$' 29 | else: 30 | return s + '%' 31 | 32 | 33 | def get_single_acts(acts, thresh, filter_i, up_size=(224,224), thresh_first=True): 34 | upsample = nn.Upsample(size=up_size, mode='bilinear') 35 | if thresh_first: 36 | return upsample(Variable(torch.Tensor((acts > thresh).astype(float)))).data.cpu().numpy()[0,filter_i] 37 | else: 38 | return (upsample(Variable(torch.Tensor(acts))).data.cpu().numpy()[0] > thresh).astype(float)[filter_i] 39 | 40 | 41 | def get_weighted_acts(acts, label_weights, thresh=None, label_bias=None,up_size=(224,224), act=True, 42 | positive=False, bias=False, thresh_first=True): 43 | layer = CustomLayer(len(label_weights), upsample=thresh_first, up_size=up_size, act=act, positive=positive, bias=bias) 44 | layer.weight.data[...] = torch.Tensor(label_weights) 45 | if bias: 46 | assert(label_bias is not None) 47 | layer.bias.data[...] = torch.Tensor(label_bias) 48 | upsample = nn.Upsample(size=up_size, mode='bilinear') 49 | if thresh is None: 50 | return layer(upsample(Variable(torch.Tensor(acts)))).data.cpu().numpy() 51 | else: 52 | if thresh_first: 53 | return layer(Variable(torch.Tensor((acts > thresh).astype(float)))).data.cpu().numpy() 54 | else: 55 | return layer((upsample(Variable(torch.Tensor(acts))) > Variable(torch.Tensor(thresh))).type(torch.FloatTensor)).data.cpu().numpy() 56 | 57 | 58 | def show_examples_for_unit(ds, blobdata, thresh, blob, unit_i, examples_idx, up_size=(224,224), num_to_show=8, mask_alpha=0.2, thresh_first=False): 59 | f, ax = plt.subplots(1,num_to_show,figsize=(4*num_to_show,4)) 60 | for i in range(num_to_show): 61 | ax[i].set_xticks([]) 62 | ax[i].set_yticks([]) 63 | if i == 0: 64 | pass 65 | #ax[i].set_ylabel('unit %d' % unit_i) 66 | #try: 67 | up = get_single_acts(np.expand_dims(blobdata[blob][examples_idx[i]], axis=0), thresh[blob], unit_i, up_size=up_size, thresh_first=thresh_first) 68 | ax[i].imshow((np.expand_dims((up > 0.5), axis=2) * (1-mask_alpha) + mask_alpha) * imread(ds.filename(examples_idx[i]))/255.) 69 | # #ax[i].imshow(probe.activation_visualization(blob, unit_i, top[unit_i][i], normalize=True, use_fieldmap=False)) 70 | #except: 71 | # break 72 | f.subplots_adjust(wspace=0.0,hspace=0.0) 73 | #plt.tight_layout() 74 | plt.show() 75 | 76 | 77 | def show_examples_for_linear(ds, blobdata, label_weights, examples_idx, thresh=None, up_size=(224,224), num_to_show=8, mask_alpha=0.2, 78 | thresh_first=False, soft_mask=False): 79 | f, ax = plt.subplots(1,num_to_show,figsize=(4*num_to_show,4)) 80 | for i in range(num_to_show): 81 | ax[i].set_xticks([]) 82 | ax[i].set_yticks([]) 83 | if i == 0: 84 | pass 85 | #ax[i].set_ylabel('unit %d' % unit_i) 86 | #try: 87 | if thresh is None: 88 | up = get_weighted_acts(np.expand_dims(blobdata[examples_idx[i]], axis=0), label_weights, thresh=thresh, label_bias=None, 89 | up_size=up_size, act=True, positive=False, bias=False, thresh_first=thresh_first) 90 | else: 91 | up = get_weighted_acts(np.expand_dims(blobdata[examples_idx[i]], axis=0), label_weights, thresh=thresh, label_bias=None, 92 | up_size=up_size, act=True, positive=False, bias=False, thresh_first=thresh_first) 93 | #except: 94 | # break 95 | if soft_mask: 96 | ax[i].imshow(imread(ds.filename(examples_idx[i]))/255.) 97 | ax[i].imshow(up, cmap='jet', alpha=mask_alpha) 98 | #ax[i].imshow((np.expand_dims(up, axis=2) * (1-mask_alpha) + mask_alpha) * imread(ds.filename(examples_idx[i]))/255.) 99 | else: 100 | ax[i].imshow((np.expand_dims((up > 0.5), axis=2) * (1-mask_alpha) + mask_alpha) * imread(ds.filename(examples_idx[i]))/255.) 101 | 102 | f.subplots_adjust(wspace=0.0,hspace=0.0) 103 | plt.show() 104 | 105 | def find_label(phrase, label_names): 106 | return np.array([i for i in range(len(label_names)) if phrase in label_names[i]]) 107 | 108 | def find_exact_label(phrase, label_names): 109 | for i in range(len(label_names)): 110 | if phrase == label_names[i]: 111 | return i 112 | else: 113 | return -1 114 | -------------------------------------------------------------------------------- /src/graphprobe.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | from collections import OrderedDict, defaultdict 4 | import expdir 5 | 6 | category_colors = OrderedDict([ 7 | ('scene', '#3288bd'), 8 | ('object', '#99d594'), 9 | ('part', '#e6f598'), 10 | ('material', '#fee08b'), 11 | ('texture', '#fc8d59'), 12 | ('color', '#d53e4f'), 13 | ('total', '#aaaaaa') 14 | ]) 15 | 16 | 17 | def loadviz(directory, blob): 18 | ed = expdir.ExperimentDirectory(directory) 19 | html_fn = ed.filename(['html', '%s.html' % expdir.fn_safe(blob)]) 20 | with open(html_fn) as f: 21 | lines = f.readlines() 22 | result = OrderedDict() 23 | for line in lines: 24 | if 'unit ' in line: 25 | u, v = line.split(':', 1) 26 | unit = int(re.search(r'unit (\d+)', u).group(1)) - 1 27 | u_result = [] 28 | for w in v.split(';'): 29 | m = re.search(r'(\w.*\w)? \((\w+), ([.\d]+)\)', w) 30 | if not m: 31 | print 'On line', v 32 | print 'Error with', w 33 | label = m.group(1) 34 | category = m.group(2) 35 | score = float(m.group(3)) 36 | u_result.append((label, category, score)) 37 | result[unit] = u_result 38 | return result 39 | 40 | def summarize(scores, threshold, top_only=True): 41 | result = defaultdict(float) 42 | denom = len(scores) 43 | for k, v in scores.items(): 44 | got = False 45 | for val in v: 46 | label, category, score = val 47 | if score >= threshold: 48 | result[category] += 1.0 / denom 49 | got = True 50 | if top_only: 51 | break 52 | if got: 53 | result['total'] += 1.0 / denom 54 | return result 55 | 56 | if __name__ == '__main__': 57 | import argparse 58 | import sys 59 | import traceback 60 | 61 | try: 62 | import matplotlib 63 | matplotlib.use('Agg') 64 | import matplotlib.pyplot as plt 65 | 66 | 67 | parser = argparse.ArgumentParser( 68 | description='Generate visualization for probed activation data.') 69 | parser.add_argument( 70 | '--directories', 71 | nargs='*', 72 | help='directories to graph') 73 | parser.add_argument( 74 | '--blobs', 75 | nargs='*', 76 | help='blobs to graph') 77 | parser.add_argument( 78 | '--threshold', 79 | type=float, default=0.05, 80 | help='score above which to count items') 81 | parser.add_argument( 82 | '--top_only', 83 | type=lambda s: s.lower() in ['true', 't', 'yes', '1'], 84 | default=True, 85 | help='include only the top values') 86 | parser.add_argument( 87 | '--include_total', 88 | type=lambda s: s.lower() in ['true', 't', 'yes', '1'], 89 | default=False, 90 | help='include total value line') 91 | parser.add_argument( 92 | '--labels', 93 | nargs='*', 94 | help='tick labels') 95 | parser.add_argument( 96 | '--title', 97 | help='graph title') 98 | parser.add_argument( 99 | '--legend', 100 | default='upper right', 101 | help='location of legend') 102 | parser.add_argument( 103 | '--maxy', 104 | type=float, default=None, 105 | help='y axis range to apply') 106 | parser.add_argument( 107 | '--out', 108 | help='output filename for graph') 109 | args = parser.parse_args() 110 | data = [] 111 | categories = set(category_colors.keys()) 112 | for directory in args.directories: 113 | for blob in args.blobs: 114 | stats = summarize(loadviz(directory, blob), args.threshold, 115 | top_only=args.top_only) 116 | data.append(stats) 117 | categories.update(stats.keys()) 118 | x = range(1, len(data) + 1) 119 | maxval = 0 120 | plt.figure(num=None, figsize=(7,4), dpi=300) 121 | for cat in category_colors.keys(): 122 | if cat not in categories: 123 | continue 124 | if not args.include_total and cat == 'total': 125 | continue 126 | dat = [d[cat] for d in data] 127 | maxval = max(maxval, max(dat)) 128 | plt.plot(x, dat, 'o-' if cat != 'total' else 's--', 129 | color=category_colors[cat], label=cat) 130 | if args.labels: 131 | plt.xticks(x, args.labels) 132 | plt.margins(0.1) 133 | plt.legend(loc=args.legend) 134 | if args.maxy is not None: 135 | plt.ylim(-0.01, args.maxy) 136 | else: 137 | plt.ylim(-maxval * 0.05, maxval * 1.5) 138 | ax = plt.gca() 139 | ax.yaxis.grid(True) 140 | for side in ['top', 'bottom', 'right', 'left']: 141 | ax.spines[side].set_visible(False) 142 | ax.xaxis.set_ticks_position('bottom') 143 | if args.title: 144 | plt.title(args.title) 145 | plt.ylabel('portion of units alinged to a category concept') 146 | plt.savefig(args.out) 147 | 148 | except: 149 | traceback.print_exc(file=sys.stdout) 150 | sys.exit(1) 151 | -------------------------------------------------------------------------------- /src/indexdata.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | import loadseg 4 | import expdir 5 | 6 | from torchvision.datasets.folder import find_classes, make_dataset 7 | 8 | def has_image_to_label(directory): 9 | ed = expdir.ExperimentDirectory(directory) 10 | return ed.has_mmap(part='image_to_label') 11 | 12 | 13 | def load_image_to_label(directory, blob=None): 14 | ed = expdir.ExperimentDirectory(directory) 15 | info = ed.load_info() 16 | if 'broden' in info.dataset: 17 | ds = loadseg.SegmentationData(info.dataset) 18 | shape = (ds.size(), len(ds.label)) 19 | elif 'imagenet' in info.dataset or 'ILSVRC' in info.dataset: 20 | assert(blob is not None) 21 | blob_info = ed.load_info(blob=blob) 22 | shape = (blob_info.shape[0], 1000) 23 | else: 24 | assert(False) 25 | return ed.open_mmap(part='image_to_label', mode='r', dtype=bool, 26 | shape=shape) 27 | 28 | 29 | def create_image_to_label(directory, batch_size=16, ahead=4): 30 | ed = expdir.ExperimentDirectory(directory) 31 | info = ed.load_info() 32 | 33 | print info.dataset 34 | if 'broden' in info.dataset: 35 | ds = loadseg.SegmentationData(info.dataset) 36 | categories = ds.category_names() 37 | shape = (ds.size(), len(ds.label)) 38 | 39 | pf = loadseg.SegmentationPrefetcher(ds, categories=categories, once=True, 40 | batch_size=batch_size, ahead=ahead, thread=False) 41 | 42 | image_to_label = np.zeros(shape, dtype='int32') 43 | 44 | batch_count = 0 45 | for batch in pf.batches(): 46 | if batch_count % 100 == 0: 47 | print('Processing batch %d ...' % batch_count) 48 | for rec in batch: 49 | image_index = rec['i'] 50 | for cat in categories: 51 | if ((type(rec[cat]) is np.ndarray and rec[cat].size > 0) or 52 | type(rec[cat]) is list and len(rec[cat]) > 0): 53 | image_to_label[image_index][np.unique(rec[cat])] = True 54 | batch_count += 1 55 | elif 'imagenet' in info.dataset or 'ILSVRC' in info.dataset: 56 | classes, class_to_idx = find_classes(info.dataset) 57 | imgs = make_dataset(info.dataset, class_to_idx) 58 | _, labels = zip(*imgs) 59 | labels = np.array(labels) 60 | 61 | L = 1000 62 | shape = (len(labels), L) 63 | 64 | image_to_label = np.zeros(shape) 65 | 66 | for i in range(L): 67 | image_to_label[labels == i, i] = 1 68 | else: 69 | assert(False) 70 | 71 | 72 | mmap = ed.open_mmap(part='image_to_label', mode='w+', dtype=bool, 73 | shape=shape) 74 | mmap[:] = image_to_label[:] 75 | ed.finish_mmap(mmap) 76 | f = ed.mmap_filename(part='image_to_label') 77 | 78 | print('Finished and saved index_to_label at %s' % f) 79 | 80 | 81 | if __name__ == '__main__': 82 | import argparse 83 | import sys 84 | import traceback 85 | 86 | try: 87 | parser = argparse.ArgumentParser( 88 | description='Generate an image-to-label array that maps \ 89 | each image in the dataset to the labels included in it.') 90 | parser.add_argument('--directory', help='experiment directory for net probe') 91 | parser.add_argument('--ahead', type=int, default=4, 92 | help='the prefetch lookahead size') 93 | parser.add_argument('--batch_size', type=int, default=16, 94 | help='the batch size to use') 95 | args = parser.parse_args() 96 | create_image_to_label(args.directory, batch_size=args.batch_size, 97 | ahead=args.ahead) 98 | except: 99 | traceback.print_exc(file=sys.stdout) 100 | sys.exit(1) 101 | -------------------------------------------------------------------------------- /src/indexdataset.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | import loadseg 4 | import expdir 5 | 6 | 7 | DATASETS = ['pascal', 'ade20k', 'opensurfaces', 'dtd'] 8 | 9 | 10 | def get_dataset_index(fn): 11 | for i in range(len(DATASETS)): 12 | if DATASETS[i] in fn: 13 | return i 14 | 15 | def load_image_dataset_label_index(directory): 16 | ed = expdir.ExperimentDirectory(directory) 17 | info = ed.load_info() 18 | ds = loadseg.SegmentationData(info.dataset) 19 | shape = (ds.size(), len(DATASETS), len(ds.label)) 20 | return ed.open_mmap(part='image_dataset_label', mode='r', dtype=bool, 21 | shape=shape) 22 | 23 | def create_image_dataset_label_index(directory, batch_size=64, ahead=16): 24 | ed = expdir.ExperimentDirectory(directory) 25 | info = ed.load_info() 26 | ds = loadseg.SegmentationData(info.dataset) 27 | categories = ds.category_names() 28 | shape = (ds.size(), len(DATASETS), len(ds.label)) 29 | index = np.zeros(shape) 30 | pf = loadseg.SegmentationPrefetcher(ds, categories=categories, 31 | once=True, batch_size=batch_size, 32 | ahead=ahead, thread=True) 33 | batch_count = 0 34 | for batch in pf.batches(): 35 | if batch_count % 100 == 0: 36 | print('Processing batch %d ...' % batch_count) 37 | for rec in batch: 38 | dataset_index = get_dataset_index(rec['fn']) 39 | image_index = rec['i'] 40 | for cat in categories: 41 | if ((type(rec[cat]) is np.ndarray and rec[cat].size > 0) or 42 | type(rec[cat]) is list and len(rec[cat]) > 0): 43 | index[image_index][dataset_index][np.unique(rec[cat])] = True 44 | batch_count += 1 45 | 46 | mmap = ed.open_mmap(part='image_dataset_label', mode='w+', dtype=bool, 47 | shape=shape) 48 | mmap[:] = index[:] 49 | ed.finish_mmap(mmap) 50 | print('Finished and saved at %s' % ed.mmap_filename(part='image_dataset_label')) 51 | 52 | 53 | if __name__ == '__main__': 54 | import argparse 55 | import sys 56 | import traceback 57 | 58 | try: 59 | parser = argparse.ArgumentParser( 60 | description='Generate an image-dataset-label matrix index') 61 | parser.add_argument('--directory', help='experiment directory for net probe') 62 | parser.add_argument('--ahead', type=int, default=16, 63 | help='the prefetch lookahead size') 64 | parser.add_argument('--batch_size', type=int, default=64, 65 | help='the batch size to use') 66 | args = parser.parse_args() 67 | create_image_dataset_label_index(args.directory, batch_size=args.batch_size, 68 | ahead=args.ahead) 69 | except: 70 | traceback.print_exc(file=sys.stdout) 71 | sys.exit(1) 72 | 73 | -------------------------------------------------------------------------------- /src/invert.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | from torch.autograd import Variable 5 | 6 | import torchvision.transforms as transforms 7 | import torchvision.models as models 8 | 9 | import numpy as np 10 | 11 | import os 12 | 13 | import matplotlib.pyplot as plt 14 | 15 | from PIL import Image 16 | 17 | def alpha_prior(x, alpha=2.): 18 | return torch.abs(x.view(-1)**alpha).sum() 19 | 20 | 21 | def tv_norm(x, beta=2.): 22 | assert(x.size(0) == 1) 23 | img = x[0] 24 | dy = -img[:,:-1,:] + img[:,1:,:] 25 | dx = -img[:,:,:-1] + img[:,:,1:] 26 | return ((dx.pow(2) + dy.pow(2)).pow(beta/2.)).sum() 27 | 28 | 29 | def norm_loss(input, target): 30 | return torch.div(alpha_prior(input - target, alpha=2.), alpha_prior(target, alpha=2.)) 31 | 32 | 33 | class Denormalize(object): 34 | def __init__(self, mean, std): 35 | self.mean = mean 36 | self.std = std 37 | 38 | def __call__(self, tensor): 39 | for t, m, s in zip(tensor, self.mean, self.std): 40 | t.mul_(s).add_(m) 41 | return tensor 42 | 43 | 44 | class Clip(object): 45 | def __init__(self): 46 | return 47 | 48 | def __call__(self, tensor): 49 | t = tensor.clone() 50 | t[t>1] = 1 51 | t[t<0] = 0 52 | return t 53 | 54 | 55 | #function to decay the learning rate 56 | def decay_lr(optimizer, factor): 57 | for param_group in optimizer.param_groups: 58 | param_group['lr'] *= factor 59 | 60 | 61 | def get_pytorch_module(net, blob): 62 | modules = blob.split('.') 63 | if len(modules) == 1: 64 | return net._modules.get(blob) 65 | else: 66 | curr_m = net 67 | for m in modules: 68 | curr_m = curr_m._modules.get(m) 69 | return curr_m 70 | 71 | 72 | def invert(image, network='alexnet', size=227, layer='features.4', alpha=6, beta=2, 73 | alpha_lambda=1e-5, tv_lambda=1e-5, epochs=200, learning_rate=1e2, 74 | momentum=0.9, decay_iter=100, decay_factor=1e-1, print_iter=25, 75 | cuda=False): 76 | 77 | mu = [0.485, 0.456, 0.406] 78 | sigma = [0.229, 0.224, 0.225] 79 | 80 | transform = transforms.Compose([ 81 | transforms.Scale(size=size), 82 | transforms.CenterCrop(size=size), 83 | transforms.ToTensor(), 84 | transforms.Normalize(mu, sigma), 85 | ]) 86 | 87 | detransform = transforms.Compose([ 88 | Denormalize(mu, sigma), 89 | Clip(), 90 | transforms.ToPILImage(), 91 | ]) 92 | 93 | model = models.__dict__[network](pretrained=True) 94 | model.eval() 95 | if cuda: 96 | model.cuda() 97 | 98 | img_ = transform(Image.open(image)).unsqueeze(0) 99 | 100 | activations = [] 101 | 102 | def hook_acts(module, input, output): 103 | activations.append(output) 104 | 105 | def get_acts(model, input): 106 | del activations[:] 107 | _ = model(input) 108 | assert(len(activations) == 1) 109 | return activations[0] 110 | 111 | _ = get_pytorch_module(model, layer).register_forward_hook(hook_acts) 112 | input_var = Variable(img_.cuda() if cuda else img_) 113 | ref_acts = get_acts(model, input_var).detach() 114 | 115 | x_ = Variable((1e-3 * torch.randn(*img_.size()).cuda() if cuda else 116 | 1e-3 * torch.randn(*img_.size())), requires_grad=True) 117 | 118 | alpha_f = lambda x: alpha_prior(x, alpha=alpha) 119 | tv_f = lambda x: tv_norm(x, beta=beta) 120 | loss_f = lambda x: norm_loss(x, ref_acts) 121 | 122 | optimizer = torch.optim.SGD([x_], lr=learning_rate, momentum=momentum) 123 | 124 | for i in range(epochs): 125 | acts = get_acts(model, x_) 126 | 127 | alpha_term = alpha_f(x_) 128 | tv_term = tv_f(x_) 129 | loss_term = loss_f(acts) 130 | 131 | tot_loss = alpha_lambda*alpha_term + tv_lambda*tv_term + loss_term 132 | 133 | if (i+1) % print_iter == 0: 134 | print('Epoch %d:\tAlpha: %f\tTV: %f\tLoss: %f\tTot Loss: %f' % (i+1, 135 | alpha_term.data.cpu().numpy()[0], tv_term.data.cpu().numpy()[0], 136 | loss_term.data.cpu().numpy()[0], tot_loss.data.cpu().numpy()[0])) 137 | 138 | optimizer.zero_grad() 139 | tot_loss.backward() 140 | optimizer.step() 141 | 142 | if (i+1) % decay_iter == 0: 143 | decay_lr(optimizer, decay_factor) 144 | 145 | f, ax = plt.subplots(1,2) 146 | ax[0].imshow(detransform(img_[0])) 147 | ax[1].imshow(detransform(x_[0].data.cpu())) 148 | for a in ax: 149 | a.set_xticks([]) 150 | a.set_yticks([]) 151 | plt.show() 152 | 153 | 154 | if __name__ == '__main__': 155 | import argparse 156 | import sys 157 | import traceback 158 | 159 | try: 160 | parser = argparse.ArgumentParser() 161 | parser.add_argument('--image', type=str, 162 | default='grace_hopper.jpg') 163 | parser.add_argument('--network', type=str, default='alexnet') 164 | parser.add_argument('--size', type=int, default=227) 165 | parser.add_argument('--layer', type=str, default='features.4') 166 | parser.add_argument('--alpha', type=float, default=6.) 167 | parser.add_argument('--beta', type=float, default=2.) 168 | parser.add_argument('--alpha_lambda', type=float, default=1e-5) 169 | parser.add_argument('--tv_lambda', type=float, default=1e-5) 170 | parser.add_argument('--epochs', type=int, default=200) 171 | parser.add_argument('--learning_rate', type=int, default=1e2) 172 | parser.add_argument('--momentum', type=float, default=0.9) 173 | parser.add_argument('--print_iter', type=int, default=25) 174 | parser.add_argument('--decay_iter', type=int, default=100) 175 | parser.add_argument('--decay_factor', type=float, default=1e-1) 176 | parser.add_argument('--gpu', type=int, nargs='*', default=None) 177 | 178 | args = parser.parse_args() 179 | 180 | gpu = args.gpu 181 | cuda = True if gpu is not None else False 182 | use_mult_gpu = isinstance(gpu, list) 183 | if cuda: 184 | if use_mult_gpu: 185 | os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu).strip('[').strip(']') 186 | else: 187 | os.environ['CUDA_VISIBLE_DEVICES'] = '%d' % gpu 188 | print(torch.cuda.device_count(), use_mult_gpu, cuda) 189 | 190 | invert(image=args.image, network=args.network, layer=args.layer, 191 | alpha=args.alpha, beta=args.beta, alpha_lambda=args.alpha_lambda, 192 | tv_lambda=args.tv_lambda, epochs=args.epochs, 193 | learning_rate=args.learning_rate, momentum=args.momentum, 194 | print_iter=args.print_iter, decay_iter=args.decay_iter, 195 | decay_factor=args.decay_factor, cuda=cuda) 196 | except: 197 | traceback.print_exc(file=sys.stdout) 198 | sys.exit(1) 199 | 200 | 201 | -------------------------------------------------------------------------------- /src/labelprobe_pytorch.py: -------------------------------------------------------------------------------- 1 | import loadseg 2 | import expdir 3 | 4 | from labelprobe import cached_memmap 5 | from indexdata import has_image_to_label, load_image_to_label, create_image_to_label 6 | 7 | import os 8 | import time 9 | 10 | import numpy as np 11 | 12 | import torch 13 | import torch.nn as nn 14 | from torch.autograd import Variable 15 | 16 | 17 | def iou_intersect_d(input, target, threshold = 0.5): 18 | return torch.sum(torch.sum(torch.mul((input > threshold).float(), target), dim=-1), 19 | dim=-1) 20 | 21 | 22 | def iou_union_d(input, target, threshold = 0.5): 23 | return torch.sum(torch.sum(torch.clamp(torch.add(target, (input > threshold).float()), 24 | max=1.), dim=-1), dim=-1) 25 | 26 | 27 | def get_seg_size(input_dim): 28 | if input_dim == [227, 227] or (input_dim[0] == 227 and input_dim[1] == 227): 29 | seg_size = (113, 113) 30 | elif input_dim == [224, 224] or (input_dim[0] == 224 and input_dim[1] == 224): 31 | seg_size = (112, 112) 32 | elif input_dim == [384, 384] or (input_dim[0] == 384 and input_dim[1] == 384): 33 | seg_size = (192, 192) 34 | else: 35 | print input_dim 36 | assert(False) 37 | return seg_size 38 | 39 | 40 | def label_probe(directory, blob, quantile=0.005, batch_size=16, ahead=4, start=None, 41 | end=None, suffix='', cuda=False): 42 | # Make sure we have a directory to work in 43 | qcode = ('%f' % quantile).replace('0.','').rstrip('0') 44 | ed = expdir.ExperimentDirectory(directory) 45 | # Check if label probe has already been created 46 | if (ed.has_mmap(blob=blob, part='single_set_ious%s' % suffix) and 47 | ed.has_mmap(blob=blob, part='single_ind_ious%s' % suffix)): 48 | print('label_probe_pytorch.py has already been run.') 49 | return 50 | # Load probe metadata 51 | info = ed.load_info() 52 | seg_size = get_seg_size(info.input_dim) 53 | # Load blob metadata 54 | blob_info = ed.load_info(blob=blob) 55 | shape = blob_info.shape 56 | tot_imgs = shape[0] 57 | unit_size = shape[1] 58 | # Load the blob quantile data and grab thresholds 59 | quantdata = ed.open_mmap(blob=blob, part='quant-*', shape=(unit_size, -1)) 60 | threshold = quantdata[:, int(round(quantdata.shape[1] * quantile))] 61 | thresh = threshold[:, np.newaxis, np.newaxis] 62 | # Load the dataset 63 | ds = loadseg.SegmentationData(info.dataset) 64 | # Map the blob activation data for reading 65 | #fn_read = ed.mmap_filename(blob=blob) 66 | #blobdata = cached_memmap(fn_read, mode='r', dtype='float32', shape=shape) 67 | blobdata = ed.open_mmap(blob=blob, mode='r', shape=shape) 68 | # Get image-to-labels mapping 69 | if not has_image_to_label(directory): 70 | print('image_to_label does not exist in %s; creating it now...' % directory) 71 | create_image_to_label(directory, batch_size=batch_size, ahead=ahead) 72 | image_to_label = load_image_to_label(directory) 73 | 74 | num_labels = ds.label_size() 75 | upsample = nn.Upsample(size=seg_size, mode='bilinear') 76 | 77 | set_ious_train_mmap = ed.open_mmap(blob=blob, part='single_set_train_ious%s' % suffix, 78 | mode='w+', dtype='float32', shape=(num_labels, unit_size)) 79 | set_ious_val_mmap = ed.open_mmap(blob=blob, part='single_set_val_ious%s' % suffix, 80 | mode='w+', dtype='float32', shape=(num_labels, unit_size)) 81 | set_ious_mmap = ed.open_mmap(blob=blob, part='single_set_ious%s' % suffix, mode='w+', 82 | dtype='float32', shape=(num_labels, unit_size)) 83 | ind_ious_mmap = ed.open_mmap(blob=blob, part='single_ind_ious%s' % suffix, mode='w+', 84 | dtype='float32', shape=(num_labels, tot_imgs, unit_size)) 85 | 86 | if start is None: 87 | start = 1 88 | if end is None: 89 | end = num_labels 90 | #for label_i in range(1, num_labels): 91 | for label_i in range(start, end): 92 | print('Starting for label %d (%s)' % (label_i, ds.name(category=None, 93 | j=label_i))) 94 | label_categories = ds.label[label_i]['category'].keys() 95 | num_cats = len(label_categories) 96 | label_idx = np.where(image_to_label[:, label_i])[0] 97 | loader = loadseg.SegmentationPrefetcher(ds, categories=label_categories, 98 | indexes=label_idx, once=False, batch_size=batch_size, 99 | ahead=ahead, thread=True) 100 | loader_idx = loader.indexes 101 | N = len(loader_idx) 102 | iou_intersects = np.zeros((N, unit_size)) 103 | iou_unions = np.zeros((N, unit_size)) 104 | 105 | if num_cats > 1: 106 | rec_labcat = [] 107 | for batch in loader.batches(): 108 | for rec in batch: 109 | for cat in label_categories: 110 | if rec[cat] != []: 111 | rec_labcat.append(cat) 112 | break 113 | else: 114 | rec_labcat = [label_categories[0] for i in range(N)] 115 | 116 | 117 | i = 0 118 | for batch in loader.batches(): 119 | start_t = time.time() 120 | if (i+1)*batch_size < N: 121 | idx = range(i*batch_size, (i+1)*batch_size) 122 | else: 123 | idx = range(i*batch_size, N) 124 | i += 1 125 | input = torch.Tensor((blobdata[loader_idx[idx]] > thresh).astype(float)) 126 | input_var = upsample(Variable(input.cuda()) if cuda else 127 | Variable(input)) 128 | target = torch.Tensor([np.max((rec[rec_labcat[j]] 129 | == label_i).astype(float), axis=0) 130 | if type(rec[rec_labcat[j]]) is np.ndarray 131 | else np.ones(seg_size) for j, rec in enumerate(batch)]) 132 | target_var = Variable(target.unsqueeze(1).expand_as( 133 | input_var).cuda() if cuda 134 | else target.unsqueeze(1).expand_as(input_var)) 135 | iou_intersects[idx] = np.squeeze(iou_intersect_d(input_var, 136 | target_var).data.cpu().numpy()) 137 | iou_unions[idx] = np.squeeze(iou_union_d(input_var, 138 | target_var).data.cpu().numpy()) 139 | print('Batch %d/%d\tTime %f secs\tAvg Ind IOU %f\t' % (i, N/batch_size, 140 | time.time()-start_t, np.mean(np.true_divide(iou_intersects[idx], 141 | iou_unions[idx] + 1e-20)))) 142 | 143 | set_ious = np.true_divide(np.sum(iou_intersects, axis=0), 144 | np.sum(iou_unions, axis=0) + 1e-20) 145 | loader.close() 146 | best_filter = np.argmax(set_ious) 147 | print('Label %d (%s): best set IOU = %f (filter %d)' % (label_i, 148 | ds.name(category=None,j=label_i), set_ious[best_filter], best_filter)) 149 | ind_ious = np.true_divide(iou_intersects, iou_unions + 1e-20) 150 | 151 | set_ious_mmap[label_i] = set_ious 152 | ind_ious_mmap[label_i, loader_idx] = ind_ious 153 | train_idx = [i for i in range(len(loader_idx)) if ds.split(loader_idx[i]) == 'train'] 154 | val_idx = [i for i in range(len(loader_idx)) if ds.split(loader_idx[i]) == 'val'] 155 | set_ious_train_mmap[label_i] = np.true_divide(np.sum(iou_intersects[train_idx], axis=0), 156 | np.sum(iou_unions[train_idx], axis=0) + 1e-20) 157 | set_ious_val_mmap[label_i] = np.true_divide(np.sum(iou_intersects[val_idx], axis=0), 158 | np.sum(iou_unions[val_idx], axis=0) + 1e-20) 159 | 160 | #set_ious_mmap.flush() 161 | #ind_ious_mmap.flush() 162 | 163 | ed.finish_mmap(set_ious_train_mmap) 164 | ed.finish_mmap(set_ious_val_mmap) 165 | ed.finish_mmap(set_ious_mmap) 166 | ed.finish_mmap(ind_ious_mmap) 167 | 168 | 169 | if __name__ == '__main__': 170 | import argparse 171 | import sys 172 | import traceback 173 | try: 174 | parser = argparse.ArgumentParser( 175 | description='Compute set IOUs for single filters') 176 | 177 | parser.add_argument('--directory', default='.', 178 | help='output directory for the net probe') 179 | parser.add_argument('--blobs', nargs='*', 180 | help='network blob names to tally') 181 | parser.add_argument('--quantile', type=float, default=0.005, 182 | help='the quantile cutoff to use') 183 | parser.add_argument('--batch_size', default=16, type=int, 184 | help='the batch size to use') 185 | parser.add_argument('--gpu', type=int, default=None, 186 | help='use GPU for training') 187 | parser.add_argument('--suffix', type=str, default='') 188 | parser.add_argument('--start', type=int, default=1) 189 | parser.add_argument('--end', type=int, default=1198) 190 | 191 | args = parser.parse_args() 192 | 193 | gpu = args.gpu 194 | cuda = True if gpu is not None else False 195 | use_mult_gpu = isinstance(gpu, list) 196 | if cuda: 197 | if use_mult_gpu: 198 | os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu).strip('[').strip(']') 199 | else: 200 | os.environ['CUDA_VISIBLE_DEVICES'] = '%d' % gpu 201 | print torch.cuda.device_count(), use_mult_gpu, cuda 202 | 203 | for blob in args.blobs: 204 | label_probe(args.directory, blob, quantile=args.quantile, 205 | batch_size=args.batch_size, start=args.start, 206 | end=args.end, suffix=args.suffix, cuda=cuda) 207 | except: 208 | traceback.print_exc(file=sys.stdout) 209 | sys.exit(1) 210 | -------------------------------------------------------------------------------- /src/linearmaxprobe.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | import re 4 | import shutil 5 | import time 6 | import expdir 7 | import sys 8 | from scipy.io import savemat 9 | import pickle as pkl 10 | 11 | import loadseg 12 | 13 | def max_probe(directory, blob, batch_size=None, quantile=0.005, results=None, num_components=None, 14 | use_y_weights=False, suffix='', new_suffix='', normalize=True, pool='max_pool', should_thresh=False, disc=False): 15 | # Make sure we have a directory to work in 16 | ed = expdir.ExperimentDirectory(directory) 17 | 18 | # If it's already computed, then skip it!!! 19 | if disc: 20 | suffix = '_disc%s' % suffix 21 | print "Checking", ed.mmap_filename(blob=blob, part='linear_imgmax%s%s' % (suffix, new_suffix)) 22 | if ed.has_mmap(blob=blob, part='linear_imgmax%s%s' % (suffix, new_suffix)): 23 | print "Already have %s-imgmax.mmap, skipping." % (blob) 24 | return 25 | 26 | info = ed.load_info() 27 | ds = loadseg.SegmentationData(info.dataset) 28 | 29 | # Read about the blob shape 30 | blob_info = ed.load_info(blob=blob) 31 | shape = blob_info.shape 32 | N = shape[0] 33 | K = shape[1] 34 | L = ds.label_size() 35 | 36 | if should_thresh: 37 | if quantile == 1: 38 | thresh = np.zeros((1,1,K,1,1)) 39 | else: 40 | quantdata = ed.open_mmap(blob=blob, part='quant-*', shape=(K, -1)) 41 | threshold = quantdata[:, int(round(quantdata.shape[1] * quantile))] 42 | thresh = threshold[numpy.newaxis,numpy.newaxis,:,numpy.newaxis,numpy.newaxis] 43 | 44 | if use_y_weights: 45 | shape=(N,L,113,113) # TODO: don't hardcode segmentation size, this only works for alexnet 46 | data = ed.open_mmap(part='concept_data', mode='r', shape=shape) 47 | else: 48 | data = ed.open_mmap(blob=blob, shape=shape) 49 | print 'Computing imgmax for %s shape %r' % (blob, shape) 50 | if results is not None: 51 | imgmax = ed.open_mmap(blob=blob, part='linear_imgmax%s%s' % (suffix, new_suffix), 52 | mode='w+', shape=(N,num_components)) 53 | if use_y_weights: 54 | all_weights = pkl.load(open(results, 'rb'))['model'].y_weights_.T 55 | else: 56 | all_weights = pkl.load(open(results, 'rb'))['model'].x_weights_.T 57 | else: 58 | imgmax = ed.open_mmap(blob=blob, part='linear_imgmax%s%s' % (suffix, new_suffix), 59 | mode='w+', shape=(N,L)) 60 | if disc: 61 | assert(ed.has_mmap(blob=blob, part='linear_weights%s' % suffix)) 62 | all_weights = ed.open_mmap(blob=blob, part='linear_weights%s' % suffix, 63 | mode='r', dtype='float32', shape=(L,2,K)) 64 | all_weights = all_weights[:,-1,:] 65 | else: 66 | assert(ed.has_mmap(blob=blob, part='linear_weights%s' % suffix)) 67 | all_weights = ed.open_mmap(blob=blob, part='linear_weights%s' % suffix, 68 | mode='r', dtype='float32', shape=(L,K)) 69 | if normalize: 70 | all_weights = numpy.array([numpy.true_divide(all_weights[i], numpy.linalg.norm(all_weights[i])) for i in range(L)]) 71 | 72 | # Automatic batch size selection: 64mb batches 73 | if batch_size is None: 74 | batch_size = max(1, int(128 * 1024 * 1024 / numpy.prod(shape[1:]))) 75 | 76 | # Algorithm: one pass over the data 77 | start_time = time.time() 78 | last_batch_time = start_time 79 | for i in range(0, data.shape[0], batch_size): 80 | batch_time = time.time() 81 | rate = i / (batch_time - start_time + 1e-15) 82 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 83 | last_batch_time = batch_time 84 | print 'Imgmax %s index %d: %f %f' % (blob, i, rate, batch_rate) 85 | sys.stdout.flush() 86 | batch = data[i:i+batch_size][:,numpy.newaxis,:,:,:] # (batch_size, L, K, S, S) 87 | if should_thresh: 88 | batch = (batch > thresh).astype(float) 89 | if pool == 'max_pool': 90 | imgmax[i:i+batch_size,:] = (batch * all_weights[numpy.newaxis,:,:,numpy.newaxis,numpy.newaxis]).sum(axis=2).max(axis=(2,3)) 91 | elif pool == 'avg_pool': 92 | imgmax[i:i+batch_size,:] = (batch * all_weights[numpy.newaxis,:,:,numpy.newaxis,numpy.newaxis]).sum(axis=2).mean(axis=(2,3)) 93 | print 'Writing imgmax' 94 | sys.stdout.flush() 95 | # Save as mat file 96 | filename = ed.filename('linear_imgmax%s.mat' % suffix, blob=blob) 97 | savemat(filename, { 'linear_imgmax': imgmax }) 98 | # And as mmap 99 | ed.finish_mmap(imgmax) 100 | 101 | 102 | if __name__ == '__main__': 103 | import argparse 104 | import sys 105 | import traceback 106 | 107 | try: 108 | import loadseg 109 | 110 | parser = argparse.ArgumentParser( 111 | description='Generate sorted files for probed activation data.') 112 | parser.add_argument( 113 | '--directory', 114 | default='.', 115 | help='output directory for the net probe') 116 | parser.add_argument( 117 | '--blobs', 118 | nargs='*', 119 | help='network blob names to sort') 120 | parser.add_argument( 121 | '--batch_size', 122 | type=int, default=None, 123 | help='the batch size to use') 124 | parser.add_argument( 125 | '--normalize', 126 | action='store_true', 127 | default=False) 128 | parser.add_argument( 129 | '--results', 130 | type=str, 131 | default=None) 132 | parser.add_argument( 133 | '--num_components', 134 | type=int, 135 | default=None) 136 | parser.add_argument( 137 | '--use_y_weights', 138 | action='store_true', 139 | default=False) 140 | parser.add_argument( 141 | '--disc', 142 | action='store_true', 143 | default=False) 144 | parser.add_argument( 145 | '--thresh', 146 | action='store_true', 147 | default=False) 148 | parser.add_argument( 149 | '--quantile', 150 | type=float, 151 | default=0.005) 152 | parser.add_argument( 153 | '--suffix', 154 | default='', 155 | type=str) 156 | parser.add_argument( 157 | '--new_suffix', 158 | default='', 159 | type=str) 160 | parser.add_argument( 161 | '--pool', 162 | default='max_pool', 163 | type=str) 164 | 165 | args = parser.parse_args() 166 | for blob in args.blobs: 167 | max_probe(args.directory, blob, args.batch_size, suffix=args.suffix, 168 | normalize=args.normalize, results=args.results, num_components=args.num_components, 169 | use_y_weights=args.use_y_weights, disc=args.disc, quantile=args.quantile, 170 | should_thresh=args.thresh, new_suffix=args.new_suffix, 171 | pool=args.pool) 172 | except: 173 | traceback.print_exc(file=sys.stdout) 174 | sys.exit(1) 175 | -------------------------------------------------------------------------------- /src/maxprobe.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | import re 4 | import shutil 5 | import time 6 | import expdir 7 | import sys 8 | from scipy.io import savemat 9 | 10 | def max_probe(directory, blob, batch_size=None): 11 | # Make sure we have a directory to work in 12 | ed = expdir.ExperimentDirectory(directory) 13 | 14 | # If it's already computed, then skip it!!! 15 | print "Checking", ed.mmap_filename(blob=blob, part='imgmax') 16 | if ed.has_mmap(blob=blob, part='imgmax'): 17 | print "Already have %s-imgmax.mmap, skipping." % (blob) 18 | return 19 | 20 | # Read about the blob shape 21 | blob_info = ed.load_info(blob=blob) 22 | shape = blob_info.shape 23 | 24 | print 'Computing imgmax for %s shape %r' % (blob, shape) 25 | data = ed.open_mmap(blob=blob, shape=shape) 26 | imgmax = ed.open_mmap(blob=blob, part='imgmax', 27 | mode='w+', shape=data.shape[:2]) 28 | 29 | # Automatic batch size selection: 64mb batches 30 | if batch_size is None: 31 | batch_size = max(1, int(128 * 1024 * 1024 / numpy.prod(shape[1:]))) 32 | 33 | # Algorithm: one pass over the data 34 | start_time = time.time() 35 | last_batch_time = start_time 36 | for i in range(0, data.shape[0], batch_size): 37 | batch_time = time.time() 38 | rate = i / (batch_time - start_time + 1e-15) 39 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 40 | last_batch_time = batch_time 41 | print 'Imgmax %s index %d: %f %f' % (blob, i, rate, batch_rate) 42 | sys.stdout.flush() 43 | batch = data[i:i+batch_size] 44 | imgmax[i:i+batch_size] = batch.max(axis=(2,3)) 45 | print 'Writing imgmax' 46 | sys.stdout.flush() 47 | # Save as mat file 48 | filename = ed.filename('imgmax.mat', blob=blob) 49 | savemat(filename, { 'imgmax': imgmax }) 50 | # And as mmap 51 | ed.finish_mmap(imgmax) 52 | 53 | 54 | if __name__ == '__main__': 55 | import argparse 56 | import sys 57 | import traceback 58 | 59 | try: 60 | import loadseg 61 | 62 | parser = argparse.ArgumentParser( 63 | description='Generate sorted files for probed activation data.') 64 | parser.add_argument( 65 | '--directory', 66 | default='.', 67 | help='output directory for the net probe') 68 | parser.add_argument( 69 | '--blobs', 70 | nargs='*', 71 | help='network blob names to sort') 72 | parser.add_argument( 73 | '--batch_size', 74 | type=int, default=None, 75 | help='the batch size to use') 76 | args = parser.parse_args() 77 | for blob in args.blobs: 78 | max_probe(args.directory, blob, args.batch_size) 79 | except: 80 | traceback.print_exc(file=sys.stdout) 81 | sys.exit(1) 82 | -------------------------------------------------------------------------------- /src/multi_filter_multi_concept.py: -------------------------------------------------------------------------------- 1 | import expdir 2 | import loadseg 3 | import time 4 | 5 | import numpy as np 6 | import pickle as pkl 7 | 8 | from sklearn.cross_decomposition import CCA 9 | from upsample_blob_data import get_seg_size 10 | 11 | 12 | def compute_correlation(directory, blob, num_samples=None, num_components=1, out_file=None, 13 | verbose=False): 14 | ed = expdir.ExperimentDirectory(directory) 15 | 16 | info = ed.load_info() 17 | ds = loadseg.SegmentationData(info.dataset) 18 | 19 | L = ds.label_size() 20 | N = ds.size() 21 | 22 | blob_info = ed.load_info(blob=blob) 23 | shape = blob_info.shape 24 | K = shape[1] 25 | 26 | categories = np.array(ds.category_names()) 27 | label_names = np.array([ds.label[i]['name'] for i in range(L)]) 28 | 29 | (Hs, Ws) = get_seg_size(info.input_dim) 30 | 31 | if verbose: 32 | start = time.time() 33 | print 'Loading data...' 34 | upsampled_data = ed.open_mmap(blob=blob, part='upsampled', mode='r', 35 | shape=(N,K,Hs,Ws)) 36 | concept_data = ed.open_mmap(part='concept_data', mode='r', 37 | shape=(N,L,Hs,Ws)) 38 | if verbose: 39 | print 'Finished loading data in %d secs.' % (time.time() - start) 40 | 41 | if verbose: 42 | start = time.time() 43 | print 'Selecting data...' 44 | 45 | if num_samples is not None: 46 | rand_idx = np.random.choice(N, num_samples, replace=False) 47 | X = upsampled_data[rand_idx,:,Hs/2,Ws/2] 48 | Y = concept_data[rand_idx,:,Hs/2,Ws/2] 49 | else: 50 | X = upsampled_data[:,:,Hs/2,Ws/2] 51 | Y = concept_data[:,:,Hs/2,Ws/2] 52 | 53 | if verbose: 54 | print 'Finished selecting data in %d secs.' % (time.time() - start) 55 | 56 | cca = CCA(n_components=num_components) 57 | 58 | if verbose: 59 | start = time.time() 60 | if num_samples is None: 61 | num_samples = N 62 | print 'Fitting %d-component CCA with N = %d samples...' % (num_components, num_samples) 63 | cca.fit(X,Y) 64 | if verbose: 65 | print 'Fitted %d-component CCA with N = %d samples in %d secs.' % (num_components, 66 | num_samples, time.time() - start) 67 | 68 | X_c, Y_c = cca.transform(X,Y) 69 | score = cca.score(X,Y) 70 | 71 | results = {} 72 | if out_file is not None: 73 | if verbose: 74 | start = time.time() 75 | print 'Saving results...' 76 | results['model'] = cca 77 | try: 78 | results['idx'] = rand_idx 79 | except: 80 | results['idx'] = None 81 | results['directory'] = directory 82 | results['blob'] = blob 83 | results['num_samples'] = num_samples 84 | results['num_components'] = num_components 85 | results['score'] = score 86 | 87 | pkl.dump(results, open(out_file, 'wb')) 88 | if verbose: 89 | print 'Saved results at %s in %d secs.' % (out_file, time.time() - start) 90 | 91 | return results 92 | 93 | 94 | if __name__ == '__main__': 95 | import sys 96 | import argparse 97 | import traceback 98 | 99 | try: 100 | parser = argparse.ArgumentParser(description='TODO') 101 | parser.add_argument('--directory', 102 | help='TODO') 103 | parser.add_argument('--blob', 104 | type=str, 105 | help='TODO') 106 | parser.add_argument('--num_samples', 107 | type=int, 108 | default=None, 109 | help='TODO') 110 | parser.add_argument('--num_components', 111 | type=int, 112 | default=1, 113 | help='TODO') 114 | parser.add_argument('--out_file', 115 | type=str, 116 | default=None, 117 | help='TODO') 118 | 119 | args = parser.parse_args() 120 | compute_correlation(args.directory, args.blob, num_samples=args.num_samples, 121 | num_components=args.num_components, out_file=args.out_file, verbose=True) 122 | except: 123 | traceback.print_exc(file=sys.stdout) 124 | sys.exit(1) 125 | -------------------------------------------------------------------------------- /src/netprobe.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import os 4 | import numpy 5 | import glob 6 | import shutil 7 | import codecs 8 | import time 9 | import sys 10 | 11 | os.environ['GLOG_minloglevel'] = '2' 12 | import caffe 13 | from caffe.proto import caffe_pb2 14 | from google.protobuf import text_format 15 | from scipy.misc import imresize, imread 16 | from scipy.ndimage.filters import gaussian_filter 17 | from scipy.ndimage.interpolation import zoom 18 | from tempfile import NamedTemporaryFile 19 | from contextlib import contextmanager 20 | from collections import namedtuple 21 | import upsample 22 | import rotate 23 | import expdir 24 | 25 | caffe.set_mode_gpu() 26 | caffe.set_device(0) 27 | 28 | def create_probe( 29 | directory, dataset, definition, weights, mean, blobs, 30 | colordepth=3, 31 | rotation_seed=None, rotation_power=1, rotation_unpermute=False, 32 | limit=None, split=None, 33 | batch_size=16, ahead=4, 34 | cl_args=None, verbose=True): 35 | # If we're already done, skip it! 36 | ed = expdir.ExperimentDirectory(directory) 37 | if all(ed.has_mmap(blob=b) for b in blobs): 38 | return 39 | 40 | ''' 41 | directory: where to place the probe_conv5.mmap files. 42 | data: the AbstractSegmentation data source to draw upon 43 | definition: the filename for the caffe prototxt 44 | weights: the filename for the caffe model weights 45 | mean: to use to normalize rgb values for the network 46 | blobs: ['conv3', 'conv4', 'conv5'] to probe 47 | ''' 48 | if verbose: 49 | print 'Opening dataset', dataset 50 | data = loadseg.SegmentationData(args.dataset) 51 | if verbose: 52 | print 'Opening network', definition 53 | np = caffe_pb2.NetParameter() 54 | with open(definition, 'r') as dfn_file: 55 | text_format.Merge(dfn_file.read(), np) 56 | net = caffe.Net(definition, weights, caffe.TEST) 57 | input_blob = net.inputs[0] 58 | input_dim = net.blobs[input_blob].data.shape[2:] 59 | data_size = data.size(split) 60 | if limit is not None: 61 | data_size = min(data_size, limit) 62 | 63 | # Make sure we have a directory to work in 64 | ed.ensure_dir() 65 | 66 | # Step 0: write a README file with generated information. 67 | ed.save_info(dict( 68 | dataset=dataset, 69 | split=split, 70 | definition=definition, 71 | weights=weights, 72 | mean=mean, 73 | blobs=blobs, 74 | input_dim=input_dim, 75 | rotation_seed=rotation_seed, 76 | rotation_power=rotation_power)) 77 | 78 | # Clear old probe data 79 | ed.remove_all('*.mmap*') 80 | 81 | # Create new (empty) mmaps 82 | if verbose: 83 | print 'Creating new mmaps.' 84 | out = {} 85 | rot = None 86 | if rotation_seed is not None: 87 | rot = {} 88 | for blob in blobs: 89 | shape = (data_size, ) + net.blobs[blob].data.shape[1:] 90 | out[blob] = ed.open_mmap(blob=blob, mode='w+', shape=shape) 91 | # Find the shortest path through the network to the target blob 92 | fieldmap, _ = upsample.composed_fieldmap(np.layer, blob) 93 | # Compute random rotation for each blob, if needed 94 | if rot is not None: 95 | rot[blob] = rotate.randomRotationPowers( 96 | shape[1], [rotation_power], rotation_seed, 97 | unpermute=rotation_unpermute)[0] 98 | ed.save_info(blob=blob, data=dict( 99 | name=blob, shape=shape, fieldmap=fieldmap)) 100 | 101 | # The main loop 102 | if verbose: 103 | print 'Beginning work.' 104 | pf = loadseg.SegmentationPrefetcher(data, categories=['image'], 105 | split=split, once=True, batch_size=batch_size, ahead=ahead) 106 | index = 0 107 | start_time = time.time() 108 | last_batch_time = start_time 109 | batch_size = 0 110 | for batch in pf.tensor_batches(bgr_mean=mean): 111 | batch_time = time.time() 112 | rate = index / (batch_time - start_time + 1e-15) 113 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 114 | last_batch_time = batch_time 115 | if verbose: 116 | print 'netprobe index', index, 'items per sec', batch_rate, rate 117 | sys.stdout.flush() 118 | inp = batch[0] 119 | batch_size = len(inp) 120 | if limit is not None and index + batch_size > limit: 121 | # Truncate last if limited 122 | batch_size = limit - index 123 | inp = inp[:batch_size] 124 | if colordepth == 1: 125 | inp = numpy.mean(inp, axis=1, keepdims=True) 126 | net.blobs[input_blob].reshape(*(inp.shape)) 127 | net.blobs[input_blob].data[...] = inp 128 | result = net.forward(blobs=blobs) 129 | if rot is not None: 130 | for key in out.keys(): 131 | result[key] = numpy.swapaxes(numpy.tensordot( 132 | rot[key], result[key], axes=((1,), (1,))), 0, 1) 133 | # print 'Computation done' 134 | for key in out.keys(): 135 | out[key][index:index + batch_size] = result[key] 136 | # print 'Recording data in mmap done' 137 | index += batch_size 138 | if index >= data_size: 139 | break 140 | assert index == data_size, ( 141 | "Data source should return evey item once %d %d." % 142 | (index, data_size)) 143 | if verbose: 144 | print 'Renaming mmaps.' 145 | for blob in blobs: 146 | ed.finish_mmap(out[blob]) 147 | 148 | # Final step: write the README file 149 | write_readme_file([ 150 | ('cl_args', cl_args), 151 | ('data', data), 152 | ('definition', definition), 153 | ('weight', weights), 154 | ('mean', mean), 155 | ('blobs', blobs)], ed, verbose=verbose) 156 | 157 | 158 | def ensure_dir(targetdir): 159 | if not os.path.isdir(targetdir): 160 | try: 161 | os.makedirs(targetdir) 162 | except: 163 | print 'Could not create', targetdir 164 | pass 165 | 166 | def write_readme_file(args, ed, verbose): 167 | ''' 168 | Writes a README.txt that describes the settings used to geenrate the ds. 169 | ''' 170 | with codecs.open(ed.filename('README.txt'), 'w', 'utf-8') as f: 171 | def report(txt): 172 | f.write('%s\n' % txt) 173 | if verbose: 174 | print txt 175 | title = '%s network probe' % ed.basename() 176 | report('%s\n%s' % (title, '=' * len(title))) 177 | for key, val in args: 178 | if key == 'cl_args': 179 | if val is not None: 180 | report('Command-line args:') 181 | for ck, cv in vars(val).items(): 182 | report(' %s: %r' % (ck, cv)) 183 | report('%s: %r' % (key, val)) 184 | report('\ngenerated at: %s' % time.strftime("%Y-%m-%d %H:%M")) 185 | try: 186 | label = subprocess.check_output(['git', 'rev-parse', 'HEAD']) 187 | report('git label: %s' % label) 188 | except: 189 | pass 190 | 191 | if __name__ == '__main__': 192 | import sys 193 | import traceback 194 | import argparse 195 | try: 196 | import loadseg 197 | 198 | parser = argparse.ArgumentParser(description= 199 | 'Probe a caffe network and save results in a directory.') 200 | parser.add_argument( 201 | '--directory', 202 | default='.', 203 | help='output directory for the net probe') 204 | parser.add_argument( 205 | '--blobs', 206 | nargs='*', 207 | help='network blob names to collect') 208 | parser.add_argument( 209 | '--definition', 210 | help='the deploy prototext defining the net') 211 | parser.add_argument( 212 | '--weights', 213 | help='the caffemodel file of weights for the net') 214 | parser.add_argument( 215 | '--mean', 216 | nargs='*', type=float, 217 | help='mean values to subtract from input') 218 | parser.add_argument( 219 | '--dataset', 220 | help='the directory containing the dataset to use') 221 | parser.add_argument( 222 | '--split', 223 | help='the split of the dataset to use') 224 | parser.add_argument( 225 | '--limit', 226 | type=int, default=None, 227 | help='limit dataset to this size') 228 | parser.add_argument( 229 | '--batch_size', 230 | type=int, default=256, 231 | help='the batch size to use') 232 | parser.add_argument( 233 | '--ahead', 234 | type=int, default=4, 235 | help='number of batches to prefetch') 236 | parser.add_argument( 237 | '--rotation_seed', 238 | type=int, default=None, 239 | help='the seed for the random rotation to apply') 240 | parser.add_argument( 241 | '--rotation_power', 242 | type=float, default=1.0, 243 | help='the power of the random rotation') 244 | parser.add_argument( 245 | '--rotation_unpermute', 246 | type=int, default=0, 247 | help='set to 1 to unpermute random rotation') 248 | parser.add_argument( 249 | '--colordepth', 250 | type=int, default=3, 251 | help='set to 1 for grayscale') 252 | args = parser.parse_args() 253 | 254 | create_probe( 255 | args.directory, args.dataset, args.definition, args.weights, 256 | numpy.array(args.mean, dtype=numpy.float32), args.blobs, 257 | batch_size=args.batch_size, ahead=args.ahead, limit=args.limit, 258 | colordepth=args.colordepth, 259 | rotation_seed=args.rotation_seed, 260 | rotation_power=args.rotation_power, 261 | rotation_unpermute=args.rotation_unpermute, 262 | split=args.split, cl_args=args, verbose=True) 263 | except: 264 | traceback.print_exc(file=sys.stdout) 265 | sys.exit(1) 266 | -------------------------------------------------------------------------------- /src/osseg.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | from unicsv import DictUnicodeReader 4 | import colorname 5 | from loadseg import AbstractSegmentation 6 | from scipy.misc import imread 7 | 8 | class OpenSurfaceSegmentation(AbstractSegmentation): 9 | def __init__(self, directory=None, supply=None): 10 | directory = os.path.expanduser(directory) 11 | self.directory = directory 12 | self.supply = supply 13 | # Process open surfaces labels: open label-name-colors.csv 14 | object_name_map = {} 15 | with open(os.path.join(directory, 'label-name-colors.csv')) as f: 16 | for row in DictUnicodeReader(f): 17 | object_name_map[row['name_name']] = int( 18 | row['green_color']) 19 | self.object_names = ['-'] * (1 + max(object_name_map.values())) 20 | for k, v in object_name_map.items(): 21 | self.object_names[v] = k 22 | # Process material labels: open label-substance-colors.csv 23 | subst_name_map = {} 24 | with open(os.path.join(directory, 'label-substance-colors.csv')) as f: 25 | for row in DictUnicodeReader(f): 26 | subst_name_map[row['substance_name']] = int( 27 | row['red_color']) 28 | self.substance_names = ['-'] * (1 + max(subst_name_map.values())) 29 | for k, v in subst_name_map.items(): 30 | self.substance_names[v] = k 31 | # Now load the metadata about images from photos.csv 32 | with open(os.path.join(directory, 'photos.csv')) as f: 33 | self.image_meta = list(DictUnicodeReader(f)) 34 | scenes = set(row['scene_category_name'] for row in self.image_meta) 35 | self.scenes = ['-'] + sorted(list(scenes)) 36 | self.scene_map = dict((s, i) for i, s in enumerate(self.scenes)) 37 | # Zero out special ~ scene names, which mean unlabeled. 38 | for k in self.scene_map: 39 | if k.startswith('~'): 40 | self.scene_map[k] = 0 41 | 42 | def all_names(self, category, j): 43 | if j == 0: 44 | return [] 45 | if category == 'color': 46 | return [colorname.color_names[j - 1] + '-c'] 47 | if category == 'scene': 48 | return [self.scenes[j].lower() + '-s'] 49 | if category == 'material': 50 | return [norm_name(n) for n in self.substance_names[j].split('/')] 51 | if category == 'object': 52 | return [norm_name(n) for n in self.object_names[j].split('/')] 53 | return [] 54 | 55 | def size(self): 56 | '''Returns the number of images in this dataset.''' 57 | return len(self.image_meta) 58 | 59 | def filename(self, i): 60 | '''Returns the filename for the nth dataset image.''' 61 | photo_id = int(self.image_meta[i]['photo_id']) 62 | return os.path.join(self.directory, 'photos', '%d.jpg' % photo_id) 63 | 64 | def metadata(self, i): 65 | '''Returns an object that can be used to create all segmentations.''' 66 | row = self.image_meta[i] 67 | return (self.directory, 68 | row, 69 | self.scene_map[row['scene_category_name']], 70 | self.supply) 71 | 72 | @classmethod 73 | def resolve_segmentation(cls, m, categories=None): 74 | directory, row, scene, supply = m 75 | photo_id = int(row['photo_id']) 76 | fnpng = os.path.join(directory, 'photos-labels', '%d.png' % photo_id) 77 | fnjpg = os.path.join(directory, 'photos', '%d.jpg' % photo_id) 78 | 79 | if wants('material', categories) or wants('object', categories): 80 | labels = imread(fnpng) 81 | # Opensurfaces has some completely unlabled images; omit them. 82 | # TODO: figure out how to do this nicely in the joiner instead. 83 | # if labels.max() == 0: 84 | # return {} 85 | result = {} 86 | if wants('scene', categories) and wants('scene', supply): 87 | result['scene'] = scene 88 | if wants('material', categories) and wants('material', supply): 89 | result['material'] = labels[:,:,0] 90 | if wants('object', categories) and wants('object', supply): 91 | result['object'] = labels[:,:,1] 92 | if wants('color', categories) and wants('color', supply): 93 | result['color'] = colorname.label_major_colors(imread(fnjpg)) + 1 94 | arrs = [a for a in result.values() if numpy.shape(a) >= 2] 95 | shape = arrs[0].shape[-2:] if arrs else (1, 1) 96 | return result, shape 97 | 98 | def norm_name(s): 99 | return s.replace(' - ', '-').replace('/', '-').strip().lower() 100 | 101 | def wants(what, option): 102 | if option is None: 103 | return True 104 | return what in option 105 | -------------------------------------------------------------------------------- /src/printmean.py: -------------------------------------------------------------------------------- 1 | import caffe 2 | import numpy as np 3 | import sys 4 | 5 | # if len(sys.argv) != 3: 6 | # print "Usage: python convert_protomean.py proto.mean out.npy" 7 | # sys.exit() 8 | 9 | blob = caffe.proto.caffe_pb2.BlobProto() 10 | data = open( sys.argv[1] , 'rb' ).read() 11 | blob.ParseFromString(data) 12 | arr = np.array( caffe.io.blobproto_to_array(blob) ) 13 | print arr, arr.shape 14 | print arr.mean(axis=(2,3)) 15 | -------------------------------------------------------------------------------- /src/quantile.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | 3 | class QuantileCounter: 4 | """ 5 | Streaming randomized quantile computation for numpy. 6 | 7 | Add any amount of data repeatedly via add(data). At any time, 8 | quantile estimates (or old-style percentiles) can be read out using 9 | quantiles(q) or percentiles(p). 10 | 11 | Accuracy scales according to resolution: the default is to 12 | set resolution to be almost-always accurate to approximately 0.1%, 13 | limiting storage to about 100,000 samples. 14 | 15 | Good for computing quantiles of huge data without using much memory. 16 | Works well on arbitrary data with probability near 1. 17 | 18 | Based on the optimal KLL quantile algorithm by Karnin, Lang, and Liberty 19 | from FOCS 2016. http://ieee-focs.org/FOCS-2016-Papers/3933a071.pdf 20 | """ 21 | 22 | def __init__(self, resolution=32 * 1024, buffersize=None, 23 | dtype=None, seed=None): 24 | self.resolution = resolution 25 | # Default buffersize: 4096 samples (and smaller than resolution). 26 | if buffersize is None: 27 | buffersize = min(512, (resolution + 3) // 4) 28 | self.buffersize = buffersize 29 | self.samplerate = 1.0 30 | self.data = [numpy.zeros(shape=resolution, dtype=dtype)] 31 | self.firstfree = [0] 32 | # The 0th buffer is left marked full always 33 | self.full = numpy.ones(shape=1, dtype='bool') 34 | self.random = numpy.random.RandomState(seed) 35 | 36 | def add(self, incoming): 37 | # Convert to a flat numpy array. 38 | incoming = numpy.ravel(incoming) 39 | if self.samplerate >= 1.0: 40 | self._add_every(incoming) 41 | return 42 | # If we are sampling, then subsample a large chunk at a time. 43 | chunksize = numpy.ceil[self.buffersize / self.samplerate] 44 | for index in xrange(0, len(incoming), chunksize): 45 | batch = incoming[index:index+chunksize] 46 | sample = batch[self.random.binomial(1, self.samplerate, len(batch))] 47 | self._add_every(sample) 48 | 49 | def _add_every(self, incoming): 50 | supplied = len(incoming) 51 | index = 0 52 | while index < supplied: 53 | ff = self.firstfree[0] 54 | available = len(self.data[0]) - ff 55 | if available == 0: 56 | if not self._shift(): 57 | # If we shifted by subsampling, then subsample. 58 | incoming = incoming[index:][self.random.binomial(1, 0.5, 59 | len(incoming - index))] 60 | index = 0 61 | supplied = len(incoming) 62 | ff = self.firstfree[0] 63 | available = len(self.data[0]) - ff 64 | copycount = min(available, supplied - index) 65 | self.data[0][ff:ff + copycount] = incoming[index:index + copycount] 66 | self.firstfree[0] += copycount 67 | index += copycount 68 | 69 | def _shift(self): 70 | index = 0 71 | while self.firstfree[index] * 2 > len(self.data[index]): 72 | if index + 1 >= len(self.data): 73 | return self._expand() 74 | data = self.data[index][0:self.firstfree[index]] 75 | data.sort() 76 | offset = self.random.binomial(1, 0.5) 77 | position = self.firstfree[index + 1] 78 | subset = data[offset::2] 79 | self.data[index + 1][position:position + len(subset)] = subset 80 | self.firstfree[index] = 0 81 | self.firstfree[index + 1] += len(subset) 82 | index += 1 83 | return True 84 | 85 | def _expand(self): 86 | cap = self._next_capacity() 87 | if cap > 0: 88 | # First, make a new layer of the proper capacity. 89 | self.data.insert(0, 90 | numpy.empty(shape=cap, dtype=self.data[-1].dtype)) 91 | self.firstfree.insert(0, 0) 92 | else: 93 | # Unless we're so big we are just subsampling. 94 | assert self.firstfree[0] == 0 95 | self.samplerate *= 0.5 96 | for index in range(1, len(self.data)): 97 | # Scan for existing data that needs to be moved down a level. 98 | amount = self.firstfree[index] 99 | if amount == 0: 100 | continue 101 | position = self.firstfree[index - 1] 102 | if (amount + position) * 2 <= len(self.data[index - 1]): 103 | # Move the data down if it would leave things half-empty. 104 | self.data[index - 1][position:position + amount] = ( 105 | self.data[index][:amount]) 106 | self.firstfree[index - 1] += amount 107 | self.firstfree[index] = 0 108 | else: 109 | # Scrunch the data if it would not. 110 | data = self.data[index][:amount] 111 | data.sort() 112 | offset = self.random.binomial(1, 0.5) 113 | scrunched = data[offset::2] 114 | self.data[index][:len(scrunched)] = scrunched 115 | self.firstfree[index] = len(scrunched) 116 | return cap > 0 117 | 118 | def _next_capacity(self): 119 | cap = numpy.ceil(self.resolution * numpy.power(0.67, len(self.data))) 120 | if cap < 2: 121 | return 0 122 | return max(self.buffersize, int(cap)) 123 | 124 | def quantiles(self, quantiles, old_style=False): 125 | size = sum(self.firstfree) 126 | weights = numpy.empty(shape=size, dtype='float32') # floating point 127 | summary = numpy.empty(shape=size, dtype=self.data[-1].dtype) 128 | index = 0 129 | for level, ff in enumerate(self.firstfree): 130 | if ff == 0: 131 | continue 132 | summary[index:index + ff] = self.data[level][:ff] 133 | weights[index:index + ff] = numpy.power(2.0, level) 134 | index += ff 135 | assert index == len(summary) 136 | order = numpy.argsort(summary) 137 | summary = summary[order] 138 | weights = weights[order] 139 | cumweights = numpy.cumsum(weights) - weights / 2 140 | if old_style: 141 | # To be convenient with numpy.percentile 142 | cumweights -= cumweights[0] 143 | cumweights /= cumweights[-1] 144 | else: 145 | cumweights /= numpy.sum(weights) 146 | return numpy.interp(quantiles, cumweights, summary) 147 | 148 | def percentiles(self, percentiles): 149 | return self.quantiles(percentiles, old_style=True) 150 | 151 | def readout(self, count, old_style=True): 152 | return self.quantiles( 153 | numpy.linspace(0.0, 1.0, count), old_style=old_style) 154 | 155 | 156 | if __name__ == '__main__': 157 | # An adverarial case: we keep finding more numbers in the middle 158 | # as the stream goes on. 159 | qc = QuantileCounter() 160 | amount = 50000000 161 | percentiles = 1000 162 | data = numpy.arange(amount) 163 | data[1::2] = data[-1::-2] + (len(data) - 1) 164 | data /= 2 165 | # data[::2] = data[-2::-2] 166 | # numpy.random.shuffle(data) 167 | qc.add(data) 168 | ro = qc.readout(1001) 169 | # print ro - numpy.linspace(0, amount, percentiles+1) 170 | print "Maximum relative deviation among %d perentiles:" % percentiles, max( 171 | abs(ro - numpy.linspace(0, amount, percentiles+1)) 172 | / amount) * percentiles 173 | 174 | -------------------------------------------------------------------------------- /src/quantprobe.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | import re 4 | import shutil 5 | import time 6 | import expdir 7 | import sys 8 | import vecquantile 9 | 10 | def quant_probe(directory, blob, quantiles=None, batch_size=None): 11 | ''' 12 | Adds a [blob]-sort directory to a probe such as conv5-sort, 13 | where the directory contains [blob]-[unit].mmap files for 14 | each unit such as conv5-0143.mmap, where the contents are 15 | sorted. Also creates a [blob]-quant-[num].mmap file 16 | such as conv5-quantile-1000.mmap, where the higest seen 17 | value for each quantile of each unit is summarized in a 18 | single (units, quantiles) matrix. 19 | ''' 20 | # Make sure we have a directory to work in 21 | ed = expdir.ExperimentDirectory(directory) 22 | 23 | # If it's already computed, then skip it!!! 24 | print "Checking", ed.mmap_filename(blob=blob, part='quant-%d' % quantiles) 25 | if ed.has_mmap(blob=blob, part='quant-%d' % quantiles): 26 | print "Already have %s-quant-%d.mmap, skipping." % (blob, quantiles) 27 | return 28 | 29 | # Read about the blob shape 30 | blob_info = ed.load_info(blob=blob) 31 | shape = blob_info.shape 32 | 33 | print 'Computing quantiles for %s shape %r' % (blob, shape) 34 | data = ed.open_mmap(blob=blob, shape=shape) 35 | qmap = ed.open_mmap(blob=blob, part='quant-%d' % quantiles, 36 | mode='w+', shape=(data.shape[1], quantiles)) 37 | linmap = ed.open_mmap(blob=blob, part='minmax', 38 | mode='w+', shape=(data.shape[1], 2)) 39 | 40 | # Automatic batch size selection: 64mb batches 41 | if batch_size is None: 42 | batch_size = max(1, int(16 * 1024 * 1024 / numpy.prod(shape[1:]))) 43 | # Algorithm: one pass over the data with a quantile counter for each unit 44 | quant = vecquantile.QuantileVector(depth=data.shape[1], seed=1) 45 | start_time = time.time() 46 | last_batch_time = start_time 47 | for i in range(0, data.shape[0], batch_size): 48 | batch_time = time.time() 49 | rate = i / (batch_time - start_time + 1e-15) 50 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 51 | last_batch_time = batch_time 52 | print 'Processing %s index %d: %f %f' % (blob, i, rate, batch_rate) 53 | sys.stdout.flush() 54 | batch = data[i:i+batch_size] 55 | if len(batch.shape) == 4: 56 | batch = numpy.transpose(batch,axes=(0,2,3,1)).reshape(-1, data.shape[1]) 57 | elif len(batch.shape) != 2: 58 | assert(False) 59 | quant.add(batch) 60 | print 'Writing quantiles' 61 | sys.stdout.flush() 62 | # Reverse the quantiles, largest first. 63 | qmap[...] = quant.readout(quantiles)[:,::-1] 64 | linmap[...] = quant.minmax() 65 | 66 | if qmap is not None: 67 | ed.finish_mmap(qmap) 68 | if linmap is not None: 69 | ed.finish_mmap(linmap) 70 | 71 | 72 | if __name__ == '__main__': 73 | import argparse 74 | import sys 75 | import traceback 76 | 77 | try: 78 | import loadseg 79 | 80 | parser = argparse.ArgumentParser( 81 | description='Generate sorted files for probed activation data.') 82 | parser.add_argument( 83 | '--directory', 84 | default='.', 85 | help='output directory for the net probe') 86 | parser.add_argument( 87 | '--blobs', 88 | nargs='*', 89 | help='network blob names to sort') 90 | parser.add_argument( 91 | '--batch_size', 92 | type=int, default=None, 93 | help='the batch size to use') 94 | parser.add_argument( 95 | '--quantiles', 96 | type=int, default=1000, 97 | help='the number of quantiles to summarize') 98 | args = parser.parse_args() 99 | for blob in args.blobs: 100 | quant_probe(args.directory, blob, args.quantiles, args.batch_size) 101 | except: 102 | traceback.print_exc(file=sys.stdout) 103 | sys.exit(1) 104 | -------------------------------------------------------------------------------- /src/rotate.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | import scipy.linalg 3 | from tempfile import NamedTemporaryFile 4 | 5 | # Generates a random orthogonal matrix correctly. 6 | # See http://www.ams.org/notices/200511/what-is.pdf 7 | # Also http://home.lu.lv/~sd20008/papers/essays/Random%20unitary%20[paper].pdf 8 | # Equation 35 9 | 10 | def randomRotation(n, seed=None): 11 | if seed is None: 12 | NR = numpy.random.normal(size=(n, n)) 13 | else: 14 | NR = numpy.random.RandomState(seed).normal(size=(n, n)) 15 | Q, R = numpy.linalg.qr(NR) 16 | # Eliminate negative diagonals 17 | D = numpy.diagonal(R) 18 | L = numpy.diagflat(D / abs(D)) 19 | # The replacement R would be LR and have a positive diagonal. 20 | # This allows det(QL) to be random sign whereas Householder Q is fixed. 21 | result = numpy.dot(Q, L) 22 | # Now negate the first column if this result is a reflection 23 | if numpy.linalg.det(result) < 0: 24 | result[0] = -result[0] 25 | return result 26 | 27 | def randomRotationPowers(n, powers, seed=None, unpermute=False): 28 | RR = randomRotation(n, seed) 29 | # Reduce the matrix to canonical (block-diag) form using schur decomp 30 | # (TODO: Consider implementing the stabilized variant of the algorithm 31 | # by Gragg called UHQR which is tailored to solve this decomposition for 32 | # unitary RR - or 'OHQR' by Ammar for the real orthogonal case.) 33 | T, W = scipy.linalg.schur(RR, output='real') 34 | # Now T is not exactly block diagonal due to numerical approximations. 35 | # However, its eigenvalues should have the right distribution: let's 36 | # read the angles off the diagonal and use them to construct a perfectly 37 | # block diagonal form. 38 | RA = numpy.arccos(numpy.clip(numpy.diag(T), -1, 1))[0:n:2] 39 | if unpermute: 40 | # If unpermuting, first figure out full rotation matrix 41 | B = [numpy.cos([[a, a + numpy.pi/2], [a - numpy.pi/2, a]]) for a in RA] 42 | BD = scipy.linalg.block_diag(*B) 43 | fullrot = numpy.dot(numpy.dot(W, BD), W.transpose()) 44 | # Then sort the rows by max value 45 | biggest_first = numpy.argsort(-numpy.amax(fullrot, axis=1)) 46 | # In greedy order, assign permutation order 47 | cols_seen = set() 48 | permutation = numpy.zeros((n, n)) 49 | for row in biggest_first: 50 | # For each row, find the max col to which it is closest 51 | biggest_cols = numpy.argsort(-fullrot[row]) 52 | for col in biggest_cols: 53 | if col not in cols_seen: 54 | # Bring the row back to that slot if not taken 55 | permutation[col,row] = 1 56 | cols_seen.add(col) 57 | break 58 | else: 59 | permutation = numpy.eye(n) 60 | # Now form the requested powers of W * (RA^p) * W' 61 | result = [] 62 | for p in powers: 63 | A = [a * p for a in RA] 64 | B = [numpy.cos([[a, a + numpy.pi/2], [a - numpy.pi/2, a]]) for a in A] 65 | BD = scipy.linalg.block_diag(*B) 66 | result.append(numpy.dot(permutation, 67 | numpy.dot(numpy.dot(W, BD), W.transpose()))) 68 | return result 69 | 70 | def randomNearIdentity(n, scale, seed=None): 71 | if numpy.isinf(scale): 72 | return randomOrthogonal(n, seed) 73 | sigma = scale / numpy.sqrt(2 * n) 74 | if seed is None: 75 | RR = numpy.random.normal(scale=sigma, size=(n, n)) 76 | else: 77 | RR = numpy.random.RandomState(seed).normal(scale=sigma, size=(n, n)) 78 | U, _, V = numpy.linalg.svd(numpy.eye(n) + RR) 79 | return numpy.dot(U, V) 80 | 81 | def deviation(rotation): 82 | return numpy.linalg.norm(rotation - numpy.eye(len(rotation)), 2) 83 | 84 | def temparray_like(arr): 85 | fs = NamedTemporaryFile() 86 | result = numpy.memmap(fs.name, dtype=arr.dtype, mode='w+', 87 | shape=arr.shape) 88 | fs.close() 89 | return result 90 | 91 | def dumpinfo(fn, arr, limit=5): 92 | text = [] 93 | for row in range(len(arr)): 94 | line = [] 95 | for col in numpy.argsort(-arr[row])[:limit]: 96 | line.append('%.2fa%d' % (arr[row,col], col+1)) 97 | text.append(('s%d = ' % (row+1)) + ' + '.join(line) + '...\n') 98 | with open(fn, 'w') as f: 99 | f.writelines(text) 100 | 101 | if __name__ == '__main__': 102 | from scipy.io import loadmat, savemat 103 | import os 104 | directory = 'randrot' 105 | useed = 1000 106 | try: 107 | os.makedirs(directory) 108 | except: 109 | pass 110 | dims = 256 111 | seed = 1 112 | alpha = numpy.arange(0.1, 1.0 + 1e-15, 0.1) 113 | rots = randomRotationPowers(dims, alpha, seed=seed, unpermute=True) 114 | for rot, a in zip(rots, alpha): 115 | fn = 'rotpow-%d-%.1f-%.2f-p%d.mat' % (dims, a, deviation(rot), seed) 116 | savemat(os.path.join(directory, fn), {'rotation': rot}) 117 | fn = 'rotpow-%d-%.1f-%.2f-p%d.info' % (dims, a, deviation(rot), seed) 118 | dumpinfo(os.path.join(directory, fn), rot) 119 | 120 | # for dims in [128, 256, 384, 512]: 121 | # for sigma in list(numpy.arange(0.2, 2.0, 0.2) 122 | # ) + list(numpy.arange(2.0, 4.0, 0.4)) + [float('inf')]: 123 | # r1 = randomNearIdentity(dims, sigma, seed=1) 124 | # fn = 'randrot-%d-%.2f-%.1f-1.mat' % (dims, deviation(r1), sigma) 125 | # print fn 126 | # savemat(os.path.join(directory, fn), {'rotation': r1}) 127 | # r0 = randomNearIdentity(dims, sigma, seed=useed) 128 | # useed += 1 129 | # fn = 'randrot-%d-%.2f-%.1f-0.mat' % (dims, deviation(r1), sigma) 130 | # print fn 131 | # savemat(os.path.join(directory, fn), {'rotation': r1}) 132 | -------------------------------------------------------------------------------- /src/saveneuralstatistics.py: -------------------------------------------------------------------------------- 1 | import expdir 2 | import loadseg 3 | from indexdata import load_image_to_label 4 | 5 | import numpy as np 6 | 7 | def save_neural_statistics(directory, blob, split=None): 8 | ed = expdir.ExperimentDirectory(directory) 9 | info = ed.load_info() 10 | dataset = info.dataset 11 | blob_info = ed.load_info(blob=blob) 12 | 13 | if 'broden' in dataset: 14 | assert(split is not None) 15 | suffix = '_%s' % split 16 | else: 17 | suffix = '' 18 | if ed.has_mmap(blob=blob, part='act_probs%s' % suffix): 19 | print('%s already exists' % ed.mmap_filename(blob=blob, part='act_probs%s' % suffix)) 20 | return 21 | 22 | acts = ed.open_mmap(blob=blob, mode='r', dtype='float32', shape=blob_info.shape) 23 | 24 | if 'broden' in dataset: 25 | ds = loadseg.SegmentationData(dataset) 26 | split_ind = np.array([True if ds.split(i) == split else False for i in range(ds.size())]) 27 | L = ds.label_size() 28 | elif 'imagenet' in dataset or 'ILSVRC' in dataset: 29 | assert(split is None) 30 | L = 1000 31 | split_ind = True 32 | image_to_label = load_image_to_label(directory, blob=blob) 33 | 34 | K = blob_info.shape[1] 35 | 36 | probs = np.zeros((L,K)) 37 | mus = np.zeros((L,K)) 38 | sigmas = np.zeros((L,K)) 39 | 40 | for class_i in range(L): 41 | class_idx = np.where(split_ind & image_to_label[:,class_i])[0] 42 | if len(class_idx) == 0: 43 | probs[class_i,:] = np.nan 44 | mus[class_i,:] = np.nan 45 | sigmas[class_i,:] = np.nan 46 | print class_i, 'no examples' 47 | continue 48 | print class_i 49 | max_acts = np.amax(acts[class_idx], axis=(2,3)) 50 | for filter_i in range(blob_info.shape[1]): 51 | nz_idx = np.where(max_acts[:,filter_i] > 0)[0] 52 | probs[class_i][filter_i] = len(nz_idx)/float(len(max_acts[:,filter_i])) 53 | try: 54 | mus[class_i][filter_i] = np.mean(max_acts[nz_idx,filter_i]) 55 | sigmas[class_i][filter_i] = np.std(max_acts[nz_idx,filter_i]) 56 | except: 57 | mus[class_i][filter_i] = np.nan 58 | sigmas[class_i][filter_i] = np.nan 59 | print class_i, filter_i, 'no nonzero idx' 60 | 61 | probs_mmap = ed.open_mmap(blob=blob, part='act_probs%s' % suffix, mode='w+', dtype='float32', 62 | shape=probs.shape) 63 | mus_mmap = ed.open_mmap(blob=blob, part='act_mus%s' % suffix, mode='w+', dtype='float32', 64 | shape=mus.shape) 65 | sigmas_mmap = ed.open_mmap(blob=blob, part='act_sigmas%s' % suffix, mode='w+', dtype='float32', 66 | shape=sigmas.shape) 67 | 68 | probs_mmap[:] = probs[:] 69 | mus_mmap[:] = mus[:] 70 | sigmas_mmap[:] = sigmas[:] 71 | 72 | ed.finish_mmap(probs_mmap) 73 | ed.finish_mmap(mus_mmap) 74 | ed.finish_mmap(sigmas_mmap) 75 | 76 | print('Finished saving neural statistics for %s' % blob) 77 | 78 | 79 | if __name__ == '__main__': 80 | import argparse 81 | import sys 82 | import traceback 83 | 84 | try: 85 | parser = argparse.ArgumentParser() 86 | parser.add_argument('--directory', default='.') 87 | parser.add_argument('--blobs', nargs='*') 88 | parser.add_argument('--split', type=str, default=None) 89 | 90 | args = parser.parse_args() 91 | 92 | for blob in args.blobs: 93 | save_neural_statistics(args.directory, blob, args.split) 94 | 95 | except: 96 | traceback.print_exc(file=sys.stdout) 97 | sys.exit(1) 98 | -------------------------------------------------------------------------------- /src/synonym.py: -------------------------------------------------------------------------------- 1 | '''Contains a list of synonyms to normalize out of ADE20K data.''' 2 | 3 | def synonyms(words): 4 | ''' 5 | Transforms a list of words to a better list of synonyms. 6 | Preferred names are listed first, and 7 | ''' 8 | # Common case: no common or confusing words 9 | if not any(w in confusing_names or w in common_names for w in words): 10 | return words 11 | # The list needs to be changed. First, accumulate better names. 12 | full_list = [] 13 | for word in words: 14 | full_list.extend(common_names.get(word, [])) 15 | full_list.extend(words) 16 | # Now accumulate confusing names to omit. 17 | omit = set() 18 | for word in full_list: 19 | if word in confusing_names: 20 | omit.update(confusing_names[word]) 21 | # Now produce a filtered list 22 | seen = set() 23 | see = seen.add 24 | return [w for w in full_list if not (w in omit or w in seen or see(w))] 25 | 26 | # These potential confusing names are drawn from aliases created by 27 | # joining ade20k data with opensurfaces. Keys are more specific names 28 | # that should not be aliased with more generic names in values. 29 | 30 | # Read this list as, { 'flowerpot': ['pot'] } "A flowerpot should not be 31 | # called a pot, because that might confuse it with other pots." 32 | 33 | # 29 confusing synonyms to avoid 34 | confusing_names ={ 35 | 'flowerpot': ['pot'], 36 | 'toilet': ['throne', 'can'], 37 | 'curtain': ['mantle'], 38 | 'fabric': ['material'], 39 | 'file cabinet': ['file'], 40 | 'chest of drawers': ['chest'], 41 | 'fire hydrant': ['plug'], 42 | 'car': ['machine'], 43 | 'closet': ['press'], 44 | 'bicycle': ['wheel'], 45 | 'brochure': ['folder'], 46 | 'filing cabinet': ['file'], 47 | 'paper': ['tissue'], # Opensurfaces groups these materials; call it paper. 48 | 'exhaust hood': ['hood'], 49 | 'blanket': ['cover'], 50 | 'carapace': ['shield'], 51 | 'cellular phone': ['cell'], 52 | 'handbag': ['bag'], 53 | 'land': ['soil'], 54 | 'sidewalk': ['pavement'], 55 | 'poster': ['card', 'bill'], 56 | 'paper towel': ['tissue'], 57 | 'computer mouse': ['mouse'], 58 | 'steering wheel': ['wheel'], 59 | 'lighthouse': ['beacon'], 60 | 'basketball hoop': ['hoop'], 61 | 'bus': ['passenger vehicle'], 62 | 'head': ['caput'], 63 | # Do not use left/right/person to qualify body parts 64 | 'arm': ['left arm', 'right arm', 'person arm'], 65 | 'foot': ['left foot', 'right foot', 'person foot'], 66 | 'hand': ['left hand', 'right hand', 'person hand'], 67 | 'leg': ['left leg', 'right leg', 'person leg'], 68 | 'shoulder': ['left shoulder', 'right shoulder', 'person shoulder'], 69 | 'torso': ['person torso'], 70 | 'head': ['person head'], 71 | 'hair': ['person hair'], 72 | 'nose': ['person nose'], 73 | 'ear': ['person ear'], 74 | 'neck': ['person neck'], 75 | 'eye': ['person eye'], 76 | 'eyebrow': ['person eyebrow'] 77 | } 78 | 79 | # These potential synonyms are drawn from the raw ADE20K data, with 80 | # shorter and more common names listed first. I have manually uncommented 81 | # word pairs that seem unambiguously the same, where the most common uses 82 | # of the second word would allow the first word to be substituted without 83 | # changing meaning. 84 | 85 | common_names = { 86 | # We do not distinguish between left+right parts for our purposes. 87 | 'left arm': ['arm'], 88 | 'right arm': ['arm'], 89 | 'left foot': ['foot'], 90 | 'right foot': ['foot'], 91 | 'left hand': ['hand'], 92 | 'right hand': ['hand'], 93 | 'left leg': ['leg'], 94 | 'right leg': ['leg'], 95 | 'left shoulder': ['shoulder'], 96 | 'right shoulder': ['shoulder'], 97 | # And we assume that human parts do not need to say 'person' 98 | 'person torso': ['torso'], 99 | 'person head': ['head'], 100 | 'person arm': ['arm'], 101 | 'person hand': ['hand'], 102 | 'person hair': ['hair'], 103 | 'person nose': ['nose'], 104 | 'person leg': ['leg'], 105 | 'person mouth': ['mouth'], 106 | 'person ear': ['ear'], 107 | 'person neck': ['neck'], 108 | 'person eye': ['eye'], 109 | 'person eyebrow': ['eyebrow'], 110 | 'person foot': ['foot'], 111 | 112 | # why is this word airport-airport? 113 | 'airport-airport-s': ['airport-s'], 114 | # This is the preferred spelling for us. 115 | 'aeroplane': ['airplane'], 116 | 'airplane': ['airplane', 'aeroplane', 'plane'], 117 | 'spectacles': ['eyeglasses'], 118 | 'windopane': ['window'], # ade20k calls windows windowpanes. 119 | 'dog': ['dog', 'domestic dog', 'canis familiaris'], 120 | 'alga': ['algae'], 121 | 'bicycle': ['bicycle', 'bike', 'cycle'], 122 | 'food': ['food', 'solid food'], 123 | 'caput': ['head'], 124 | 'route': ['road'], 125 | 'fencing': ['fence'], 126 | 'flooring': ['floor'], 127 | 'carpet': ['carpet', 'carpeting'], 128 | 'shrub': ['bush'], 129 | 'armour': ['armor'], 130 | 'pail': ['bucket'], 131 | 'spigot': ['faucet'], 132 | 'faucet': ['faucet', 'spigot'], 133 | 'crt screen': ['screen'], 134 | 'cistern': ['water tank'], 135 | 'video display': ['display'], 136 | 'lift': ['elevator'], 137 | 'hydroplane': ['seaplane'], 138 | 'microwave oven': ['microwave'], 139 | 'falls': ['waterfall'], 140 | 'mike': ['microphone'], 141 | 'windscreen': ['windshield'], 142 | 'fluorescent fixture': ['fluorescent'], 143 | 'water vapour': ['water vapor'], 144 | 'numberplate': ['license plate'], 145 | 'tin can': ['can'], 146 | 'cow': ['cow', 'moo-cow'], 147 | 'horse': ['horse', 'equus caballus'], 148 | 'kerb': ['curb'], 149 | 'filing cabinet': ['file cabinet'], 150 | 'electrical switch': ['switch'], 151 | 'telephone set': ['telephone'], 152 | 'totaliser': ['adding machine'], 153 | 'television receiver': ['television'], 154 | 'fabric': ['fabric', 'cloth', 'textile'], 155 | 'textile': ['fabric'], 156 | 'attack aircraft carrier': ['aircraft carrier'], 157 | 'cooking stove': ['stove'], 158 | 'electric-light bulb': ['light bulb'], 159 | } 160 | -------------------------------------------------------------------------------- /src/unicsv.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | ''' 3 | File: unicsv.py 4 | Author: Spencer Rathbun 5 | Date: 05/30/2012 6 | Description: Unicode wrapper classes around csv reader/writers 7 | ''' 8 | import csv, codecs, cStringIO 9 | 10 | class UTF8Recoder: 11 | """ 12 | Iterator that reads an encoded stream and reencodes the input to UTF-8 13 | """ 14 | def __init__(self, f, encoding): 15 | self.reader = codecs.getreader(encoding)(f) 16 | 17 | def __iter__(self): 18 | return self 19 | 20 | def next(self): 21 | return self.reader.next().encode("utf-8") 22 | 23 | class UnicodeReader: 24 | """ 25 | A CSV reader which will iterate over lines in the CSV file "f", 26 | which is encoded in the given encoding. 27 | """ 28 | 29 | def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds): 30 | f = UTF8Recoder(f, encoding) 31 | self.reader = csv.reader(f, dialect=dialect, **kwds) 32 | 33 | def next(self): 34 | row = self.reader.next() 35 | return [unicode(s, "utf-8") for s in row] 36 | 37 | def __iter__(self): 38 | return self 39 | 40 | class DictUnicodeReader: 41 | """ 42 | A dict-based CSV reader which will iterate over lines in the CSV 43 | file "f", which is encoded in the given encoding. 44 | """ 45 | 46 | def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds): 47 | f = UTF8Recoder(f, encoding) 48 | self.reader = csv.DictReader(f, dialect=dialect, **kwds) 49 | 50 | def next(self): 51 | row = self.reader.next() 52 | outrow = dict([(k,unicode(v, "utf-8")) for k,v in row.iteritems()]) 53 | return outrow 54 | 55 | def __iter__(self): 56 | return self 57 | 58 | class UnicodeWriter: 59 | """ 60 | A CSV writer which will write rows to CSV file "f", 61 | which is encoded in the given encoding. 62 | """ 63 | 64 | def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds): 65 | # Redirect output to a queue 66 | self.queue = cStringIO.StringIO() 67 | self.writer = csv.writer(self.queue, dialect=dialect, **kwds) 68 | self.stream = f 69 | self.encoder = codecs.getincrementalencoder(encoding)() 70 | 71 | def writerow(self, row): 72 | self.writer.writerow([unicode(s).encode("utf-8") for s in row]) 73 | # Fetch UTF-8 output from the queue ... 74 | data = self.queue.getvalue() 75 | data = data.decode("utf-8") 76 | # ... and reencode it into the target encoding 77 | data = self.encoder.encode(data) 78 | # write to the target stream 79 | self.stream.write(data) 80 | # empty queue 81 | self.queue.truncate(0) 82 | 83 | def writerows(self, rows): 84 | for row in rows: 85 | self.writerow(row) 86 | 87 | class DictUnicodeWriter(object): 88 | """ 89 | A dict-based CSV writer which will write rows to CSV file "f", 90 | which is encoded in the given encoding. 91 | """ 92 | 93 | def __init__(self, f, fieldnames, dialect=csv.excel, 94 | encoding="utf-8", **kwds): 95 | # Redirect output to a queue 96 | self.queue = cStringIO.StringIO() 97 | self.writer = csv.DictWriter(self.queue, fieldnames, dialect=dialect, **kwds) 98 | self.stream = f 99 | self.encoder = codecs.getincrementalencoder(encoding)() 100 | 101 | def writerow(self, D): 102 | self.writer.writerow( 103 | {k:unicode(v).encode("utf-8") for k,v in D.items()}) 104 | 105 | # Fetch UTF-8 output from the queue ... 106 | data = self.queue.getvalue() 107 | data = data.decode("utf-8") 108 | # ... and reencode it into the target encoding 109 | data = self.encoder.encode(data) 110 | # write to the target stream 111 | self.stream.write(data) 112 | # empty queue 113 | self.queue.truncate(0) 114 | 115 | def writerows(self, rows): 116 | for D in rows: 117 | self.writerow(D) 118 | 119 | def writeheader(self): 120 | self.writer.writeheader() 121 | 122 | # Example code: 123 | ''' 124 | D1 = {'name':u'马克','pinyin':u'Mǎkè'} 125 | D2 = {'name':u'美国','pinyin':u'Měiguó'} 126 | f = open('out.csv','wb') 127 | f.write(u'\ufeff'.encode('utf8')) 128 | w = csv.DictUnicodeWriter(f,sorted(D1.keys())) 129 | w.writeheader() 130 | w.writerows([D1,D2]) 131 | f.close() 132 | ''' 133 | -------------------------------------------------------------------------------- /src/upsample_blob_data.py: -------------------------------------------------------------------------------- 1 | import expdir 2 | 3 | import torch 4 | import torch.nn as nn 5 | from torch.autograd import Variable 6 | import loadseg 7 | import time 8 | 9 | import numpy as np 10 | 11 | 12 | def get_seg_size(input_dim): 13 | if input_dim == [227, 227]: 14 | seg_size = (113, 113) 15 | elif input_dim == [224, 224]: 16 | seg_size = (112, 112) 17 | elif input_dim == [384, 384]: 18 | seg_size = (192, 192) 19 | else: 20 | print input_dim 21 | assert(False) 22 | return seg_size 23 | 24 | 25 | def upsample_blob_data(directory, blob, batch_size=64, verbose=True): 26 | ed = expdir.ExperimentDirectory(directory) 27 | if ed.has_mmap(blob=blob, part='upsampled'): 28 | print('%s already has %s, so skipping.' % (directory, 29 | ed.mmap_filename(blob=blob, part='upsampled'))) 30 | info = ed.load_info() 31 | blob_info = ed.load_info(blob=blob) 32 | shape = blob_info.shape 33 | N = shape[0] 34 | seg_size = get_seg_size(info.input_dim) 35 | 36 | blobdata = ed.open_mmap(blob=blob, mode='r', shape=shape) 37 | if verbose: 38 | print 'Creating new mmap at %s' % ed.mmap_filename(blob=blob, part='upsampled') 39 | upsampled_data = ed.open_mmap(blob=blob, part='upsampled', mode='w+', 40 | shape=(shape[0], shape[1], seg_size[0], seg_size[1])) 41 | 42 | up = nn.Upsample(size=seg_size, mode='bilinear') 43 | 44 | start_time = time.time() 45 | last_batch_time = start_time 46 | for i in range(int(np.ceil(np.true_divide(N, batch_size)))): 47 | if (i+1)*batch_size < N: 48 | idx = range(i*batch_size, (i+1)*batch_size) 49 | else: 50 | idx = range(i*batch_size, N) 51 | batch_time = time.time() 52 | rate = idx[-1] / (batch_time - start_time + 1e-15) 53 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 54 | last_batch_time = batch_time 55 | if verbose: 56 | print 'upsample_blob_data %d/%d (%.2f)\titems per sec %.2f\t%.2f' % ( 57 | idx[-1], N, idx[-1]/float(N), batch_rate, rate) 58 | inp = Variable(torch.Tensor(blobdata[idx])) 59 | out = up(inp).data.cpu().numpy() 60 | upsampled_data[idx] = np.copy(out) 61 | 62 | if verbose: 63 | print 'Renaming mmap.' 64 | ed.finish_mmap(upsampled_data) 65 | 66 | 67 | if __name__ == '__main__': 68 | import sys 69 | import traceback 70 | import argparse 71 | 72 | try: 73 | parser = argparse.ArgumentParser(description='TODO') 74 | parser.add_argument( 75 | '--directory', 76 | default='.', 77 | help='output directory for the net probe') 78 | parser.add_argument( 79 | '--blobs', 80 | nargs='*', 81 | help='network blob names to collect' 82 | ) 83 | parser.add_argument( 84 | '--batch_size', 85 | type=int, 86 | default=64, 87 | help='TODO') 88 | 89 | args = parser.parse_args() 90 | for blob in args.blobs: 91 | upsample_blob_data(args.directory, blob, batch_size=args.batch_size, verbose=True) 92 | except: 93 | traceback.print_exc(file=sys.stdout) 94 | sys.exit(1) 95 | -------------------------------------------------------------------------------- /src/voclabels.py: -------------------------------------------------------------------------------- 1 | from collections import OrderedDict 2 | # 259 is autobus, more for car, table, plant, train 3 | voc_labels = OrderedDict([('airplane', 154), ('bicycle', 115), ('bird', 128), ('boat', 123), ('bottle', 70), 4 | ('bus', 203), ('car', 38), ('cat', 105), ('chair', 36), ('cow', 251), ('table', 27), 5 | ('dog', 93), ( 'horse', 183), ('motorbike', 184), ( 'person', 19), ( 'pottedplant', 145), 6 | ('sheep', 243), ('sofa', 63), ('train', 185), ('tvmonitor', 168)]) 7 | -------------------------------------------------------------------------------- /src/w2color.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ruthcfong/net2vec/4d314fe2975762532afde7f7cf1c7f4a71ce1045/src/w2color.npy -------------------------------------------------------------------------------- /zoo/README.md: -------------------------------------------------------------------------------- 1 | Model Zoo directory 2 | =================== 3 | 4 | Add caffemodel and deploy prototxt files to this directory. 5 | Scripts are set up assuming that you follow the naming convention 6 | 7 | [modelname].caffemodel 8 | [modelname].prototxt 9 | --------------------------------------------------------------------------------