├── .gitignore ├── LICENSE.md ├── README.md ├── dataset └── README.md ├── dissection └── README.md ├── plot ├── ResNet-152-model_imagenet.csv ├── ResNet-152-model_places365.csv ├── caffe_reference_imagenet.csv ├── caffe_reference_places205.csv ├── caffe_reference_places365.csv ├── extract_csv.m ├── getPrintName.m ├── network2layer.m ├── plot_histogram.m ├── plot_semantics.m ├── printLabels.m ├── result.txt ├── return_annotation.m ├── semantics_allnetwork.pdf ├── semantics_cvpr_release.mat └── semantics_samples.mat ├── script ├── dlbroden.sh ├── dlbroden_227.sh ├── dlzoo.sh ├── dlzoo_example.sh ├── makebroden.sh ├── makelinks.sh ├── rundissect.sh ├── rundissect_pytorch.sh └── rundissect_pytorch_external.sh ├── src ├── ade20k.py ├── adeseg.py ├── bargraph.py ├── colorname.py ├── dtdseg.py ├── expdir.py ├── fieldmap.py ├── graphprobe.py ├── intersect.py ├── joinseg.py ├── labelprobe.py ├── loadseg.py ├── makeresult.py ├── maxprobe.py ├── netprobe.py ├── netprobe_pytorch.py ├── osseg.py ├── pascalseg.py ├── printmean.py ├── quantile.py ├── quantprobe.py ├── report.py ├── rotate.py ├── synonym.py ├── unicsv.py ├── upsample.py ├── vecquantile.py ├── viewprobe.py └── w2color.npy └── zoo └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .* 2 | *.pyc 3 | dataset 4 | dissection 5 | zoo 6 | !/dataset/README.md 7 | !/dissection/README.md 8 | !/zoo/README.md 9 | !/.gitignore 10 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT LICENSE 2 | =========== 3 | 4 | Copyright (c) 2017 MIT, Computer Science and Artificial Intelligence Laboratory, David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 7 | 8 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 11 | 12 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Network Dissection 2 | 3 | ## News (Jan.18, 2018)! 4 | 5 | **We release a light and portable version of Network Dissection in pyTorch at [NetDissect-Lite](https://github.com/CSAILVision/NetDissect-Lite). It is much faster than this first version and the code structure is cleaned up, without any complex shell commands. It takes about 30 min for a resnet18 model and 2 hours for a densenet161. If you have questions, please open issues at NetDissect-Lite** 6 | 7 | ## Introduction 8 | This repository contains the demo code for the [CVPR'17 paper](http://netdissect.csail.mit.edu/final-network-dissection.pdf) Network Dissection: Quantifying Interpretability of Deep Visual Representations. You can use this code with naive [Caffe](https://github.com/BVLC/caffe), with matcaffe and pycaffe compiled. We also provide a [PyTorch wrapper](script/rundissect_pytorch.sh) to apply NetDissect to probe networks in PyTorch format. There are dissection results for several networks at the [project page](http://netdissect.csail.mit.edu/). 9 | 10 | This code includes 11 | 12 | * Code to run network dissection on an arbitrary deep convolutional 13 | neural network provided as a Caffe deploy.prototxt and .caffemodel. 14 | The script `rundissect.sh` runs all the needed phases. 15 | 16 | * Code to create the merged Broden dataset from constituent datasets 17 | ADE, PASCAL, PASCAL Parts, PASCAL Context, OpenSurfaces, and DTD. 18 | The script `makebroden.sh` runs all the needed steps. 19 | 20 | 21 | ## Download 22 | * Clone the code of Network Dissection from github 23 | ``` 24 | https://github.com/CSAILVision/NetDissect.git 25 | cd NetDissect 26 | ``` 27 | * Download the Broden dataset (~1GB space) and the example pretrained models. 28 | ``` 29 | script/dlbroden_227.sh 30 | script/dlzoo_example.sh 31 | ``` 32 | 33 | Note that you can run ```script/dlbroden.sh``` to download Broden dataset with images in all three resolution (227x227,224x224,384x384), or run ```script/dlzoo.sh``` to download more CNN models. AlexNet models work with 227x227 image input, while VGG, ResNet, GoogLeNet works with 224x224 image input. 34 | 35 | ## Run in Caffe 36 | * Run Network Dissection in Caffe to probe the conv5 layer of the AlexNet trained on Places365. Results will be saved to ```dissection/caffe_reference_model_places365/```, in which ```html``` contains the visualization of all the units in a html page and ```conv5-result.csv``` contains the raw predicted labels for each unit. The code takes about 40 mintues to run, and it will generate about 1.5GB intermediate results (mmap) for one layer, which you could delete after the code finishes running. 37 | 38 | ``` 39 | script/rundissect.sh --model caffe_reference_places365 --layers "conv5" --dataset dataset/broden1_227 --resolution 227 40 | ``` 41 | 42 | * Run Network Dissection to compare three layers of AlexNet trained on ImageNet. Results will be saved to ```dissection/caffe_reference_model_imagenet/```. 43 | 44 | ``` 45 | script/rundissect.sh --model caffe_reference_imagenet --layers "conv3 conv4 conv5" --dataset dataset/broden1_227 --resolution 227 46 | ``` 47 | 48 | * If you need to regenerate the Broden dataset from scratch, you can run ```script/makebroden.sh```. The script will download the pieces and merge them. 49 | 50 | * Network dissection depends on scipy as well as pycaffe. [Details on installing pycaffe can be found here](http://caffe.berkeleyvision.org/tutorial/interfaces.html#python). 51 | 52 | ## Run in PyTorch 53 | 54 | * Run Network Dissection in PyTorch. Please install [PyTorch](http://pytorch.org/) and [Torchvision](https://github.com/pytorch/vision) first. We provide a [feature extract wrapper](src/netprobe_pytorch.py) for PyTorch. So you could run ```script/rundissect_pytorch.sh``` to probe the existing networks trained on ImageNet in [Torchvision](https://github.com/pytorch/vision/tree/master/torchvision/models). 55 | 56 | ``` 57 | script/rundissect_pytorch.sh 58 | ``` 59 | 60 | * Or try ```script/rundissect_pytorch_external.sh``` on a resnet18 trained on [Places365](https://github.com/CSAILVision/places365). 61 | 62 | ``` 63 | script/rundissect_pytorch_external.sh 64 | ``` 65 | 66 | ## Report 67 | * At the end of the dissection script, a report will be generated that summarizes the semantics of the networks. For example, after you have tested the conv5 layer of caffe_reference_places365, you will have: 68 | 69 | ``` 70 | dissection/caffe_reference_places365/html/conv5.html 71 | dissection/caffe_reference_places365/html/image/conv5-bargraph.svg 72 | dissection/caffe_reference_places365/html/image/conv5-0[###].png 73 | dissection/caffe_reference_places365/conv5-result.csv 74 | ``` 75 | 76 | These are, respectively, the HTML-formatted report, the semantics of the units of the layer summarized as a bar graph, visualizations of all the units of the layer (using zero-indexed unit numbers), and a CSV file containing raw scores of the top matching semantic concepts in each category for each unit of the layer. 77 | 78 | * Dissect results of all the existing networks in mat format. After the csv file containing the raw data of the unit semantics is generated, you could use the sample scripts in ```plot/extract_csv.m``` to plot the figure. ```plot/semantics_cvpr_release.mat``` contains the semantics of all the networks analyzed in the CVPR paper. It will generate a [figure](plot/semantics_allnetwork.pdf) showing the number of unique detectors across different networks. 79 | 80 | 81 | ## Reference 82 | If you find the codes useful, please cite this paper 83 | ``` 84 | @inproceedings{netdissect2017, 85 | title={Network Dissection: Quantifying Interpretability of Deep Visual Representations}, 86 | author={Bau, David and Zhou, Bolei and Khosla, Aditya and Oliva, Aude and Torralba, Antonio}, 87 | booktitle={Computer Vision and Pattern Recognition}, 88 | year={2017} 89 | } 90 | ``` 91 | 92 | -------------------------------------------------------------------------------- /dataset/README.md: -------------------------------------------------------------------------------- 1 | Dataset directory 2 | ================= 3 | 4 | Install or link datasets at this directory. 5 | 6 | The script dlbroden.sh will download the Broden dataset here. 7 | -------------------------------------------------------------------------------- /dissection/README.md: -------------------------------------------------------------------------------- 1 | Dissection directory 2 | ==================== 3 | 4 | Indermediate and final results of network dissection are written 5 | into this directory, with final output files of the form: 6 | 7 | [modelname]/[layername].csv 8 | [modelname]/html/[layername].html 9 | 10 | -------------------------------------------------------------------------------- /plot/extract_csv.m: -------------------------------------------------------------------------------- 1 | % loading the csv files generate by NetDissect 2 | disp('extract semantics from raw data') 3 | params = {}; 4 | params.networks_name = {'ResNet-152-model_places365','ResNet-152-model_imagenet','caffe_reference_places365','caffe_reference_places205','caffe_reference_imagenet'}; 5 | params.csv_files = {'ResNet-152-model_places365.csv','ResNet-152-model_imagenet.csv','caffe_reference_places365.csv','caffe_reference_places205.csv','caffe_reference_imagenet.csv'}; 6 | 7 | params.layers_name = network2layer(params.networks_name, 'networkprobe'); 8 | 9 | 10 | [unit_semantics] = return_annotation( params ); 11 | 12 | 13 | thresh = 0.04; % IoU threshold to decide if an unit is a interpretable detector. 14 | concepts = {'object','part','scene','material','texture', 'color'}; 15 | indices_concepts = [1,6,4,3,5,2]; % (objet,part,scene,material,texture,color) as the column index inside the csv file 16 | stat = {}; 17 | stat.networks_name = params.networks_name; 18 | stat.layers_name = params.layers_name; 19 | stat.ratio_detectors = zeros(numel(params.networks_name), numel(concepts)+1); 20 | stat.num_uniquedetectors = zeros(numel(params.networks_name), numel(concepts)+1); 21 | stat.num_detectors = zeros(numel(params.networks_name), numel(concepts)+1); 22 | stat.concepts = ['all' concepts]; 23 | stat.thresh = thresh; 24 | stat.indices_concepts = indices_concepts; 25 | 26 | 27 | for netID = 1:numel(params.networks_name) 28 | semantics_network = unit_semantics{netID,3}; 29 | 30 | scores_allconcept = str2double(semantics_network(:,2:2:end)); 31 | [max_scores, max_idx] = max(scores_allconcept,[],2); 32 | num_unit = size(max_scores,1); 33 | 34 | num_detector_concepts = zeros(1,6); 35 | num_uniquedetector_concepts = zeros(1,6); 36 | for conceptID = 1:numel(indices_concepts) 37 | num_detector_overall = sum(scores_allconcept(:, indices_concepts(conceptID))>thresh); 38 | num_uniquedetector_overall = numel(unique(semantics_network(scores_allconcept(:,indices_concepts(conceptID))>thresh,indices_concepts(conceptID)*2-1))); 39 | num_detector_concepts(conceptID) = num_detector_overall; 40 | num_uniquedetector_concepts(conceptID) = num_uniquedetector_overall; 41 | end 42 | 43 | num_detector_overall = sum(max_scores>thresh); 44 | num_uniquedetector_overall = sum(num_detector_concepts);%num_detector_object_unique+num_detector_objectpart_unique+num_detector_texture_unique+num_detector_material_unique+num_detector_color_unique+num_detector_scene_unique; 45 | stat.num_detectors(netID,:) = [num_detector_overall num_detector_concepts];%[num_detector, num_detector_object, num_detector_objectpart, num_detector_scene, num_detector_material, num_detector_texture, num_detector_color]; 46 | stat.ratio_detectors(netID,:) = stat.num_detectors(netID,:)./num_unit; %[num_detector/num_unit num_detector_object/num_unit num_detector_objectpart/num_unit num_detector_scene/num_unit num_detector_material/num_unit num_detector_texture/num_unit num_detector_color/num_unit]; 47 | stat.num_uniquedetectors(netID,:) = [num_uniquedetector_overall num_uniquedetector_concepts];%[num_uniquedetector num_detector_object_unique num_detector_objectpart_unique num_detector_scene_unique num_detector_material_unique num_detector_texture_unique num_detector_color_unique]; 48 | end 49 | save('semantics_samples.mat','stat','unit_semantics'); 50 | -------------------------------------------------------------------------------- /plot/getPrintName.m: -------------------------------------------------------------------------------- 1 | function list_output = getPrintName( list_featurename, type_print) 2 | if strcmp(type_print, 'feature') 3 | printMap = construct_dictionary_feature(); 4 | elseif strcmp(type_print, 'semantics') 5 | printMap = construct_dictionary_semantics(); 6 | elseif strcmp(type_print,'width') 7 | printMap = construct_dictionary_width(); 8 | end 9 | list_output = cell(numel(list_featurename),1); 10 | 11 | for i = 1:numel(list_featurename) 12 | try 13 | list_output{i} = printMap(list_featurename{i}); 14 | catch exception 15 | error(list_featurename{i}) 16 | end 17 | end 18 | 19 | 20 | end 21 | 22 | function printMap = construct_dictionary_width() 23 | printMap = containers.Map(); 24 | printMap('caffe_reference_places365_GAP_gap') = 'AlexNet-Places365-GAP'; 25 | 26 | end 27 | 28 | function printMap = construct_dictionary_feature() 29 | 30 | printMap = containers.Map(); 31 | printMap('ResNet-152-model_imagenet-pool5') = 'ResNet152-ImageNet'; 32 | printMap('ResNet-50-model_imagenet-pool5') = 'ResNet50-ImageNet'; 33 | printMap('googlenet_imagenet-pool5/7x7_s1') = 'GoogLeNet-ImageNet'; 34 | printMap('ResNet-152-model_places365-pool5') = 'ResNet152-Places365'; 35 | printMap('vgg16_hybrid1365-fc7') = 'VGG-Hybrid'; 36 | printMap('vgg16_imagenet-fc7') = 'VGG-ImageNet'; 37 | printMap('vgg16_places205-fc7') = 'VGG-Places205'; 38 | printMap('vgg16_places365-fc7') = 'VGG-Places365'; 39 | printMap('caffe_reference_imagenetplaces205-pool5') = 'AlexNet-Hybrid'; 40 | printMap('googlenet_places205-pool5/7x7_s1') = 'GoogLeNet-Places205'; 41 | printMap('googlenet_places365-pool5/7x7_s1') = 'GoogLeNet-Places365'; 42 | printMap('caffe_reference_imagenet-pool5') = 'AlexNet-ImageNet'; 43 | printMap('caffe_reference_places205_batchnorm-pool5') = 'AlexNet-Places205-BN'; 44 | printMap('caffe_reference_places365_GAP-pool5_gap') = 'AlexNet-Places365-GAP'; 45 | printMap('caffe_reference_places365-pool5') = 'AlexNet-Places365'; 46 | printMap('caffe_reference_places205-pool5') = 'AlexNet-Places205'; 47 | printMap('weakly_deepcontext-pool5') = 'context'; 48 | printMap('weakly_colorization-pool5') = 'colorization'; 49 | printMap('weakly_audio-pool5') = 'audio'; 50 | printMap('weakly_splitbrain-pool5') = 'crosschannel'; 51 | printMap('weakly_videotracking-pool5') = 'tracking'; 52 | printMap('weakly_solvingpuzzle-pool5') = 'puzzle'; 53 | printMap('weakly_objectcentric-pool5') = 'objectcentric'; 54 | printMap('weakly_egomotion-cls_pool5') = 'egomotion'; 55 | printMap('weakly_learningbymoving-pool5') = 'moving'; 56 | printMap('caffenet_random-pool5') = 'AlexNet-random'; 57 | printMap('weakly_videoorder-pool5') = 'frameorder'; 58 | 59 | end 60 | 61 | function printMap = construct_dictionary_semantics() 62 | 63 | printMap = containers.Map(); 64 | printMap('ResNet-152-model_imagenet') = 'ResNet152-ImageNet'; 65 | printMap('ResNet-152-model_places365') = 'ResNet152-Places365'; 66 | printMap('ResNet-50-model_imagenet') = 'ResNet50-ImageNet'; 67 | printMap('resnet-152-torch-places365') = 'ResNet152-Places365'; 68 | printMap('resnet-152-torch-imagenet') = 'ResNet152-ImageNet'; 69 | printMap('vgg16_imagenet') = 'VGG-ImageNet'; 70 | printMap('vgg16_places205') = 'VGG-Places205'; 71 | printMap('vgg16_places365') = 'VGG-Places365'; 72 | printMap('vgg16_hybrid1365') = 'VGG-Hybrid'; 73 | printMap('googlenet_imagenet') = 'GoogLeNet-ImageNet'; 74 | printMap('googlenet_places205') = 'GoogLeNet-Places205'; 75 | printMap('googlenet_places365') = 'GoogLeNet-Places365'; 76 | printMap('caffenet_random') = 'AlexNet-random'; 77 | printMap('caffe_reference_imagenet') = 'AlexNet-ImageNet'; 78 | printMap('caffe_reference_places205') = 'AlexNet-Places205'; 79 | printMap('caffe_reference_imagenetplaces205') = 'AlexNet-Hybrid'; 80 | printMap('caffe_reference_places205_batchnorm') = 'AlexNet-Places205-BatchNorm'; 81 | printMap('caffe_reference_places365') = 'AlexNet-Places365'; 82 | printMap('caffe_reference_places365_GAP') = 'AlexNet-Places365-GAP'; 83 | printMap('caffe_reference_places365_GAP1024') = 'AlexNet-Places365-GAP1024'; 84 | printMap('caffe_reference_places205_nodropout') = 'AlexNet-Places205-NoDropout'; 85 | printMap('caffe_reference_places205_repeat1') = 'AlexNet-Places205-repeat1'; 86 | printMap('caffe_reference_places205_repeat2') = 'AlexNet-Places205-repeat2'; 87 | printMap('caffe_reference_places205_repeat3') = 'AlexNet-Places205-repeat3'; 88 | printMap('caffe_reference_places205_repeat4') = 'AlexNet-Places205-repeat4'; 89 | 90 | printMap('weakly_deepcontext') = 'context'; 91 | printMap('weakly_audio') = 'audio'; 92 | printMap('weakly_videoorder') = 'frameorder'; 93 | printMap('weakly_videotracking') = 'tracking'; 94 | printMap('weakly_egomotion') = 'egomotion'; 95 | printMap('weakly_learningbymoving') = 'moving'; 96 | printMap('weakly_objectcentric') = 'objectcentric'; 97 | printMap('weakly_solvingpuzzle') = 'puzzle'; 98 | printMap('weakly_colorization') = 'colorization'; 99 | printMap('weakly_splitbrain') = 'crosschannel'; 100 | 101 | end -------------------------------------------------------------------------------- /plot/network2layer.m: -------------------------------------------------------------------------------- 1 | function [layers] = network2layer( networks, type_networks) 2 | %NETWORK2FEATURE Summary of this function goes here 3 | % Detailed explanation goes here 4 | 5 | layers = cell(numel(networks),1); 6 | if strcmp(type_networks,'genericfeature') 7 | layerMap = get_layerMap_genericfeature(); 8 | elseif strcmp(type_networks, 'networkprobe') 9 | layerMap = get_layerMap_networkprobe(); 10 | end 11 | 12 | try 13 | for netID = 1:numel(networks) 14 | if strcmp(networks{netID}(1:11),'places_iter') 15 | layers{netID} = 'conv5'; 16 | else 17 | layers{netID} = layerMap(networks{netID}); 18 | end 19 | end 20 | catch exception 21 | 22 | error(['no info for ' networks{netID}]); 23 | end 24 | 25 | end 26 | 27 | function layerMap = get_layerMap_networkprobe() 28 | layerMap = containers.Map(); 29 | layerMap('caffenet_random') = 'conv5'; 30 | layerMap('caffe_reference_imagenet') = 'conv5'; 31 | layerMap('caffe_reference_places205_bn') = 'conv5'; 32 | layerMap('caffe_reference_places205') = 'conv5'; 33 | layerMap('caffe_reference_places365') = 'conv5'; 34 | layerMap('caffe_reference_places205_batchnorm') = 'conv5'; 35 | layerMap('caffe_reference_imagenetplaces205') = 'conv5'; 36 | layerMap('caffe_reference_places365_GAP') = 'conv5'; 37 | layerMap('caffe_reference_places365_GAPplus') = 'conv5'; 38 | layerMap('caffe_reference_places365_GAP1024') = 'conv5'; 39 | layerMap('caffe_reference_places205_repeat1') = 'conv5'; 40 | layerMap('caffe_reference_places205_repeat2') = 'conv5'; 41 | layerMap('caffe_reference_places205_repeat3') = 'conv5'; 42 | layerMap('caffe_reference_places205_repeat4') = 'conv5'; 43 | layerMap('caffe_reference_places205_nodropout') = 'conv5'; 44 | 45 | layerMap('vgg16_hybrid1365') = 'conv5_3'; 46 | layerMap('vgg16_imagenet') = 'conv5_3'; 47 | layerMap('vgg16_places205') = 'conv5_3'; 48 | layerMap('vgg16_places365') = 'conv5_3'; 49 | 50 | layerMap('weakly_audio') = 'conv5'; 51 | layerMap('weakly_deepcontext') = 'conv5'; 52 | layerMap('weakly_videoorder') = 'conv5'; 53 | layerMap('weakly_videotracking') = 'conv5'; 54 | layerMap('weakly_egomotion') = 'cls_conv5'; 55 | layerMap('weakly_learningbymoving') = 'conv5'; 56 | layerMap('weakly_objectcentric') = 'conv5'; 57 | layerMap('weakly_solvingpuzzle') = 'conv5_s1'; 58 | layerMap('weakly_colorization') = 'conv5'; 59 | layerMap('weakly_splitbrain') = 'conv5'; 60 | 61 | layerMap('googlenet_imagenet') = 'inception_5b-output'; 62 | layerMap('googlenet_places205') = 'inception_5b-output'; 63 | layerMap('googlenet_places365') = 'inception_5b-output'; 64 | 65 | layerMap('ResNet-152-model_imagenet') = 'res5c'; 66 | layerMap('ResNet-152-model_places365') = 'res5c'; 67 | layerMap('ResNet-50-model_imagenet') = 'res5c'; 68 | layerMap('resnet-152-torch-places365') = 'caffe.Eltwise_510'; 69 | layerMap('resnet-152-torch-imagenet') = 'caffe.Eltwise_510'; 70 | layerMap('rotation_020') = 'conv5'; 71 | layerMap('rotation_040') = 'conv5'; 72 | layerMap('rotation_060') = 'conv5'; 73 | layerMap('rotation_080') = 'conv5'; 74 | layerMap('rotation_100') = 'conv5'; 75 | end 76 | 77 | function layerMap = get_layerMap_genericfeature() 78 | layerMap = containers.Map(); 79 | layerMap('caffenet_random') = 'pool5'; 80 | layerMap('caffe_reference_imagenet') = 'pool5'; 81 | layerMap('caffe_reference_places205_bn') = 'pool5'; 82 | layerMap('caffe_reference_places205') = 'pool5'; 83 | layerMap('caffe_reference_places365') = 'pool5'; 84 | layerMap('caffe_reference_places365_GAP') = 'pool5_gap'; 85 | layerMap('caffe_reference_places205_batchnorm') = 'pool5'; 86 | 87 | layerMap('caffe_reference_imagenetplaces205') = 'pool5'; 88 | layerMap('vgg16_hybrid1365') = 'fc7'; 89 | layerMap('vgg16_imagenet') = 'fc7'; 90 | layerMap('vgg16_places205') = 'fc7'; 91 | layerMap('vgg16_places365') = 'fc7'; 92 | 93 | layerMap('weakly_audio') = 'pool5'; 94 | layerMap('weakly_deepcontext') = 'pool5'; 95 | layerMap('weakly_videoorder') = 'pool5'; 96 | layerMap('weakly_videotracking') = 'pool5'; 97 | layerMap('weakly_egomotion') = 'cls_pool5'; 98 | layerMap('weakly_learningbymoving') = 'pool5'; 99 | layerMap('weakly_objectcentric') = 'pool5'; 100 | layerMap('weakly_solvingpuzzle') = 'pool5'; 101 | layerMap('colorization_berkeley') = 'pool5'; 102 | layerMap('googlenet_imagenet') = 'pool5/7x7_s1'; 103 | layerMap('googlenet_places205') = 'pool5/7x7_s1'; 104 | layerMap('googlenet_places365') = 'pool5/7x7_s1'; 105 | layerMap('ResNet-152-model_imagenet') = 'pool5'; 106 | layerMap('ResNet-152-model_places365') = 'pool5'; 107 | layerMap('ResNet-50-model_imagenet') = 'pool5'; 108 | layerMap('weakly_colorization') = 'pool5'; 109 | layerMap('weakly_colorization_nors') = 'pool5'; 110 | layerMap('weakly_splitbrain') = 'pool5'; 111 | layerMap('weakly_splitbrain_nors') = 'pool5'; 112 | end 113 | -------------------------------------------------------------------------------- /plot/plot_histogram.m: -------------------------------------------------------------------------------- 1 | % plot the histogram of all the detectors the network dissection identifies 2 | % for the given networks. 3 | clear 4 | load('semantics_samples.mat'); 5 | thresh = stat.thresh; 6 | concepts = stat.concepts(2:end);% the first one is the all the detector 7 | indices_concepts = stat.indices_concepts; 8 | 9 | concepts_select = {'object','texture','scene'}; % select the concepts to plot 10 | [concepts_select, ia, ib] = intersect(concepts_select, concepts); 11 | indices_concepts_select = indices_concepts(ib); 12 | 13 | for netID = 1:numel(stat.networks_name) 14 | figure 15 | semantics_network = unit_semantics{netID,3}; 16 | num_unit = size(semantics_network,1); 17 | scores_allconcept = str2double(semantics_network(:,2:2:end)); 18 | 19 | for conceptID = 1:numel(concepts_select) 20 | detector_concept = semantics_network(scores_allconcept(:,indices_concepts_select(conceptID))>thresh, indices_concepts_select(conceptID)*2-1); 21 | 22 | [unique_data,junk,ind] = unique(detector_concept); 23 | freq_unique_data = histc(ind,1:numel(unique_data)); 24 | [value_sort, idx_sort] = sort(freq_unique_data,'descend'); 25 | uniquedetectors = unique_data(idx_sort); 26 | freq_detectors = value_sort; 27 | 28 | subplot(numel(concepts_select),1,conceptID); 29 | set(gcf,'Color',[1 1 1]); 30 | 31 | bar( freq_detectors,'stacked'), 32 | title(sprintf('Histogram of %s detectors', concepts_select{conceptID})); 33 | xticks([1:numel(uniquedetectors)]) 34 | xticklabels(printLabels(uniquedetectors)), xtickangle(45) 35 | 36 | set(gca,'FontSize',20); 37 | xlim(gca,[0 numel(uniquedetectors)+1]) 38 | end 39 | xlabel(strrep(stat.networks_name{netID},'_','-')); 40 | end -------------------------------------------------------------------------------- /plot/plot_semantics.m: -------------------------------------------------------------------------------- 1 | % sample script to plot the summary of detector numbers for all the 2 | % networks 3 | % run extract_csv.m first to extract the semantics from the raw csv if you 4 | % jush finish running network dissection 5 | 6 | clear 7 | load('semantics_cvpr_release.mat'); 8 | 9 | selectIDX_networks = [1:numel(stat.networks_name)]; 10 | 11 | stat.networks_name = stat.networks_name(selectIDX_networks); 12 | stat.layers_name = stat.layers_name(selectIDX_networks); 13 | stat.ratio_detectors = stat.ratio_detectors(selectIDX_networks,:); 14 | stat.num_uniquedetectors = stat.num_uniquedetectors(selectIDX_networks,:); 15 | stat.num_detectors = stat.num_detectors(selectIDX_networks,:); 16 | 17 | sum_uniquedetectors = sum(stat.num_uniquedetectors(:,2:end),2); 18 | [value_sort, idx_sort] = sort(sum_uniquedetectors,'descend'); 19 | stat.networks_name = stat.networks_name(idx_sort); 20 | stat.layers_name = stat.layers_name(idx_sort); 21 | stat.ratio_detectors = stat.ratio_detectors(idx_sort,:); 22 | stat.num_uniquedetectors = stat.num_uniquedetectors(idx_sort,:); 23 | stat.num_detectors = stat.num_detectors(idx_sort,:); 24 | 25 | disp(stat); 26 | networks_print = getPrintName(stat.networks_name,'semantics'); 27 | 28 | % figure, 29 | % %plot([1:size(stat.ratio_detectors,1)], stat.ratio_detectors', '--o'), 30 | % bar(stat.ratio_detectors, 'stacked'), 31 | % legend(stat.concepts),title('Ratio of detectors'); 32 | % xticks([1:size(stat.ratio_detectors,1)]) 33 | % xticklabels(networks_print),xtickangle(45) 34 | 35 | figure, 36 | subplot(1,2,1); 37 | bar( stat.num_uniquedetectors(:,2:end),'stacked'), 38 | legend(stat.concepts(2:end)),title('Number of unique detectors'); 39 | xticks([1:size(stat.ratio_detectors,1)]) 40 | xticklabels(networks_print), xtickangle(45) 41 | 42 | subplot(1,2,2); 43 | bar( stat.num_detectors(:,2:end),'stacked'), 44 | legend(stat.concepts(2:end)),title('Number of detectors'); 45 | xticks([1:size(stat.num_detectors,1)]) 46 | xticklabels(networks_print), xtickangle(45) -------------------------------------------------------------------------------- /plot/printLabels.m: -------------------------------------------------------------------------------- 1 | function [list_output] = printLabels( list_input) 2 | % remove some legacy signs in the names of concepts 3 | list_output = cell(numel(list_input),1); 4 | for i = 1:numel(list_input) 5 | tmp = list_input{i}; 6 | tmp(1:end-2) = strrep(tmp(1:end-2),'_',' '); 7 | tmp(1:end-2) = strrep(tmp(1:end-2),'-',' '); 8 | tmp = strrep(tmp,'-s',''); 9 | tmp = strrep(tmp,'-c',''); 10 | list_output{i} = tmp; 11 | end 12 | 13 | 14 | end 15 | 16 | -------------------------------------------------------------------------------- /plot/result.txt: -------------------------------------------------------------------------------- 1 | ./vgg16_places205/backup/conv5_3-result.csv 2 | ./vgg16_places205/conv3_3-result.csv 3 | ./vgg16_places205/conv5_3-result.csv 4 | ./vgg16_places205/conv4_3-result.csv 5 | ./places_iter_492/conv5-result.csv 6 | ./places_iter_492/conv3-result.csv 7 | ./places_iter_492/conv4-result.csv 8 | ./googlenet_imagenet/inception_4e-output-result.csv 9 | ./googlenet_imagenet/inception_5b-output-backup.csv 10 | ./googlenet_imagenet/inception_3b-output-result.csv 11 | ./googlenet_imagenet/inception_5b-output-result.csv 12 | ./googlenet_imagenet/conv2-norm2-result.csv 13 | ./places_iter_27396/conv5-result.csv 14 | ./places_iter_27396/conv3-result.csv 15 | ./places_iter_27396/conv4-result.csv 16 | ./caffe_reference_places205_batchnorm/conv5-result.csv 17 | ./caffe_reference_places205_batchnorm/conv2-result.csv 18 | ./caffe_reference_places205_batchnorm/conv3-result.csv 19 | ./caffe_reference_places205_batchnorm/conv4-result.csv 20 | ./caffe_reference_places205_batchnorm/conv1-result.csv 21 | ./places_iter_1/conv5-result.csv 22 | ./places_iter_1/conv3-result.csv 23 | ./places_iter_1/conv4-result.csv 24 | ./weakly_objectcentric/conv5-result.csv 25 | ./places_iter_600818/conv5-result.csv 26 | ./places_iter_600818/conv3-result.csv 27 | ./places_iter_600818/conv4-result.csv 28 | ./colorization_berkeley/conv5-result.csv 29 | ./places_iter_60491/conv3-result.csv 30 | ./places_iter_60491/conv4-result.csv 31 | ./places_iter_60491/conv5-result.csv 32 | ./places_iter_12164/conv5-result.csv 33 | ./places_iter_12164/conv4-result.csv 34 | ./places_iter_12164/conv3-result.csv 35 | ./rotation_060/conv5-result.csv 36 | ./rotation_060/conv2-result.csv 37 | ./rotation_060/conv1-result.csv 38 | ./rotation_060/conv3-result.csv 39 | ./rotation_060/conv4-result.csv 40 | ./vgg16_imagenet/conv4_3-result.csv 41 | ./vgg16_imagenet/conv5_3-result.csv 42 | ./vgg16_imagenet/conv3_3-result.csv 43 | ./places_iter_601603/conv5-result.csv 44 | ./places_iter_601603/conv3-result.csv 45 | ./places_iter_601603/conv4-result.csv 46 | ./caffe_reference_places205_repeat2/conv2-result.csv 47 | ./caffe_reference_places205_repeat2/conv5-result.csv 48 | ./caffe_reference_places205_repeat2/conv4-result.csv 49 | ./caffe_reference_places205_repeat2/conv3-result.csv 50 | ./caffe_reference_places205_repeat2/conv1-result.csv 51 | ./weakly_splitbrain/conv5-result.csv 52 | ./vgg16_hybrid1365/conv3_3-result.csv 53 | ./vgg16_hybrid1365/conv5_3-result.csv 54 | ./vgg16_hybrid1365/conv4_3-result.csv 55 | ./weakly_audio/conv5-result.csv 56 | ./places_iter_1200818/conv5-result.csv 57 | ./places_iter_1200818/conv3-result.csv 58 | ./places_iter_1200818/conv4-result.csv 59 | ./rotation_080/conv3-result.csv 60 | ./rotation_080/conv4-result.csv 61 | ./rotation_080/conv1-result.csv 62 | ./rotation_080/conv5-result.csv 63 | ./rotation_080/conv2-result.csv 64 | ./places_iter_300818/conv5-result.csv 65 | ./places_iter_300818/conv3-result.csv 66 | ./places_iter_300818/conv4-result.csv 67 | ./vgg16_places365/conv4_3-result.csv 68 | ./vgg16_places365/conv5_3-result.csv 69 | ./vgg16_places365/conv3_3-result.csv 70 | ./places_iter_99/conv4-result.csv 71 | ./places_iter_99/conv3-result.csv 72 | ./places_iter_99/conv5-result.csv 73 | ./places_iter_2/conv3-result.csv 74 | ./places_iter_2/conv4-result.csv 75 | ./places_iter_2/conv5-result.csv 76 | ./rotation_100/conv1-result.csv 77 | ./rotation_100/conv3-result.csv 78 | ./rotation_100/conv4-result.csv 79 | ./rotation_100/conv5-result.csv 80 | ./rotation_100/conv2-result.csv 81 | ./caffe_reference_imagenetplaces205/conv2-result.csv 82 | ./caffe_reference_imagenetplaces205/conv5-result.csv 83 | ./caffe_reference_imagenetplaces205/conv1-result.csv 84 | ./caffe_reference_imagenetplaces205/conv4-result.csv 85 | ./caffe_reference_imagenetplaces205/conv3-result.csv 86 | ./caffe_reference_places365_GAP/conv5-result.csv 87 | ./caffe_reference_places205_repeat1/conv5-result.csv 88 | ./caffe_reference_places205_repeat1/conv2-result.csv 89 | ./caffe_reference_places205_repeat1/conv1-result.csv 90 | ./caffe_reference_places205_repeat1/conv3-result.csv 91 | ./caffe_reference_places205_repeat1/conv4-result.csv 92 | ./caffe_reference_places205_repeat3/conv2-result.csv 93 | ./caffe_reference_places205_repeat3/conv5-result.csv 94 | ./caffe_reference_places205_repeat3/conv1-result.csv 95 | ./caffe_reference_places205_repeat3/conv4-result.csv 96 | ./caffe_reference_places205_repeat3/conv3-result.csv 97 | ./places_iter_136238/conv5-result.csv 98 | ./places_iter_136238/conv3-result.csv 99 | ./places_iter_136238/conv4-result.csv 100 | ./places_iter_1108/conv5-result.csv 101 | ./places_iter_1108/conv3-result.csv 102 | ./places_iter_1108/conv4-result.csv 103 | ./resnet-152-torch-imagenet/caffe.Eltwise_34-result.csv 104 | ./resnet-152-torch-imagenet/caffe.Eltwise_510-result.csv 105 | ./resnet-152-torch-imagenet/caffe.Eltwise_238-result.csv 106 | ./resnet-152-torch-imagenet/caffe.BN_1-result.csv 107 | ./resnet-152-torch-imagenet/caffe.Eltwise_116-result.csv 108 | ./resnet-152-torch-imagenet/caffe.Eltwise_478-result.csv 109 | ./resnet-152-torch-imagenet/caffe.Eltwise_358-result.csv 110 | ./rotation_040/conv4-result.csv 111 | ./rotation_040/conv3-result.csv 112 | ./rotation_040/conv1-result.csv 113 | ./rotation_040/conv2-result.csv 114 | ./rotation_040/conv5-result.csv 115 | ./weakly_videoorder/conv5-result.csv 116 | ./caffe_reference_places365_GAPplus/conv5-result.csv 117 | ./caffe_reference_places365_GAPplus/conv6-result.csv 118 | ./caffenet_random/conv5-result.csv 119 | ./ResNet-152-model_imagenet/res4b24-result.csv 120 | ./ResNet-152-model_imagenet/res5c-result.csv 121 | ./ResNet-152-model_imagenet/res4b12-result.csv 122 | ./ResNet-152-model_imagenet/res4b35-result.csv 123 | ./ResNet-152-model_imagenet/res3b7-result.csv 124 | ./weakly_colorization/conv5-result.csv 125 | ./googlenet_places205/inception_5b-output-result.csv 126 | ./googlenet_places205/inception_4e-output-result.csv 127 | ./googlenet_places205/conv2-norm2-result.csv 128 | ./googlenet_places205/inception_3b-output-result.csv 129 | ./caffe_reference_places205_nodropout/conv1-result.csv 130 | ./caffe_reference_places205_nodropout/conv4-result.csv 131 | ./caffe_reference_places205_nodropout/conv3-result.csv 132 | ./caffe_reference_places205_nodropout/conv2-result.csv 133 | ./caffe_reference_places205_nodropout/conv5-result.csv 134 | ./places_iter_5509/conv5-result.csv 135 | ./places_iter_5509/conv4-result.csv 136 | ./places_iter_5509/conv3-result.csv 137 | ./caffe_reference_places205_repeat4/conv3-result.csv 138 | ./caffe_reference_places205_repeat4/conv4-result.csv 139 | ./caffe_reference_places205_repeat4/conv1-result.csv 140 | ./caffe_reference_places205_repeat4/conv5-result.csv 141 | ./caffe_reference_places205_repeat4/conv2-result.csv 142 | ./weakly_egomotion/cls_conv5-result.csv 143 | ./caffe_reference_places205/conv1-result.csv 144 | ./caffe_reference_places205/conv3-result.csv 145 | ./caffe_reference_places205/conv4-result.csv 146 | ./caffe_reference_places205/conv5-result.csv 147 | ./caffe_reference_places205/conv2-result.csv 148 | ./ResNet-152-model_places365/res3b7-result.csv 149 | ./ResNet-152-model_places365/res4b24-result.csv 150 | ./ResNet-152-model_places365/res4b12-result.csv 151 | ./ResNet-152-model_places365/res4b35-result.csv 152 | ./ResNet-152-model_places365/res5c-result.csv 153 | ./weakly_deepcontext/conv5-result.csv 154 | ./places_iter_223/conv5-result.csv 155 | ./places_iter_223/conv3-result.csv 156 | ./places_iter_223/conv4-result.csv 157 | ./places_iter_2446/conv4-result.csv 158 | ./places_iter_2446/conv3-result.csv 159 | ./places_iter_2446/conv5-result.csv 160 | ./resnet-152-torch-places365/caffe.Eltwise_238-result.csv 161 | ./resnet-152-torch-places365/caffe.Eltwise_34-result.csv 162 | ./resnet-152-torch-places365/caffe.BN_1-result.csv 163 | ./resnet-152-torch-places365/caffe.Eltwise_478-result.csv 164 | ./resnet-152-torch-places365/caffe.Eltwise_358-result.csv 165 | ./resnet-152-torch-places365/caffe.Eltwise_116-result.csv 166 | ./resnet-152-torch-places365/caffe.Eltwise_510-result.csv 167 | ./places_iter_4/conv5-result.csv 168 | ./places_iter_4/conv3-result.csv 169 | ./places_iter_4/conv4-result.csv 170 | ./weakly_videotracking/conv5-result.csv 171 | ./weakly_solvingpuzzle/conv5_s1-result.csv 172 | ./ResNet-50-model_imagenet/res4f-result.csv 173 | ./ResNet-50-model_imagenet/res3d-result.csv 174 | ./ResNet-50-model_imagenet/res5c-result.csv 175 | ./caffe_reference_places365/conv5-result.csv 176 | ./places_iter_44/conv4-result.csv 177 | ./places_iter_44/conv3-result.csv 178 | ./places_iter_44/conv5-result.csv 179 | ./places_iter_9/conv3-result.csv 180 | ./places_iter_9/conv4-result.csv 181 | ./places_iter_9/conv5-result.csv 182 | ./weakly_learningbymoving/conv5-result.csv 183 | ./places_iter_2400818/conv4-result.csv 184 | ./places_iter_2400818/conv3-result.csv 185 | ./places_iter_2400818/conv5-result.csv 186 | ./rotation_020/conv1-result.csv 187 | ./rotation_020/conv4-result.csv 188 | ./rotation_020/conv3-result.csv 189 | ./rotation_020/conv2-result.csv 190 | ./rotation_020/conv5-result.csv 191 | ./caffe_reference_imagenet/conv3-result.csv 192 | ./caffe_reference_imagenet/conv4-result.csv 193 | ./caffe_reference_imagenet/conv1-result.csv 194 | ./caffe_reference_imagenet/conv5-result.csv 195 | ./caffe_reference_imagenet/conv2-result.csv 196 | ./places_iter_20/conv5-result.csv 197 | ./places_iter_20/conv4-result.csv 198 | ./places_iter_20/conv3-result.csv 199 | ./googlenet_places365/inception_5b-output-result.csv 200 | ./googlenet_places365/inception_3b-output-result.csv 201 | ./googlenet_places365/inception_4e-output-result.csv 202 | -------------------------------------------------------------------------------- /plot/return_annotation.m: -------------------------------------------------------------------------------- 1 | function [unit_activations] = return_annotation(params) 2 | % extract semantics from the raw csv file. 3 | 4 | unit_activations = cell(numel(params.networks_name),4); 5 | 6 | for i = 1:numel(params.csv_files) 7 | 8 | semantics_file = fullfile(params.csv_files{i}); 9 | fprintf('processing %s\n', semantics_file) 10 | 11 | data = fopen(semantics_file); 12 | csv_data = textscan(data,'%s','Delimiter','\n'); 13 | semantics_headers = textscan(csv_data{1}{1},'%s','Delimiter',','); 14 | semantics_headers = semantics_headers{1}; 15 | num_units = numel(csv_data{1})-1; 16 | semantics_values = cell(num_units, numel(semantics_headers)); 17 | for unitID = 1:num_units 18 | C = textscan(csv_data{1}{unitID+1},'%s','Delimiter',','); 19 | D = C{1}; 20 | semantics_values(str2double(D{1}),:) = D; 21 | end 22 | unit_activations{i, 1} = params.networks_name{i}; 23 | unit_activations{i, 2} = params.layers_name{i}; 24 | unit_activations{i, 3} = select_semantics(semantics_values); 25 | unit_activations{i, 4} = semantics_headers; 26 | fclose(data); 27 | end 28 | 29 | end 30 | 31 | function semantics_new = select_semantics(semantics) 32 | tmp_1 = [5:5:size(semantics,2)]; 33 | tmp_2 = [9:5:size(semantics,2)]; 34 | index_concepts = [tmp_1, tmp_2]; 35 | index_concepts = sort(index_concepts,'ascend'); 36 | semantics_new = semantics(:, index_concepts); 37 | end 38 | -------------------------------------------------------------------------------- /plot/semantics_allnetwork.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CSAILVision/NetDissect/20ce6f887607aadc04ab9fb3d452e4723685cbbb/plot/semantics_allnetwork.pdf -------------------------------------------------------------------------------- /plot/semantics_cvpr_release.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CSAILVision/NetDissect/20ce6f887607aadc04ab9fb3d452e4723685cbbb/plot/semantics_cvpr_release.mat -------------------------------------------------------------------------------- /plot/semantics_samples.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CSAILVision/NetDissect/20ce6f887607aadc04ab9fb3d452e4723685cbbb/plot/semantics_samples.mat -------------------------------------------------------------------------------- /script/dlbroden.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | # Download broden1_224 8 | if [ ! -f dataset/broden1_224/index.csv ] 9 | then 10 | 11 | echo "Downloading broden1_224" 12 | mkdir -p dataset 13 | pushd dataset 14 | wget --progress=bar \ 15 | http://netdissect.csail.mit.edu/data/broden1_224.zip \ 16 | -O broden1_224.zip 17 | unzip broden1_224.zip 18 | rm broden1_224.zip 19 | popd 20 | 21 | fi 22 | 23 | # Download broden1_227 24 | if [ ! -f dataset/broden1_227/index.csv ] 25 | then 26 | 27 | echo "Downloading broden1_227" 28 | mkdir -p dataset 29 | pushd dataset 30 | wget --progress=bar \ 31 | http://netdissect.csail.mit.edu/data/broden1_227.zip \ 32 | -O broden1_227.zip 33 | unzip broden1_227.zip 34 | rm broden1_227.zip 35 | popd 36 | 37 | fi 38 | 39 | # Download broden1_384 40 | if [ ! -f dataset/broden1_384/index.csv ] 41 | then 42 | 43 | echo "Downloading broden1_384" 44 | mkdir -p dataset 45 | pushd dataset 46 | wget --progress=bar \ 47 | http://netdissect.csail.mit.edu/data/broden1_384.zip \ 48 | -O broden1_384.zip 49 | unzip broden1_384.zip 50 | rm broden1_384.zip 51 | popd 52 | 53 | fi 54 | 55 | -------------------------------------------------------------------------------- /script/dlbroden_227.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | # Download broden1_227 8 | if [ ! -f dataset/broden1_227/index.csv ] 9 | then 10 | 11 | echo "Downloading broden1_227" 12 | mkdir -p dataset 13 | pushd dataset 14 | wget --progress=bar \ 15 | http://netdissect.csail.mit.edu/data/broden1_227.zip \ 16 | -O broden1_227.zip 17 | unzip broden1_227.zip 18 | rm broden1_227.zip 19 | popd 20 | 21 | fi 22 | 23 | 24 | -------------------------------------------------------------------------------- /script/dlzoo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | declare -a MODELS=( 8 | "caffe_reference_imagenet" # alexnet-imagenet 9 | "caffe_reference_places205" # alexnet-places205 10 | "caffe_reference_places365" # alexnet-places365 11 | "vgg16_imagenet" # vgg-imagenet 12 | "vgg16_places205" # vgg-places205 13 | "vgg16_places365" # vgg-places365 14 | "googlenet_imagenet" # googlenet-imagenet 15 | "googlenet_places205" # googlenet-places205 16 | "googlenet_places365" # googlenet-places365 17 | "resnet-152-torch-imagenet" # resnet-imagenet 18 | "resnet-152-torch-places365" # resnet-places365 19 | ) 20 | 21 | for MODEL in "${MODELS[@]}" 22 | do 23 | 24 | if [ ! -f zoo/${MODEL}.prototxt ] || [ ! -f zoo/${MODEL}.caffemodel ] 25 | then 26 | 27 | echo "Downloading $MODEL" 28 | mkdir -p zoo 29 | pushd zoo 30 | wget --progress=bar \ 31 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.prototxt 32 | wget --progress=bar \ 33 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.caffemodel 34 | popd 35 | 36 | fi 37 | 38 | done 39 | -------------------------------------------------------------------------------- /script/dlzoo_example.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | echo "Download AlexNet trained on Places365" 8 | declare -a MODELS=( 9 | "caffe_reference_places365" # alexnet-places365 10 | ) 11 | 12 | for MODEL in "${MODELS[@]}" 13 | do 14 | 15 | if [ ! -f zoo/${MODEL}.prototxt ] || [ ! -f zoo/${MODEL}.caffemodel ] 16 | then 17 | 18 | echo "Downloading $MODEL" 19 | mkdir -p zoo 20 | pushd zoo 21 | wget --progress=bar \ 22 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.prototxt 23 | wget --progress=bar \ 24 | http://netdissect.csail.mit.edu/dissect/zoo/$MODEL.caffemodel 25 | popd 26 | 27 | fi 28 | 29 | done 30 | -------------------------------------------------------------------------------- /script/makebroden.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # Start from parent directory of script 5 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 6 | 7 | # PASCAL 2010 Images 8 | if [ ! -f dataset/pascal/VOC2010/ImageSets/Segmentation/train.txt ] 9 | then 10 | 11 | echo "Downloading Pascal VOC2010 images" 12 | mkdir -p dataset/pascal 13 | pushd dataset/pascal 14 | wget --progress=bar \ 15 | http://host.robots.ox.ac.uk/pascal/VOC/voc2010/VOCtrainval_03-May-2010.tar \ 16 | -O VOCtrainval_03-May-2010.tar 17 | tar xvf VOCtrainval_03-May-2010.tar 18 | rm VOCtrainval_03-May-2010.tar 19 | mv VOCdevkit/* . 20 | rmdir VOCdevkit 21 | popd 22 | 23 | fi 24 | 25 | 26 | # PASCAL Part dataset 27 | if [ ! -f dataset/pascal/part/part2ind.m ] 28 | then 29 | 30 | echo "Downloading Pascal Part Dataset" 31 | mkdir -p dataset/pascal/part 32 | pushd dataset/pascal/part 33 | wget --progress=bar \ 34 | http://www.stat.ucla.edu/~xianjie.chen/pascal_part_dataset/trainval.tar.gz \ 35 | -O trainval.tar.gz 36 | tar xvfz trainval.tar.gz 37 | rm trainval.tar.gz 38 | popd 39 | 40 | fi 41 | 42 | 43 | # PASCAL Context dataset 44 | if [ ! -f dataset/pascal/context/labels.txt ] 45 | then 46 | 47 | echo "Downloading Pascal Context Dataset" 48 | mkdir -p dataset/pascal/context 49 | pushd dataset/pascal/context 50 | wget --progress=bar \ 51 | http://www.cs.stanford.edu/~roozbeh/pascal-context/trainval.tar.gz \ 52 | -O trainval.tar.gz 53 | tar xvfz trainval.tar.gz 54 | rm trainval.tar.gz 55 | popd 56 | 57 | fi 58 | 59 | 60 | # DTD 61 | if [ ! -f dataset/dtd/dtd-r1.0.1/imdb/imdb.mat ] 62 | then 63 | 64 | echo "Downloading Describable Textures Dataset" 65 | mkdir -p dataset/dtd 66 | pushd dataset/dtd 67 | wget --progress=bar \ 68 | https://www.robots.ox.ac.uk/~vgg/data/dtd/download/dtd-r1.0.1.tar.gz \ 69 | -O dtd-r1.0.1.tar.gz 70 | tar xvzf dtd-r1.0.1.tar.gz 71 | mv dtd dtd-r1.0.1 72 | rm dtd-r1.0.1.tar.gz 73 | popd 74 | 75 | fi 76 | 77 | 78 | # OpenSurfaces 79 | if [ ! -f dataset/opensurfaces/photos.csv ] 80 | then 81 | 82 | echo "Downloading OpenSurfaces Dataset" 83 | mkdir -p dataset/opensurfaces 84 | pushd dataset/opensurfaces 85 | wget --progress=bar \ 86 | http://labelmaterial.s3.amazonaws.com/release/opensurfaces-release-0.zip \ 87 | -O opensurfaces-release-0.zip 88 | unzip opensurfaces-release-0.zip 89 | rm opensurfaces-release-0.zip 90 | PROCESS=process_opensurfaces_release_0.py 91 | wget --progress=bar \ 92 | http://labelmaterial.s3.amazonaws.com/release/$PROCESS \ 93 | -O $PROCESS 94 | python $PROCESS 95 | popd 96 | 97 | fi 98 | 99 | 100 | # ADE20K 101 | if [ ! -f dataset/ade20k/index_ade20k.mat ] 102 | then 103 | 104 | echo "Downloading ADE20K Dataset" 105 | mkdir -p dataset/ade20k 106 | pushd dataset/ade20k 107 | wget --progress=bar \ 108 | http://groups.csail.mit.edu/vision/datasets/ADE20K/ADE20K_2016_07_26.zip \ 109 | -O ADE20K_2016_07_26.zip 110 | unzip ADE20K_2016_07_26.zip 111 | rm ADE20K_2016_07_26.zip 112 | popd 113 | 114 | fi 115 | 116 | 117 | # Now make broden in various sizes 118 | if [ ! -f dataset/broden1_224/index.csv ] 119 | then 120 | echo "Building Broden1 224" 121 | python src/joinseg.py --size=224 122 | fi 123 | 124 | if [ ! -f dataset/broden1_227/index.csv ] 125 | then 126 | echo "Building Broden1 227" 127 | python src/joinseg.py --size=227 128 | fi 129 | 130 | if [ ! -f dataset/broden1_384/index.csv ] 131 | then 132 | echo "Building Broden1 384" 133 | python src/joinseg.py --size=384 134 | fi 135 | -------------------------------------------------------------------------------- /script/makelinks.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This script replaces dummy directories with symbolic links to your 4 | # own directories ~/ndlinks/[dataset|dissection|sourcedata|zoo], which 5 | # of course may be further symbolic links to wherever you choose 6 | # to keep your datasets, models, and dissections. 7 | 8 | # Start from parent directory of script 9 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 10 | 11 | # Remove dummy directories 12 | for DUMMY in dataset dissection zoo 13 | do 14 | if [ -h ${DUMMY} ] 15 | then 16 | rm ${DUMMY} 17 | echo "Removed link ${DUMMY}" 18 | fi 19 | if [ -d ${DUMMY} ] 20 | then 21 | rm -f ${DUMMY}/README.md 22 | rmdir ${DUMMY} 23 | echo "Removed dummy directory ${DUMMY}" 24 | fi 25 | ln -s --target-directory=. ~/ndlinks/${DUMMY} 26 | echo "Created link ${DUMMY}" 27 | done 28 | 29 | # Remove dummy directories from git using sparse-checkout 30 | if [ -e .git/info ] 31 | then 32 | git config core.sparsecheckout true 33 | cat << EOF >> .git/info/sparse-checkout 34 | !dataset/* 35 | !dissection/* 36 | !zoo/* 37 | /* 38 | EOF 39 | git read-tree -mu HEAD 40 | # git checkout dataset dissection zoo 41 | fi 42 | -------------------------------------------------------------------------------- /script/rundissect.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # To use this, put the caffe model to be tested in the "zoo" directory 4 | # the following naming convention for a model called "vgg16_places365": 5 | # 6 | # zoo/caffe_reference_places365.caffemodel 7 | # zoo/caffe_reference_places365.prototxt 8 | # 9 | # and then, with scipy and pycaffe available in your python, run: 10 | # 11 | # ./rundissect.sh --model caffe_reference_places365 --layers "conv4 conv5" 12 | # 13 | # the output will be placed in a directory dissection/caffe_reference_places365/ 14 | # 15 | # More options are listed below. 16 | 17 | # Defaults 18 | THRESHOLD="0.04" 19 | WORKDIR="dissection" 20 | TALLYDEPTH=2048 21 | PARALLEL=4 22 | TALLYBATCH=16 23 | PROBEBATCH=64 24 | QUANTILE="0.005" 25 | COLORDEPTH="3" 26 | CENTERED="c" 27 | MEAN="0 0 0" 28 | FORCE="none" 29 | ENDAFTER="none" 30 | MODELDIR="zoo" 31 | ROTATION_SEED="" 32 | ROTATION_POWER="1" 33 | 34 | # Start from parent directory of script 35 | cd "$(dirname "$(dirname "$(readlink -f "$0")")")" 36 | 37 | # Parse command-line arguments. http://stackoverflow.com/questions/192249 38 | 39 | while [[ $# -gt 1 ]] 40 | do 41 | key="$1" 42 | 43 | case $key in 44 | -d|--model) 45 | DIR="$2" 46 | shift 47 | ;; 48 | -w|--weights) 49 | WEIGHTS="$2" 50 | shift 51 | ;; 52 | -p|--proto) 53 | PROTO="$2" 54 | shift 55 | ;; 56 | -l|--layers) 57 | LAYERS="$2" 58 | shift 59 | ;; 60 | -s|--dataset) 61 | DATASET="$2" 62 | shift 63 | ;; 64 | --colordepth) 65 | COLORDEPTH="$2" 66 | shift 67 | ;; 68 | --resolution) 69 | RESOLUTION="$2" 70 | shift 71 | ;; 72 | --probebatch) 73 | PROBEBATCH="$2" 74 | shift 75 | ;; 76 | -t|--threshold) 77 | THRESHOLD="$2" 78 | shift 79 | ;; 80 | --tallydepth) 81 | TALLYDEPTH="$2" 82 | shift 83 | ;; 84 | --tallybatch) 85 | TALLYBATCH="$2" 86 | shift 87 | ;; 88 | --mean) 89 | MEAN="$2" 90 | shift 91 | ;; 92 | --rotation_seed) 93 | ROTATION_SEED="$2" 94 | shift 95 | ;; 96 | --rotation_power) 97 | ROTATION_POWER="$2" 98 | shift 99 | ;; 100 | -w|--workdir) 101 | WORKDIR="$2" 102 | shift 103 | ;; 104 | -f|--force) 105 | FORCE="$2" 106 | shift 107 | ;; 108 | --endafter) 109 | ENDAFTER="$2" 110 | shift 111 | ;; 112 | --gpu) 113 | export CUDA_VISIBLE_DEVICES="$2" 114 | shift 115 | ;; 116 | *) 117 | echo "Unknown option" $key 118 | exit 3 119 | # unknown option 120 | ;; 121 | esac 122 | shift # past argument or value 123 | done 124 | 125 | # Get rid of slashes in layer names for directory purposes 126 | LAYERA=(${LAYERS//\//-}) 127 | 128 | # For expanding globs http://stackoverflow.com/questions/2937407 129 | function globexists { 130 | set +f 131 | test -e "$1" -o -L "$1";set -f 132 | } 133 | 134 | if [ -z $DIR ]; then 135 | echo '--model directory' must be specified 136 | exit 1 137 | fi 138 | 139 | if [ -z "${LAYERS}" ]; then 140 | echo '--layers layers' must be specified 141 | exit 1 142 | fi 143 | 144 | # Set up directory to work in, and lay down pid file etc. 145 | mkdir -p $WORKDIR/$DIR 146 | if [ -z "${FORCE##*pid*}" ] || [ ! -e $WORKDIR/$DIR/job.pid ] 147 | then 148 | exec &> >(tee -a "$WORKDIR/$DIR/job.log") 149 | echo "Beginning pid $$ on host $(hostname) at $(date)" 150 | trap "rm -rf $WORKDIR/$DIR/job.pid" EXIT 151 | echo $(hostname) $$ > $WORKDIR/$DIR/job.pid 152 | else 153 | echo "Already running $DIR at $(cat $WORKDIR/$DIR/job.pid)" 154 | exit 1 155 | fi 156 | 157 | if [ "$COLORDEPTH" -le 0 ] 158 | then 159 | (( COLORDEPTH = -COLORDEPTH )) 160 | CENTERED="" 161 | fi 162 | 163 | if [ -z "${CENTERED##*c*}" ] 164 | then 165 | MEAN="109.5388 118.6897 124.6901" 166 | fi 167 | 168 | # Convention: dir, weights, and proto all have the same name 169 | if [[ -z "${WEIGHTS}" && -z "${PROTO}" ]] 170 | then 171 | WEIGHTS="zoo/$DIR.caffemodel" 172 | PROTO="zoo/$DIR.prototxt" 173 | fi 174 | 175 | echo DIR = "${DIR}" 176 | echo LAYERS = "${LAYERS}" 177 | echo DATASET = "${DATASET}" 178 | echo COLORDEPTH = "${COLORDEPTH}" 179 | echo RESOLUTION = "${RESOLUTION}" 180 | echo WORKDIR = "${WORKDIR}" 181 | echo WEIGHTS = "${WEIGHTS}" 182 | echo PROTO = "${PROTO}" 183 | echo THRESHOLD = "${THRESHOLD}" 184 | echo PROBEBATCH = "${PROBEBATCH}" 185 | echo TALLYDEPTH = "${TALLYDEPTH}" 186 | echo TALLYBATCH = "${TALLYBATCH}" 187 | echo MEAN = "${MEAN}" 188 | echo FORCE = "${FORCE}" 189 | echo ENDAFTER = "${ENDAFTER}" 190 | 191 | # Set up rotation flag if rotation is selected 192 | ROTATION_FLAG="" 193 | if [ ! -z "${ROTATION_SEED}" ] 194 | then 195 | ROTATION_FLAG=" --rotation_seed ${ROTATION_SEED} 196 | --rotation_unpermute 1 197 | --rotation_power ${ROTATION_POWER} " 198 | fi 199 | 200 | # Step 1: do a forward pass over the network specified by the model files. 201 | # The output is e.g.,: "conv5.mmap", "conv5-info.txt" 202 | if [ -z "${FORCE##*probe*}" ] || \ 203 | ! ls $(printf " $WORKDIR/$DIR/%s.mmap" "${LAYERA[@]}") 2>/dev/null 204 | then 205 | 206 | echo 'Testing activations' 207 | python src/netprobe.py \ 208 | --directory $WORKDIR/$DIR \ 209 | --blobs $LAYERS \ 210 | --weights $WEIGHTS \ 211 | --definition $PROTO \ 212 | --batch_size $PROBEBATCH \ 213 | --mean $MEAN \ 214 | --colordepth $COLORDEPTH \ 215 | ${ROTATION_FLAG} \ 216 | --dataset $DATASET 217 | 218 | [[ $? -ne 0 ]] && exit $? 219 | echo netprobe > $WORKDIR/$DIR/job.done 220 | fi 221 | 222 | if [ -z "${ENDAFTER##*probe*}" ] 223 | then 224 | exit 0 225 | fi 226 | 227 | # Step 2: compute quantiles 228 | # saves results in conv5-quant-1000.mmap; conv5-rank-1001.mmap inverts this too. 229 | #echo 'Computing quantiles' 230 | if [ -z "${FORCE##*sort*}" ] || \ 231 | ! ls $(printf " $WORKDIR/$DIR/%s-quant-*.mmap" "${LAYERA[@]}") 2>/dev/null 232 | then 233 | 234 | echo 'Collecting quantiles of activations' 235 | python src/quantprobe.py \ 236 | --directory $WORKDIR/$DIR \ 237 | --blobs $LAYERS 238 | [[ $? -ne 0 ]] && exit $? 239 | 240 | echo quantprobe > $WORKDIR/$DIR/job.done 241 | fi 242 | 243 | if [ -z "${ENDAFTER##*quant*}" ] 244 | then 245 | exit 0 246 | fi 247 | 248 | # Step 3: the output here is the ~1G file called "conv5-tally-005.mmap". 249 | # It contains all the I/O/U etc counts for every label and unit (at the 0.5% 250 | # top activtation mask) for EVERY image. I.e., for image #n, we can read 251 | # out the number of pixels that light up both unit #u and label #l for 252 | # any combination of (n, u, l). That is a a very large sparse matrix, so 253 | # we encode that matrix specially. This is a 254 | # (#images, 2048, 3) dimensional file with entries such as this: 255 | # E.g., for image 412, we will have a list of up to 2048 triples. 256 | # Each triple has (count, #unit, #label) in it. 257 | if [ -z "${FORCE##*tally*}" ] || \ 258 | ! ls $(printf " $WORKDIR/$DIR/%s-tally-*.mmap" "${LAYERA[@]}") 2>/dev/null 259 | then 260 | 261 | echo 'Tallying counts' 262 | python src/labelprobe.py \ 263 | --directory $WORKDIR/$DIR \ 264 | --quantile $QUANTILE \ 265 | --tally_depth $TALLYDEPTH \ 266 | --blobs $LAYERS \ 267 | --parallel $PARALLEL \ 268 | --batch_size $TALLYBATCH \ 269 | --ahead 4 270 | 271 | [[ $? -ne 0 ]] && exit $? 272 | 273 | echo tallyprobe > $WORKDIR/$DIR/job.done 274 | fi 275 | 276 | if [ -z "${ENDAFTER##*tally*}" ] 277 | then 278 | exit 0 279 | fi 280 | 281 | # Step 4: compute conv5-imgmax.mmap / conv5-imgmax.mat 282 | # This contains the per-imgae maximum activation for every unit. 283 | if [ -z "${FORCE##*imgmax*}" ] || \ 284 | ! ls $(printf " $WORKDIR/$DIR/%s-imgmax.mmap" "${LAYERA[@]}") 2>/dev/null 285 | then 286 | 287 | echo 'Computing imgmax' 288 | python src/maxprobe.py \ 289 | --directory $WORKDIR/$DIR \ 290 | --blobs $LAYERS 291 | 292 | [[ $? -ne 0 ]] && exit $? 293 | echo maxprobe > $WORKDIR/$DIR/job.done 294 | fi 295 | 296 | if [ -z "${ENDAFTER##*imgmax*}" ] 297 | then 298 | exit 0 299 | fi 300 | 301 | 302 | # Step 5: we just run over the tally file to extract whatever score we 303 | # want to derive. That gets summarized in a [layer]-result.csv file. 304 | if [ -z "${FORCE##*result*}" ] || \ 305 | ! ls $(printf " $WORKDIR/$DIR/%s-result.csv" "${LAYERA[@]}") 306 | then 307 | 308 | echo 'Generating result.csv' 309 | python src/makeresult.py \ 310 | --directory $WORKDIR/$DIR \ 311 | --blobs $LAYERS 312 | 313 | [[ $? -ne 0 ]] && exit $? 314 | 315 | echo makeresult > $WORKDIR/$DIR/job.done 316 | fi 317 | 318 | if [ -z "${ENDAFTER##*result*}" ] 319 | then 320 | exit 0 321 | fi 322 | 323 | # Step 6: now generate the HTML visualization and images. 324 | if [ -z "${FORCE##*report*}" ] || \ 325 | ! ls $(printf " $WORKDIR/$DIR/html/%s.html" "${LAYERA[@]}") || \ 326 | ! ls $(printf " $WORKDIR/$DIR/html/image/%s-bargraph.svg" "${LAYERA[@]}") 327 | then 328 | 329 | echo 'Generating report' 330 | python src/report.py \ 331 | --directory $WORKDIR/$DIR \ 332 | --blobs $LAYERS \ 333 | --threshold ${THRESHOLD} 334 | 335 | [[ $? -ne 0 ]] && exit $? 336 | 337 | echo viewprobe > $WORKDIR/$DIR/job.done 338 | fi 339 | 340 | if [ -z "${ENDAFTER##*view*}" ] 341 | then 342 | exit 0 343 | fi 344 | 345 | # 346 | # Step 6: graph the results. 347 | if [ -z "${FORCE##*graph*}" ] || ! globexists $WORKDIR/$DIR/html/*-graph.png 348 | then 349 | 350 | # Compute text for labeling graphs. 351 | PERCENT=$(printf '%g%%' $(echo "scale=0; $THRESHOLD * 100" | bc)) 352 | 353 | echo 'Generating graph' 354 | python src/graphprobe.py \ 355 | --directories $WORKDIR/$DIR \ 356 | --blobs $LAYERS \ 357 | --labels $LAYERS \ 358 | --threshold $THRESHOLD \ 359 | --include_total true \ 360 | --title "Interpretability By Layer ($PERCENT IOU)" \ 361 | --out $WORKDIR/$DIR/html/layer-graph.png 362 | 363 | [[ $? -ne 0 ]] && exit $? 364 | echo graphprobe > $WORKDIR/$DIR/job.done 365 | fi 366 | 367 | if [ "$WORKDIR" != "$WORKDIR" ] 368 | then 369 | rm -rf $WORKDIR/$DIR/ 370 | [[ $? -ne 0 ]] && exit $? 371 | fi 372 | 373 | echo finished > $WORKDIR/$DIR/job.done 374 | 375 | if [ -z "${ENDAFTER##*graph*}" ] 376 | then 377 | exit 0 378 | fi 379 | 380 | -------------------------------------------------------------------------------- /script/rundissect_pytorch.sh: -------------------------------------------------------------------------------- 1 | # pre-defined setting 2 | 3 | WORKDIR=probes 4 | DIR=pytorch_alexnet_imagenet 5 | ARCH='alexnet' # [alexnet,squeezenet1_1,resnet18,...]. It should work for all the models in https://github.com/pytorch/vision/tree/master/torchvision/models 6 | LAYERS="features" 7 | NUMCLASSES=1000 8 | DATASET=dataset/broden1_227 9 | WEIGHTS="none" 10 | 11 | # default setting 12 | THRESHOLD=0.04 13 | TALLYDEPTH=2048 14 | PARALLEL=4 15 | TALLYBATCH=16 16 | PROBEBATCH=64 17 | QUANTILE="0.005" 18 | COLORDEPTH="3" 19 | CENTERED="c" 20 | MEAN="0 0 0" 21 | FORCE="none" 22 | ENDAFTER="none" 23 | RESOLUTION="120" 24 | 25 | 26 | # Set up directory to work in, and lay down pid file etc. 27 | mkdir -p $WORKDIR/$DIR 28 | if [ -z "${FORCE##*pid*}" ] || [ ! -e $WORKDIR/$DIR/job.pid ] 29 | then 30 | exec &> >(tee -a "$WORKDIR/$DIR/job.log") 31 | echo "Beginning pid $$ on host $(hostname) at $(date)" 32 | trap "rm -rf $WORKDIR/$DIR/job.pid $WORKDIR/$DIR/job.host" EXIT 33 | echo $$ > $WORKDIR/$DIR/job.pid 34 | echo $(hostname) > $WORKDIR/$DIR/job.host 35 | else 36 | echo "Already running $DIR under pid $(cat $WORKDIR/$DIR/job.pid)" 37 | exit 1 38 | fi 39 | 40 | if [ "$COLORDEPTH" -le 0 ] 41 | then 42 | (( COLORDEPTH = -COLORDEPTH )) 43 | CENTERED="" 44 | fi 45 | 46 | if [ -z "${CENTERED##*c*}" ] 47 | then 48 | MEAN="109.5388 118.6897 124.6901" 49 | fi 50 | 51 | # Get rid of slashes in layer names for directory purposes 52 | LAYERA=(${LAYERS//\//-}) 53 | 54 | # Step 1: do a forward pass over the network specified by the model files. 55 | # The output is e.g.,: "conv5.mmap", "conv5-info.txt" 56 | if [ -z "${FORCE##*probe*}" ] || \ 57 | ! ls $(printf " $WORKDIR/$DIR/%s.mmap" "${LAYERA[@]}") 2>/dev/null 58 | then 59 | 60 | echo 'Testing activations' 61 | python src/netprobe_pytorch.py \ 62 | --directory $WORKDIR/$DIR \ 63 | --blobs $LAYERS \ 64 | --mean $MEAN \ 65 | --definition $ARCH \ 66 | --weights $WEIGHTS \ 67 | --num_classes $NUMCLASSES \ 68 | --dataset $DATASET 69 | 70 | [[ $? -ne 0 ]] && exit $? 71 | echo netprobe > $WORKDIR/$DIR/job.done 72 | fi 73 | 74 | if [ -z "${ENDAFTER##*probe*}" ] 75 | then 76 | exit 0 77 | fi 78 | 79 | # Step 2: compute quantiles 80 | # saves results in conv5-quant-1000.mmap; conv5-rank-1001.mmap inverts this too. 81 | #echo 'Computing quantiles' 82 | if [ -z "${FORCE##*sort*}" ] || \ 83 | ! ls $(printf " $WORKDIR/$DIR/%s-quant-*.mmap" "${LAYERA[@]}") 2>/dev/null 84 | then 85 | 86 | echo 'Collecting quantiles of activations' 87 | python src/quantprobe.py \ 88 | --directory $WORKDIR/$DIR \ 89 | --blobs $LAYERS 90 | [[ $? -ne 0 ]] && exit $? 91 | 92 | echo quantprobe > $WORKDIR/$DIR/job.done 93 | fi 94 | 95 | if [ -z "${ENDAFTER##*quant*}" ] 96 | then 97 | exit 0 98 | fi 99 | 100 | # Step 3: the output here is the ~1G file called "conv5-tally-005.mmap". 101 | # It contains all the I/O/U etc counts for every label and unit (at the 0.5% 102 | # top activtation mask) for EVERY image. I.e., for image #n, we can read 103 | # out the number of pixels that light up both unit #u and label #l for 104 | # any combination of (n, u, l). That is a a very large sparse matrix, so 105 | # we encode that matrix specially. This is a 106 | # (#images, 2048, 3) dimensional file with entries such as this: 107 | # E.g., for image 412, we will have a list of up to 2048 triples. 108 | # Each triple has (count, #unit, #label) in it. 109 | if [ -z "${FORCE##*tally*}" ] || \ 110 | ! ls $(printf " $WORKDIR/$DIR/%s-tally-*.mmap" "${LAYERA[@]}") 2>/dev/null 111 | then 112 | 113 | echo 'Tallying counts' 114 | python src/labelprobe.py \ 115 | --directory $WORKDIR/$DIR \ 116 | --quantile $QUANTILE \ 117 | --tally_depth $TALLYDEPTH \ 118 | --blobs $LAYERS \ 119 | --parallel $PARALLEL \ 120 | --batch_size $TALLYBATCH \ 121 | --ahead 4 122 | 123 | [[ $? -ne 0 ]] && exit $? 124 | 125 | echo tallyprobe > $WORKDIR/$DIR/job.done 126 | fi 127 | 128 | if [ -z "${ENDAFTER##*tally*}" ] 129 | then 130 | exit 0 131 | fi 132 | 133 | # Step 4: compute conv5-imgmax.mmap / conv5-imgmax.mat 134 | # This contains the per-imgae maximum activation for every unit. 135 | if [ -z "${FORCE##*imgmax*}" ] || \ 136 | ! ls $(printf " $WORKDIR/$DIR/%s-imgmax.mmap" "${LAYERA[@]}") 2>/dev/null 137 | then 138 | 139 | echo 'Computing imgmax' 140 | python src/maxprobe.py \ 141 | --directory $WORKDIR/$DIR \ 142 | --blobs $LAYERS 143 | 144 | [[ $? -ne 0 ]] && exit $? 145 | echo maxprobe > $WORKDIR/$DIR/job.done 146 | fi 147 | 148 | if [ -z "${ENDAFTER##*imgmax*}" ] 149 | then 150 | exit 0 151 | fi 152 | 153 | 154 | # Step 5: we just run over the tally file to extract whatever score we 155 | # want to derive. That gets summarized in a [layer]-result.csv file. 156 | if [ -z "${FORCE##*result*}" ] || \ 157 | ! ls $(printf " $WORKDIR/$DIR/%s-result.csv" "${LAYERA[@]}") 158 | then 159 | 160 | echo 'Generating result.csv' 161 | python src/makeresult.py \ 162 | --directory $WORKDIR/$DIR \ 163 | --blobs $LAYERS 164 | 165 | [[ $? -ne 0 ]] && exit $? 166 | 167 | echo makeresult > $WORKDIR/$DIR/job.done 168 | fi 169 | 170 | if [ -z "${ENDAFTER##*result*}" ] 171 | then 172 | exit 0 173 | fi 174 | 175 | # Step 6: now generate the HTML visualization and images. 176 | if [ -z "${FORCE##*report*}" ] || \ 177 | ! ls $(printf " $WORKDIR/$DIR/html/%s.html" "${LAYERA[@]}") || \ 178 | ! ls $(printf " $WORKDIR/$DIR/html/image/%s-bargraph.svg" "${LAYERA[@]}") 179 | then 180 | 181 | echo 'Generating report' 182 | python src/report.py \ 183 | --directory $WORKDIR/$DIR \ 184 | --blobs $LAYERS \ 185 | --threshold ${THRESHOLD} 186 | 187 | [[ $? -ne 0 ]] && exit $? 188 | 189 | echo viewprobe > $WORKDIR/$DIR/job.done 190 | fi 191 | 192 | if [ -z "${ENDAFTER##*view*}" ] 193 | then 194 | exit 0 195 | fi 196 | 197 | # 198 | # Step 6: graph the results. 199 | if [ -z "${FORCE##*graph*}" ] || ! globexists $WORKDIR/$DIR/html/*-graph.png 200 | then 201 | 202 | # Compute text for labeling graphs. 203 | PERCENT=$(printf '%g%%' $(echo "scale=0; $THRESHOLD * 100" | bc)) 204 | 205 | echo 'Generating graph' 206 | python src/graphprobe.py \ 207 | --directories $WORKDIR/$DIR \ 208 | --blobs $LAYERS \ 209 | --labels $LAYERS \ 210 | --threshold $THRESHOLD \ 211 | --include_total true \ 212 | --title "Interpretability By Layer ($PERCENT IOU)" \ 213 | --out $WORKDIR/$DIR/html/layer-graph.png 214 | 215 | [[ $? -ne 0 ]] && exit $? 216 | echo graphprobe > $WORKDIR/$DIR/job.done 217 | fi 218 | 219 | if [ "$WORKDIR" != "$WORKDIR" ] 220 | then 221 | rm -rf $WORKDIR/$DIR/ 222 | [[ $? -ne 0 ]] && exit $? 223 | fi 224 | 225 | echo finished > $WORKDIR/$DIR/job.done 226 | 227 | if [ -z "${ENDAFTER##*graph*}" ] 228 | then 229 | exit 0 230 | fi 231 | 232 | -------------------------------------------------------------------------------- /script/rundissect_pytorch_external.sh: -------------------------------------------------------------------------------- 1 | # pre-defined setting 2 | WORKDIR=probes 3 | DIR=pytorch_alexnet_imagenet 4 | ARCH='alexnet' # [alexnet,squeezenet1_1,resnet18,...]. It should work for all the models in https://github.com/pytorch/vision/tree/master/torchvision/models 5 | LAYERS="features" 6 | DATASET=dataset/broden1_224 7 | NUMCLASSES=1000 8 | 9 | # default setting 10 | THRESHOLD=0.04 11 | TALLYDEPTH=2048 12 | PARALLEL=4 13 | TALLYBATCH=16 14 | PROBEBATCH=64 15 | QUANTILE="0.005" 16 | COLORDEPTH="3" 17 | CENTERED="c" 18 | MEAN="0 0 0" 19 | FORCE="none" 20 | ENDAFTER="none" 21 | RESOLUTION=100 22 | 23 | # Download the external model and reset the parameters 24 | DIR="pytorch_resnet18_places365" 25 | ARCH="resnet18" 26 | NUMCLASSES=365 27 | LAYERS="layer4" 28 | WEIGHTS=zoo/resnet18_places365.pth.tar 29 | if [ ! -e $WEIGHTS ] 30 | then 31 | echo "Download the resnet18 trained on Places365" 32 | wget http://places2.csail.mit.edu/models_places365/resnet18_places365.pth.tar -O $WEIGHTS 33 | fi 34 | 35 | # Set up directory to work in, and lay down pid file etc. 36 | mkdir -p $WORKDIR/$DIR 37 | if [ -z "${FORCE##*pid*}" ] || [ ! -e $WORKDIR/$DIR/job.pid ] 38 | then 39 | exec &> >(tee -a "$WORKDIR/$DIR/job.log") 40 | echo "Beginning pid $$ on host $(hostname) at $(date)" 41 | trap "rm -rf $WORKDIR/$DIR/job.pid $WORKDIR/$DIR/job.host" EXIT 42 | echo $$ > $WORKDIR/$DIR/job.pid 43 | echo $(hostname) > $WORKDIR/$DIR/job.host 44 | else 45 | echo "Already running $DIR under pid $(cat $WORKDIR/$DIR/job.pid)" 46 | exit 1 47 | fi 48 | 49 | if [ "$COLORDEPTH" -le 0 ] 50 | then 51 | (( COLORDEPTH = -COLORDEPTH )) 52 | CENTERED="" 53 | fi 54 | 55 | if [ -z "${CENTERED##*c*}" ] 56 | then 57 | MEAN="109.5388 118.6897 124.6901" 58 | fi 59 | 60 | # Get rid of slashes in layer names for directory purposes 61 | LAYERA=(${LAYERS//\//-}) 62 | 63 | # Step 1: do a forward pass over the network specified by the model files. 64 | # The output is e.g.,: "conv5.mmap", "conv5-info.txt" 65 | if [ -z "${FORCE##*probe*}" ] || \ 66 | ! ls $(printf " $WORKDIR/$DIR/%s.mmap" "${LAYERA[@]}") 2>/dev/null 67 | then 68 | 69 | echo 'Testing activations' 70 | python src/netprobe_pytorch.py \ 71 | --directory $WORKDIR/$DIR \ 72 | --blobs $LAYERS \ 73 | --mean $MEAN \ 74 | --definition $ARCH \ 75 | --weights $WEIGHTS \ 76 | --num_classes $NUMCLASSES \ 77 | --dataset $DATASET 78 | 79 | 80 | [[ $? -ne 0 ]] && exit $? 81 | echo netprobe > $WORKDIR/$DIR/job.done 82 | fi 83 | 84 | if [ -z "${ENDAFTER##*probe*}" ] 85 | then 86 | exit 0 87 | fi 88 | 89 | # Step 2: compute quantiles 90 | # saves results in conv5-quant-1000.mmap; conv5-rank-1001.mmap inverts this too. 91 | #echo 'Computing quantiles' 92 | if [ -z "${FORCE##*sort*}" ] || \ 93 | ! ls $(printf " $WORKDIR/$DIR/%s-quant-*.mmap" "${LAYERA[@]}") 2>/dev/null 94 | then 95 | 96 | echo 'Collecting quantiles of activations' 97 | python src/quantprobe.py \ 98 | --directory $WORKDIR/$DIR \ 99 | --blobs $LAYERS 100 | [[ $? -ne 0 ]] && exit $? 101 | 102 | echo quantprobe > $WORKDIR/$DIR/job.done 103 | fi 104 | 105 | if [ -z "${ENDAFTER##*quant*}" ] 106 | then 107 | exit 0 108 | fi 109 | 110 | # Step 3: the output here is the ~1G file called "conv5-tally-005.mmap". 111 | # It contains all the I/O/U etc counts for every label and unit (at the 0.5% 112 | # top activtation mask) for EVERY image. I.e., for image #n, we can read 113 | # out the number of pixels that light up both unit #u and label #l for 114 | # any combination of (n, u, l). That is a a very large sparse matrix, so 115 | # we encode that matrix specially. This is a 116 | # (#images, 2048, 3) dimensional file with entries such as this: 117 | # E.g., for image 412, we will have a list of up to 2048 triples. 118 | # Each triple has (count, #unit, #label) in it. 119 | if [ -z "${FORCE##*tally*}" ] || \ 120 | ! ls $(printf " $WORKDIR/$DIR/%s-tally-*.mmap" "${LAYERA[@]}") 2>/dev/null 121 | then 122 | 123 | echo 'Tallying counts' 124 | python src/labelprobe.py \ 125 | --directory $WORKDIR/$DIR \ 126 | --quantile $QUANTILE \ 127 | --tally_depth $TALLYDEPTH \ 128 | --blobs $LAYERS \ 129 | --parallel $PARALLEL \ 130 | --batch_size $TALLYBATCH \ 131 | --ahead 4 132 | 133 | [[ $? -ne 0 ]] && exit $? 134 | 135 | echo tallyprobe > $WORKDIR/$DIR/job.done 136 | fi 137 | 138 | if [ -z "${ENDAFTER##*tally*}" ] 139 | then 140 | exit 0 141 | fi 142 | 143 | # Step 4: compute conv5-imgmax.mmap / conv5-imgmax.mat 144 | # This contains the per-imgae maximum activation for every unit. 145 | if [ -z "${FORCE##*imgmax*}" ] || \ 146 | ! ls $(printf " $WORKDIR/$DIR/%s-imgmax.mmap" "${LAYERA[@]}") 2>/dev/null 147 | then 148 | 149 | echo 'Computing imgmax' 150 | python src/maxprobe.py \ 151 | --directory $WORKDIR/$DIR \ 152 | --blobs $LAYERS 153 | 154 | [[ $? -ne 0 ]] && exit $? 155 | echo maxprobe > $WORKDIR/$DIR/job.done 156 | fi 157 | 158 | if [ -z "${ENDAFTER##*imgmax*}" ] 159 | then 160 | exit 0 161 | fi 162 | 163 | 164 | # Step 5: we just run over the tally file to extract whatever score we 165 | # want to derive. That gets summarized in a [layer]-result.csv file. 166 | if [ -z "${FORCE##*result*}" ] || \ 167 | ! ls $(printf " $WORKDIR/$DIR/%s-result.csv" "${LAYERA[@]}") 168 | then 169 | 170 | echo 'Generating result.csv' 171 | python src/makeresult.py \ 172 | --directory $WORKDIR/$DIR \ 173 | --blobs $LAYERS 174 | 175 | [[ $? -ne 0 ]] && exit $? 176 | 177 | echo makeresult > $WORKDIR/$DIR/job.done 178 | fi 179 | 180 | if [ -z "${ENDAFTER##*result*}" ] 181 | then 182 | exit 0 183 | fi 184 | 185 | # Step 6: now generate the HTML visualization and images. 186 | if [ -z "${FORCE##*report*}" ] || \ 187 | ! ls $(printf " $WORKDIR/$DIR/html/%s.html" "${LAYERA[@]}") || \ 188 | ! ls $(printf " $WORKDIR/$DIR/html/image/%s-bargraph.svg" "${LAYERA[@]}") 189 | then 190 | 191 | echo 'Generating report' 192 | python src/report.py \ 193 | --directory $WORKDIR/$DIR \ 194 | --blobs $LAYERS \ 195 | --threshold ${THRESHOLD} 196 | 197 | [[ $? -ne 0 ]] && exit $? 198 | 199 | echo viewprobe > $WORKDIR/$DIR/job.done 200 | fi 201 | 202 | if [ -z "${ENDAFTER##*view*}" ] 203 | then 204 | exit 0 205 | fi 206 | 207 | # 208 | # Step 6: graph the results. 209 | if [ -z "${FORCE##*graph*}" ] || ! globexists $WORKDIR/$DIR/html/*-graph.png 210 | then 211 | 212 | # Compute text for labeling graphs. 213 | PERCENT=$(printf '%g%%' $(echo "scale=0; $THRESHOLD * 100" | bc)) 214 | 215 | echo 'Generating graph' 216 | python src/graphprobe.py \ 217 | --directories $WORKDIR/$DIR \ 218 | --blobs $LAYERS \ 219 | --labels $LAYERS \ 220 | --threshold $THRESHOLD \ 221 | --include_total true \ 222 | --title "Interpretability By Layer ($PERCENT IOU)" \ 223 | --out $WORKDIR/$DIR/html/layer-graph.png 224 | 225 | [[ $? -ne 0 ]] && exit $? 226 | echo graphprobe > $WORKDIR/$DIR/job.done 227 | fi 228 | 229 | if [ "$WORKDIR" != "$WORKDIR" ] 230 | then 231 | rm -rf $WORKDIR/$DIR/ 232 | [[ $? -ne 0 ]] && exit $? 233 | fi 234 | 235 | echo finished > $WORKDIR/$DIR/job.done 236 | 237 | if [ -z "${ENDAFTER##*graph*}" ] 238 | then 239 | exit 0 240 | fi 241 | 242 | -------------------------------------------------------------------------------- /src/ade20k.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import os, glob 4 | import re 5 | import numpy 6 | from scipy.io import loadmat 7 | from scipy.misc import imread, imsave 8 | from collections import namedtuple 9 | from scipy.misc import imresize 10 | from scipy.ndimage.interpolation import zoom 11 | 12 | ADE_ROOT = '/home/davidbau/bulk/ade20k/' 13 | ADE_VER = 'ADE20K_2016_07_26' 14 | 15 | def decodeClassMask(im): 16 | '''Decodes pixel-level object/part class and instance data from 17 | the given image, previously encoded into RGB channels.''' 18 | # Classes are a combination of RG channels (dividing R by 10) 19 | return (im[:,:,0] // 10) * 256 + im[:,:,1] 20 | 21 | def decodeInstanceMask(im): 22 | # Instance number is scattered, so we renumber them 23 | (orig, instances) = numpy.unique(im[:,:,2], return_inverse=True) 24 | return instances.reshape(classes.shape) 25 | 26 | def encodeClassMask(im, offset=0): 27 | result = numpy.zeros(im.shape + (3,), dtype=numpy.uint8) 28 | if offset: 29 | support = im > offset 30 | mapped = (im + support) * offset 31 | else: 32 | mapped = im 33 | result[:,:,1] = mapped % 256 34 | result[:,:,0] = (mapped // 256) * 10 35 | return result 36 | 37 | class Dataset: 38 | def __init__(self, directory=None, version=None): 39 | # Default to value of ADE20_ROOT env variable 40 | if directory is None: 41 | directory = os.environ['ADE20K_ROOT'] 42 | directory = os.path.expanduser(directory) 43 | # Default to the latest version present in the directory 44 | if version is None: 45 | contents = os.listdir(directory) 46 | if not list(c for c in contents if re.match('^index.*mat$', c)): 47 | version = sorted(c for c in contents if os.path.isdir( 48 | os.path.join(directory, c)))[-1] 49 | else: 50 | version = '' 51 | self.root = directory 52 | self.version = version 53 | 54 | mat = loadmat(self.expand(self.version, 'index*.mat'), squeeze_me=True) 55 | index = mat['index'] 56 | Ade20kIndex = namedtuple('Ade20kIndex', index.dtype.names) 57 | # for name in index.dtype.names: 58 | # setattr(self, name, index[name][()]) 59 | self.index = Ade20kIndex( 60 | **{name: index[name][()] for name in index.dtype.names}) 61 | self.raw_mat = mat 62 | 63 | def expand(self, *path): 64 | '''Expands a filename and directories with the ADE dataset''' 65 | result = os.path.join(self.root, *path) 66 | if '*' in result or '?' in result: 67 | globbed = glob.glob(result) 68 | if len(globbed): 69 | return globbed[0] 70 | return result 71 | 72 | def filename(self, n): 73 | '''Returns the filename for the nth dataset image.''' 74 | filename = self.index.filename[n] 75 | folder = self.index.folder[n] 76 | return self.expand(folder, filename) 77 | 78 | def short_filename(self, n): 79 | '''Returns the filename for the nth dataset image, without folder.''' 80 | return self.index.filename[n] 81 | 82 | def size(self): 83 | '''Returns the number of images in this dataset.''' 84 | return len(self.index.filename) 85 | 86 | def num_object_types(self): 87 | return len(self.index.objectnames) 88 | 89 | def seg_filename(self, n): 90 | '''Returns the segmentation filename for the nth dataset image.''' 91 | return re.sub(r'\.jpg$', '_seg.png', self.filename(n)) 92 | 93 | def part_filenames(self, n): 94 | '''Returns all the subpart images for the nth dataset image.''' 95 | filename = self.filename(n) 96 | level = 1 97 | result = [] 98 | while True: 99 | probe = re.sub(r'\.jpg$', '_parts_%d.png' % level, filename) 100 | if not os.path.isfile(probe): 101 | break 102 | result.append(probe) 103 | level += 1 104 | return result 105 | 106 | def part_levels(self): 107 | return max([len(self.part_filenames(n)) for n in range(self.size())]) 108 | 109 | def image(self, n): 110 | '''Returns the nth dataset image as a numpy array.''' 111 | return imread(self.filename(n)) 112 | 113 | def segmentation(self, n, include_instances=False): 114 | '''Returns the nth dataset segmentation as a numpy array, 115 | where each entry at a pixel is an object class value. 116 | 117 | If include_instances is set, returns a pair where the second 118 | array labels each instance with a unique number.''' 119 | data = imread(self.seg_filename(n)) 120 | if include_instances: 121 | return (decodeClassMask(data), decodeInstanceMask(data)) 122 | else: 123 | return decodeClassMask(data) 124 | 125 | def parts(self, n, include_instances=False): 126 | '''Returns an list of part segmentations for the nth dataset item, 127 | with one array for each level available. If included_instances is 128 | set, the list contains pairs of numpy arrays (c, i) where i 129 | represents instances.''' 130 | result = [] 131 | for fn in self.part_filenames(n): 132 | data = imread(fn) 133 | if include_instances: 134 | result.append((decodeClassMask(data), decodeInstanceMask(data))) 135 | else: 136 | result.append(decodeClassMask(data)) 137 | return result 138 | 139 | def full_segmentation(self, n, include_instances=False): 140 | '''Returns a single tensor with all levels of segmentations included 141 | in the channels, one channel per level. If include_instances is 142 | requested, a parallel tensor with instance labels is returned in 143 | a tuple.''' 144 | full = [self.segmentation(n, include_instances) 145 | ] + self.parts(n, include_instances) 146 | if include_instances: 147 | return tuple(numpy.concatenate(tuple(m[numpy.newaxis] for m in d) 148 | for d in zip(full))) 149 | return numpy.concatenate(tuple(m[numpy.newaxis] for m in full)) 150 | 151 | def object_name(self, c): 152 | '''Returns a short English name for the object class c.''' 153 | # Off by one due to use of 1-based indexing in matlab. 154 | if c == 0: 155 | return '-' 156 | result = self.index.objectnames[c - 1] 157 | return re.split(',\s*', result, 1)[0] 158 | 159 | def object_count(self, c): 160 | '''Returns a count of the object over the whole dataset.''' 161 | # Off by one due to use of 1-based indexing in matlab. 162 | return self.index.objectcounts[c - 1] 163 | 164 | def object_presence(self, c): 165 | '''Returns a per-dataset-item count of the object.''' 166 | # Off by one due to use of 1-based indexing in matlab. 167 | return self.index.objectPresence[c - 1] 168 | 169 | def scale_image(self, im, dims, crop=False): 170 | if len(im.shape) == 2: 171 | # Handle grayscale images by adding an RGB channel 172 | im = numpy.repeat(im[numpy.newaxis], 3, axis=0) 173 | if im.shape[0:2] != dims: 174 | if not crop: 175 | im = imresize(im, dims) 176 | else: 177 | source = im.shape[0:2] 178 | aspect = float(dims[1]) / dims[0] 179 | if aspect * source[0] > source[1]: 180 | width = int(dims[1] / aspect) 181 | margin = (width - dims[0]) // 2 182 | im = imresize(im, (width, dims[1]))[ 183 | margin:margin+dims[0],:,:] 184 | else: 185 | height = int(dims[0] * aspect) 186 | margin = (height - dims[1]) // 2 187 | im = imresize(im, (dims[0], height))[ 188 | margin:margin+dims[1],:,:] 189 | return im 190 | 191 | def scale_segmentation(self, segmentation, dims, crop=False): 192 | if segmentation.shape[1:] == dims: 193 | return segmentation 194 | levels = segmentation.shape[0] 195 | result = numpy.zeros((levels, ) + dims, 196 | dtype=segmentation.dtype) 197 | ratio = (1,) + tuple(res / float(orig) 198 | for res, orig in zip(result.shape[1:], segmentation.shape[1:])) 199 | if not crop: 200 | safezoom(segmentation, ratio, output=result, order=0) 201 | else: 202 | ratio = max(ratio[1:]) 203 | height = int(round(dims[0] / ratio)) 204 | hmargin = (segmentation.shape[0] - height) // 2 205 | width = int(round(dims[1] / ratio)) 206 | wmargin = (segmentation.shape[1] - height) // 2 207 | safezoom(segmentation[:, hmargin:hmargin+height, 208 | wmargin:wmargin+width], 209 | (1, ratio, ratio), output=result, order=0) 210 | return result 211 | 212 | def save_image(self, im, filename, folder): 213 | imsave(os.path.join(folder, filename), im) 214 | 215 | def save_segmentation(self, seg, filename, folder, offset=0): 216 | for channel in range(seg.shape[0]): 217 | im = encodeClassMask(seg[channel], offset=offset) 218 | if channel == 0: 219 | fn = re.sub('\.jpg$', '_seg.png', filename) 220 | else: 221 | fn = re.sub('\.jpg$', '_parts_%s.png' % channel, filename) 222 | imsave(os.path.join(folder, fn), im) 223 | 224 | def save_sample(self, folder, size=None, indexes=None, crop=False, 225 | offset=0, reduction=1, progress=False): 226 | if indexes is None: 227 | indexes = range(self.size()) 228 | count = len(indexes) 229 | test_dim = None 230 | if size is not None: 231 | test_dim = tuple(int(d / reduction) for d in size) 232 | for i, index in enumerate(indexes): 233 | filename = self.short_filename(index) 234 | print 'Proessing %s (%d of %d)' % (filename, i, count) 235 | im = self.image(index) 236 | if size is not None: 237 | im = self.scale_image(im, size, crop=crop) 238 | self.save_image(im, filename, folder) 239 | seg = self.full_segmentation(index) 240 | if test_dim is not None: 241 | seg = self.scale_segmentation(seg, test_dim, crop=crop) 242 | self.save_segmentation(seg, filename, folder, offset=offset) 243 | print 'Processed %d images' % count 244 | 245 | def save_object_names(self, folder, offset=0): 246 | with file(os.path.join(folder, 'object_names.txt'), 'w') as f: 247 | for index in range(offset, self.num_object_types()): 248 | f.write('%s\t%d\n' % (self.object_name(index), index - offset)) 249 | 250 | def safezoom(array, ratio, output=None, order=0): 251 | '''Like numpy.zoom, but does not crash when the first dimension 252 | of the array is of size 1, as happens often with segmentations''' 253 | dtype = array.dtype 254 | if array.dtype == numpy.float16: 255 | array = array.astype(numpy.float32) 256 | if array.shape[0] == 1: 257 | if output is not None: 258 | output = output[0,...] 259 | result = zoom(array[0,...], ratio[1:], 260 | output=output, order=order) 261 | if output is None: 262 | output = result[numpy.newaxis] 263 | else: 264 | result = zoom(array, ratio, output=output, order=order) 265 | if output is None: 266 | output = result 267 | return output.astype(dtype) 268 | 269 | 270 | -------------------------------------------------------------------------------- /src/adeseg.py: -------------------------------------------------------------------------------- 1 | import colorname 2 | import glob 3 | import os 4 | import re 5 | import numpy 6 | from loadseg import AbstractSegmentation 7 | from scipy.io import loadmat 8 | from scipy.misc import imread 9 | from collections import namedtuple 10 | 11 | class AdeSegmentation(AbstractSegmentation): 12 | def __init__(self, directory=None, version=None): 13 | # Default to value of ADE20_ROOT env variable 14 | if directory is None: 15 | directory = os.environ['ADE20K_ROOT'] 16 | directory = os.path.expanduser(directory) 17 | # Default to the latest version present in the directory 18 | if version is None: 19 | contents = os.listdir(directory) 20 | if not list(c for c in contents if re.match('^index.*mat$', c)): 21 | version = sorted(c for c in contents if os.path.isdir( 22 | os.path.join(directory, c)))[-1] 23 | else: 24 | version = '' 25 | self.root = directory 26 | self.version = version 27 | mat = loadmat(self.expand(self.version, 'index*.mat'), squeeze_me=True) 28 | index = mat['index'] 29 | Ade20kIndex = namedtuple('Ade20kIndex', index.dtype.names) 30 | self.index = Ade20kIndex( 31 | **{name: index[name][()] for name in index.dtype.names}) 32 | self.scenes = ['-'] + [ 33 | norm_name(s) for s in sorted(set(self.index.scene))] 34 | self.scene_map = dict((s, i) for i, s in enumerate(self.scenes)) 35 | # Zero out special ~ scene names, which mean unlabeled. 36 | for k in self.scene_map: 37 | if k.startswith('~'): 38 | self.scene_map[k] = 0 39 | self.raw_mat = mat 40 | 41 | def all_names(self, category, j): 42 | if j == 0: 43 | return [] 44 | if category == 'color': 45 | return [colorname.color_names[j - 1] + '-c'] 46 | if category == 'scene': 47 | return [self.scenes[j] + '-s'] 48 | result = self.index.objectnames[j - 1] 49 | return re.split(',\s*', result) 50 | 51 | def size(self): 52 | '''Returns the number of images in this dataset.''' 53 | return len(self.index.filename) 54 | 55 | def filename(self, n): 56 | '''Returns the filename for the nth dataset image.''' 57 | filename = self.index.filename[n] 58 | folder = self.index.folder[n] 59 | return self.expand(folder, filename) 60 | 61 | def metadata(self, i): 62 | '''Returns an object that can be used to create all segmentations.''' 63 | return dict( 64 | filename=self.filename(i), 65 | seg_filename=self.seg_filename(i), 66 | part_filenames=self.part_filenames(i), 67 | scene=self.scene_map[norm_name(self.index.scene[i])] 68 | ) 69 | 70 | @classmethod 71 | def resolve_segmentation(cls, m, categories=None): 72 | result = {} 73 | if wants('scene', categories): 74 | result['scene'] = m['scene'] 75 | if wants('part', categories): 76 | result['part'] = load_parts(m) 77 | if wants('object', categories): 78 | result['object'] = load_segmentation(m) 79 | if wants('color', categories): 80 | result['color'] = colorname.label_major_colors(load_image(m)) + 1 81 | arrs = [a for a in result.values() if numpy.shape(a) >= 2] 82 | shape = arrs[0].shape[-2:] if arrs else (1, 1) 83 | return result, shape 84 | 85 | ### End of contract for AbstractSegmentation 86 | 87 | def seg_filename(self, n): 88 | '''Returns the segmentation filename for the nth dataset image.''' 89 | return re.sub(r'\.jpg$', '_seg.png', self.filename(n)) 90 | 91 | def part_filenames(self, n): 92 | '''Returns all the subpart images for the nth dataset image.''' 93 | filename = self.filename(n) 94 | level = 1 95 | result = [] 96 | while True: 97 | probe = re.sub(r'\.jpg$', '_parts_%d.png' % level, filename) 98 | if not os.path.isfile(probe): 99 | break 100 | result.append(probe) 101 | level += 1 102 | return result 103 | 104 | def expand(self, *path): 105 | '''Expands a filename and directories with the ADE dataset''' 106 | result = os.path.join(self.root, *path) 107 | if '*' in result or '?' in result: 108 | globbed = glob.glob(result) 109 | if len(globbed): 110 | return globbed[0] 111 | return result 112 | 113 | def norm_name(s): 114 | return s.replace(' - ', '-').replace('/', '-') 115 | 116 | def load_image(m): 117 | '''Returns the nth dataset image as a numpy array.''' 118 | return imread(m['filename']) 119 | 120 | def load_segmentation(m): 121 | '''Returns the nth dataset segmentation as a numpy array, 122 | where each entry at a pixel is an object class value. 123 | ''' 124 | data = imread(m['seg_filename']) 125 | return decodeClassMask(data) 126 | 127 | def load_parts(m): 128 | '''Returns an list of part segmentations for the nth dataset item, 129 | with one array for each level available. 130 | ''' 131 | result = [] 132 | for fn in m['part_filenames']: 133 | data = imread(fn) 134 | result.append(decodeClassMask(data)) 135 | if not result: 136 | return [] 137 | return numpy.concatenate(tuple(m[numpy.newaxis] for m in result)) 138 | 139 | def decodeClassMask(im): 140 | '''Decodes pixel-level object/part class and instance data from 141 | the given image, previously encoded into RGB channels.''' 142 | # Classes are a combination of RG channels (dividing R by 10) 143 | return (im[:,:,0] // 10) * 256 + im[:,:,1] 144 | 145 | def wants(what, option): 146 | if option is None: 147 | return True 148 | return what in option 149 | -------------------------------------------------------------------------------- /src/bargraph.py: -------------------------------------------------------------------------------- 1 | import re 2 | import expdir 3 | import itertools 4 | import operator 5 | from xml.etree import ElementTree as et 6 | 7 | default_category_order = [ 8 | 'object', 9 | 'scene', 10 | 'part', 11 | 'material', 12 | 'texture', 13 | 'color' 14 | ] 15 | 16 | palette = [ 17 | ('#4B4CBF', '#B6B6F2'), 18 | ('#55B05B', '#B6F2BA'), 19 | ('#50BDAC', '#A5E5DB'), 20 | ('#D4CF24', '#F2F1B6'), 21 | ('#F0883B', '#F2CFB6'), 22 | ('#D92E2B', '#F2B6B6') 23 | ] 24 | 25 | def most_common(L): 26 | # get an iterable of (item, iterable) pairs 27 | SL = sorted((x, i) for i, x in enumerate(L)) 28 | groups = itertools.groupby(SL, key=operator.itemgetter(0)) 29 | # auxiliary function to get "quality" for an item 30 | def _auxfun(g): 31 | item, iterable = g 32 | count = 0 33 | min_index = len(L) 34 | for _, where in iterable: 35 | count += 1 36 | min_index = min(min_index, where) 37 | return count, -min_index 38 | # pick the highest-count/earliest item 39 | return max(groups, key=_auxfun)[0] 40 | 41 | def bar_graph_svg(ed, blob, barheight, barwidth, 42 | order=None, 43 | show_labels=True, 44 | threshold=0.04, 45 | rendered_order=None, 46 | save=None): 47 | records = ed.load_csv(blob=blob, part='result') 48 | # ['unit', 'category', 'label', 'score'] 49 | # Examine each label 50 | label_cats = {} 51 | label_score = {} 52 | for record in records: 53 | if float(record['score']) < threshold: 54 | continue 55 | label = record['label'] 56 | if label not in label_cats: 57 | label_cats[label] = [] 58 | label_cats[label].append(record['category']) 59 | if (label not in label_score 60 | or label_score[label] < float(record['score'])): 61 | label_score[label] = float(record['score']) 62 | # Count each label, and note its cateogry 63 | label_counts = {} 64 | for label, cats in label_cats.items(): 65 | label_counts[label] = len(cats) 66 | label_cats[label] = most_common(cats) 67 | # Sort labels by frequency and max score 68 | sorted_labels = sorted(label_counts.keys(), 69 | key=lambda x: (-label_counts[x], -label_score[x])) 70 | category_order = order 71 | if not category_order: 72 | # Default category order: broden order plus any missing categories 73 | category_order = list(default_category_order) 74 | for label in sorted_labels: 75 | if label_cats[label] not in category_order: 76 | category_order.append(label_cats[label]) 77 | # Now make a plot 78 | heights = [] 79 | colors = [] 80 | categories = [] 81 | labels = [] 82 | for cat in category_order: 83 | filtered = [label for label in sorted_labels 84 | if label_cats[label] == cat] 85 | labels.extend(filtered) 86 | heights.extend([label_counts[label] for label in filtered]) 87 | categories.append((cat, len(filtered))) 88 | # Sort records in histogram order and output them if requested 89 | if rendered_order is not None: 90 | rendered_order.extend(sorted(records, key=lambda record: ( 91 | # Items below score threshold are sorted last, by score 92 | (len(category_order), 0, 0, -float(record['score'])) 93 | if float(record['score']) < threshold else 94 | # Others are sorted by category, label count/score, and score 95 | (category_order.index(label_cats[record['label']]), 96 | -label_counts[record['label']], -label_score[record['label']], 97 | -float(record['score']))))) 98 | filename = None 99 | if save: 100 | if save == True: 101 | filename = ed.filename('bargraph.svg', blob=blob, directory='html') 102 | else: 103 | filename = save 104 | ed.ensure_dir('html') 105 | return make_svg_bargraph(labels, heights, categories, 106 | barheight, barwidth, show_labels, filename) 107 | 108 | def make_svg_bargraph(labels, heights, categories, 109 | barheight=100, barwidth=12, show_labels=True, filename=None): 110 | unitheight = float(barheight) / max(heights) 111 | textheight = barheight if show_labels else 0 112 | labelsize = float(barwidth) 113 | gap = float(barwidth) / 4 114 | textsize = barwidth + gap 115 | rollup = max(heights) 116 | textmargin = float(labelsize) * 2 / 3 117 | leftmargin = 32 118 | rightmargin = 8 119 | svgwidth = len(heights) * (barwidth + gap) + 2 * leftmargin + rightmargin 120 | svgheight = barheight + textheight 121 | 122 | # create an SVG XML element 123 | svg = et.Element('svg', width=str(svgwidth), height=str(svgheight), 124 | version='1.1', xmlns='http://www.w3.org/2000/svg') 125 | 126 | # Draw the bar graph 127 | basey = svgheight - textheight 128 | x = leftmargin 129 | # Add units scale on left 130 | for h in [1, (max(heights) + 1) // 2, max(heights)]: 131 | et.SubElement(svg, 'text', x='0', y='0', 132 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:end;'+ 133 | 'alignment-baseline:hanging;' + 134 | 'transform:translate(%dpx, %dpx);') % 135 | (textsize, x - gap, basey - h * unitheight)).text = str(h) 136 | et.SubElement(svg, 'text', x='0', y='0', 137 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:middle;'+ 138 | 'transform:translate(%dpx, %dpx) rotate(-90deg)') % 139 | (textsize, x - gap - 1.2 * textsize, basey - h * unitheight / 2) 140 | ).text = 'units' 141 | # Draw big category background rectangles 142 | for catindex, (cat, catcount) in enumerate(categories): 143 | if not catcount: 144 | continue 145 | et.SubElement(svg, 'rect', x=str(x), y=str(basey - rollup * unitheight), 146 | width=(str((barwidth + gap) * catcount - gap)), 147 | height = str(rollup*unitheight), 148 | fill=palette[catindex % len(palette)][1]) 149 | x += (barwidth + gap) * catcount 150 | # Draw small bars as well as 45degree text labels 151 | x = leftmargin 152 | catindex = -1 153 | catcount = 0 154 | for label, height in zip(labels, heights): 155 | while not catcount and catindex <= len(categories): 156 | catindex += 1 157 | catcount = categories[catindex][1] 158 | color = palette[catindex % len(palette)][0] 159 | et.SubElement(svg, 'rect', x=str(x), y=str(basey-(height * unitheight)), 160 | width=str(barwidth), height=str(height * unitheight), 161 | fill=color) 162 | x += barwidth 163 | if show_labels: 164 | et.SubElement(svg, 'text', x='0', y='0', 165 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:end;'+ 166 | 'transform:translate(%dpx, %dpx) rotate(-45deg);') % 167 | (labelsize, x, basey + textmargin)).text = fix(label) 168 | x += gap 169 | catcount -= 1 170 | # Text lables for each category 171 | x = leftmargin 172 | for cat, catcount in categories: 173 | if not catcount: 174 | continue 175 | et.SubElement(svg, 'text', x='0', y='0', 176 | style=('font-family:sans-serif;font-size:%dpx;text-anchor:end;'+ 177 | 'transform:translate(%dpx, %dpx) rotate(-90deg);') % 178 | (textsize, x + (barwidth + gap) * catcount - gap, 179 | basey - rollup * unitheight + gap)).text = '%d %s' % ( 180 | catcount, fix(cat + ('s' if catcount != 1 else ''))) 181 | x += (barwidth + gap) * catcount 182 | # Output - this is the bare svg. 183 | result = et.tostring(svg) 184 | if filename: 185 | f = open(filename, 'w') 186 | # When writing to a file a special header is needed. 187 | f.write('\n') 188 | f.write('\n') 190 | f.write(result) 191 | f.close() 192 | return result 193 | 194 | replacements = [(re.compile(r[0]), r[1]) for r in [ 195 | (r'-[sc]$', ''), 196 | (r'_', ' '), 197 | ]] 198 | 199 | def fix(s): 200 | for pattern, subst in replacements: 201 | s = re.sub(pattern, subst, s) 202 | return s 203 | 204 | # Usage: 205 | # bargraph.py --directory dir --blob layer --barheight h --barwidth w 206 | if __name__ == '__main__': 207 | import argparse 208 | import sys 209 | import traceback 210 | 211 | parser = argparse.ArgumentParser( 212 | description='Plot graph of unique interpretations.') 213 | parser.add_argument( 214 | '--directory', 215 | help='directory to graph') 216 | parser.add_argument( 217 | '--blob', 218 | nargs='*', 219 | help='blob to graph') 220 | parser.add_argument( 221 | '--barheight', 222 | type=int, default=100, 223 | help='graph big color bar height') 224 | parser.add_argument( 225 | '--barwidth', 226 | type=int, default=12, 227 | help='graph little color bar width') 228 | args = parser.parse_args() 229 | 230 | bar_graph_svg(args.directory, args.blob, args.barheight, args.barwidth, 231 | save=True) 232 | -------------------------------------------------------------------------------- /src/colorname.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Discretizes colors to eleven color names as described in 3 | 4 | Learning Color Names for Real-World Applications 5 | J. van de Weijer, C. Schmid, J. Verbeek, D. Larlus. 6 | IEEE Transactions in Image Processing 2009. 7 | 8 | The work above uses color names observed in the wild (on Google Images) 9 | to derive probabilities for eleven color names for every point in RGB space. 10 | 11 | The function provided here simply assigns each pixel according to its 12 | maximum probability label based on probabilities from van de Weijer. 13 | ''' 14 | 15 | from scipy.io import loadmat 16 | import numpy 17 | 18 | # The 11 fundamental named colors in English. 19 | color_names = [ 20 | 'black', 21 | 'blue', 22 | 'brown', 23 | 'grey', 24 | 'green', 25 | 'orange', 26 | 'pink', 27 | 'purple', 28 | 'red', 29 | 'white', 30 | 'yellow'] 31 | 32 | color_values = numpy.array([ 33 | [0, 0, 0], # black 34 | [64, 64, 255], # blue 35 | [127, 101, 63], # brown 36 | [127, 127, 127], # grey 37 | [0, 192, 0], # green 38 | [255, 203, 0], # orange 39 | [192, 64, 192], # pink 40 | [255, 64, 255], # purple 41 | [255, 0, 0], # red 42 | [255, 255, 255], # white 43 | [255, 255, 0]], # yellow 44 | dtype='uint8') 45 | 46 | # Flat weighted mean of RGB values, generated from van de Weijer's data via: 47 | # w2c = numpy.swapaxes(scipy.io.loadmat( 48 | # 'w2c.mat')['w2c'].reshape(32,32,32,11), 0, 2) 49 | # r = numpy.tile(numpy.arange(0,256,8), (32,32,1)) 50 | # rr = numpy.stack([numpy.swapaxes(r, 2, n) 51 | # for n in range(3)])[:,:,:,:,None] 52 | # ((w2c[None] * rr).sum(axis=(1,2,3)) / 53 | # w2c.sum(axis=(0,1,2))[None]).transpose().astype('uint8') 54 | # What we actually need is weightings by real 55 | # color ocurrences. 56 | mean_color_values = numpy.array( 57 | [[ 97, 107, 113], # black 58 | [ 64, 119, 180], # blue 59 | [140, 116, 88], # brown 60 | [114, 125, 125], # grey 61 | [ 84, 186, 90], # green 62 | [182, 106, 65], # orange 63 | [188, 90, 136], # pink 64 | [138, 65, 172], # purple 65 | [155, 84, 77], # red 66 | [130, 166, 170], 67 | [162, 173, 75]], 68 | dtype='uint8') 69 | 70 | # Load 71 | _cached_w2color = None 72 | def get_color_map(): 73 | global _cached_w2color 74 | if _cached_w2color is None: 75 | # w2c = loadmat('w2c.mat')['w2c'] 76 | # _cached_w2color = numpy.swapaxes( 77 | # w2c.argmax(axis=1).reshape(32, 32, 32), 0, 2) 78 | # numpy.save('w2color', _cached_w2color, False) 79 | # instead just load this from the saved .npy file 80 | from inspect import getsourcefile 81 | import os.path 82 | _cached_w2color = numpy.load(os.path.join( 83 | os.path.dirname(getsourcefile(lambda:0)), 'w2color.npy')) 84 | return _cached_w2color 85 | 86 | def label_colors(im): 87 | try: 88 | if len(im.shape) == 2: 89 | im = numpy.repeat(im[:,:,numpy.newaxis], 3, axis=2) 90 | return get_color_map()[tuple(im[:,:,c] // 8 for c in range(3))] 91 | except: 92 | print im.shape 93 | print (im // 8).max() 94 | raise 95 | 96 | def label_major_colors(im, threshold=0): 97 | raw = label_colors(im) 98 | if not threshold: 99 | return raw 100 | pixel_threshold = int(im.shape[0] * im.shape[1] * threshold) 101 | counts = numpy.bincount(raw.ravel(), minlength=11) 102 | mask = (counts > pixel_threshold) 103 | return (raw + 1).astype('int') * mask[raw] - 1 104 | 105 | def posterize(im, use_mean_colors=False): 106 | if use_mean_colors: 107 | return mean_color_values[label_colors(im)] 108 | else: 109 | return color_values[label_colors(im)] 110 | 111 | if __name__ == '__main__': 112 | print get_color_map() 113 | -------------------------------------------------------------------------------- /src/dtdseg.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | import colorname 4 | from loadseg import AbstractSegmentation 5 | from scipy.misc import imread 6 | 7 | class DtdSegmentation(AbstractSegmentation): 8 | def __init__(self, directory=None): 9 | directory = os.path.expanduser(directory) 10 | self.directory = directory 11 | with open(os.path.join(directory, 'labels', 12 | 'labels_joint_anno.txt')) as f: 13 | self.dtd_meta = [line.split(None, 1) for line in f.readlines()] 14 | self.textures = ['-'] + sorted(list(set(sum( 15 | [c.split() for f, c in self.dtd_meta], [])))) 16 | self.texture_map = dict((t, i) for i, t in enumerate(self.textures)) 17 | 18 | def all_names(self, category, j): 19 | if j == 0: 20 | return [] 21 | if category == 'color': 22 | return [colorname.color_names[j - 1] + '-c'] 23 | if category == 'texture': 24 | return [self.textures[j]] 25 | return [] 26 | 27 | def size(self): 28 | '''Returns the number of images in this dataset.''' 29 | return len(self.dtd_meta) 30 | 31 | def filename(self, i): 32 | '''Returns the filename for the nth dataset image.''' 33 | return os.path.join(self.directory, 'images', self.dtd_meta[i][0]) 34 | 35 | def metadata(self, i): 36 | '''Returns an object that can be used to create all segmentations.''' 37 | fn, tnames = self.dtd_meta[i] 38 | tnumbers = [self.texture_map[n] for n in tnames.split()] 39 | return os.path.join(self.directory, 'images', fn), tnumbers 40 | 41 | @classmethod 42 | def resolve_segmentation(cls, m, categories=None): 43 | filename, tnumbers = m 44 | result = {} 45 | if wants('texture', categories): 46 | result['texture'] = tnumbers 47 | if wants('color', categories): 48 | result['color'] = colorname.label_major_colors(imread(filename)) + 1 49 | arrs = [a for a in result.values() if numpy.shape(a) >= 2] 50 | shape = arrs[0].shape[-2:] if arrs else (1, 1) 51 | return result, shape 52 | 53 | def wants(what, option): 54 | if option is None: 55 | return True 56 | return what in option 57 | -------------------------------------------------------------------------------- /src/fieldmap.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import os 4 | 5 | os.environ['GLOG_minloglevel'] = '2' 6 | import caffe 7 | from caffe.proto import caffe_pb2 8 | from google.protobuf import text_format 9 | import upsample 10 | import rotate 11 | 12 | def print_fieldmap(definition, blob): 13 | ''' 14 | definition: the filename for the caffe prototxt 15 | blobs: 'conv3' to probe 16 | ''' 17 | np = caffe_pb2.NetParameter() 18 | with open(definition, 'r') as dfn_file: 19 | text_format.Merge(dfn_file.read(), np) 20 | # Compute the input-to-layer fieldmap for each blob 21 | # ind = max(i for i, lay in enumerate(np.layer) if blob in lay.top) 22 | # path = upsample.shortest_layer_path(np.input, blob, np.layer) 23 | net = caffe.Net(definition, caffe.TEST) 24 | fieldmap, path = upsample.composed_fieldmap(np.layer, blob) 25 | for b, layer in path: 26 | f, _ = upsample.composed_fieldmap(np.layer, b) 27 | a = net.blobs[b].data.shape[-2:] 28 | s = upsample.upsampled_shape(f, a) 29 | print b, f, a, '->', s 30 | print blob, fieldmap 31 | 32 | if __name__ == '__main__': 33 | import argparse 34 | import loadseg 35 | 36 | parser = argparse.ArgumentParser( 37 | description='Calculate fieldmap for a blob in a caffe network.') 38 | parser.add_argument( 39 | '--blob', 40 | help='network blob name to probe') 41 | parser.add_argument( 42 | '--definition', 43 | help='the deploy prototext defining the net') 44 | args = parser.parse_args() 45 | 46 | print_fieldmap(args.definition, args.blob) 47 | -------------------------------------------------------------------------------- /src/graphprobe.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | from collections import OrderedDict, defaultdict 4 | import expdir 5 | 6 | category_colors = OrderedDict([ 7 | ('scene', '#3288bd'), 8 | ('object', '#99d594'), 9 | ('part', '#e6f598'), 10 | ('material', '#fee08b'), 11 | ('texture', '#fc8d59'), 12 | ('color', '#d53e4f'), 13 | ('total', '#aaaaaa') 14 | ]) 15 | 16 | 17 | def loadviz(directory, blob): 18 | ed = expdir.ExperimentDirectory(directory) 19 | html_fn = ed.filename(['html', '%s.html' % expdir.fn_safe(blob)]) 20 | with open(html_fn) as f: 21 | lines = f.readlines() 22 | result = OrderedDict() 23 | for line in lines: 24 | if 'unit ' in line: 25 | u, v = line.split(':', 1) 26 | unit = int(re.search(r'unit (\d+)', u).group(1)) - 1 27 | u_result = [] 28 | for w in v.split(';'): 29 | m = re.search(r'(\w.*\w)? \((\w+), ([.\d]+)\)', w) 30 | if not m: 31 | print 'On line', v 32 | print 'Error with', w 33 | label = m.group(1) 34 | category = m.group(2) 35 | score = float(m.group(3)) 36 | u_result.append((label, category, score)) 37 | result[unit] = u_result 38 | return result 39 | 40 | def summarize(scores, threshold, top_only=True): 41 | result = defaultdict(float) 42 | denom = len(scores) 43 | for k, v in scores.items(): 44 | got = False 45 | for val in v: 46 | label, category, score = val 47 | if score >= threshold: 48 | result[category] += 1.0 / denom 49 | got = True 50 | if top_only: 51 | break 52 | if got: 53 | result['total'] += 1.0 / denom 54 | return result 55 | 56 | if __name__ == '__main__': 57 | import argparse 58 | import sys 59 | import traceback 60 | 61 | try: 62 | import matplotlib 63 | matplotlib.use('Agg') 64 | import matplotlib.pyplot as plt 65 | 66 | 67 | parser = argparse.ArgumentParser( 68 | description='Generate visualization for probed activation data.') 69 | parser.add_argument( 70 | '--directories', 71 | nargs='*', 72 | help='directories to graph') 73 | parser.add_argument( 74 | '--blobs', 75 | nargs='*', 76 | help='blobs to graph') 77 | parser.add_argument( 78 | '--threshold', 79 | type=float, default=0.05, 80 | help='score above which to count items') 81 | parser.add_argument( 82 | '--top_only', 83 | type=lambda s: s.lower() in ['true', 't', 'yes', '1'], 84 | default=True, 85 | help='include only the top values') 86 | parser.add_argument( 87 | '--include_total', 88 | type=lambda s: s.lower() in ['true', 't', 'yes', '1'], 89 | default=False, 90 | help='include total value line') 91 | parser.add_argument( 92 | '--labels', 93 | nargs='*', 94 | help='tick labels') 95 | parser.add_argument( 96 | '--title', 97 | help='graph title') 98 | parser.add_argument( 99 | '--legend', 100 | default='upper right', 101 | help='location of legend') 102 | parser.add_argument( 103 | '--maxy', 104 | type=float, default=None, 105 | help='y axis range to apply') 106 | parser.add_argument( 107 | '--out', 108 | help='output filename for graph') 109 | args = parser.parse_args() 110 | data = [] 111 | categories = set(category_colors.keys()) 112 | for directory in args.directories: 113 | for blob in args.blobs: 114 | stats = summarize(loadviz(directory, blob), args.threshold, 115 | top_only=args.top_only) 116 | data.append(stats) 117 | categories.update(stats.keys()) 118 | x = range(1, len(data) + 1) 119 | maxval = 0 120 | plt.figure(num=None, figsize=(7,4), dpi=300) 121 | for cat in category_colors.keys(): 122 | if cat not in categories: 123 | continue 124 | if not args.include_total and cat == 'total': 125 | continue 126 | dat = [d[cat] for d in data] 127 | maxval = max(maxval, max(dat)) 128 | plt.plot(x, dat, 'o-' if cat != 'total' else 's--', 129 | color=category_colors[cat], label=cat) 130 | if args.labels: 131 | plt.xticks(x, args.labels) 132 | plt.margins(0.1) 133 | plt.legend(loc=args.legend) 134 | if args.maxy is not None: 135 | plt.ylim(-0.01, args.maxy) 136 | else: 137 | plt.ylim(-maxval * 0.05, maxval * 1.5) 138 | ax = plt.gca() 139 | ax.yaxis.grid(True) 140 | for side in ['top', 'bottom', 'right', 'left']: 141 | ax.spines[side].set_visible(False) 142 | ax.xaxis.set_ticks_position('bottom') 143 | if args.title: 144 | plt.title(args.title) 145 | plt.ylabel('portion of units alinged to a category concept') 146 | plt.savefig(args.out) 147 | 148 | except: 149 | traceback.print_exc(file=sys.stdout) 150 | sys.exit(1) 151 | -------------------------------------------------------------------------------- /src/labelprobe.py: -------------------------------------------------------------------------------- 1 | ''' 2 | labelprobe does the following: 3 | For every sample, 4 | - Get the ground-truth labels. 5 | - Upsample all activations, and make a mask at a .01 top cutoff. 6 | - Accumulate counts of pixels of ground-truth labels (count_g) 7 | - Accumulate counts of pixels of all activations per-image (count_a) 8 | - Accumulate counts of intersections between g and a (count_i) 9 | - Optionally count intesections between each category and a (count_c) 10 | - Tally top 500 intersection counts per unit per sample (count_t). 11 | ''' 12 | 13 | import glob 14 | import os 15 | import numpy 16 | import re 17 | import upsample 18 | import time 19 | import loadseg 20 | import signal 21 | import multiprocessing 22 | import multiprocessing.pool 23 | import expdir 24 | import sys 25 | import intersect 26 | 27 | # Should we tally intersections between each category (as a whole) 28 | # and each units' activations? This can be useful if we wish to 29 | # ignore pixels that are unlabeled in a category when computing 30 | # precision (i.e., count cross-label and not label-to-unlabeled 31 | # confusion). This does not work that well, so we do not tally these. 32 | COUNT_CAT_INTERSECTIONS = False 33 | 34 | class NoDaemonProcess(multiprocessing.Process): 35 | # make 'daemon' attribute always return False 36 | def _get_daemon(self): 37 | return False 38 | def _set_daemon(self, value): 39 | pass 40 | daemon = property(_get_daemon, _set_daemon) 41 | 42 | # We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool 43 | # because the latter is only a wrapper function, not a proper class. 44 | class NoDaemonPool(multiprocessing.pool.Pool): 45 | Process = NoDaemonProcess 46 | 47 | def label_probe(directory, blob, 48 | batch_size=5, ahead=4, tally_depth=2048, 49 | quantile=0.01, categories=None, verbose=False, 50 | parallel=0): 51 | ''' 52 | Evaluates the given blob from the probe stored in the given directory. 53 | Note that as part of the evaluation, the dataset is loaded again. 54 | ''' 55 | # Make sure we have a directory to work in 56 | qcode = ('%f' % quantile).replace('0.', '').rstrip('0') 57 | ed = expdir.ExperimentDirectory(directory) 58 | # If the tally file already exists, return. 59 | if ed.has_mmap(blob=blob, part='tally-%s' % qcode): 60 | if verbose: 61 | print 'Have', (ed.mmap_filename(blob=blob, part='tally-%s' % qcode) 62 | ), 'already. Skipping.' 63 | return 64 | # Load probe metadata 65 | info = ed.load_info() 66 | ih, iw = info.input_dim 67 | # Load blob metadata 68 | blob_info = ed.load_info(blob=blob) 69 | shape = blob_info.shape 70 | unit_size = shape[1] 71 | fieldmap = blob_info.fieldmap 72 | # Load the blob quantile data and grab thresholds 73 | quantdata = ed.open_mmap(blob=blob, part='quant-*', 74 | shape=(shape[1], -1)) 75 | threshold = quantdata[:, int(round(quantdata.shape[1] * quantile))] 76 | thresh = threshold[:, numpy.newaxis, numpy.newaxis] 77 | # Map the blob activation data for reading 78 | fn_read = ed.mmap_filename(blob=blob) 79 | # Load the dataset 80 | ds = loadseg.SegmentationData(info.dataset) 81 | if categories is None: 82 | categories = ds.category_names() 83 | labelcat = onehot(primary_categories_per_index(ds, categories)) 84 | # Be sure to zero out the background label - it belongs to no category. 85 | labelcat[0,:] = 0 86 | # Set up output data 87 | # G = ground truth counts 88 | # A = activation counts 89 | # I = intersection counts 90 | # TODO: consider activation_quantile and intersect_quantile 91 | # fn_s = os.path.join(directory, '%s-%s-scores.txt' % (blob, qcode)) 92 | # T = tally, a compressed per-sample record of intersection counts. 93 | fn_t = ed.mmap_filename(blob=blob, part='tally-%s' % qcode, 94 | inc=True) 95 | # Create the file so that it can be mapped 'r+' 96 | ed.open_mmap(blob=blob, part='tally-%s' % qcode, mode='w+', 97 | dtype='int32', shape=(ds.size(), tally_depth, 3)) 98 | if parallel > 1: 99 | fast_process(fn_t, fn_read, shape, tally_depth, ds, iw, ih, 100 | categories, fieldmap, 101 | thresh, labelcat, batch_size, ahead, verbose, parallel) 102 | else: 103 | process_data(fn_t, fn_read, shape, tally_depth, ds, iw, ih, 104 | categories, fieldmap, 105 | thresh, labelcat, batch_size, ahead, verbose, False, 106 | 0, ds.size()) 107 | fn_final = re.sub('-inc$', '', fn_t) 108 | if verbose: 109 | print 'Renaming', fn_t, 'to', fn_final 110 | os.rename(fn_t, fn_final) 111 | 112 | def fast_process(fn_t, fn_read, shape, tally_depth, ds, iw, ih, 113 | categories, fieldmap, 114 | thresh, labelcat, batch_size, ahead, verbose, parallel): 115 | psize = int(numpy.ceil(float(ds.size()) / parallel)) 116 | ranges = [(s, min(ds.size(), s + psize)) 117 | for s in range(0, ds.size(), psize) if s < ds.size()] 118 | parallel = len(ranges) 119 | original_sigint_handler = setup_sigint() 120 | # pool = multiprocessing.Pool(processes=parallel, initializer=setup_sigint) 121 | pool = multiprocessing.pool.ThreadPool(processes=parallel) 122 | restore_sigint(original_sigint_handler) 123 | # Precache memmaped files 124 | blobdata = cached_memmap(fn_read, mode='r', dtype='float32', 125 | shape=shape) 126 | count_t = cached_memmap(fn_t, mode='r+', dtype='int32', 127 | shape=(ds.size(), tally_depth, 3)) 128 | data = [ 129 | (fn_t, fn_read, shape, tally_depth, ds, iw, ih, categories, fieldmap, 130 | thresh, labelcat, batch_size, ahead, verbose, True) + r 131 | for r in ranges] 132 | try: 133 | result = pool.map_async(individual_process, data) 134 | result.get(31536000) 135 | except KeyboardInterrupt: 136 | print("Caught KeyboardInterrupt, terminating workers") 137 | pool.terminate() 138 | raise 139 | else: 140 | pool.close() 141 | pool.join() 142 | 143 | def individual_process(args): 144 | process_data(*args) 145 | 146 | global_memmap_cache = {} 147 | 148 | def cached_memmap(fn, mode, dtype, shape): 149 | global global_memmap_cache 150 | if fn not in global_memmap_cache: 151 | global_memmap_cache[fn] = numpy.memmap( 152 | fn, mode=mode, dtype=dtype, shape=shape) 153 | return global_memmap_cache[fn] 154 | 155 | def process_data(fn_t, fn_read, shape, tally_depth, ds, iw, ih, 156 | categories, fieldmap, 157 | thresh, labelcat, batch_size, ahead, verbose, thread, 158 | start, end): 159 | unit_size = len(thresh) 160 | blobdata = cached_memmap(fn_read, mode='r', dtype='float32', 161 | shape=shape) 162 | count_t = cached_memmap(fn_t, mode='r+', dtype='int32', 163 | shape=(ds.size(), tally_depth, 3)) 164 | count_t[...] = 0 165 | # The main loop 166 | if verbose: 167 | print 'Beginning work for evaluating', blob 168 | pf = loadseg.SegmentationPrefetcher(ds, categories=categories, 169 | start=start, end=end, once=True, 170 | batch_size=batch_size, ahead=ahead, thread=False) 171 | index = start 172 | start_time = time.time() 173 | last_batch_time = start_time 174 | batch_size = 0 175 | for batch in pf.batches(): 176 | batch_time = time.time() 177 | rate = (index - start) / (batch_time - start_time + 1e-15) 178 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 179 | last_batch_time = batch_time 180 | if verbose: 181 | print 'labelprobe index', index, 'items per sec', batch_rate, rate 182 | sys.stdout.flush() 183 | for rec in batch: 184 | sw, sh = [rec[k] for k in ['sw', 'sh']] 185 | reduction = int(round(iw / float(sw))) 186 | up = upsample.upsampleL(fieldmap, blobdata[index], 187 | shape=(sh,sw), reduction=reduction) 188 | mask = up > thresh 189 | accumulate_counts( 190 | mask, 191 | [rec[cat] for cat in categories], 192 | count_t[index], 193 | unit_size, 194 | labelcat) 195 | index += 1 196 | batch_size = len(batch) 197 | count_t.flush() 198 | 199 | def accumulate_counts(masks, label_list, tally_i, unit_size, labelcat): 200 | ''' 201 | masks shape (256, 192, 192) - 1/0 for each pixel 202 | label_list is a list of items whose shape varies: 203 | () - scalar label for the whole image 204 | (2) - multiple scalar whole-image labels 205 | (192, 192) - scalar per-point 206 | (3, 192, 192) - multiple scalar per-point 207 | tally_i = (32, 3) - 32x3 list of (count, label, unit) (one per sample) 208 | ''' 209 | # First convert label_list to scalar-matrix pair 210 | scalars = [] 211 | pixels = [] 212 | for label_group in label_list: 213 | shape = numpy.shape(label_group) 214 | if len(shape) % 2 == 0: 215 | label_group = [label_group] 216 | if len(shape) < 2: 217 | scalars.append(label_group) 218 | else: 219 | pixels.append(label_group) 220 | labels = ( 221 | numpy.concatenate(pixels) if pixels else numpy.array([], dtype=int), 222 | numpy.concatenate(scalars) if scalars else numpy.array([], dtype=int)) 223 | intersect.tallyMaskLabel(masks, labels, out=tally_i) 224 | 225 | def primary_categories_per_index(ds, categories): 226 | ''' 227 | Returns an array of primary category numbers for each label, where the 228 | first category listed in ds.category_names is given category number 0. 229 | ''' 230 | catmap = {} 231 | for cat in categories: 232 | imap = ds.category_index_map(cat) 233 | if len(imap) < ds.label_size(None): 234 | imap = numpy.concatenate((imap, numpy.zeros( 235 | ds.label_size(None) - len(imap), dtype=imap.dtype))) 236 | catmap[cat] = imap 237 | result = [] 238 | for i in range(ds.label_size(None)): 239 | maxcov, maxcat = max( 240 | (ds.coverage(cat, catmap[cat][i]) if catmap[cat][i] else 0, ic) 241 | for ic, cat in enumerate(categories)) 242 | result.append(maxcat) 243 | return numpy.array(result) 244 | 245 | def onehot(arr, minlength=None): 246 | ''' 247 | Expands an array of integers in one-hot encoding by adding a new last 248 | dimension, leaving zeros everywhere except for the nth dimension, where 249 | the original array contained the integer n. The minlength parameter is 250 | used to indcate the minimum size of the new dimension. 251 | ''' 252 | length = numpy.amax(arr) + 1 253 | if minlength is not None: 254 | length = max(minlength, length) 255 | result = numpy.zeros(arr.shape + (length,), dtype='int64') 256 | result[list(numpy.indices(arr.shape)) + [arr]] = 1 257 | return result 258 | 259 | def setup_sigint(): 260 | return signal.signal(signal.SIGINT, signal.SIG_IGN) 261 | 262 | def restore_sigint(original): 263 | if original is None: 264 | original = signal.SIG_DFL 265 | signal.signal(signal.SIGINT, original) 266 | 267 | if __name__ == '__main__': 268 | import argparse 269 | import sys 270 | import traceback 271 | 272 | try: 273 | parser = argparse.ArgumentParser( 274 | description='Generate evaluation files for probed activation data.') 275 | parser.add_argument( 276 | '--directory', 277 | default='.', 278 | help='output directory for the net probe') 279 | parser.add_argument( 280 | '--blobs', 281 | nargs='*', 282 | help='network blob names to tally') 283 | parser.add_argument( 284 | '--batch_size', 285 | type=int, default=5, 286 | help='the batch size to use') 287 | parser.add_argument( 288 | '--ahead', 289 | type=int, default=4, 290 | help='the prefetch lookahead size') 291 | parser.add_argument( 292 | '--quantile', 293 | type=float, default=0.01, 294 | help='the quantile cutoff to use') 295 | parser.add_argument( 296 | '--tally_depth', 297 | type=int, default=2048, 298 | help='the number of top label counts to tally for each sample') 299 | parser.add_argument( 300 | '--parallel', 301 | type=int, default=0, 302 | help='the number of parallel processes to apply') 303 | args = parser.parse_args() 304 | for blob in args.blobs: 305 | label_probe(args.directory, blob, batch_size=args.batch_size, 306 | ahead=args.ahead, quantile=args.quantile, 307 | tally_depth=args.tally_depth, verbose=True, 308 | parallel=args.parallel) 309 | except: 310 | traceback.print_exc(file=sys.stdout) 311 | sys.exit(1) 312 | -------------------------------------------------------------------------------- /src/makeresult.py: -------------------------------------------------------------------------------- 1 | ''' 2 | viewprobe creates visualizations for a certain eval. 3 | ''' 4 | 5 | import os 6 | import numpy 7 | import upsample 8 | import loadseg 9 | from scipy.misc import imread, imresize, imsave 10 | from loadseg import normalize_label 11 | import expdir 12 | 13 | def open_dataset(ed): 14 | return loadseg.SegmentationData(ed.load_info().dataset) 15 | 16 | def score_tally_stats(ed, ds, layerprobe, verbose=False): 17 | # First, score every unit 18 | if verbose: 19 | print 'Adding tallys of unit/label alignments.' 20 | sys.stdout.flush() 21 | categories = ds.category_names() 22 | # Provides a category for each label in the dataset. 23 | primary_categories = primary_categories_per_index(ds) 24 | # tally activations, groundtruth, and intersection 25 | ta, tg, ti = summarize_tally(ds, layerprobe) 26 | labelcat = onehot(primary_categories) 27 | # also tally category-groundtruth-presence 28 | tc = count_act_with_labelcat(ds, layerprobe) 29 | # If we were doing per-category activations p, then: 30 | # c = numpy.dot(p, labelcat.transpose()) 31 | epsilon = 1e-20 # avoid division-by-zero 32 | # If we were counting activations on non-category examples then: 33 | # iou = i / (a[:,numpy.newaxis] + g[numpy.newaxis,:] - i + epsilon) 34 | iou = ti / (tc + tg[numpy.newaxis,:] - ti + epsilon) 35 | # Let's tally by primary-category. 36 | ar = numpy.arange(iou.shape[1]) 37 | # actually - let's get the top iou for every category 38 | # per-category-iou. 39 | pciou = numpy.array([iou * (primary_categories[ar] == ci)[numpy.newaxis,:] 40 | for ci in range(len(categories))]) 41 | label_pciou = pciou.argmax(axis=2) 42 | # name_iou = [ds.name(None, i) for i in label_iou] 43 | name_pciou = [ 44 | [ds.name(None, j) for j in label_pciou[ci]] 45 | for ci in range(len(label_pciou))] 46 | # score_iou = iou[numpy.arange(iou.shape[0]), label_iou] 47 | score_pciou = pciou[ 48 | numpy.arange(pciou.shape[0])[:,numpy.newaxis], 49 | numpy.arange(pciou.shape[1])[numpy.newaxis,:], 50 | label_pciou] 51 | bestcat_pciou = score_pciou.argsort(axis=0)[::-1] 52 | # Assign category for each label 53 | # cat_per_label = primary_categories_per_index(ds) 54 | # cat_iou = [categories[cat_per_label[i]] for i in label_iou] 55 | # Now sort units by score and visulize each one 56 | return bestcat_pciou, name_pciou, score_pciou, label_pciou, tc, tg, ti 57 | 58 | def generate_csv_summary( 59 | layer, csvfile, tally_stats, 60 | ds, order=None, verbose=False): 61 | if verbose: 62 | print 'Generating csv summary', csvfile 63 | sys.stdout.flush() 64 | bestcat_pciou, name_pciou, score_pciou, label_pciou, tc, tg, ti = ( 65 | tally_stats) 66 | 67 | # For each unit in a layer, outputs the following information: 68 | # - label: best interpretation 69 | # - object-label: top ranked interpretation for scene/object/color/etc 70 | # - object-truth: ground truth pixels 71 | # - object-activation: activating pixels 72 | # - object-intersect: intersecting pixels 73 | # - object-iou: iou score 74 | # - etc, for each category. 75 | categories = ds.category_names() 76 | csv_fields = sum([[ 77 | '%s-label' % cat, 78 | '%s-truth' % cat, 79 | '%s-activation' % cat, 80 | '%s-intersect' % cat, 81 | '%s-iou' % cat] for cat in categories], 82 | ['unit', 'category', 'label', 'score']) 83 | 84 | if order is not None: 85 | csv_fields = order 86 | 87 | if verbose: 88 | print 'Sorting units by score.' 89 | sys.stdout.flush() 90 | ordering = score_pciou.max(axis=0).argsort()[::-1] 91 | 92 | import csv 93 | with open(csvfile, 'w') as f: 94 | writer = csv.DictWriter(open(csvfile, 'w'), csv_fields) 95 | writer.writeheader() 96 | 97 | for unit in ordering: 98 | # Top images are top[unit] 99 | bestcat = bestcat_pciou[0, unit] 100 | data = { 101 | 'unit': (unit + 1), 102 | 'category': categories[bestcat], 103 | 'label': name_pciou[bestcat][unit], 104 | 'score': score_pciou[bestcat][unit] 105 | } 106 | for ci, cat in enumerate(categories): 107 | label = label_pciou[ci][unit] 108 | data.update({ 109 | '%s-label' % cat: name_pciou[ci][unit], 110 | '%s-truth' % cat: tg[label], 111 | '%s-activation' % cat: tc[unit, label], 112 | '%s-intersect' % cat: ti[unit, label], 113 | '%s-iou' % cat: score_pciou[ci][unit] 114 | }) 115 | writer.writerow(data) 116 | 117 | def summarize_tally(ds, layerprobe): 118 | cat_count = len(ds.category_names()) 119 | tally = layerprobe.tally 120 | unit_size = layerprobe.shape[1] 121 | label_size = ds.label_size() 122 | count = numpy.zeros( 123 | (unit_size + 1, label_size + 1 + cat_count), dtype='int64') 124 | for i in range(len(tally)): 125 | t = tally[i] 126 | count[t[:,0]+1, t[:,1]+1+cat_count] += t[:,2] 127 | # count_a.shape = (unit size,) 128 | count_a = count[1:,cat_count] 129 | # this would summarize category intersections if we tallied them 130 | # count_c.shape = (unit_size, cat_size) 131 | # count_c = count[1:,0:cat_count] 132 | # count_g.shape = (label_size,) 133 | count_g = count[0,1+cat_count:] 134 | # count_i.shape = (unit_size, label_size) 135 | count_i = count[1:,1+cat_count:] 136 | # return count_a, count_c, count_g, count_i 137 | return count_a, count_g, count_i 138 | 139 | def count_act_with_labelcat(ds, layerprobe): 140 | # Because our dataaset is sparse, instead of using count_a to count 141 | # all activations, we can compute count_act_with_labelcat to count 142 | # activations only within those images which contain an instance 143 | # of a given label category. 144 | labelcat = onehot(primary_categories_per_index(ds)) 145 | # Be sure to zero out the background label - it belongs to no category. 146 | labelcat[0,:] = 0 147 | tally = layerprobe.tally 148 | unit_size = layerprobe.shape[1] 149 | label_size = ds.label_size() 150 | count = numpy.zeros((unit_size, labelcat.shape[1]), dtype='int64') 151 | for i in range(len(tally)): 152 | c1 = numpy.zeros((unit_size + 1, label_size + 1), dtype='int64') 153 | t = tally[i] 154 | c1[t[:,0]+1, t[:,1]+1] = t[:,2] 155 | count += c1[1:,0][:,numpy.newaxis] * ( 156 | numpy.dot(c1[0,1:], labelcat) > 0) 157 | # retval: (unit_size, label_size) 158 | return numpy.dot(count, numpy.transpose(labelcat)) 159 | 160 | class LayerProbe: 161 | def __init__(self, ed, ds, blob): 162 | info = ed.load_info(blob=blob) 163 | self.shape = info.shape 164 | self.fieldmap = info.fieldmap 165 | # Load the raw activation data 166 | if ed.has_mmap(blob=blob): 167 | self.blobdata = ed.open_mmap(blob=blob, shape=self.shape, mode='r') 168 | # Load the blob quantile data and grab thresholds 169 | if ed.has_mmap(blob=blob, part='quant-*'): 170 | self.quantdata = ed.open_mmap(blob=blob, part='quant-*', 171 | shape=(self.shape[1], -1), mode='r') 172 | # Load tally too; tally_depth is inferred from file size. 173 | self.tally = ed.open_mmap(blob=blob, part='tally-*', decimal=True, 174 | shape=(ds.size(), -1, 3), dtype='int32', mode='r') 175 | # And load imgmax 176 | if ed.has_mmap(blob=blob, part='imgmax'): 177 | self.imgmax = ed.open_mmap(blob=blob, part='imgmax', 178 | shape=(ds.size(), self.shape[1]), mode='r') 179 | # Figure out tally level that was used. 180 | self.level = ed.glob_number( 181 | 'tally-*.mmap', blob=blob, decimal=True) 182 | 183 | def primary_categories_per_index(ds, categories=None): 184 | ''' 185 | Returns an array of primary category numbers for each label, where the 186 | first category listed in ds.category_names is given category number 0. 187 | ''' 188 | if categories is None: 189 | categories = ds.category_names() 190 | catmap = {} 191 | for cat in categories: 192 | imap = ds.category_index_map(cat) 193 | if len(imap) < ds.label_size(None): 194 | imap = numpy.concatenate((imap, numpy.zeros( 195 | ds.label_size(None) - len(imap), dtype=imap.dtype))) 196 | catmap[cat] = imap 197 | result = [] 198 | for i in range(ds.label_size(None)): 199 | maxcov, maxcat = max( 200 | (ds.coverage(cat, catmap[cat][i]) if catmap[cat][i] else 0, ic) 201 | for ic, cat in enumerate(categories)) 202 | result.append(maxcat) 203 | return numpy.array(result) 204 | 205 | def onehot(arr, minlength=None): 206 | ''' 207 | Expands an array of integers in one-hot encoding by adding a new last 208 | dimension, leaving zeros everywhere except for the nth dimension, where 209 | the original array contained the integer n. The minlength parameter is 210 | used to indcate the minimum size of the new dimension. 211 | ''' 212 | length = numpy.amax(arr) + 1 213 | if minlength is not None: 214 | length = max(minlength, length) 215 | result = numpy.zeros(arr.shape + (length,)) 216 | result[list(numpy.indices(arr.shape)) + [arr]] = 1 217 | return result 218 | 219 | if __name__ == '__main__': 220 | import argparse 221 | import sys 222 | import traceback 223 | 224 | try: 225 | parser = argparse.ArgumentParser( 226 | description='Generate blob-result.csv for probed activation data.') 227 | parser.add_argument( 228 | '--directory', 229 | default='.', 230 | help='output directory for the net probe') 231 | parser.add_argument( 232 | '--csvorder', 233 | help='csv header order') 234 | parser.add_argument( 235 | '--blobs', 236 | nargs='*', 237 | help='network blob names to visualize') 238 | args = parser.parse_args() 239 | ed = expdir.ExperimentDirectory(args.directory) 240 | ds = open_dataset(ed) 241 | for blob in args.blobs: 242 | layerprobe = LayerProbe(ed, ds, blob) 243 | tally_stats = score_tally_stats(ed, ds, layerprobe, verbose=True) 244 | filename = ed.csv_filename(blob=blob, part='result') 245 | generate_csv_summary(blob, filename, tally_stats, ds, 246 | order=(args.csvorder.split(',') 247 | if args.csvorder else None), verbose=True) 248 | except: 249 | traceback.print_exc(file=sys.stdout) 250 | sys.exit(1) 251 | -------------------------------------------------------------------------------- /src/maxprobe.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | import re 4 | import shutil 5 | import time 6 | import expdir 7 | import sys 8 | from scipy.io import savemat 9 | 10 | def max_probe(directory, blob, batch_size=None): 11 | # Make sure we have a directory to work in 12 | ed = expdir.ExperimentDirectory(directory) 13 | 14 | # If it's already computed, then skip it!!! 15 | print "Checking", ed.mmap_filename(blob=blob, part='imgmax') 16 | if ed.has_mmap(blob=blob, part='imgmax'): 17 | print "Already have %s-imgmax.mmap, skipping." % (blob) 18 | return 19 | 20 | # Read about the blob shape 21 | blob_info = ed.load_info(blob=blob) 22 | shape = blob_info.shape 23 | 24 | print 'Computing imgmax for %s shape %r' % (blob, shape) 25 | data = ed.open_mmap(blob=blob, shape=shape) 26 | imgmax = ed.open_mmap(blob=blob, part='imgmax', 27 | mode='w+', shape=data.shape[:2]) 28 | 29 | # Automatic batch size selection: 64mb batches 30 | if batch_size is None: 31 | batch_size = max(1, int(128 * 1024 * 1024 / numpy.prod(shape[1:]))) 32 | 33 | # Algorithm: one pass over the data 34 | start_time = time.time() 35 | last_batch_time = start_time 36 | for i in range(0, data.shape[0], batch_size): 37 | batch_time = time.time() 38 | rate = i / (batch_time - start_time + 1e-15) 39 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 40 | last_batch_time = batch_time 41 | print 'Imgmax %s index %d: %f %f' % (blob, i, rate, batch_rate) 42 | sys.stdout.flush() 43 | batch = data[i:i+batch_size] 44 | imgmax[i:i+batch_size] = batch.max(axis=(2,3)) 45 | print 'Writing imgmax' 46 | sys.stdout.flush() 47 | # Save as mat file 48 | filename = ed.filename('imgmax.mat', blob=blob) 49 | savemat(filename, { 'imgmax': imgmax }) 50 | # And as mmap 51 | ed.finish_mmap(imgmax) 52 | 53 | 54 | if __name__ == '__main__': 55 | import argparse 56 | import sys 57 | import traceback 58 | 59 | try: 60 | import loadseg 61 | 62 | parser = argparse.ArgumentParser( 63 | description='Generate sorted files for probed activation data.') 64 | parser.add_argument( 65 | '--directory', 66 | default='.', 67 | help='output directory for the net probe') 68 | parser.add_argument( 69 | '--blobs', 70 | nargs='*', 71 | help='network blob names to sort') 72 | parser.add_argument( 73 | '--batch_size', 74 | type=int, default=None, 75 | help='the batch size to use') 76 | args = parser.parse_args() 77 | for blob in args.blobs: 78 | max_probe(args.directory, blob, args.batch_size) 79 | except: 80 | traceback.print_exc(file=sys.stdout) 81 | sys.exit(1) 82 | -------------------------------------------------------------------------------- /src/netprobe.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import os 4 | import numpy 5 | import glob 6 | import shutil 7 | import codecs 8 | import time 9 | import sys 10 | 11 | os.environ['GLOG_minloglevel'] = '2' 12 | import caffe 13 | from caffe.proto import caffe_pb2 14 | from google.protobuf import text_format 15 | from scipy.misc import imresize, imread 16 | from scipy.ndimage.filters import gaussian_filter 17 | from scipy.ndimage.interpolation import zoom 18 | from tempfile import NamedTemporaryFile 19 | from contextlib import contextmanager 20 | from collections import namedtuple 21 | import upsample 22 | import rotate 23 | import expdir 24 | 25 | caffe.set_mode_gpu() 26 | caffe.set_device(0) 27 | 28 | def create_probe( 29 | directory, dataset, definition, weights, mean, blobs, 30 | colordepth=3, 31 | rotation_seed=None, rotation_power=1, rotation_unpermute=False, 32 | limit=None, split=None, 33 | batch_size=16, ahead=4, 34 | cl_args=None, verbose=True): 35 | # If we're already done, skip it! 36 | ed = expdir.ExperimentDirectory(directory) 37 | if all(ed.has_mmap(blob=b) for b in blobs): 38 | return 39 | 40 | ''' 41 | directory: where to place the probe_conv5.mmap files. 42 | data: the AbstractSegmentation data source to draw upon 43 | definition: the filename for the caffe prototxt 44 | weights: the filename for the caffe model weights 45 | mean: to use to normalize rgb values for the network 46 | blobs: ['conv3', 'conv4', 'conv5'] to probe 47 | ''' 48 | if verbose: 49 | print 'Opening dataset', dataset 50 | data = loadseg.SegmentationData(args.dataset) 51 | if verbose: 52 | print 'Opening network', definition 53 | np = caffe_pb2.NetParameter() 54 | with open(definition, 'r') as dfn_file: 55 | text_format.Merge(dfn_file.read(), np) 56 | net = caffe.Net(definition, weights, caffe.TEST) 57 | input_blob = net.inputs[0] 58 | input_dim = net.blobs[input_blob].data.shape[2:] 59 | data_size = data.size(split) 60 | if limit is not None: 61 | data_size = min(data_size, limit) 62 | 63 | # Make sure we have a directory to work in 64 | ed.ensure_dir() 65 | 66 | # Step 0: write a README file with generated information. 67 | ed.save_info(dict( 68 | dataset=dataset, 69 | split=split, 70 | definition=definition, 71 | weights=weights, 72 | mean=mean, 73 | blobs=blobs, 74 | input_dim=input_dim, 75 | rotation_seed=rotation_seed, 76 | rotation_power=rotation_power)) 77 | 78 | # Clear old probe data 79 | ed.remove_all('*.mmap*') 80 | 81 | # Create new (empty) mmaps 82 | if verbose: 83 | print 'Creating new mmaps.' 84 | out = {} 85 | rot = None 86 | if rotation_seed is not None: 87 | rot = {} 88 | for blob in blobs: 89 | shape = (data_size, ) + net.blobs[blob].data.shape[1:] 90 | out[blob] = ed.open_mmap(blob=blob, mode='w+', shape=shape) 91 | # Find the shortest path through the network to the target blob 92 | fieldmap, _ = upsample.composed_fieldmap(np.layer, blob) 93 | # Compute random rotation for each blob, if needed 94 | if rot is not None: 95 | rot[blob] = rotate.randomRotationPowers( 96 | shape[1], [rotation_power], rotation_seed, 97 | unpermute=rotation_unpermute)[0] 98 | ed.save_info(blob=blob, data=dict( 99 | name=blob, shape=shape, fieldmap=fieldmap)) 100 | 101 | # The main loop 102 | if verbose: 103 | print 'Beginning work.' 104 | pf = loadseg.SegmentationPrefetcher(data, categories=['image'], 105 | split=split, once=True, batch_size=batch_size, ahead=ahead) 106 | index = 0 107 | start_time = time.time() 108 | last_batch_time = start_time 109 | batch_size = 0 110 | for batch in pf.tensor_batches(bgr_mean=mean): 111 | batch_time = time.time() 112 | rate = index / (batch_time - start_time + 1e-15) 113 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 114 | last_batch_time = batch_time 115 | if verbose: 116 | print 'netprobe index', index, 'items per sec', batch_rate, rate 117 | sys.stdout.flush() 118 | inp = batch[0] 119 | batch_size = len(inp) 120 | if limit is not None and index + batch_size > limit: 121 | # Truncate last if limited 122 | batch_size = limit - index 123 | inp = inp[:batch_size] 124 | if colordepth == 1: 125 | inp = numpy.mean(inp, axis=1, keepdims=True) 126 | net.blobs[input_blob].reshape(*(inp.shape)) 127 | net.blobs[input_blob].data[...] = inp 128 | result = net.forward(blobs=blobs) 129 | if rot is not None: 130 | for key in out.keys(): 131 | result[key] = numpy.swapaxes(numpy.tensordot( 132 | rot[key], result[key], axes=((1,), (1,))), 0, 1) 133 | # print 'Computation done' 134 | for key in out.keys(): 135 | out[key][index:index + batch_size] = result[key] 136 | # print 'Recording data in mmap done' 137 | index += batch_size 138 | if index >= data_size: 139 | break 140 | assert index == data_size, ( 141 | "Data source should return evey item once %d %d." % 142 | (index, data_size)) 143 | if verbose: 144 | print 'Renaming mmaps.' 145 | for blob in blobs: 146 | ed.finish_mmap(out[blob]) 147 | 148 | # Final step: write the README file 149 | write_readme_file([ 150 | ('cl_args', cl_args), 151 | ('data', data), 152 | ('definition', definition), 153 | ('weight', weights), 154 | ('mean', mean), 155 | ('blobs', blobs)], ed, verbose=verbose) 156 | 157 | 158 | def ensure_dir(targetdir): 159 | if not os.path.isdir(targetdir): 160 | try: 161 | os.makedirs(targetdir) 162 | except: 163 | print 'Could not create', targetdir 164 | pass 165 | 166 | def write_readme_file(args, ed, verbose): 167 | ''' 168 | Writes a README.txt that describes the settings used to geenrate the ds. 169 | ''' 170 | with codecs.open(ed.filename('README.txt'), 'w', 'utf-8') as f: 171 | def report(txt): 172 | f.write('%s\n' % txt) 173 | if verbose: 174 | print txt 175 | title = '%s network probe' % ed.basename() 176 | report('%s\n%s' % (title, '=' * len(title))) 177 | for key, val in args: 178 | if key == 'cl_args': 179 | if val is not None: 180 | report('Command-line args:') 181 | for ck, cv in vars(val).items(): 182 | report(' %s: %r' % (ck, cv)) 183 | report('%s: %r' % (key, val)) 184 | report('\ngenerated at: %s' % time.strftime("%Y-%m-%d %H:%M")) 185 | try: 186 | label = subprocess.check_output(['git', 'rev-parse', 'HEAD']) 187 | report('git label: %s' % label) 188 | except: 189 | pass 190 | 191 | if __name__ == '__main__': 192 | import sys 193 | import traceback 194 | import argparse 195 | try: 196 | import loadseg 197 | 198 | parser = argparse.ArgumentParser(description= 199 | 'Probe a caffe network and save results in a directory.') 200 | parser.add_argument( 201 | '--directory', 202 | default='.', 203 | help='output directory for the net probe') 204 | parser.add_argument( 205 | '--blobs', 206 | nargs='*', 207 | help='network blob names to collect') 208 | parser.add_argument( 209 | '--definition', 210 | help='the deploy prototext defining the net') 211 | parser.add_argument( 212 | '--weights', 213 | help='the caffemodel file of weights for the net') 214 | parser.add_argument( 215 | '--mean', 216 | nargs='*', type=float, 217 | help='mean values to subtract from input') 218 | parser.add_argument( 219 | '--dataset', 220 | help='the directory containing the dataset to use') 221 | parser.add_argument( 222 | '--split', 223 | help='the split of the dataset to use') 224 | parser.add_argument( 225 | '--limit', 226 | type=int, default=None, 227 | help='limit dataset to this size') 228 | parser.add_argument( 229 | '--batch_size', 230 | type=int, default=256, 231 | help='the batch size to use') 232 | parser.add_argument( 233 | '--ahead', 234 | type=int, default=4, 235 | help='number of batches to prefetch') 236 | parser.add_argument( 237 | '--rotation_seed', 238 | type=int, default=None, 239 | help='the seed for the random rotation to apply') 240 | parser.add_argument( 241 | '--rotation_power', 242 | type=float, default=1.0, 243 | help='the power of the random rotation') 244 | parser.add_argument( 245 | '--rotation_unpermute', 246 | type=int, default=0, 247 | help='set to 1 to unpermute random rotation') 248 | parser.add_argument( 249 | '--colordepth', 250 | type=int, default=3, 251 | help='set to 1 for grayscale') 252 | args = parser.parse_args() 253 | 254 | create_probe( 255 | args.directory, args.dataset, args.definition, args.weights, 256 | numpy.array(args.mean, dtype=numpy.float32), args.blobs, 257 | batch_size=args.batch_size, ahead=args.ahead, limit=args.limit, 258 | colordepth=args.colordepth, 259 | rotation_seed=args.rotation_seed, 260 | rotation_power=args.rotation_power, 261 | rotation_unpermute=args.rotation_unpermute, 262 | split=args.split, cl_args=args, verbose=True) 263 | except: 264 | traceback.print_exc(file=sys.stdout) 265 | sys.exit(1) 266 | -------------------------------------------------------------------------------- /src/netprobe_pytorch.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Bolei added 4 | import pdb 5 | import torch 6 | import torchvision 7 | from torch.autograd import Variable as V 8 | from torchvision import transforms as trn 9 | 10 | import os 11 | import numpy 12 | import glob 13 | import shutil 14 | import codecs 15 | import time 16 | import sys 17 | 18 | os.environ['GLOG_minloglevel'] = '2' 19 | from scipy.misc import imresize, imread 20 | from scipy.ndimage.filters import gaussian_filter 21 | from scipy.ndimage.interpolation import zoom 22 | from tempfile import NamedTemporaryFile 23 | from contextlib import contextmanager 24 | from collections import namedtuple 25 | import upsample 26 | import rotate 27 | import expdir 28 | 29 | 30 | def create_probe( 31 | directory, dataset, definition, weights, mean, blobs, 32 | colordepth=3, 33 | rotation_seed=None, rotation_power=1, 34 | limit=None, split=None, 35 | batch_size=16, ahead=4, 36 | cl_args=None, verbose=True): 37 | # If we're already done, skip it! 38 | ed = expdir.ExperimentDirectory(directory) 39 | if all(ed.has_mmap(blob=b) for b in blobs): 40 | return 41 | 42 | ''' 43 | directory: where to place the probe_conv5.mmap files. 44 | data: the AbstractSegmentation data source to draw upon 45 | definition: the filename for the pytorch 46 | weights: the filename for the weights 47 | mean: to use to normalize rgb values for the network 48 | blobs: ['conv3', 'conv4', 'conv5'] to probe 49 | ''' 50 | if verbose: 51 | print 'Opening dataset', dataset 52 | data = loadseg.SegmentationData(args.dataset) 53 | 54 | # the network to dissect 55 | if args.weights == 'none': 56 | # load the imagenet pretrained model 57 | net = torchvision.models.__dict__[args.definition](pretrained=True) 58 | else: 59 | # load your own model 60 | net = torchvision.models.__dict__[args.definition](num_classes=args.num_classes) 61 | checkpoint = torch.load(args.weights) 62 | state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].iteritems()} # the data parallel layer will add 'module' before each layer name 63 | net.load_state_dict(state_dict) 64 | net.eval() 65 | # hook up to get the information for each selected layer 66 | layers = net._modules.keys() 67 | size_blobs_output = [] 68 | def hook_size(module, input, output): 69 | size_blobs_output.append(output.data.size()) 70 | input_sample = V(torch.randn(1,3,args.input_size,args.input_size)) 71 | for blob in blobs: 72 | net._modules.get(blob).register_forward_hook(hook_size) 73 | 74 | output_sample = net(input_sample) 75 | 76 | input_dim = [args.input_size, args.input_size] 77 | data_size = data.size(split) # the image size 78 | if limit is not None: 79 | data_size = min(data_size, limit) 80 | 81 | # Make sure we have a directory to work in 82 | ed.ensure_dir() 83 | 84 | # Step 0: write a README file with generated information. 85 | ed.save_info(dict( 86 | dataset=dataset, 87 | split=split, 88 | definition=definition, 89 | weights=weights, 90 | mean=mean, 91 | blobs=blobs, 92 | input_dim=input_dim, 93 | rotation_seed=rotation_seed, 94 | rotation_power=rotation_power)) 95 | 96 | # Clear old probe data 97 | ed.remove_all('*.mmap*') 98 | 99 | # Create new (empty) mmaps 100 | if verbose: 101 | print 'Creating new mmaps.' 102 | out = {} 103 | rot = None 104 | if rotation_seed is not None: 105 | rot = {} 106 | for idx, blob in enumerate(blobs): 107 | #shape = (data_size, ) + net.blobs[blob].data.shape[1:] 108 | shape = (data_size, int(size_blobs_output[idx][1]), int(size_blobs_output[idx][2]),int(size_blobs_output[idx][3])) 109 | out[blob] = ed.open_mmap(blob=blob, mode='w+', shape=shape) 110 | 111 | # Rather than use the exact RF, here we use some heuristics to compute the approximate RF 112 | size_RF = (args.input_size/size_blobs_output[idx][2], args.input_size/size_blobs_output[idx][3]) 113 | fieldmap = ((0, 0), size_RF, size_RF) 114 | 115 | ed.save_info(blob=blob, data=dict( 116 | name=blob, shape=shape, fieldmap=fieldmap)) 117 | 118 | # The main loop 119 | if verbose: 120 | print 'Beginning work.' 121 | pf = loadseg.SegmentationPrefetcher(data, categories=['image'], 122 | split=split, once=True, batch_size=batch_size, ahead=ahead) 123 | index = 0 124 | start_time = time.time() 125 | last_batch_time = start_time 126 | batch_size = 0 127 | net.cuda() 128 | 129 | # hook the feature extractor 130 | features_blobs = [] 131 | def hook_feature(module, input, output): 132 | features_blobs.append(output.data.cpu().numpy()) 133 | for blob in blobs: 134 | net._modules.get(blob).register_forward_hook(hook_feature) 135 | 136 | for batch in pf.tensor_batches(bgr_mean=mean): 137 | del features_blobs[:] # clear up the feature basket 138 | batch_time = time.time() 139 | rate = index / (batch_time - start_time + 1e-15) 140 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 141 | last_batch_time = batch_time 142 | if verbose: 143 | print 'netprobe index', index, 'items per sec', batch_rate, rate 144 | sys.stdout.flush() 145 | inp = batch[0] 146 | batch_size = len(inp) 147 | if limit is not None and index + batch_size > limit: 148 | # Truncate last if limited 149 | batch_size = limit - index 150 | inp = inp[:batch_size] 151 | if colordepth == 1: 152 | inp = numpy.mean(inp, axis=1, keepdims=True) 153 | # previous feedforward case 154 | inp = inp[:,::-1,:,:] 155 | inp_tensor = V(torch.from_numpy(inp.copy())) 156 | inp_tensor.div_(255.0*0.224) # hack: approximately normalize the input to make the images scaled at around 1. 157 | inp_tensor = inp_tensor.cuda() 158 | result = net.forward(inp_tensor) 159 | # output the hooked feature 160 | for i, key in enumerate(blobs): 161 | out[key][index:index+batch_size] = numpy.copy(features_blobs[i][:batch_size]) 162 | # print 'Recording data in mmap done' 163 | index += batch_size 164 | if index >= data_size: 165 | break 166 | assert index == data_size, ( 167 | "Data source should return evey item once %d %d." % 168 | (index, data_size)) 169 | if verbose: 170 | print 'Renaming mmaps.' 171 | for blob in blobs: 172 | ed.finish_mmap(out[blob]) 173 | 174 | # Final step: write the README file 175 | write_readme_file([ 176 | ('cl_args', cl_args), 177 | ('data', data), 178 | ('definition', definition), 179 | ('weight', weights), 180 | ('mean', mean), 181 | ('blobs', blobs)], ed, verbose=verbose) 182 | 183 | 184 | def ensure_dir(targetdir): 185 | if not os.path.isdir(targetdir): 186 | try: 187 | os.makedirs(targetdir) 188 | except: 189 | print 'Could not create', targetdir 190 | pass 191 | 192 | def write_readme_file(args, ed, verbose): 193 | ''' 194 | Writes a README.txt that describes the settings used to geenrate the ds. 195 | ''' 196 | with codecs.open(ed.filename('README.txt'), 'w', 'utf-8') as f: 197 | def report(txt): 198 | f.write('%s\n' % txt) 199 | if verbose: 200 | print txt 201 | title = '%s network probe' % ed.basename() 202 | report('%s\n%s' % (title, '=' * len(title))) 203 | for key, val in args: 204 | if key == 'cl_args': 205 | if val is not None: 206 | report('Command-line args:') 207 | for ck, cv in vars(val).items(): 208 | report(' %s: %r' % (ck, cv)) 209 | report('%s: %r' % (key, val)) 210 | report('\ngenerated at: %s' % time.strftime("%Y-%m-%d %H:%M")) 211 | try: 212 | label = subprocess.check_output(['git', 'rev-parse', 'HEAD']) 213 | report('git label: %s' % label) 214 | except: 215 | pass 216 | 217 | if __name__ == '__main__': 218 | import sys 219 | import traceback 220 | import argparse 221 | try: 222 | import loadseg 223 | 224 | parser = argparse.ArgumentParser(description= 225 | 'Probe a pytorch network and save results in a directory.') 226 | parser.add_argument( 227 | '--directory', 228 | default='.', 229 | help='output directory for the net probe') 230 | parser.add_argument( 231 | '--blobs', 232 | nargs='*', 233 | help='network blob names to collect') 234 | parser.add_argument( 235 | '--definition', 236 | help='the deploy prototext defining the net') 237 | parser.add_argument( 238 | '--weights', 239 | default=None, 240 | help='the pretrained weight') 241 | parser.add_argument( 242 | '--mean', 243 | nargs='*', type=float, 244 | help='mean values to subtract from input') 245 | parser.add_argument( 246 | '--dataset', 247 | help='the directory containing the dataset to use') 248 | parser.add_argument( 249 | '--split', 250 | help='the split of the dataset to use') 251 | parser.add_argument( 252 | '--limit', 253 | type=int, default=None, 254 | help='limit dataset to this size') 255 | parser.add_argument( 256 | '--batch_size', 257 | type=int, default=64, 258 | help='the batch size to use') 259 | parser.add_argument( 260 | '--input_size', 261 | type=int, default=224, 262 | help='the image size input to the network(usually it is 224x224, but alexnet uses 227x227)') 263 | parser.add_argument( 264 | '--ahead', 265 | type=int, default=4, 266 | help='number of batches to prefetch') 267 | parser.add_argument( 268 | '--rotation_seed', 269 | type=int, default=None, 270 | help='the seed for the random rotation to apply') 271 | parser.add_argument( 272 | '--rotation_power', 273 | type=float, default=1.0, 274 | help='the power of hte random rotation') 275 | parser.add_argument( 276 | '--colordepth', 277 | type=int, default=3, 278 | help='set to 1 for grayscale') 279 | parser.add_argument( 280 | '--num_classes', 281 | type=int, default=365, 282 | help='the number of classes for the network output(default is 365)') 283 | 284 | args = parser.parse_args() 285 | create_probe( 286 | args.directory, args.dataset, args.definition, args.weights, 287 | numpy.array(args.mean, dtype=numpy.float32), args.blobs, 288 | batch_size=args.batch_size, ahead=args.ahead, limit=args.limit, 289 | colordepth=args.colordepth, 290 | rotation_seed=args.rotation_seed, rotation_power=args.rotation_power, 291 | split=args.split, cl_args=args, verbose=True) 292 | except: 293 | traceback.print_exc(file=sys.stdout) 294 | sys.exit(1) 295 | -------------------------------------------------------------------------------- /src/osseg.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | from unicsv import DictUnicodeReader 4 | import colorname 5 | from loadseg import AbstractSegmentation 6 | from scipy.misc import imread 7 | 8 | class OpenSurfaceSegmentation(AbstractSegmentation): 9 | def __init__(self, directory=None, supply=None): 10 | directory = os.path.expanduser(directory) 11 | self.directory = directory 12 | self.supply = supply 13 | # Process open surfaces labels: open label-name-colors.csv 14 | object_name_map = {} 15 | with open(os.path.join(directory, 'label-name-colors.csv')) as f: 16 | for row in DictUnicodeReader(f): 17 | object_name_map[row['name_name']] = int( 18 | row['green_color']) 19 | self.object_names = ['-'] * (1 + max(object_name_map.values())) 20 | for k, v in object_name_map.items(): 21 | self.object_names[v] = k 22 | # Process material labels: open label-substance-colors.csv 23 | subst_name_map = {} 24 | with open(os.path.join(directory, 'label-substance-colors.csv')) as f: 25 | for row in DictUnicodeReader(f): 26 | subst_name_map[row['substance_name']] = int( 27 | row['red_color']) 28 | self.substance_names = ['-'] * (1 + max(subst_name_map.values())) 29 | for k, v in subst_name_map.items(): 30 | self.substance_names[v] = k 31 | # Now load the metadata about images from photos.csv 32 | with open(os.path.join(directory, 'photos.csv')) as f: 33 | self.image_meta = list(DictUnicodeReader(f)) 34 | scenes = set(row['scene_category_name'] for row in self.image_meta) 35 | self.scenes = ['-'] + sorted(list(scenes)) 36 | self.scene_map = dict((s, i) for i, s in enumerate(self.scenes)) 37 | # Zero out special ~ scene names, which mean unlabeled. 38 | for k in self.scene_map: 39 | if k.startswith('~'): 40 | self.scene_map[k] = 0 41 | 42 | def all_names(self, category, j): 43 | if j == 0: 44 | return [] 45 | if category == 'color': 46 | return [colorname.color_names[j - 1] + '-c'] 47 | if category == 'scene': 48 | return [self.scenes[j].lower() + '-s'] 49 | if category == 'material': 50 | return [norm_name(n) for n in self.substance_names[j].split('/')] 51 | if category == 'object': 52 | return [norm_name(n) for n in self.object_names[j].split('/')] 53 | return [] 54 | 55 | def size(self): 56 | '''Returns the number of images in this dataset.''' 57 | return len(self.image_meta) 58 | 59 | def filename(self, i): 60 | '''Returns the filename for the nth dataset image.''' 61 | photo_id = int(self.image_meta[i]['photo_id']) 62 | return os.path.join(self.directory, 'photos', '%d.jpg' % photo_id) 63 | 64 | def metadata(self, i): 65 | '''Returns an object that can be used to create all segmentations.''' 66 | row = self.image_meta[i] 67 | return (self.directory, 68 | row, 69 | self.scene_map[row['scene_category_name']], 70 | self.supply) 71 | 72 | @classmethod 73 | def resolve_segmentation(cls, m, categories=None): 74 | directory, row, scene, supply = m 75 | photo_id = int(row['photo_id']) 76 | fnpng = os.path.join(directory, 'photos-labels', '%d.png' % photo_id) 77 | fnjpg = os.path.join(directory, 'photos', '%d.jpg' % photo_id) 78 | 79 | if wants('material', categories) or wants('object', categories): 80 | labels = imread(fnpng) 81 | # Opensurfaces has some completely unlabled images; omit them. 82 | # TODO: figure out how to do this nicely in the joiner instead. 83 | # if labels.max() == 0: 84 | # return {} 85 | result = {} 86 | if wants('scene', categories) and wants('scene', supply): 87 | result['scene'] = scene 88 | if wants('material', categories) and wants('material', supply): 89 | result['material'] = labels[:,:,0] 90 | if wants('object', categories) and wants('object', supply): 91 | result['object'] = labels[:,:,1] 92 | if wants('color', categories) and wants('color', supply): 93 | result['color'] = colorname.label_major_colors(imread(fnjpg)) + 1 94 | arrs = [a for a in result.values() if numpy.shape(a) >= 2] 95 | shape = arrs[0].shape[-2:] if arrs else (1, 1) 96 | return result, shape 97 | 98 | def norm_name(s): 99 | return s.replace(' - ', '-').replace('/', '-').strip().lower() 100 | 101 | def wants(what, option): 102 | if option is None: 103 | return True 104 | return what in option 105 | -------------------------------------------------------------------------------- /src/printmean.py: -------------------------------------------------------------------------------- 1 | import caffe 2 | import numpy as np 3 | import sys 4 | 5 | # if len(sys.argv) != 3: 6 | # print "Usage: python convert_protomean.py proto.mean out.npy" 7 | # sys.exit() 8 | 9 | blob = caffe.proto.caffe_pb2.BlobProto() 10 | data = open( sys.argv[1] , 'rb' ).read() 11 | blob.ParseFromString(data) 12 | arr = np.array( caffe.io.blobproto_to_array(blob) ) 13 | print arr, arr.shape 14 | print arr.mean(axis=(2,3)) 15 | -------------------------------------------------------------------------------- /src/quantile.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | 3 | class QuantileCounter: 4 | """ 5 | Streaming randomized quantile computation for numpy. 6 | 7 | Add any amount of data repeatedly via add(data). At any time, 8 | quantile estimates (or old-style percentiles) can be read out using 9 | quantiles(q) or percentiles(p). 10 | 11 | Accuracy scales according to resolution: the default is to 12 | set resolution to be almost-always accurate to approximately 0.1%, 13 | limiting storage to about 100,000 samples. 14 | 15 | Good for computing quantiles of huge data without using much memory. 16 | Works well on arbitrary data with probability near 1. 17 | 18 | Based on the optimal KLL quantile algorithm by Karnin, Lang, and Liberty 19 | from FOCS 2016. http://ieee-focs.org/FOCS-2016-Papers/3933a071.pdf 20 | """ 21 | 22 | def __init__(self, resolution=32 * 1024, buffersize=None, 23 | dtype=None, seed=None): 24 | self.resolution = resolution 25 | # Default buffersize: 4096 samples (and smaller than resolution). 26 | if buffersize is None: 27 | buffersize = min(512, (resolution + 3) // 4) 28 | self.buffersize = buffersize 29 | self.samplerate = 1.0 30 | self.data = [numpy.zeros(shape=resolution, dtype=dtype)] 31 | self.firstfree = [0] 32 | # The 0th buffer is left marked full always 33 | self.full = numpy.ones(shape=1, dtype='bool') 34 | self.random = numpy.random.RandomState(seed) 35 | 36 | def add(self, incoming): 37 | # Convert to a flat numpy array. 38 | incoming = numpy.ravel(incoming) 39 | if self.samplerate >= 1.0: 40 | self._add_every(incoming) 41 | return 42 | # If we are sampling, then subsample a large chunk at a time. 43 | chunksize = numpy.ceil[self.buffersize / self.samplerate] 44 | for index in xrange(0, len(incoming), chunksize): 45 | batch = incoming[index:index+chunksize] 46 | sample = batch[self.random.binomial(1, self.samplerate, len(batch))] 47 | self._add_every(sample) 48 | 49 | def _add_every(self, incoming): 50 | supplied = len(incoming) 51 | index = 0 52 | while index < supplied: 53 | ff = self.firstfree[0] 54 | available = len(self.data[0]) - ff 55 | if available == 0: 56 | if not self._shift(): 57 | # If we shifted by subsampling, then subsample. 58 | incoming = incoming[index:][self.random.binomial(1, 0.5, 59 | len(incoming - index))] 60 | index = 0 61 | supplied = len(incoming) 62 | ff = self.firstfree[0] 63 | available = len(self.data[0]) - ff 64 | copycount = min(available, supplied - index) 65 | self.data[0][ff:ff + copycount] = incoming[index:index + copycount] 66 | self.firstfree[0] += copycount 67 | index += copycount 68 | 69 | def _shift(self): 70 | index = 0 71 | while self.firstfree[index] * 2 > len(self.data[index]): 72 | if index + 1 >= len(self.data): 73 | return self._expand() 74 | data = self.data[index][0:self.firstfree[index]] 75 | data.sort() 76 | offset = self.random.binomial(1, 0.5) 77 | position = self.firstfree[index + 1] 78 | subset = data[offset::2] 79 | self.data[index + 1][position:position + len(subset)] = subset 80 | self.firstfree[index] = 0 81 | self.firstfree[index + 1] += len(subset) 82 | index += 1 83 | return True 84 | 85 | def _expand(self): 86 | cap = self._next_capacity() 87 | if cap > 0: 88 | # First, make a new layer of the proper capacity. 89 | self.data.insert(0, 90 | numpy.empty(shape=cap, dtype=self.data[-1].dtype)) 91 | self.firstfree.insert(0, 0) 92 | else: 93 | # Unless we're so big we are just subsampling. 94 | assert self.firstfree[0] == 0 95 | self.samplerate *= 0.5 96 | for index in range(1, len(self.data)): 97 | # Scan for existing data that needs to be moved down a level. 98 | amount = self.firstfree[index] 99 | if amount == 0: 100 | continue 101 | position = self.firstfree[index - 1] 102 | if (amount + position) * 2 <= len(self.data[index - 1]): 103 | # Move the data down if it would leave things half-empty. 104 | self.data[index - 1][position:position + amount] = ( 105 | self.data[index][:amount]) 106 | self.firstfree[index - 1] += amount 107 | self.firstfree[index] = 0 108 | else: 109 | # Scrunch the data if it would not. 110 | data = self.data[index][:amount] 111 | data.sort() 112 | offset = self.random.binomial(1, 0.5) 113 | scrunched = data[offset::2] 114 | self.data[index][:len(scrunched)] = scrunched 115 | self.firstfree[index] = len(scrunched) 116 | return cap > 0 117 | 118 | def _next_capacity(self): 119 | cap = numpy.ceil(self.resolution * numpy.power(0.67, len(self.data))) 120 | if cap < 2: 121 | return 0 122 | return max(self.buffersize, int(cap)) 123 | 124 | def quantiles(self, quantiles, old_style=False): 125 | size = sum(self.firstfree) 126 | weights = numpy.empty(shape=size, dtype='float32') # floating point 127 | summary = numpy.empty(shape=size, dtype=self.data[-1].dtype) 128 | index = 0 129 | for level, ff in enumerate(self.firstfree): 130 | if ff == 0: 131 | continue 132 | summary[index:index + ff] = self.data[level][:ff] 133 | weights[index:index + ff] = numpy.power(2.0, level) 134 | index += ff 135 | assert index == len(summary) 136 | order = numpy.argsort(summary) 137 | summary = summary[order] 138 | weights = weights[order] 139 | cumweights = numpy.cumsum(weights) - weights / 2 140 | if old_style: 141 | # To be convenient with numpy.percentile 142 | cumweights -= cumweights[0] 143 | cumweights /= cumweights[-1] 144 | else: 145 | cumweights /= numpy.sum(weights) 146 | return numpy.interp(quantiles, cumweights, summary) 147 | 148 | def percentiles(self, percentiles): 149 | return self.quantiles(percentiles, old_style=True) 150 | 151 | def readout(self, count, old_style=True): 152 | return self.quantiles( 153 | numpy.linspace(0.0, 1.0, count), old_style=old_style) 154 | 155 | 156 | if __name__ == '__main__': 157 | # An adverarial case: we keep finding more numbers in the middle 158 | # as the stream goes on. 159 | qc = QuantileCounter() 160 | amount = 50000000 161 | percentiles = 1000 162 | data = numpy.arange(amount) 163 | data[1::2] = data[-1::-2] + (len(data) - 1) 164 | data /= 2 165 | # data[::2] = data[-2::-2] 166 | # numpy.random.shuffle(data) 167 | qc.add(data) 168 | ro = qc.readout(1001) 169 | # print ro - numpy.linspace(0, amount, percentiles+1) 170 | print "Maximum relative deviation among %d perentiles:" % percentiles, max( 171 | abs(ro - numpy.linspace(0, amount, percentiles+1)) 172 | / amount) * percentiles 173 | 174 | -------------------------------------------------------------------------------- /src/quantprobe.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy 3 | import re 4 | import shutil 5 | import time 6 | import expdir 7 | import sys 8 | import vecquantile 9 | 10 | def quant_probe(directory, blob, quantiles=None, batch_size=None): 11 | ''' 12 | Adds a [blob]-sort directory to a probe such as conv5-sort, 13 | where the directory contains [blob]-[unit].mmap files for 14 | each unit such as conv5-0143.mmap, where the contents are 15 | sorted. Also creates a [blob]-quant-[num].mmap file 16 | such as conv5-quantile-1000.mmap, where the higest seen 17 | value for each quantile of each unit is summarized in a 18 | single (units, quantiles) matrix. 19 | ''' 20 | # Make sure we have a directory to work in 21 | ed = expdir.ExperimentDirectory(directory) 22 | 23 | # If it's already computed, then skip it!!! 24 | print "Checking", ed.mmap_filename(blob=blob, part='quant-%d' % quantiles) 25 | if ed.has_mmap(blob=blob, part='quant-%d' % quantiles): 26 | print "Already have %s-quant-%d.mmap, skipping." % (blob, quantiles) 27 | return 28 | 29 | # Read about the blob shape 30 | blob_info = ed.load_info(blob=blob) 31 | shape = blob_info.shape 32 | 33 | print 'Computing quantiles for %s shape %r' % (blob, shape) 34 | data = ed.open_mmap(blob=blob, shape=shape) 35 | qmap = ed.open_mmap(blob=blob, part='quant-%d' % quantiles, 36 | mode='w+', shape=(data.shape[1], quantiles)) 37 | linmap = ed.open_mmap(blob=blob, part='minmax', 38 | mode='w+', shape=(data.shape[1], 2)) 39 | 40 | # Automatic batch size selection: 64mb batches 41 | if batch_size is None: 42 | batch_size = max(1, int(16 * 1024 * 1024 / numpy.prod(shape[1:]))) 43 | # Algorithm: one pass over the data with a quantile counter for each unit 44 | quant = vecquantile.QuantileVector(depth=data.shape[1], seed=1) 45 | start_time = time.time() 46 | last_batch_time = start_time 47 | for i in range(0, data.shape[0], batch_size): 48 | batch_time = time.time() 49 | rate = i / (batch_time - start_time + 1e-15) 50 | batch_rate = batch_size / (batch_time - last_batch_time + 1e-15) 51 | last_batch_time = batch_time 52 | print 'Processing %s index %d: %f %f' % (blob, i, rate, batch_rate) 53 | sys.stdout.flush() 54 | batch = data[i:i+batch_size] 55 | batch = numpy.transpose(batch,axes=(0,2,3,1)).reshape(-1, data.shape[1]) 56 | quant.add(batch) 57 | print 'Writing quantiles' 58 | sys.stdout.flush() 59 | # Reverse the quantiles, largest first. 60 | qmap[...] = quant.readout(quantiles)[:,::-1] 61 | linmap[...] = quant.minmax() 62 | 63 | if qmap is not None: 64 | ed.finish_mmap(qmap) 65 | if linmap is not None: 66 | ed.finish_mmap(linmap) 67 | 68 | 69 | if __name__ == '__main__': 70 | import argparse 71 | import sys 72 | import traceback 73 | 74 | try: 75 | import loadseg 76 | 77 | parser = argparse.ArgumentParser( 78 | description='Generate sorted files for probed activation data.') 79 | parser.add_argument( 80 | '--directory', 81 | default='.', 82 | help='output directory for the net probe') 83 | parser.add_argument( 84 | '--blobs', 85 | nargs='*', 86 | help='network blob names to sort') 87 | parser.add_argument( 88 | '--batch_size', 89 | type=int, default=None, 90 | help='the batch size to use') 91 | parser.add_argument( 92 | '--quantiles', 93 | type=int, default=1000, 94 | help='the number of quantiles to summarize') 95 | args = parser.parse_args() 96 | for blob in args.blobs: 97 | quant_probe(args.directory, blob, args.quantiles, args.batch_size) 98 | except: 99 | traceback.print_exc(file=sys.stdout) 100 | sys.exit(1) 101 | -------------------------------------------------------------------------------- /src/rotate.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | import scipy.linalg 3 | from tempfile import NamedTemporaryFile 4 | 5 | # Generates a random orthogonal matrix correctly. 6 | # See http://www.ams.org/notices/200511/what-is.pdf 7 | # Also http://home.lu.lv/~sd20008/papers/essays/Random%20unitary%20[paper].pdf 8 | # Equation 35 9 | 10 | def randomRotation(n, seed=None): 11 | if seed is None: 12 | NR = numpy.random.normal(size=(n, n)) 13 | else: 14 | NR = numpy.random.RandomState(seed).normal(size=(n, n)) 15 | Q, R = numpy.linalg.qr(NR) 16 | # Eliminate negative diagonals 17 | D = numpy.diagonal(R) 18 | L = numpy.diagflat(D / abs(D)) 19 | # The replacement R would be LR and have a positive diagonal. 20 | # This allows det(QL) to be random sign whereas Householder Q is fixed. 21 | result = numpy.dot(Q, L) 22 | # Now negate the first column if this result is a reflection 23 | if numpy.linalg.det(result) < 0: 24 | result[0] = -result[0] 25 | return result 26 | 27 | def randomRotationPowers(n, powers, seed=None, unpermute=False): 28 | RR = randomRotation(n, seed) 29 | # Reduce the matrix to canonical (block-diag) form using schur decomp 30 | # (TODO: Consider implementing the stabilized variant of the algorithm 31 | # by Gragg called UHQR which is tailored to solve this decomposition for 32 | # unitary RR - or 'OHQR' by Ammar for the real orthogonal case.) 33 | T, W = scipy.linalg.schur(RR, output='real') 34 | # Now T is not exactly block diagonal due to numerical approximations. 35 | # However, its eigenvalues should have the right distribution: let's 36 | # read the angles off the diagonal and use them to construct a perfectly 37 | # block diagonal form. 38 | RA = numpy.arccos(numpy.clip(numpy.diag(T), -1, 1))[0:n:2] 39 | if unpermute: 40 | # If unpermuting, first figure out full rotation matrix 41 | B = [numpy.cos([[a, a + numpy.pi/2], [a - numpy.pi/2, a]]) for a in RA] 42 | BD = scipy.linalg.block_diag(*B) 43 | fullrot = numpy.dot(numpy.dot(W, BD), W.transpose()) 44 | # Then sort the rows by max value 45 | biggest_first = numpy.argsort(-numpy.amax(fullrot, axis=1)) 46 | # In greedy order, assign permutation order 47 | cols_seen = set() 48 | permutation = numpy.zeros((n, n)) 49 | for row in biggest_first: 50 | # For each row, find the max col to which it is closest 51 | biggest_cols = numpy.argsort(-fullrot[row]) 52 | for col in biggest_cols: 53 | if col not in cols_seen: 54 | # Bring the row back to that slot if not taken 55 | permutation[col,row] = 1 56 | cols_seen.add(col) 57 | break 58 | else: 59 | permutation = numpy.eye(n) 60 | # Now form the requested powers of W * (RA^p) * W' 61 | result = [] 62 | for p in powers: 63 | A = [a * p for a in RA] 64 | B = [numpy.cos([[a, a + numpy.pi/2], [a - numpy.pi/2, a]]) for a in A] 65 | BD = scipy.linalg.block_diag(*B) 66 | result.append(numpy.dot(permutation, 67 | numpy.dot(numpy.dot(W, BD), W.transpose()))) 68 | return result 69 | 70 | def randomNearIdentity(n, scale, seed=None): 71 | if numpy.isinf(scale): 72 | return randomOrthogonal(n, seed) 73 | sigma = scale / numpy.sqrt(2 * n) 74 | if seed is None: 75 | RR = numpy.random.normal(scale=sigma, size=(n, n)) 76 | else: 77 | RR = numpy.random.RandomState(seed).normal(scale=sigma, size=(n, n)) 78 | U, _, V = numpy.linalg.svd(numpy.eye(n) + RR) 79 | return numpy.dot(U, V) 80 | 81 | def deviation(rotation): 82 | return numpy.linalg.norm(rotation - numpy.eye(len(rotation)), 2) 83 | 84 | def temparray_like(arr): 85 | fs = NamedTemporaryFile() 86 | result = numpy.memmap(fs.name, dtype=arr.dtype, mode='w+', 87 | shape=arr.shape) 88 | fs.close() 89 | return result 90 | 91 | def dumpinfo(fn, arr, limit=5): 92 | text = [] 93 | for row in range(len(arr)): 94 | line = [] 95 | for col in numpy.argsort(-arr[row])[:limit]: 96 | line.append('%.2fa%d' % (arr[row,col], col+1)) 97 | text.append(('s%d = ' % (row+1)) + ' + '.join(line) + '...\n') 98 | with open(fn, 'w') as f: 99 | f.writelines(text) 100 | 101 | if __name__ == '__main__': 102 | from scipy.io import loadmat, savemat 103 | import os 104 | directory = 'randrot' 105 | useed = 1000 106 | try: 107 | os.makedirs(directory) 108 | except: 109 | pass 110 | dims = 256 111 | seed = 1 112 | alpha = numpy.arange(0.1, 1.0 + 1e-15, 0.1) 113 | rots = randomRotationPowers(dims, alpha, seed=seed, unpermute=True) 114 | for rot, a in zip(rots, alpha): 115 | fn = 'rotpow-%d-%.1f-%.2f-p%d.mat' % (dims, a, deviation(rot), seed) 116 | savemat(os.path.join(directory, fn), {'rotation': rot}) 117 | fn = 'rotpow-%d-%.1f-%.2f-p%d.info' % (dims, a, deviation(rot), seed) 118 | dumpinfo(os.path.join(directory, fn), rot) 119 | 120 | # for dims in [128, 256, 384, 512]: 121 | # for sigma in list(numpy.arange(0.2, 2.0, 0.2) 122 | # ) + list(numpy.arange(2.0, 4.0, 0.4)) + [float('inf')]: 123 | # r1 = randomNearIdentity(dims, sigma, seed=1) 124 | # fn = 'randrot-%d-%.2f-%.1f-1.mat' % (dims, deviation(r1), sigma) 125 | # print fn 126 | # savemat(os.path.join(directory, fn), {'rotation': r1}) 127 | # r0 = randomNearIdentity(dims, sigma, seed=useed) 128 | # useed += 1 129 | # fn = 'randrot-%d-%.2f-%.1f-0.mat' % (dims, deviation(r1), sigma) 130 | # print fn 131 | # savemat(os.path.join(directory, fn), {'rotation': r1}) 132 | -------------------------------------------------------------------------------- /src/synonym.py: -------------------------------------------------------------------------------- 1 | '''Contains a list of synonyms to normalize out of ADE20K data.''' 2 | 3 | def synonyms(words): 4 | ''' 5 | Transforms a list of words to a better list of synonyms. 6 | Preferred names are listed first, and 7 | ''' 8 | # Common case: no common or confusing words 9 | if not any(w in confusing_names or w in common_names for w in words): 10 | return words 11 | # The list needs to be changed. First, accumulate better names. 12 | full_list = [] 13 | for word in words: 14 | full_list.extend(common_names.get(word, [])) 15 | full_list.extend(words) 16 | # Now accumulate confusing names to omit. 17 | omit = set() 18 | for word in full_list: 19 | if word in confusing_names: 20 | omit.update(confusing_names[word]) 21 | # Now produce a filtered list 22 | seen = set() 23 | see = seen.add 24 | return [w for w in full_list if not (w in omit or w in seen or see(w))] 25 | 26 | # These potential confusing names are drawn from aliases created by 27 | # joining ade20k data with opensurfaces. Keys are more specific names 28 | # that should not be aliased with more generic names in values. 29 | 30 | # Read this list as, { 'flowerpot': ['pot'] } "A flowerpot should not be 31 | # called a pot, because that might confuse it with other pots." 32 | 33 | # 29 confusing synonyms to avoid 34 | confusing_names ={ 35 | 'flowerpot': ['pot'], 36 | 'toilet': ['throne', 'can'], 37 | 'curtain': ['mantle'], 38 | 'fabric': ['material'], 39 | 'file cabinet': ['file'], 40 | 'chest of drawers': ['chest'], 41 | 'fire hydrant': ['plug'], 42 | 'car': ['machine'], 43 | 'closet': ['press'], 44 | 'bicycle': ['wheel'], 45 | 'brochure': ['folder'], 46 | 'filing cabinet': ['file'], 47 | 'paper': ['tissue'], # Opensurfaces groups these materials; call it paper. 48 | 'exhaust hood': ['hood'], 49 | 'blanket': ['cover'], 50 | 'carapace': ['shield'], 51 | 'cellular phone': ['cell'], 52 | 'handbag': ['bag'], 53 | 'land': ['soil'], 54 | 'sidewalk': ['pavement'], 55 | 'poster': ['card', 'bill'], 56 | 'paper towel': ['tissue'], 57 | 'computer mouse': ['mouse'], 58 | 'steering wheel': ['wheel'], 59 | 'lighthouse': ['beacon'], 60 | 'basketball hoop': ['hoop'], 61 | 'bus': ['passenger vehicle'], 62 | 'head': ['caput'], 63 | # Do not use left/right/person to qualify body parts 64 | 'arm': ['left arm', 'right arm', 'person arm'], 65 | 'foot': ['left foot', 'right foot', 'person foot'], 66 | 'hand': ['left hand', 'right hand', 'person hand'], 67 | 'leg': ['left leg', 'right leg', 'person leg'], 68 | 'shoulder': ['left shoulder', 'right shoulder', 'person shoulder'], 69 | 'torso': ['person torso'], 70 | 'head': ['person head'], 71 | 'hair': ['person hair'], 72 | 'nose': ['person nose'], 73 | 'ear': ['person ear'], 74 | 'neck': ['person neck'], 75 | 'eye': ['person eye'], 76 | 'eyebrow': ['person eyebrow'] 77 | } 78 | 79 | # These potential synonyms are drawn from the raw ADE20K data, with 80 | # shorter and more common names listed first. I have manually uncommented 81 | # word pairs that seem unambiguously the same, where the most common uses 82 | # of the second word would allow the first word to be substituted without 83 | # changing meaning. 84 | 85 | common_names = { 86 | # We do not distinguish between left+right parts for our purposes. 87 | 'left arm': ['arm'], 88 | 'right arm': ['arm'], 89 | 'left foot': ['foot'], 90 | 'right foot': ['foot'], 91 | 'left hand': ['hand'], 92 | 'right hand': ['hand'], 93 | 'left leg': ['leg'], 94 | 'right leg': ['leg'], 95 | 'left shoulder': ['shoulder'], 96 | 'right shoulder': ['shoulder'], 97 | # And we assume that human parts do not need to say 'person' 98 | 'person torso': ['torso'], 99 | 'person head': ['head'], 100 | 'person arm': ['arm'], 101 | 'person hand': ['hand'], 102 | 'person hair': ['hair'], 103 | 'person nose': ['nose'], 104 | 'person leg': ['leg'], 105 | 'person mouth': ['mouth'], 106 | 'person ear': ['ear'], 107 | 'person neck': ['neck'], 108 | 'person eye': ['eye'], 109 | 'person eyebrow': ['eyebrow'], 110 | 'person foot': ['foot'], 111 | 112 | # why is this word airport-airport? 113 | 'airport-airport-s': ['airport-s'], 114 | # This is the preferred spelling for us. 115 | 'aeroplane': ['airplane'], 116 | 'airplane': ['airplane', 'aeroplane', 'plane'], 117 | 'spectacles': ['eyeglasses'], 118 | 'windopane': ['window'], # ade20k calls windows windowpanes. 119 | 'dog': ['dog', 'domestic dog', 'canis familiaris'], 120 | 'alga': ['algae'], 121 | 'bicycle': ['bicycle', 'bike', 'cycle'], 122 | 'food': ['food', 'solid food'], 123 | 'caput': ['head'], 124 | 'route': ['road'], 125 | 'fencing': ['fence'], 126 | 'flooring': ['floor'], 127 | 'carpet': ['carpet', 'carpeting'], 128 | 'shrub': ['bush'], 129 | 'armour': ['armor'], 130 | 'pail': ['bucket'], 131 | 'spigot': ['faucet'], 132 | 'faucet': ['faucet', 'spigot'], 133 | 'crt screen': ['screen'], 134 | 'cistern': ['water tank'], 135 | 'video display': ['display'], 136 | 'lift': ['elevator'], 137 | 'hydroplane': ['seaplane'], 138 | 'microwave oven': ['microwave'], 139 | 'falls': ['waterfall'], 140 | 'mike': ['microphone'], 141 | 'windscreen': ['windshield'], 142 | 'fluorescent fixture': ['fluorescent'], 143 | 'water vapour': ['water vapor'], 144 | 'numberplate': ['license plate'], 145 | 'tin can': ['can'], 146 | 'cow': ['cow', 'moo-cow'], 147 | 'horse': ['horse', 'equus caballus'], 148 | 'kerb': ['curb'], 149 | 'filing cabinet': ['file cabinet'], 150 | 'electrical switch': ['switch'], 151 | 'telephone set': ['telephone'], 152 | 'totaliser': ['adding machine'], 153 | 'television receiver': ['television'], 154 | 'fabric': ['fabric', 'cloth', 'textile'], 155 | 'textile': ['fabric'], 156 | 'attack aircraft carrier': ['aircraft carrier'], 157 | 'cooking stove': ['stove'], 158 | 'electric-light bulb': ['light bulb'], 159 | } 160 | -------------------------------------------------------------------------------- /src/unicsv.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | ''' 3 | File: unicsv.py 4 | Author: Spencer Rathbun 5 | Date: 05/30/2012 6 | Description: Unicode wrapper classes around csv reader/writers 7 | ''' 8 | import csv, codecs, cStringIO 9 | 10 | class UTF8Recoder: 11 | """ 12 | Iterator that reads an encoded stream and reencodes the input to UTF-8 13 | """ 14 | def __init__(self, f, encoding): 15 | self.reader = codecs.getreader(encoding)(f) 16 | 17 | def __iter__(self): 18 | return self 19 | 20 | def next(self): 21 | return self.reader.next().encode("utf-8") 22 | 23 | class UnicodeReader: 24 | """ 25 | A CSV reader which will iterate over lines in the CSV file "f", 26 | which is encoded in the given encoding. 27 | """ 28 | 29 | def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds): 30 | f = UTF8Recoder(f, encoding) 31 | self.reader = csv.reader(f, dialect=dialect, **kwds) 32 | 33 | def next(self): 34 | row = self.reader.next() 35 | return [unicode(s, "utf-8") for s in row] 36 | 37 | def __iter__(self): 38 | return self 39 | 40 | class DictUnicodeReader: 41 | """ 42 | A dict-based CSV reader which will iterate over lines in the CSV 43 | file "f", which is encoded in the given encoding. 44 | """ 45 | 46 | def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds): 47 | f = UTF8Recoder(f, encoding) 48 | self.reader = csv.DictReader(f, dialect=dialect, **kwds) 49 | 50 | def next(self): 51 | row = self.reader.next() 52 | outrow = dict([(k,unicode(v, "utf-8")) for k,v in row.iteritems()]) 53 | return outrow 54 | 55 | def __iter__(self): 56 | return self 57 | 58 | class UnicodeWriter: 59 | """ 60 | A CSV writer which will write rows to CSV file "f", 61 | which is encoded in the given encoding. 62 | """ 63 | 64 | def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds): 65 | # Redirect output to a queue 66 | self.queue = cStringIO.StringIO() 67 | self.writer = csv.writer(self.queue, dialect=dialect, **kwds) 68 | self.stream = f 69 | self.encoder = codecs.getincrementalencoder(encoding)() 70 | 71 | def writerow(self, row): 72 | self.writer.writerow([unicode(s).encode("utf-8") for s in row]) 73 | # Fetch UTF-8 output from the queue ... 74 | data = self.queue.getvalue() 75 | data = data.decode("utf-8") 76 | # ... and reencode it into the target encoding 77 | data = self.encoder.encode(data) 78 | # write to the target stream 79 | self.stream.write(data) 80 | # empty queue 81 | self.queue.truncate(0) 82 | 83 | def writerows(self, rows): 84 | for row in rows: 85 | self.writerow(row) 86 | 87 | class DictUnicodeWriter(object): 88 | """ 89 | A dict-based CSV writer which will write rows to CSV file "f", 90 | which is encoded in the given encoding. 91 | """ 92 | 93 | def __init__(self, f, fieldnames, dialect=csv.excel, 94 | encoding="utf-8", **kwds): 95 | # Redirect output to a queue 96 | self.queue = cStringIO.StringIO() 97 | self.writer = csv.DictWriter(self.queue, fieldnames, dialect=dialect, **kwds) 98 | self.stream = f 99 | self.encoder = codecs.getincrementalencoder(encoding)() 100 | 101 | def writerow(self, D): 102 | self.writer.writerow( 103 | {k:unicode(v).encode("utf-8") for k,v in D.items()}) 104 | 105 | # Fetch UTF-8 output from the queue ... 106 | data = self.queue.getvalue() 107 | data = data.decode("utf-8") 108 | # ... and reencode it into the target encoding 109 | data = self.encoder.encode(data) 110 | # write to the target stream 111 | self.stream.write(data) 112 | # empty queue 113 | self.queue.truncate(0) 114 | 115 | def writerows(self, rows): 116 | for D in rows: 117 | self.writerow(D) 118 | 119 | def writeheader(self): 120 | self.writer.writeheader() 121 | 122 | # Example code: 123 | ''' 124 | D1 = {'name':u'马克','pinyin':u'Mǎkè'} 125 | D2 = {'name':u'美国','pinyin':u'Měiguó'} 126 | f = open('out.csv','wb') 127 | f.write(u'\ufeff'.encode('utf8')) 128 | w = csv.DictUnicodeWriter(f,sorted(D1.keys())) 129 | w.writeheader() 130 | w.writerows([D1,D2]) 131 | f.close() 132 | ''' 133 | -------------------------------------------------------------------------------- /src/upsample.py: -------------------------------------------------------------------------------- 1 | from scipy.ndimage.filters import gaussian_filter 2 | from scipy.interpolate import RectBivariateSpline 3 | from scipy.ndimage.interpolation import zoom 4 | import numpy 5 | 6 | def upsampleL(fieldmap, activation_data, reduction=1, shape=None, 7 | scaleshape=None, out=None): 8 | ''' 9 | Applies a bilinear upsampling. 10 | ''' 11 | offset, size, step = fieldmap 12 | input_count = activation_data.shape[0] 13 | ay, ax = centered_arange(fieldmap, activation_data.shape[1:], reduction) 14 | if shape is None: 15 | shape = upsampled_shape( 16 | fieldmap, activation_data.shape[1:], reduction) 17 | if scaleshape is not None: 18 | iy, ix = full_arange(scaleshape) 19 | # TODO: consider treaing each point as a center of a pixel 20 | iy *= shape[0] / scaleshape[0] 21 | ix *= shape[1] / scaleshape[1] 22 | else: 23 | iy, ix = full_arange(shape) 24 | if out is None: 25 | out = numpy.empty((input_count, len(iy), len(ix)), 26 | dtype=activation_data.dtype) 27 | for z in range(input_count): 28 | f = RectBivariateSpline(ay, ax, activation_data[z], kx=1, ky=1) 29 | out[z] = f(iy, ix, grid=True) 30 | return out 31 | 32 | def upsampleC(fieldmap, activation_data, shape=None, out=None): 33 | ''' 34 | Applies a bicubic upsampling. 35 | ''' 36 | offset, size, step = fieldmap 37 | input_count = activation_data.shape[0] 38 | ay, ax = centered_arange(fieldmap, activation_data.shape[1:]) 39 | if shape is None: 40 | shape = upsampled_shape(fieldmap, activation_data.shape[1:]) 41 | iy, ix = full_arange(shape) 42 | if out is None: 43 | out = numpy.empty((input_count,) + shape, 44 | dtype=activation_data.dtype) 45 | for z in range(input_count): 46 | f = RectBivariateSpline(ay, ax, activation_data[z], kx=3, ky=3) 47 | out[z] = f(iy, ix, grid=True) 48 | return out 49 | 50 | def upsampleG(fieldmap, activation_data, shape=None): 51 | ''' 52 | Upsampling utility functions 53 | ''' 54 | offset, size, step = fieldmap 55 | input_count = activation_data.shape[0] 56 | if shape is None: 57 | shape = upsampled_shape(fieldmap, activation_data.shape[1:]) 58 | activations = numpy.zeros((input_count,) + shape) 59 | activations[(slice(None),) + 60 | centered_slice(fieldmap, activation_data.shape[1:])] = ( 61 | activation_data * numpy.prod(step)) 62 | blurred = gaussian_filter( 63 | activations, 64 | sigma=(0, ) + tuple(t // 1.414 for o, s, t in zip(*fieldmap)), 65 | mode='constant') 66 | return blurred 67 | 68 | def topo_sort(layers): 69 | # First, build a links-from and also a links-to graph 70 | links_from = {} 71 | links_to = {} 72 | for layer in layers: 73 | for bot in layer.bottom: 74 | if bot not in links_from: 75 | links_from[bot] = [] 76 | links_from[bot].append(layer) 77 | for top in layer.top: 78 | if top not in links_to: 79 | links_to[top] = [] 80 | links_to[top].append(layer) 81 | # Now do a DFS to figure out the ordering (using links-from) 82 | visited = set() 83 | ordering = [] 84 | stack = [] 85 | for seed in links_from: 86 | if seed not in visited: 87 | stack.append((seed, True)) 88 | stack.append((seed, False)) 89 | visited.add(seed) 90 | while stack: 91 | (blob, completed) = stack.pop() 92 | if completed: 93 | ordering.append(blob) 94 | elif blob in links_from: 95 | for layer in links_from[blob]: 96 | for t in layer.top: 97 | if t not in visited: 98 | stack.append((t, True)) 99 | stack.append((t, False)) 100 | visited.add(t) 101 | # Return a result in front-to-back order, with incoming links for each 102 | return list((blob, links_to[blob] if blob in links_to else []) 103 | for blob in reversed(ordering)) 104 | 105 | def composed_fieldmap(layers, end): 106 | ts = topo_sort(layers) 107 | fm_record = {} 108 | for blob, layers in ts: 109 | # Compute fm's on all the edges that go to this blob. 110 | all_fms = [ 111 | (compose_fieldmap(fm_record[bot][0], layer_fieldmap(layer)), 112 | fm_record[bot][1] + [(bot, layer)]) 113 | for layer in layers for bot in layer.bottom if bot != blob] 114 | # And take the max fieldmap. 115 | fm_record[blob] = max_fieldmap(all_fms) 116 | if blob == end: 117 | return fm_record[blob] 118 | 119 | def max_fieldmap(maps): 120 | biggest, bp = None, None 121 | for fm, path in maps: 122 | if biggest is None: 123 | biggest, bp = fm, path 124 | elif fm[1][0] > biggest[1][0]: 125 | biggest, bp = fm, path 126 | # When there is no biggest, for example when maps is the empty array, 127 | # use the trivial identity fieldmap with no path. 128 | if biggest is None: 129 | return ((0, 0), (1, 1), (1, 1)), [] 130 | return biggest, bp 131 | 132 | def shortest_layer_path(start, end, layers): 133 | # First, build a blob-to-outgoing-layer graph 134 | links_from = {} 135 | for layer in layers: 136 | for bot in layer.bottom: 137 | if bot not in links_from: 138 | links_from[bot] = [] 139 | links_from[bot].append(layer) 140 | # Then do a BFS on the graph to find the shortest path to 'end' 141 | queue = [(s, []) for s in start] 142 | visited = set(start) 143 | while queue: 144 | (blob, path) = queue.pop(0) 145 | for layer in links_from[blob]: 146 | for t in layer.top: 147 | if t == end: 148 | return path + [layer] 149 | if t not in visited: 150 | queue.append((t, path + [layer])) 151 | visited.add(t) 152 | return None 153 | 154 | def upsampled_shape(fieldmap, shape, reduction=1): 155 | # Given the shape of a layer's activation and a fieldmap describing 156 | # the transformation to original image space, returns the shape of 157 | # the input size 158 | return tuple(((w - 1) * t + s + 2 * o) // reduction 159 | for (o, s, t), w in zip(zip(*fieldmap), shape)) 160 | 161 | def make_mask_set(image_shape, fieldmap, activation_data, 162 | output=None, sigma=0.1, threshold=0.5, percentile=None): 163 | """Creates a set of receptive field masks with uniform thresholds 164 | over a range of inputs. 165 | """ 166 | offset, shape, step = fieldmap 167 | input_count = activation_data.shape[0] 168 | activations = numpy.zeros((input_count,) + image_shape) 169 | activations[(slice(None),) + 170 | centered_slice(fieldmap, activation_data.shape[1:])] = ( 171 | activation_data) 172 | blurred = gaussian_filter( 173 | activations, 174 | sigma=(0, ) + tuple(s * sigma for s in shape), 175 | mode='constant') 176 | if percentile is not None: 177 | limit = blurred.ravel().percentile(percentile) 178 | return blurred > limit 179 | else: 180 | maximum = blurred.ravel().max() 181 | return (blurred > maximum * threshold) 182 | 183 | def safezoom(array, ratio, output=None, order=0): 184 | '''Like numpy.zoom, but does not crash when the first dimension 185 | of the array is of size 1, as happens often with segmentations''' 186 | dtype = array.dtype 187 | if array.dtype == numpy.float16: 188 | array = array.astype(numpy.float32) 189 | if array.shape[0] == 1: 190 | if output is not None: 191 | output = output[0,...] 192 | result = zoom(array[0,...], ratio[1:], 193 | output=output, order=order) 194 | if output is None: 195 | output = result[numpy.newaxis] 196 | else: 197 | result = zoom(array, ratio, output=output, order=order) 198 | if output is None: 199 | output = result 200 | return output.astype(dtype) 201 | 202 | def receptive_field(location, fieldmap): 203 | """Computes the receptive field of a specific location. 204 | 205 | Parameters 206 | ---------- 207 | location: tuple 208 | The x-y position of the unit being queried. 209 | fieldmap: 210 | The (offset, size, step) tuple fieldmap representing the 211 | receptive field map for the layer being queried. 212 | """ 213 | return compose_fieldmap(fieldmap, (location, (1, 1), (1, 1)))[:2] 214 | 215 | 216 | def proto_getattr(p, a, d): 217 | hf = True 218 | # Try using HasField to detect the presence of a field; 219 | # if there is no HasField, then just use getattr. 220 | try: 221 | hf = p.HasField(a) 222 | except: 223 | pass 224 | if hf: 225 | return getattr(p, a, d) 226 | return d 227 | 228 | def wh_attr(layer, attrname, default=0, minval=0): 229 | if not hasattr(default, '__len__'): 230 | default = (default, default) 231 | val = proto_getattr(layer, attrname, None) 232 | if val is None or val == []: 233 | h = max(minval, getattr(layer, attrname + '_h', default[0])) 234 | w = max(minval, getattr(layer, attrname + '_w', default[1])) 235 | elif hasattr(val, '__len__'): 236 | h = val[0] 237 | w = val[1] if len(val) >= 2 else h 238 | else: 239 | h = val 240 | w = val 241 | return (h, w) 242 | 243 | def layer_fieldmap(layer): 244 | # Only convolutional and pooling layers affect geometry. 245 | if layer.type == 'Convolution' or layer.type == 'Pooling': 246 | if layer.type == 'Pooling': 247 | config = layer.pooling_param 248 | if config.global_pooling: 249 | return ((0, 0), (None, None), (1, 1)) 250 | else: 251 | config = layer.convolution_param 252 | size = wh_attr(config, 'kernel_size', wh_attr(config, 'kernel', 1)) 253 | stride = wh_attr(config, 'stride', 1, minval=1) 254 | padding = wh_attr(config, 'pad', 0) 255 | neg_padding = tuple((-x) for x in padding) 256 | return (neg_padding, size, stride) 257 | # All other layers just pass through geometry unchanged. 258 | return ((0, 0), (1, 1), (1, 1)) 259 | 260 | def layerarray_fieldmap(layerarray): 261 | fieldmap = ((0, 0), (1, 1), (1, 1)) 262 | for layer in layerarray: 263 | fieldmap = compose_fieldmap(fieldmap, layer_fieldmap(layer)) 264 | return fieldmap 265 | 266 | # rf1 is the lower layer, rf2 is the higher layer 267 | def compose_fieldmap(rf1, rf2): 268 | """Composes two stacked fieldmap maps. 269 | 270 | Field maps are represented as triples of (offset, size, step), 271 | where each is an (x, y) pair. 272 | 273 | To find the pixel range corresponding to output pixel (x, y), just 274 | do the following: 275 | start_x = x * step[0] + offset[1] 276 | limit_x = start_x + size[0] 277 | start_y = y * step[1] + offset[1] 278 | limit_y = start_y + size[1] 279 | 280 | Parameters 281 | ---------- 282 | rf1: tuple 283 | The lower-layer receptive fieldmap, a tuple of (offset, size, step). 284 | rf2: tuple 285 | The higher-layer receptive fieldmap, a tuple of (offset, size, step). 286 | """ 287 | if rf1 == None: 288 | import pdb; pdb.set_trace() 289 | offset1, size1, step1 = rf1 290 | offset2, size2, step2 = rf2 291 | 292 | size = tuple((size2c - 1) * step1c + size1c 293 | for size1c, step1c, size2c in zip(size1, step1, size2)) 294 | offset = tuple(offset2c * step1c + offset1c 295 | for offset2c, step1c, offset1c in zip(offset2, step1, offset1)) 296 | step = tuple(step2c * step1c 297 | for step1c, step2c in zip(step1, step2)) 298 | return (offset, size, step) 299 | 300 | def _cropped_slices(offset, size, limit): 301 | corner = 0 302 | if offset < 0: 303 | size += offset 304 | offset = 0 305 | if limit - offset < size: 306 | corner = limit - offset 307 | size -= corner 308 | return (slice(corner, corner + size), slice(offset, offset + size)) 309 | 310 | def crop_field(image_data, fieldmap, location): 311 | """Crops image_data to the specified receptive field. 312 | 313 | Together fieldmap and location specify a receptive field on the image, 314 | which may overlap the edge. This returns a crop to that shape, including 315 | any zero padding necessary to fill out the shape beyond the image edge. 316 | """ 317 | offset, size = receptive_field(fieldmap, location) 318 | return crop_rectangle(image_data, offset, size) 319 | 320 | def crop_rectangle(image_data, offset, size): 321 | coloraxis = 0 if image_data.size <= 2 else 1 322 | allcolors = () if not coloraxis else (slice(None),) * coloraxis 323 | colordepth = () if not coloraxis else (image_data.size[0], ) 324 | result = numpy.zeros(colordepth + size) 325 | (xto, xfrom), (yto, yfrom) = (_cropped_slices( 326 | o, s, l) for o, s, l in zip(offset, size, image_data.size[coloraxis:])) 327 | result[allcolors + (xto, yto)] = image_data[allcolors + (xfrom, yfrom)] 328 | return result 329 | 330 | def center_location(fieldmap, location): 331 | if isinstance(location, numpy.ndarray): 332 | offset, size, step = fieldmap 333 | broadcast = (numpy.newaxis, ) * (len(location.shape) - 1) + ( 334 | slice(None),) 335 | step = numpy.array(step)[broadcast] 336 | offset = numpy.array(offset)[broadcast] 337 | size = numpy.array(size)[broadcast] 338 | return location * step + offset + size // 2 339 | else: 340 | offset, shape = receptive_field(location, fieldmap) 341 | return tuple(o + s // 2 for o, s in zip(offset, shape)) 342 | 343 | def centered_slice(fieldmap, activation_shape, reduction=1): 344 | offset, size, step = fieldmap 345 | r = reduction 346 | return tuple(slice((s // 2 + o) // r, (s // 2 + o + a * t) // r, t // r) 347 | for o, s, t, a in zip(offset, size, step, activation_shape)) 348 | 349 | def centered_arange(fieldmap, activation_shape, reduction=1): 350 | offset, size, step = fieldmap 351 | r = reduction 352 | return tuple(numpy.arange( 353 | (s // 2 + o) // r, (s // 2 + o + a * t) // r, t // r)[:a] # Hack to avoid a+1 points 354 | for o, s, t, a in zip(offset, size, step, activation_shape)) 355 | 356 | def full_arange(output_shape): 357 | return tuple(numpy.arange(o) for o in output_shape) 358 | 359 | -------------------------------------------------------------------------------- /src/vecquantile.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | 3 | class QuantileVector: 4 | """ 5 | Streaming randomized quantile computation for numpy. 6 | 7 | Add any amount of data repeatedly via add(data). At any time, 8 | quantile estimates (or old-style percentiles) can be read out using 9 | quantiles(q) or percentiles(p). 10 | 11 | Accuracy scales according to resolution: the default is to 12 | set resolution to be accurate to better than 0.1%, 13 | while limiting storage to about 50,000 samples. 14 | 15 | Good for computing quantiles of huge data without using much memory. 16 | Works well on arbitrary data with probability near 1. 17 | 18 | Based on the optimal KLL quantile algorithm by Karnin, Lang, and Liberty 19 | from FOCS 2016. http://ieee-focs.org/FOCS-2016-Papers/3933a071.pdf 20 | """ 21 | 22 | def __init__(self, depth=1, resolution=24 * 1024, buffersize=None, 23 | dtype=None, seed=None): 24 | self.resolution = resolution 25 | self.depth = depth 26 | # Default buffersize: 128 samples (and smaller than resolution). 27 | if buffersize is None: 28 | buffersize = min(128, (resolution + 7) // 8) 29 | self.buffersize = buffersize 30 | self.samplerate = 1.0 31 | self.data = [numpy.zeros(shape=(depth, resolution), dtype=dtype)] 32 | self.firstfree = [0] 33 | self.random = numpy.random.RandomState(seed) 34 | self.extremes = numpy.empty(shape=(depth, 2), dtype=dtype) 35 | self.extremes.fill(numpy.NaN) 36 | self.size = 0 37 | 38 | def add(self, incoming): 39 | assert len(incoming.shape) == 2 40 | assert incoming.shape[1] == self.depth 41 | self.size += incoming.shape[0] 42 | # Convert to a flat numpy array. 43 | if self.samplerate >= 1.0: 44 | self._add_every(incoming) 45 | return 46 | # If we are sampling, then subsample a large chunk at a time. 47 | self._scan_extremes(incoming) 48 | chunksize = numpy.ceil[self.buffersize / self.samplerate] 49 | for index in xrange(0, len(incoming), chunksize): 50 | batch = incoming[index:index+chunksize] 51 | sample = batch[self.random.binomial(1, self.samplerate, len(batch))] 52 | self._add_every(sample) 53 | 54 | def _add_every(self, incoming): 55 | supplied = len(incoming) 56 | index = 0 57 | while index < supplied: 58 | ff = self.firstfree[0] 59 | available = self.data[0].shape[1] - ff 60 | if available == 0: 61 | if not self._shift(): 62 | # If we shifted by subsampling, then subsample. 63 | incoming = incoming[index:] 64 | if self.samplerate >= 0.5: 65 | print 'SAMPLING' 66 | self._scan_extremes(incoming) 67 | incoming = incoming[self.random.binomial(1, 0.5, 68 | len(incoming - index))] 69 | index = 0 70 | supplied = len(incoming) 71 | ff = self.firstfree[0] 72 | available = self.data[0].shape[1] - ff 73 | copycount = min(available, supplied - index) 74 | self.data[0][:,ff:ff + copycount] = numpy.transpose( 75 | incoming[index:index + copycount,:]) 76 | self.firstfree[0] += copycount 77 | index += copycount 78 | 79 | def _shift(self): 80 | index = 0 81 | # If remaining space at the current layer is less than half prev 82 | # buffer size (rounding up), then we need to shift it up to ensure 83 | # enough space for future shifting. 84 | while self.data[index].shape[1] - self.firstfree[index] < ( 85 | -(-self.data[index-1].shape[1] // 2) if index else 1): 86 | if index + 1 >= len(self.data): 87 | return self._expand() 88 | data = self.data[index][:,0:self.firstfree[index]] 89 | data.sort() 90 | if index == 0 and self.samplerate >= 1.0: 91 | self._update_extremes(data[:,0], data[:,-1]) 92 | offset = self.random.binomial(1, 0.5) 93 | position = self.firstfree[index + 1] 94 | subset = data[:,offset::2] 95 | self.data[index + 1][:,position:position + subset.shape[1]] = subset 96 | self.firstfree[index] = 0 97 | self.firstfree[index + 1] += subset.shape[1] 98 | index += 1 99 | return True 100 | 101 | def _scan_extremes(self, incoming): 102 | # When sampling, we need to scan every item still to get extremes 103 | self._update_extremes( 104 | numpy.nanmin(incoming, axis=0), 105 | numpy.nanmax(incoming, axis=0)) 106 | 107 | def _update_extremes(self, minr, maxr): 108 | self.extremes[:,0] = numpy.nanmin( 109 | [self.extremes[:, 0], minr], axis=0) 110 | self.extremes[:,-1] = numpy.nanmax( 111 | [self.extremes[:, -1], maxr], axis=0) 112 | 113 | def minmax(self): 114 | if self.firstfree[0]: 115 | self._scan_extremes(self.data[0][:,:self.firstfree[0]].transpose()) 116 | return self.extremes.copy() 117 | 118 | def _expand(self): 119 | cap = self._next_capacity() 120 | if cap > 0: 121 | # First, make a new layer of the proper capacity. 122 | self.data.insert(0, numpy.empty( 123 | shape=(self.depth, cap), dtype=self.data[-1].dtype)) 124 | self.firstfree.insert(0, 0) 125 | else: 126 | # Unless we're so big we are just subsampling. 127 | assert self.firstfree[0] == 0 128 | self.samplerate *= 0.5 129 | for index in range(1, len(self.data)): 130 | # Scan for existing data that needs to be moved down a level. 131 | amount = self.firstfree[index] 132 | if amount == 0: 133 | continue 134 | position = self.firstfree[index-1] 135 | # Move data down if it would leave enough empty space there 136 | # This is the key invariant: enough empty space to fit half 137 | # of the previous level's buffer size (rounding up) 138 | if self.data[index-1].shape[1] - (amount + position) >= ( 139 | -(-self.data[index-2].shape[1] // 2) if (index-1) else 1): 140 | self.data[index-1][:,position:position + amount] = ( 141 | self.data[index][:,:amount]) 142 | self.firstfree[index-1] += amount 143 | self.firstfree[index] = 0 144 | else: 145 | # Scrunch the data if it would not. 146 | data = self.data[index][:,:amount] 147 | data.sort() 148 | if index == 1: 149 | self._update_extremes(data[:,0], data[:,-1]) 150 | offset = self.random.binomial(1, 0.5) 151 | scrunched = data[:,offset::2] 152 | self.data[index][:,:scrunched.shape[1]] = scrunched 153 | self.firstfree[index] = scrunched.shape[1] 154 | return cap > 0 155 | 156 | def _next_capacity(self): 157 | cap = numpy.ceil(self.resolution * numpy.power(0.67, len(self.data))) 158 | if cap < 2: 159 | return 0 160 | return max(self.buffersize, int(cap)) 161 | 162 | def _weighted_summary(self, sort=True): 163 | if self.firstfree[0]: 164 | self._scan_extremes(self.data[0][:,:self.firstfree[0]].transpose()) 165 | size = sum(self.firstfree) + 2 166 | weights = numpy.empty( 167 | shape=(size), dtype='float32') # floating point 168 | summary = numpy.empty( 169 | shape=(self.depth, size), dtype=self.data[-1].dtype) 170 | weights[0:2] = 0 171 | summary[:,0:2] = self.extremes 172 | index = 2 173 | for level, ff in enumerate(self.firstfree): 174 | if ff == 0: 175 | continue 176 | summary[:,index:index + ff] = self.data[level][:,:ff] 177 | weights[index:index + ff] = numpy.power(2.0, level) 178 | index += ff 179 | assert index == summary.shape[1] 180 | if sort: 181 | order = numpy.argsort(summary) 182 | summary = summary[numpy.arange(self.depth)[:,None], order] 183 | weights = weights[order] 184 | return (summary, weights) 185 | 186 | def quantiles(self, quantiles, old_style=False): 187 | if self.size == 0: 188 | return numpy.full((self.depth, len(quantiles)), numpy.nan) 189 | summary, weights = self._weighted_summary() 190 | cumweights = numpy.cumsum(weights, axis=-1) - weights / 2 191 | if old_style: 192 | # To be convenient with numpy.percentile 193 | cumweights -= cumweights[:,0:1] 194 | cumweights /= cumweights[:,-1:] 195 | else: 196 | cumweights /= numpy.sum(weights, axis=-1, keepdims=True) 197 | result = numpy.empty(shape=(self.depth, len(quantiles))) 198 | for d in xrange(self.depth): 199 | result[d] = numpy.interp(quantiles, cumweights[d], summary[d]) 200 | return result 201 | 202 | def integrate(self, fun): 203 | result = None 204 | for level, ff in enumerate(self.firstfree): 205 | if ff == 0: 206 | continue 207 | term = numpy.sum( 208 | fun(self.data[level][:,:ff]) * numpy.power(2.0, level), 209 | axis=-1) 210 | if result is None: 211 | result = term 212 | else: 213 | result += term 214 | if result is not None: 215 | result /= self.samplerate 216 | return result 217 | 218 | def percentiles(self, percentiles): 219 | return self.quantiles(percentiles, old_style=True) 220 | 221 | def readout(self, count, old_style=True): 222 | return self.quantiles( 223 | numpy.linspace(0.0, 1.0, count), old_style=old_style) 224 | 225 | 226 | if __name__ == '__main__': 227 | import time 228 | # An adverarial case: we keep finding more numbers in the middle 229 | # as the stream goes on. 230 | amount = 10000000 231 | percentiles = 1000 232 | data = numpy.arange(float(amount)) 233 | data[1::2] = data[-1::-2] + (len(data) - 1) 234 | data /= 2 235 | depth = 50 236 | alldata = data[:,None] + (numpy.arange(depth) * amount)[None, :] 237 | actual_sum = numpy.sum(alldata * alldata, axis=0) 238 | amt = amount // depth 239 | for r in range(depth): 240 | numpy.random.shuffle(alldata[r*amt:r*amt+amt,r]) 241 | # data[::2] = data[-2::-2] 242 | # numpy.random.shuffle(data) 243 | starttime = time.time() 244 | qc = QuantileVector(depth=depth, resolution=8 * 1024) 245 | qc.add(alldata) 246 | ro = qc.readout(1001) 247 | endtime = time.time() 248 | # print 'ro', ro 249 | # print ro - numpy.linspace(0, amount, percentiles+1) 250 | gt = numpy.linspace(0, amount, percentiles+1)[None,:] + ( 251 | numpy.arange(qc.depth) * amount)[:,None] 252 | print "Maximum relative deviation among %d perentiles:" % percentiles, ( 253 | numpy.max(abs(ro - gt) / amount) * percentiles) 254 | print "Minmax eror %f, %f" % ( 255 | max(abs(qc.minmax()[:,0] - numpy.arange(qc.depth) * amount)), 256 | max(abs(qc.minmax()[:, -1] - (numpy.arange(qc.depth)+1) * amount + 1))) 257 | print "Integral error:", numpy.max(numpy.abs( 258 | qc.integrate(lambda x: x * x) 259 | - actual_sum) / actual_sum) 260 | print "Count error: ", (qc.integrate(lambda x: numpy.ones(x.shape[-1]) 261 | ) - qc.size) / (0.0 + qc.size) 262 | print "Time", (endtime - starttime) 263 | 264 | -------------------------------------------------------------------------------- /src/w2color.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CSAILVision/NetDissect/20ce6f887607aadc04ab9fb3d452e4723685cbbb/src/w2color.npy -------------------------------------------------------------------------------- /zoo/README.md: -------------------------------------------------------------------------------- 1 | Model Zoo directory 2 | =================== 3 | 4 | Add caffemodel and deploy prototxt files to this directory. 5 | Scripts are set up assuming that you follow the naming convention 6 | 7 | [modelname].caffemodel 8 | [modelname].prototxt 9 | --------------------------------------------------------------------------------