├── .gitignore
├── README.md
├── backend
    ├── MCR
    │   └── .gitignore
    ├── __init__.py
    ├── info.py
    ├── log
    │   └── .gitignore
    ├── networks
    │   └── .gitignore
    ├── run.py
    ├── src
    │   ├── demo.m
    │   ├── matconvnet-1.0-beta7.tar.gz
    │   ├── minConf
    │   │   ├── isLegal.m
    │   │   ├── lbfgs.m
    │   │   ├── lbfgsHvFunc2.m
    │   │   ├── lbfgsUpdate.m
    │   │   ├── minConf_PQN.m
    │   │   ├── minConf_SPG.m
    │   │   ├── minConf_TMP.m
    │   │   ├── myProcessOptions.m
    │   │   └── polyinterp.m
    │   ├── ostrichinator_lite.m
    │   └── vl_simplenn_fast.m
    └── synset_words.txt
├── frontend.py
├── static
    └── .gitignore
└── templates
    └── index.html


/.gitignore:
--------------------------------------------------------------------------------
1 | .*
2 | *.pyc
3 | keys.py
4 | !.gitignore
5 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Ostrichinator
 2 | Ostrichinator aims at providing a [web platform](http://ostrichinator.net/) for discovering adversarial examples and potentially collecting human responses, which may fundamentally help with improving current deep learning algorithms.
 3 | The frontend of this project is based on Python Flask, which handles requests from users and passes jobs to the backend, and the backend is implemented in MATLAB based on MatConvNet and minConf.
 4 | 
 5 | ## Frontend
 6 | Like most other Flask based simple web applications, the frontend of Ostrichinator is concisely implemented in two files: <code>frontend.py</code> and <code>template/index.html</code>.
 7 | User requests are indexed by UUIDs and the source (original images) and result files (hacked images and logs) are named correspondingly.
 8 | While source and resultant images are directly served in <code>static/</code>, logs are placed in <code>backend/log/</code>.
 9 | ### Configuration
10 | Except for the secret keys for Flask and Flask-WTF reCAPTCHA (i.e. <code>SECRET_KEY</code>, <code>RECAPTCHA_PUBLIC_KEY</code> and <code>RECAPTCHA_PRIVATE_KEY</code>, which in our case, are put in <code>keys.py</code>), to run the backend locally or distributedly (i.e. without or with a job queue) should be the only thing which needs to be configured here.
11 | <code>Frontend.py</code> by default is using a Celery plus Redis job queue for requests.
12 | However, users can easily comment/uncomment blocks in <code>frontend.py</code> to make it run locally following instructions in the file.
13 | ### Deployment
14 | Simply executing <code>python frontend.py</code> can launch the Flask frontend with gevent WSGI server.
15 | However, other combinations (e.g. Flask with uWSGI and Nginx, as used by the demo site) may be more favorable.
16 | 
17 | ## Backend
18 | The backend of Ostrichinator is a compiled executable which directly takes input images from the <code>static/</code> directory and parameters from the command line arguments.
19 | When the backend is running, log files stating the execution progresses and the final results are generated inside <code>backend/log/</code>, and when it finishes, result images are written into the <code>static/</code> directory as well.
20 | The log files are structurally defined with the first lines describing the tasks, the fourth- and third-to-the-last lines the original and final class labels, the second-to-the-last lines the exit flags, and the final lines "DONE".
21 | For now, there's no explicit mechanism for pushing results to the users implemented yet.
22 | ### Configuration
23 | Configuring the backend of Ostrichinator would involve compiling MATLAB codes located in <code>backend/src/</code> into an executable.
24 | We used MATLAB R2014b, while any MATLAB version after R2013a should be fine as well.
25 | First thing the users need to do would be installing and setting up [MatConvNet](http://www.vlfeat.org/matconvnet/), which is fairly simple and quick.
26 | While MatConvNet v1.0 beta-7 is used and included in this project, newer versions should work as well.
27 | The main MATLAB file is <code>backend/src/demo.m</code>, which should be ready to run after MatConvNet is correctly installed, and both MatConvNet and <code>backend/src/minConf</code> are in MATLAB’s search path.
28 | Please follow <code>demo.m</code> to compile the executable, and place the generated <code>demo</code> and <code>run_demo.sh</code> under <code>backend/</code>.
29 | If installing MATLAB is not an option, users can check out our [precompiled executables](https://drive.google.com/folderview?id=0B8LpM_21I0tYfmtjdHFoenByeVhnTkZaRWRDUkZneHQzWDVZUi1VdTFxcVRxaDQ2UnFzWnM&usp=sharing) as well.
30 | After obtaining the executable, users also need to specify the path to MATLAB runtime libraries in <code>backend/MCR/</code>.
31 | If users have a full MATLAB installation, a symbolic link, e.g. <code>backend/MCR/v84 -> /usr/local/MATLAB/R2014b/</code>, can be created to do this.
32 | If users don’t have a full MATALB and decide to use our precompiled executable, please download and install the [MATLAB Compiler Runtime](http://www.mathworks.com/products/compiler/mcr/) into <code>backend/MCR/</code>.
33 | Also remember to download the pretrained deep learning networks [[1](http://www.vlfeat.org/matconvnet/models/imagenet-caffe-ref.mat),[2](http://www.vlfeat.org/matconvnet/models/imagenet-vgg-s.mat),[3](http://www.vlfeat.org/matconvnet/models/imagenet-vgg-verydeep-19.mat)] into <code>backend/networks/</code>.
34 | Lastly, adjust <code>backend/run.py</code> and <code>backend/info.py</code> for running locally or distributedly as well.
35 | ### Deployment
36 | If users decided to run the backend distributedly, simply remember to start Redis and Celery by e.g. <code>redis-server</code> and <code>celery worker -A backend.run --loglevel=info</code> (for Celery, under the main directory, not <code>backend/</code>).
37 | Otherwise, nothing needs to be done.
38 | 
39 | 


--------------------------------------------------------------------------------
/backend/MCR/.gitignore:
--------------------------------------------------------------------------------
1 | *
2 | !.gitignore
3 | 


--------------------------------------------------------------------------------
/backend/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/coxlab/ostrichinator/24a7626b57895fa40172a922110306da4f20be3c/backend/__init__.py


--------------------------------------------------------------------------------
/backend/info.py:
--------------------------------------------------------------------------------
1 | WORKER_NUM = 128
2 | 


--------------------------------------------------------------------------------
/backend/log/.gitignore:
--------------------------------------------------------------------------------
1 | *
2 | !example*
3 | !.gitignore
4 | 


--------------------------------------------------------------------------------
/backend/networks/.gitignore:
--------------------------------------------------------------------------------
1 | *
2 | !.gitignore
3 | 


--------------------------------------------------------------------------------
/backend/run.py:
--------------------------------------------------------------------------------
 1 | import os
 2 | 
 3 | # See src/demo.m for definitions
 4 | run_inst = 'backend/run_demo.sh backend/MCR/v84/ static/{0}.png backend/networks/ 0 50 1 1 {1} {4} {2} {4} {3} {4} >> backend/log/{0}.txt 2>&1'
 5 | 
 6 | # Comment out for running locally
 7 | from celery import Celery
 8 | from keys import UPLOAD_PATH
 9 | 
10 | backend = Celery('run', broker='redis://localhost:6379/1'); backend.conf.update(CELERYD_PREFETCH_MULTIPLIER=1)
11 | 
12 | @backend.task
13 | def run_backend(taskpar):
14 | 	os.system('wget -q -P static http://localhost:8080/static/{0}.png'.format(taskpar[0]))
15 | 	os.system(run_inst.format(*taskpar))
16 | 	os.system('curl -s -X POST -F data=@static/{0}-out.png "http://localhost:8080/{1}"'.format(taskpar[0],UPLOAD_PATH))
17 | 	os.system('curl -s -X POST -F data=@static/{0}-sal.png "http://localhost:8080/{1}"'.format(taskpar[0],UPLOAD_PATH))
18 | 	os.system('curl -s -X POST -F data=@static/{0}-dff.png "http://localhost:8080/{1}"'.format(taskpar[0],UPLOAD_PATH))
19 | 	os.system('curl -s -X POST -F data=@backend/log/{0}.txt "http://localhost:8080/{1}"'.format(taskpar[0],UPLOAD_PATH))
20 | 	return None
21 | # ----------
22 | 
23 | # Uncomment for running locally
24 | #def run_backend(taskpar):
25 | #	run_inst_async = '(' + run_inst + ' &)'
26 | #	os.system(run_inst_async.format(*taskpar))
27 | #	return None
28 | # ----------
29 | 
30 | 


--------------------------------------------------------------------------------
/backend/src/demo.m:
--------------------------------------------------------------------------------
  1 | function demo(imgfile, netpath, allwgpu, iter, faststp, stpthrs, varargin)
  2 | 
  3 | % INPUT VARIABLES
  4 | % imgfile, path and filename to the target image, which must be PNG sized 227x227.
  5 | % netpath, path to the pretrained networks' mat files.
  6 | % allwgpu, enable GPU mode or not. 
  7 | % iter, maximally allowed running iterations; set to 1 for recognition only mode.
  8 | % faststp, 1 for fast stop (stopping after score of the target class becomes highest) and 0 for slow stop (stopping after probability of the target class reaches 'stpthrs').
  9 | % stpthrs, stopping threshold for probability of the target class when faststp is set to 0.
 10 | % varargin{1}, enable 'imagenet-caffe-ref' or not.
 11 | % varargin{2}, target class for 'imagenet-caffe-ref'.
 12 | % varargin{3}, enable 'imagenet-vgg-s' or not.
 13 | % varargin{4}, target class for 'imagenet-vgg-s'.
 14 | % varargin{5}, enable 'imagenet-vgg-verydeep-19' or not.
 15 | % varargin{6}, target class for 'imagenet-vgg-verydeep-19'.
 16 | 
 17 | % EXAMPLE: HACK TEST.PNG INTO CLASS 1 FOR ALL NETWORKS
 18 | % demo('static/test.png','backend/networks','0','50','1','1.0','1','1','1','1','1','1');
 19 | 
 20 | % EXAMPLE: COMPILE DEMO.M INTO EXECUTABLE
 21 | % mcc -m -R -nojvm -R -nodisplay -R -singleCompThread -v demo.m;
 22 | 
 23 | % -------------------------------------------------------------------------
 24 | 
 25 | netfile = {'imagenet-caffe-ref.mat', ...
 26 |            'imagenet-vgg-s.mat', ...
 27 |            'imagenet-vgg-verydeep-19.mat'};
 28 | 
 29 | try
 30 |     % PROCESS PARAMETERS
 31 |     [allwgpu, iter, faststp, stpthrs, varargin{:}] = cell2double(allwgpu, iter, faststp, stpthrs, varargin{:});
 32 |     
 33 |     % LOAD IMAGE (SHOULD BE PNG)
 34 |     img = double(imread(imgfile));
 35 |     switch size(img,3)
 36 |         case 1, img = repmat(img, [1 1 3]);
 37 |         case 3, % NOTHING
 38 |         otherwise, error('IMREAD ERROR');
 39 |     end
 40 |     
 41 |     % LOAD NETWORK(S)
 42 |     tic;
 43 |     net = {}; tc = {};
 44 |     for i = 1:2:numel(varargin)
 45 |         if (varargin{i} == 1)
 46 |             net(end+1) = {load(fullfile(netpath, netfile{(i-1)/2+1}), 'layers')};
 47 |             tc(end+1)  = {[(varargin{i+1}) 1000]};
 48 |             
 49 |             temp = load(fullfile(netpath, netfile{(i-1)/2+1}), 'classes');
 50 |             [~,tempidx] = sortrows(temp.classes.name');
 51 |             net{end}.layers{end-1}.filters = net{end}.layers{end-1}.filters(:,:,:,tempidx);
 52 |         end
 53 |     end
 54 |     toc;
 55 |     
 56 |     % RUN DEMO
 57 |     alg = @ostrichinator_lite;
 58 |     
 59 |     tic;
 60 |     try
 61 |         [xmin, xsal, flag, ~, ~, chist] = alg(net, img, tc, allwgpu, iter, faststp, stpthrs);
 62 |     catch ERR % RETRY CPU MODE IF GPU MODE ERROR
 63 |         if (allwgpu == 0), rethrow(ERR);
 64 |         else [xmin, xsal, flag, ~, ~, chist] = alg(net, img, tc, 0, iter, faststp, stpthrs); end
 65 |     end
 66 |     toc;
 67 |     
 68 |     % PROCESS RESULTS
 69 |     for i = 1:numel(xsal)
 70 |         xsal{i} = sum(abs(xsal{i}), 3);
 71 |         xsal{i} = xsal{i} - min(xsal{i}(:));
 72 |         xsal{i} = xsal{i} / max(max(xsal{i}(:)), eps);
 73 |     end
 74 |     xsal = cat(3, xsal{:});
 75 |     xsal = min(sum(xsal, 3), 1); 
 76 |     
 77 |     xdff = xmin - img;
 78 |     xdff = xdff - min(xdff(:));
 79 |     xdff = xdff / max(max(xdff(:)), eps);
 80 |     xdff = round(xdff * 255);
 81 |     
 82 |     chist = [chist{:}];
 83 |     
 84 |     % WRITE RESULTS
 85 |     [imgpath,imgname,~] = fileparts(imgfile);
 86 |     
 87 |     imwrite(uint8(xmin), fullfile(imgpath, [imgname '-out.png']));
 88 |     
 89 |     xsal = gray2ind(xsal, 256);
 90 |     if (exist('parula','file') > 0), cm = @parula; else cm = @gray; end
 91 |     imwrite(xsal, cm(256), fullfile(imgpath, [imgname '-sal.png']));
 92 |     
 93 |     imwrite(uint8(xdff), fullfile(imgpath, [imgname '-dff.png']));
 94 |     
 95 |     fprintf('%s\n', num2str(chist(:,1)'  ));
 96 |     fprintf('%s\n', num2str(chist(:,end)'));
 97 | catch ERR
 98 |     fprintf('%s\n', ERR.getReport);
 99 |     flag = -1;
100 | end
101 | 
102 | % 1 = SUCCESSFUL FINISH; 0 = UNSUCCESSFUL FINISH; -1 = ERROR
103 | fprintf('%d\n', flag);
104 | fprintf('DONE');
105 | 
106 | % -------------------------------------------------------------------------
107 | 
108 |     function varargout = cell2double(varargin)
109 |         
110 |         varargout = cell(size(varargin));
111 |         
112 |         for c = 1:numel(varargin)
113 |             if isa(varargin{c}, 'char'), varargout{c} = str2double(varargin{c});
114 |             else varargout{c} = double(varargin{c}); end
115 |         end
116 |     end
117 | 
118 | end
119 | 
120 | 


--------------------------------------------------------------------------------
/backend/src/matconvnet-1.0-beta7.tar.gz:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/coxlab/ostrichinator/24a7626b57895fa40172a922110306da4f20be3c/backend/src/matconvnet-1.0-beta7.tar.gz


--------------------------------------------------------------------------------
/backend/src/minConf/isLegal.m:
--------------------------------------------------------------------------------
1 | function [legal] = isLegal(v)
2 | legal = sum(any(imag(v(:))))==0 & sum(isnan(v(:)))==0 & sum(isinf(v(:)))==0;


--------------------------------------------------------------------------------
/backend/src/minConf/lbfgs.m:
--------------------------------------------------------------------------------
 1 | function [d] = lbfgs(g,s,y,Hdiag)
 2 | % BFGS Search Direction
 3 | %
 4 | % This function returns the (L-BFGS) approximate inverse Hessian,
 5 | % multiplied by the gradient
 6 | %
 7 | % If you pass in all previous directions/sizes, it will be the same as full BFGS
 8 | % If you truncate to the k most recent directions/sizes, it will be L-BFGS
 9 | %
10 | % s - previous search directions (p by k)
11 | % y - previous step sizes (p by k)
12 | % g - gradient (p by 1)
13 | % Hdiag - value of initial Hessian diagonal elements (scalar)
14 | 
15 | [p,k] = size(s);
16 | 
17 | for i = 1:k
18 |     ro(i,1) = 1/(y(:,i)'*s(:,i));
19 | end
20 | 
21 | q = zeros(p,k+1);
22 | r = zeros(p,k+1);
23 | al =zeros(k,1);
24 | be =zeros(k,1);
25 | 
26 | q(:,k+1) = g;
27 | 
28 | for i = k:-1:1
29 |     al(i) = ro(i)*s(:,i)'*q(:,i+1);
30 |     q(:,i) = q(:,i+1)-al(i)*y(:,i);
31 | end
32 | 
33 | % Multiply by Initial Hessian
34 | r(:,1) = Hdiag*q(:,1);
35 | 
36 | for i = 1:k
37 |     be(i) = ro(i)*y(:,i)'*r(:,i);
38 |     r(:,i+1) = r(:,i) + s(:,i)*(al(i)-be(i));
39 | end
40 | d=r(:,k+1);


--------------------------------------------------------------------------------
/backend/src/minConf/lbfgsHvFunc2.m:
--------------------------------------------------------------------------------
1 | function Hv = lbfgsHvFunc2(v,Hdiag,N,M)
2 | Hv = v/Hdiag - N*(M\(N'*v));


--------------------------------------------------------------------------------
/backend/src/minConf/lbfgsUpdate.m:
--------------------------------------------------------------------------------
 1 | function [old_dirs,old_stps,Hdiag] = lbfgsUpdate(y,s,corrections,debug,old_dirs,old_stps,Hdiag)
 2 | ys = y'*s;
 3 | if ys > 1e-10
 4 |     numCorrections = size(old_dirs,2);
 5 |     if numCorrections < corrections
 6 |         % Full Update
 7 |         old_dirs(:,numCorrections+1) = s;
 8 |         old_stps(:,numCorrections+1) = y;
 9 |     else
10 |         % Limited-Memory Update
11 |         old_dirs = [old_dirs(:,2:corrections) s];
12 |         old_stps = [old_stps(:,2:corrections) y];
13 |     end
14 | 
15 |     % Update scale of initial Hessian approximation
16 |     Hdiag = ys/(y'*y);
17 | else
18 |     if debug
19 |         fprintf('Skipping Update\n');
20 |     end
21 | end


--------------------------------------------------------------------------------
/backend/src/minConf/minConf_PQN.m:
--------------------------------------------------------------------------------
  1 | function [x,f,funEvals] = minConf_PQN(funObj,x,funProj,options,ckpntFun)
  2 | % function [x,f] = minConf_PQN(funObj,funProj,x,options)
  3 | %
  4 | % Function for using a limited-memory projected quasi-Newton to solve problems of the form
  5 | %   min funObj(x) s.t. x in C
  6 | %
  7 | % The projected quasi-Newton sub-problems are solved the spectral projected
  8 | % gradient algorithm
  9 | %
 10 | %   @funObj(x): function to minimize (returns gradient as second argument)
 11 | %   @funProj(x): function that returns projection of x onto C
 12 | %
 13 | %   options:
 14 | %       verbose: level of verbosity (0: no output, 1: final, 2: iter (default), 3:
 15 | %       debug)
 16 | %       optTol: tolerance used to check for optimality (default: 1e-5)
 17 | %       progTol: tolerance used to check for progress (default: 1e-9)
 18 | %       maxIter: maximum number of calls to funObj (default: 500)
 19 | %       maxProject: maximum number of calls to funProj (default: 100000)
 20 | %       numDiff: compute derivatives numerically (0: use user-supplied
 21 | %       derivatives (default), 1: use finite differences, 2: use complex
 22 | %       differentials)
 23 | %       suffDec: sufficient decrease parameter in Armijo condition (default: 1e-4)
 24 | %       corrections: number of lbfgs corrections to store (default: 10)
 25 | %       adjustStep: use quadratic initialization of line search (default: 0)
 26 | %       bbInit: initialize sub-problem with Barzilai-Borwein step (default: 1)
 27 | %       SPGoptTol: optimality tolerance for SPG direction finding (default: 1e-6)
 28 | %       SPGiters: maximum number of iterations for SPG direction finding (default:10)
 29 | 
 30 | nVars = length(x);
 31 | 
 32 | % Set Parameters
 33 | if nargin < 4
 34 |     options = [];
 35 | end
 36 | [verbose,numDiff,optTol,progTol,maxIter,maxProject,suffDec,corrections,adjustStep,bbInit,...
 37 |     SPGoptTol,SPGprogTol,SPGiters,SPGtestOpt] = ...
 38 |     myProcessOptions(...
 39 |     options,'verbose',3,'numDiff',0,'optTol',1e-5,'progTol',1e-9,'maxIter',100,'maxProject',100000,'suffDec',1e-4,...
 40 |     'corrections',10,'adjustStep',0,'bbInit',0,'SPGoptTol',1e-6,'SPGprogTol',1e-10,'SPGiters',10,'SPGtestOpt',0);
 41 | 
 42 | % Output Parameter Settings
 43 | % if verbose >= 3
 44 | %    fprintf('Running PQN...\n');
 45 | %    fprintf('Number of L-BFGS Corrections to store: %d\n',corrections);
 46 | %    fprintf('Spectral initialization of SPG: %d\n',bbInit);
 47 | %    fprintf('Maximum number of SPG iterations: %d\n',SPGiters);
 48 | %    fprintf('SPG optimality tolerance: %.2e\n',SPGoptTol);
 49 | %    fprintf('SPG progress tolerance: %.2e\n',SPGprogTol);
 50 | %    fprintf('PQN optimality tolerance: %.2e\n',optTol);
 51 | %    fprintf('PQN progress tolerance: %.2e\n',progTol);
 52 | %    fprintf('Quadratic initialization of line search: %d\n',adjustStep);
 53 | %    fprintf('Maximum number of function evaluations: %d\n',maxIter);
 54 | %    fprintf('Maximum number of projections: %d\n',maxProject);
 55 | % end
 56 | 
 57 | % Output Log
 58 | if verbose >= 2
 59 |         fprintf('%10s %10s %10s %15s %15s %15s\n','Iteration','FunEvals','Projections','Step Length','Function Val','Opt Cond');
 60 | end
 61 | 
 62 | % Make objective function (if using numerical derivatives)
 63 | funEvalMultiplier = 1;
 64 | if numDiff
 65 |     if numDiff == 2
 66 |         useComplex = 1;
 67 |     else
 68 |         useComplex = 0;
 69 |     end
 70 |     funObj = @(x)autoGrad(x,useComplex,funObj);
 71 |     funEvalMultiplier = nVars+1-useComplex;
 72 | end
 73 | 
 74 | % Project initial parameter vector
 75 | x = funProj(x);
 76 | projects = 1;
 77 | 
 78 | % Evaluate initial parameters
 79 | [f,g] = funObj(x);
 80 | funEvals = 1;
 81 | 
 82 | % Plot Progress
 83 | if (nargin == 5)
 84 |     [stop, dispstr] = ckpntFun(x);
 85 |     fprintf('\b    %s\n', dispstr);
 86 |     if stop, fprintf('Early termination condition(s) reached\n'); return; end
 87 | end
 88 | 
 89 | % Check Optimality of Initial Point
 90 | projects = projects+1;
 91 | if max(abs(funProj(x-g)-x)) < optTol
 92 |     if verbose >= 1
 93 |         fprintf('First-Order Optimality Conditions Below optTol at Initial Point\n');
 94 |     end
 95 |     return;
 96 | end
 97 | 
 98 | i = 1;
 99 | while funEvals <= maxIter
100 | 
101 |     % Compute Step Direction
102 |     if i == 1
103 |         p = funProj(x-g);
104 |         projects = projects+1;
105 |         S = zeros(nVars,0);
106 |         Y = zeros(nVars,0);
107 |         Hdiag = 1;
108 |     else
109 |         y = g-g_old;
110 |         s = x-x_old;
111 |         [S,Y,Hdiag] = lbfgsUpdate(y,s,corrections,verbose==3,S,Y,Hdiag);
112 | 
113 |         % Make Compact Representation
114 |         k = size(Y,2);
115 |         L = zeros(k);
116 |         for j = 1:k
117 |             L(j+1:k,j) = S(:,j+1:k)'*Y(:,j);
118 |         end
119 |         N = [S/Hdiag Y];
120 |         M = [S'*S/Hdiag L;L' -diag(diag(S'*Y))];
121 |         HvFunc = @(v)lbfgsHvFunc2(v,Hdiag,N,M);
122 | 
123 |         if bbInit
124 |             % Use Barzilai-Borwein step to initialize sub-problem
125 |             alpha = (s'*s)/(s'*y);
126 |             if alpha <= 1e-10 || alpha > 1e10
127 |                 alpha = min(1,1/sum(abs(g)));
128 |             end
129 | 
130 |             % Solve Sub-problem
131 |             xSubInit = x-alpha*g;
132 |             feasibleInit = 0;
133 |         else
134 |             xSubInit = x;
135 |             feasibleInit = 1;
136 |         end
137 |         % Solve Sub-problem
138 |         [p,subProjects] = solveSubProblem(x,g,HvFunc,funProj,SPGoptTol,SPGprogTol,SPGiters,SPGtestOpt,feasibleInit,xSubInit);
139 |         projects = projects+subProjects;
140 |     end
141 |     d = p-x;
142 |     g_old = g;
143 |     x_old = x;
144 | 
145 |     % Check that Progress can be made along the direction
146 |     gtd = g'*d;
147 |     if gtd > -progTol
148 |         if verbose >= 1
149 |             fprintf('Directional Derivative below progTol\n');
150 |         end
151 |         break;
152 |     end
153 | 
154 |     % Select Initial Guess to step length
155 |     if i == 1 || adjustStep == 0
156 |        t = 1; 
157 |     else
158 |         t = min(1,2*(f-f_old)/gtd);
159 |     end
160 |     
161 |     % Bound Step length on first iteration
162 |     if i == 1
163 |         t = min(1,1/sum(abs(g)));
164 |     end
165 | 
166 |     % Evaluate the Objective and Gradient at the Initial Step
167 |     if t == 1
168 |         x_new = p;
169 |     else
170 |         x_new = x + t*d;
171 |     end
172 |     [f_new,g_new] = funObj(x_new);
173 |     funEvals = funEvals+1;
174 | 
175 |     % Backtracking Line Search
176 |     f_old = f;
177 |     while f_new > f + suffDec*g'*(x_new-x) || ~isLegal(f_new)
178 |         temp = t;
179 |         
180 |         % Backtrack to next trial value
181 |         if ~isLegal(f_new) || ~isLegal(g_new)
182 |             if verbose == 3
183 |                 fprintf('Halving Step Size\n');
184 |             end
185 |             t = t/2;
186 |         else
187 |             if verbose == 3
188 |                 fprintf('Cubic Backtracking\n');
189 |             end
190 |             t = polyinterp([0 f gtd; t f_new g_new'*d]);
191 |         end
192 | 
193 |         % Adjust if change is too small/large
194 |         if t < temp*1e-3
195 |             if verbose == 3
196 |                 fprintf('Interpolated value too small, Adjusting\n');
197 |             end
198 |             t = temp*1e-3;
199 |         elseif t > temp*0.6
200 |             if verbose == 3
201 |                 fprintf('Interpolated value too large, Adjusting\n');
202 |             end
203 |             t = temp*0.6;
204 |         end
205 | 
206 |         % Check whether step has become too small
207 |         if sum(abs(t*d)) < progTol || t == 0
208 |             if verbose == 3
209 |                 fprintf('Line Search failed\n');
210 |             end
211 |             t = 0;
212 |             f_new = f;
213 |             g_new = g;
214 |             break;
215 |         end
216 | 
217 |         % Evaluate New Point
218 |         f_prev = f_new;
219 |         t_prev = temp;
220 |         x_new = x + t*d;
221 |         [f_new,g_new] = funObj(x_new);
222 |         funEvals = funEvals+1;
223 | 
224 |     end
225 | 
226 |     % Take Step
227 |     x = x_new;
228 |     f = f_new;
229 |     g = g_new;
230 |     
231 |     optCond = max(abs(funProj(x-g)-x));
232 |     projects = projects+1;
233 | 
234 |     % Output Log
235 |     if verbose >= 2
236 |             fprintf('%10d %10d %10d %15.5e %15.5e %15.5e\n',i,funEvals*funEvalMultiplier,projects,t,f,optCond);
237 |     end
238 |     
239 |     % Plot Progress
240 |     if (nargin == 5)
241 |         [stop, dispstr] = ckpntFun(x);
242 |         fprintf('\b    %s\n', dispstr);
243 |         if stop, fprintf('Early termination condition(s) reached\n'); break; end
244 |     end
245 | 
246 |     % Check optimality
247 |         if optCond < optTol
248 |             fprintf('First-Order Optimality Conditions Below optTol\n');
249 |             break;
250 |         end
251 | 
252 |     if max(abs(t*d)) < progTol
253 |         if verbose >= 1
254 |             fprintf('Step size below progTol\n');
255 |         end
256 |         break;
257 |     end
258 | 
259 |     if abs(f-f_old) < progTol
260 |         if verbose >= 1
261 |             fprintf('Function value changing by less than progTol\n');
262 |         end
263 |         break;
264 |     end
265 | 
266 |     if funEvals*funEvalMultiplier > maxIter
267 |         if verbose >= 1
268 |             fprintf('Function Evaluations exceeds maxIter\n');
269 |         end
270 |         break;
271 |     end
272 |     
273 |     if projects > maxProject
274 |         if verbose >= 1
275 |             fprintf('Number of projections exceeds maxProject\n');
276 |         end
277 |         break;
278 |     end
279 |     
280 |     i = i + 1;
281 | %    pause
282 | end
283 | end
284 | 
285 | 
286 | function [p,subProjects] = solveSubProblem(x,g,H,funProj,optTol,progTol,maxIter,testOpt,feasibleInit,x_init)
287 | % Uses SPG to solve for projected quasi-Newton direction
288 | options.verbose = 0;
289 | options.optTol = optTol;
290 | options.progTol = progTol;
291 | options.maxIter = maxIter;
292 | options.testOpt = testOpt;
293 | options.feasibleInit = feasibleInit;
294 | 
295 | funObj = @(p)subHv(p,x,g,H);
296 | [p,f,funEvals,subProjects] = minConf_SPG(funObj,x_init,funProj,options);
297 | end
298 | 
299 | function [f,g] = subHv(p,x,g,HvFunc)
300 | d = p-x;
301 | Hd = HvFunc(d);
302 | f = g'*d + (1/2)*d'*Hd;
303 | g = g + Hd;
304 | end


--------------------------------------------------------------------------------
/backend/src/minConf/minConf_SPG.m:
--------------------------------------------------------------------------------
  1 | function [x,f,funEvals,projects] = minConf_SPG(funObj,x,funProj,options)
  2 | % function [x,f] = minConF_SPG(funObj,x,funProj,options)
  3 | %
  4 | % Function for using Spectral Projected Gradient to solve problems of the form
  5 | %   min funObj(x) s.t. x in C
  6 | %
  7 | %   @funObj(x): function to minimize (returns gradient as second argument)
  8 | %   @funProj(x): function that returns projection of x onto C
  9 | %
 10 | %   options:
 11 | %       verbose: level of verbosity (0: no output, 1: final, 2: iter (default), 3:
 12 | %       debug)
 13 | %       optTol: tolerance used to check for optimality (default: 1e-5)
 14 | %       progTol: tolerance used to check for lack of progress (default: 1e-9)
 15 | %       maxIter: maximum number of calls to funObj (default: 500)
 16 | %       numDiff: compute derivatives numerically (0: use user-supplied
 17 | %       derivatives (default), 1: use finite differences, 2: use complex
 18 | %       differentials)
 19 | %       suffDec: sufficient decrease parameter in Armijo condition (default
 20 | %       : 1e-4)
 21 | %       interp: type of interpolation (0: step-size halving, 1: quadratic,
 22 | %       2: cubic)
 23 | %       memory: number of steps to look back in non-monotone Armijo
 24 | %       condition
 25 | %       useSpectral: use spectral scaling of gradient direction (default:
 26 | %       1)
 27 | %       curvilinear: backtrack along projection Arc (default: 0)
 28 | %       testOpt: test optimality condition (default: 1)
 29 | %       feasibleInit: if 1, then the initial point is assumed to be
 30 | %       feasible
 31 | %       bbType: type of Barzilai Borwein step (default: 1)
 32 | %
 33 | %   Notes: 
 34 | %       - if the projection is expensive to compute, you can reduce the
 35 | %           number of projections by setting testOpt to 0
 36 | 
 37 | 
 38 | nVars = length(x);
 39 | 
 40 | % Set Parameters
 41 | if nargin < 4
 42 |     options = [];
 43 | end
 44 | [verbose,numDiff,optTol,progTol,maxIter,suffDec,interp,memory,useSpectral,curvilinear,feasibleInit,testOpt,bbType] = ...
 45 |     myProcessOptions(...
 46 |     options,'verbose',2,'numDiff',0,'optTol',1e-5,'progTol',1e-9,'maxIter',500,'suffDec',1e-4,...
 47 |     'interp',2,'memory',10,'useSpectral',1,'curvilinear',0,'feasibleInit',0,...
 48 |     'testOpt',1,'bbType',1);
 49 | 
 50 | % Output Log
 51 | if verbose >= 2
 52 |     if testOpt
 53 |         fprintf('%10s %10s %10s %15s %15s %15s\n','Iteration','FunEvals','Projections','Step Length','Function Val','Opt Cond');
 54 |     else
 55 |         fprintf('%10s %10s %10s %15s %15s\n','Iteration','FunEvals','Projections','Step Length','Function Val');
 56 |     end
 57 | end
 58 | 
 59 | % Make objective function (if using numerical derivatives)
 60 | funEvalMultiplier = 1;
 61 | if numDiff
 62 |     if numDiff == 2
 63 |         useComplex = 1;
 64 |     else
 65 |         useComplex = 0;
 66 |     end
 67 |     funObj = @(x)autoGrad(x,useComplex,funObj);
 68 |     funEvalMultiplier = nVars+1-useComplex;
 69 | end
 70 | 
 71 | % Evaluate Initial Point
 72 | if ~feasibleInit
 73 |     x = funProj(x);
 74 | end
 75 | [f,g] = funObj(x);
 76 | projects = 1;
 77 | funEvals = 1;
 78 | 
 79 | % Optionally check optimality
 80 | if testOpt
 81 |     projects = projects+1;
 82 |     if max(abs(funProj(x-g)-x)) < optTol
 83 |         if verbose >= 1
 84 |         fprintf('First-Order Optimality Conditions Below optTol at Initial Point\n');
 85 |         end
 86 |         return;
 87 |     end
 88 | end
 89 | 
 90 | i = 1;
 91 | while funEvals <= maxIter
 92 | 
 93 |     % Compute Step Direction
 94 |     if i == 1 || ~useSpectral
 95 |         alpha = 1;
 96 |     else
 97 |         y = g-g_old;
 98 |         s = x-x_old;
 99 |         if bbType == 1
100 |             alpha = (s'*s)/(s'*y);
101 |         else
102 |             alpha = (s'*y)/(y'*y);
103 |         end
104 |         if alpha <= 1e-10 || alpha > 1e10
105 |             alpha = 1;
106 |         end
107 |     end
108 |     d = -alpha*g;
109 |     f_old = f;
110 |     x_old = x;
111 |     g_old = g;
112 |     
113 |     % Compute Projected Step
114 |     if ~curvilinear
115 |         d = funProj(x+d)-x;
116 |         projects = projects+1;
117 |     end
118 | 
119 |     % Check that Progress can be made along the direction
120 |     gtd = g'*d;
121 |     if gtd > -progTol
122 |         if verbose >= 1
123 |             fprintf('Directional Derivative below progTol\n');
124 |         end
125 |         break;
126 |     end
127 | 
128 |     % Select Initial Guess to step length
129 |     if i == 1
130 |         t = min(1,1/sum(abs(g)));
131 |     else
132 |         t = 1;
133 |     end
134 | 
135 |     % Compute reference function for non-monotone condition
136 | 
137 |     if memory == 1
138 |         funRef = f;
139 |     else
140 |         if i == 1
141 |             old_fvals = repmat(-inf,[memory 1]);
142 |         end
143 | 
144 |         if i <= memory
145 |             old_fvals(i) = f;
146 |         else
147 |             old_fvals = [old_fvals(2:end);f];
148 |         end
149 |         funRef = max(old_fvals);
150 |     end
151 | 
152 |     % Evaluate the Objective and Gradient at the Initial Step
153 |     if curvilinear
154 |         x_new = funProj(x + t*d);
155 |         projects = projects+1;
156 |     else
157 |         x_new = x + t*d;
158 |     end
159 |     [f_new,g_new] = funObj(x_new);
160 |     funEvals = funEvals+1;
161 | 
162 |     % Backtracking Line Search
163 |     lineSearchIters = 1;
164 |     while f_new > funRef + suffDec*g'*(x_new-x) || ~isLegal(f_new)
165 |         temp = t;
166 |         if interp == 0 || ~isLegal(f_new)
167 |             if verbose == 3
168 |                 fprintf('Halving Step Size\n');
169 |             end
170 | t = t/2;
171 |         elseif interp == 2 && isLegal(g_new)
172 |             if verbose == 3
173 |                 fprintf('Cubic Backtracking\n');
174 |             end
175 |             t = polyinterp([0 f gtd; t f_new g_new'*d]);
176 |         elseif lineSearchIters < 2 || ~isLegal(f_prev)
177 |             if verbose == 3
178 |                 fprintf('Quadratic Backtracking\n');
179 |             end
180 |             t = polyinterp([0 f gtd; t f_new sqrt(-1)]);
181 |         else
182 |             if verbose == 3
183 |                 fprintf('Cubic Backtracking on Function Values\n');
184 |             end
185 |             t = polyinterp([0 f gtd; t f_new sqrt(-1);t_prev f_prev sqrt(-1)]);
186 |         end
187 | 
188 |         % Adjust if change is too small
189 |         if t < temp*1e-3
190 |             if verbose == 3
191 |                 fprintf('Interpolated value too small, Adjusting\n');
192 |             end
193 |             t = temp*1e-3;
194 |         elseif t > temp*0.6
195 |             if verbose == 3
196 |                 fprintf('Interpolated value too large, Adjusting\n');
197 |             end
198 |             t = temp*0.6;
199 |         end
200 | 
201 |         % Check whether step has become too small
202 |         if max(abs(t*d)) < progTol || t == 0
203 |             if verbose == 3
204 |                 fprintf('Line Search failed\n');
205 |             end
206 |             t = 0;
207 |             f_new = f;
208 |             g_new = g;
209 |             break;
210 |         end
211 | 
212 |         % Evaluate New Point
213 |         f_prev = f_new;
214 |         t_prev = temp;
215 |         if curvilinear
216 |             x_new = funProj(x + t*d);
217 |             projects = projects+1;
218 |         else
219 |             x_new = x + t*d;
220 |         end
221 |         [f_new,g_new] = funObj(x_new);
222 |         funEvals = funEvals+1;
223 |         lineSearchIters = lineSearchIters+1;
224 | 
225 |     end
226 | 
227 |     % Take Step
228 |     x = x_new;
229 |     f = f_new;
230 |     g = g_new;
231 | 
232 |     if testOpt
233 |         optCond = max(abs(funProj(x-g)-x));
234 |         projects = projects+1;
235 |     end
236 | 
237 |     % Output Log
238 |     if verbose >= 2
239 |         if testOpt
240 |             fprintf('%10d %10d %10d %15.5e %15.5e %15.5e\n',i,funEvals*funEvalMultiplier,projects,t,f,optCond);
241 |         else
242 |             fprintf('%10d %10d %10d %15.5e %15.5e\n',i,funEvals*funEvalMultiplier,projects,t,f);
243 |         end
244 |     end
245 | 
246 |     % Check optimality
247 |     if testOpt
248 |         if optCond < optTol
249 |             if verbose >= 1
250 |             fprintf('First-Order Optimality Conditions Below optTol\n');
251 |             end
252 |             break;
253 |         end
254 |     end
255 | 
256 |     if max(abs(t*d)) < progTol
257 |         if verbose >= 1
258 |             fprintf('Step size below progTol\n');
259 |         end
260 |         break;
261 |     end
262 | 
263 |     if abs(f-f_old) < progTol
264 |         if verbose >= 1
265 |             fprintf('Function value changing by less than progTol\n');
266 |         end
267 |         break;
268 |     end
269 | 
270 |     if funEvals*funEvalMultiplier > maxIter
271 |         if verbose >= 1
272 |             fprintf('Function Evaluations exceeds maxIter\n');
273 |         end
274 |         break;
275 |     end
276 | 
277 |     i = i + 1;
278 | end
279 | end


--------------------------------------------------------------------------------
/backend/src/minConf/minConf_TMP.m:
--------------------------------------------------------------------------------
  1 | function [x,f,funEvals] = minConf_TMP(funObj,x,LB,UB,options,ckpntFun)
  2 | % function [x,f] = minConF_BC(funObj,x,LB,UB,options)
  3 | %
  4 | % Function for using Two-Metric Projection to solve problems of the form:
  5 | %   min funObj(x)
  6 | %   s.t. LB_i <= x_i <= UB_i
  7 | %
  8 | %   @funObj(x): function to minimize (returns gradient as second argument)
  9 | %
 10 | %   options:
 11 | %       verbose: level of verbosity (0: no output, 1: final, 2: iter (default), 3:
 12 | %       debug)
 13 | %       optTol: tolerance used to check for progress (default: 1e-7)
 14 | %       maxIter: maximum number of calls to funObj (default: 250)
 15 | %       numDiff: compute derivatives numerically (0: use user-supplied
 16 | %       derivatives (default), 1: use finite differences, 2: use complex
 17 | %       differentials)
 18 | %       method: 'sd', 'lbfgs', 'newton'
 19 | 
 20 | nVars = length(x);
 21 | 
 22 | % Set Parameters
 23 | if nargin < 5
 24 |     options = [];
 25 | end
 26 | [verbose,numDiff,optTol,maxIter,suffDec,interp,method,corrections,damped] = ...
 27 |     myProcessOptions(...
 28 |     options,'verbose',3,'numDiff',0,'optTol',1e-6,'maxIter',100,'suffDec',1e-4,...
 29 |     'interp',1,'method','lbfgs','corrections',10,'damped',0);
 30 | 
 31 | % Output Log
 32 | if verbose >= 3
 33 |     fprintf('%10s %10s %15s %15s %15s\n','Iteration','FunEvals','Step Length','Function Val','Opt Cond');
 34 | end
 35 | 
 36 | % Make objective function (if using numerical derivatives)
 37 | funEvalMultiplier = 1;
 38 | if numDiff
 39 |     if numDiff == 2
 40 |         useComplex = 1;
 41 |     else
 42 |         useComplex = 0;
 43 |     end
 44 |     funObj = @(x)autoGrad(x,useComplex,funObj);
 45 |     funEvalMultiplier = nVars+1-useComplex;
 46 | end
 47 | 
 48 | % Evaluate Initial Point
 49 | x = projectBounds(x,LB,UB);
 50 | if strcmp(method,'newton')
 51 |     [f,g,H] = funObj(x);
 52 |     secondOrder = 1;
 53 | else
 54 |     [f,g] = funObj(x);
 55 |     secondOrder = 0;
 56 | end
 57 | funEvals = 1;
 58 | 
 59 | % Plot Progress
 60 | if (nargin == 6)
 61 |     [stop, dispstr] = ckpntFun(x);
 62 |     fprintf('\b    %s\n', dispstr);
 63 |     if stop, fprintf('Early termination condition(s) reached\n'); return; end
 64 | end
 65 | 
 66 | % Compute Working Set
 67 | working = ones(nVars,1);
 68 | working((x < LB+optTol*2) & g >= 0) = 0;
 69 | working((x > UB-optTol*2) & g <= 0) = 0;
 70 | working = find(working);
 71 | 
 72 | % Check Optimality
 73 | if isempty(working)
 74 |     if verbose >= 1
 75 |         fprintf('All variables are at their bound and no further progress is possible at initial point\n');
 76 |     end
 77 |     return;
 78 | elseif norm(g(working)) <= optTol
 79 |     if verbose >=1
 80 |         fprintf('All working variables satisfy optimality condition at initial point\n');
 81 |     end
 82 |     return;
 83 | end
 84 | 
 85 | if verbose >= 3
 86 |     switch method
 87 |         case 'sd'
 88 |             fprintf('Steepest Descent\n');
 89 |         case 'lbfgs'
 90 |             fprintf('L-BFGS\n');
 91 |         case 'bfgs'
 92 |             fprintf('BFGS\n');
 93 |         case 'newton'
 94 |             fprintf('Newton\n');
 95 |     end
 96 | end
 97 | 
 98 | i = 1;
 99 | while funEvals <= maxIter
100 | 
101 |     % Compute Step Direction
102 |     d = zeros(nVars,1);
103 |     switch(method)
104 |         case 'sd'
105 |             d(working) = -g(working);
106 |         case 'lbfgs'
107 |             if i == 1
108 |                 d(working) = -g(working);
109 |                 old_dirs = zeros(nVars,0);
110 |                 old_stps = zeros(nVars,0);
111 |                 Hdiag = 1;
112 |             else
113 |                 if damped
114 |                     [old_dirs,old_stps,Hdiag] = dampedUpdate(g-g_old,x-x_old,corrections,verbose==3,old_dirs,old_stps,Hdiag);
115 |                 else
116 |                     [old_dirs,old_stps,Hdiag] = lbfgsUpdate(g-g_old,x-x_old,corrections,verbose==3,old_dirs,old_stps,Hdiag);
117 |                 end
118 |                     curvSat = sum(old_dirs(working,:).*old_stps(working,:)) > 1e-10;
119 |                d(working) = lbfgs(-g(working),old_dirs(working,curvSat),old_stps(working,curvSat),Hdiag);
120 |             end
121 |             g_old = g;
122 |             x_old = x;
123 |         case 'bfgs'
124 |             if i == 1
125 |                 d(working) = -g(working);
126 |                 B = eye(nVars);
127 |             else
128 |                 y = g-g_old;
129 |                 s = x-x_old;
130 | 
131 |                 ys = y'*s;
132 | 
133 |                 if i == 2
134 |                     if ys > 1e-10
135 |                         B = ((y'*y)/(y'*s))*eye(nVars);
136 |                     end
137 |                 end
138 |                 if ys > 1e-10
139 |                     B = B + (y*y')/(y'*s) - (B*s*s'*B)/(s'*B*s);
140 |                 else
141 |                     if verbose == 2
142 |                         fprintf('Skipping Update\n');
143 |                     end
144 |                 end
145 |                 d(working) = -B(working,working)\g(working);
146 |             end
147 |             g_old = g;
148 |             x_old = x;
149 | 
150 |         case 'newton'
151 |             [R,posDef] = chol(H(working,working));
152 |             
153 |             if posDef == 0
154 |                 d(working) = -R\(R'\g(working));
155 |             else
156 |                 if verbose == 3
157 |                     fprintf('Adjusting Hessian\n');
158 |                 end
159 |                 H(working,working) = H(working,working) + eye(length(working)) * max(0,1e-12 - min(real(eig(H(working,working)))));
160 |                 d(working) = -H(working,working)\g(working);
161 |             end
162 |         otherwise
163 |             fprintf('Unrecognized Method: %s\n',method);
164 |             break;
165 |     end
166 | 
167 |     % Check that Progress can be made along the direction
168 |     f_old = f;
169 |     gtd = g'*d;
170 |     if gtd > -optTol
171 |         if verbose >= 2
172 |             fprintf('Directional Derivative below optTol\n');
173 |         end
174 |         break;
175 |     end
176 | 
177 |     % Select Initial Guess to step length
178 |     if i == 1 && ~secondOrder
179 |         t = min(1,1/sum(abs(g(working))));
180 |     else
181 |         t = 1;
182 |     end
183 | 
184 |     % Evaluate the Objective and Projected Gradient at the Initial Step
185 |     x_new = projectBounds(x+t*d,LB,UB);
186 |     if secondOrder
187 |         [f_new,g_new,H] = funObj(x_new);
188 |     else
189 |         [f_new,g_new] = funObj(x_new);
190 |     end
191 |     funEvals = funEvals+1;
192 | 
193 |     % Backtracking Line Search
194 |     lineSearchIters = 1;
195 |     while f_new > f + suffDec*g'*(x_new-x) || ~isLegal(f_new)
196 |         temp = t;
197 |         if interp == 0 || ~isLegal(f_new) || ~isLegal(g_new)
198 |             if verbose == 3
199 |                 fprintf('Halving Step Size\n');
200 |             end
201 |             t = .5*t;
202 |         else
203 |             if verbose == 3
204 |                 fprintf('Cubic Backtracking\n');
205 |             end
206 |             t = polyinterp([0 f gtd; t f_new g_new'*d]);
207 |         end
208 | 
209 |         % Adjust if change is too small
210 |         if t < temp*1e-3
211 |             if verbose == 3
212 |                 fprintf('Interpolated value too small, Adjusting\n');
213 |             end
214 |             t = temp*1e-3;
215 |         elseif t > temp*0.6
216 |             if verbose == 3
217 |                 fprintf('Interpolated value too large, Adjusting\n');
218 |             end
219 |             t = temp*0.6;
220 |         end
221 | 
222 |         % Check whether step has become too small
223 |         if sum(abs(t*d)) < optTol
224 |             if verbose == 3
225 |                 fprintf('Line Search failed\n');
226 |             end
227 |             t = 0;
228 |             f_new = f;
229 |             g_new = g;
230 |             break;
231 |         end
232 | 
233 |         % Evaluate New Point
234 |         x_new = projectBounds(x+t*d,LB,UB);
235 |         [f_new,g_new] = funObj(x_new);
236 |         funEvals = funEvals+1;
237 |         lineSearchIters = lineSearchIters+1;
238 | 
239 |     end
240 | 
241 |     % Take Step
242 |     x = x_new;
243 |     f = f_new;
244 |     g = g_new;
245 | 
246 |     % Compute Working Set
247 |     working = ones(nVars,1);
248 |     working((x < LB+optTol*2) & g >= 0) = 0;
249 |     working((x > UB-optTol*2) & g <= 0) = 0;
250 |     working = find(working);
251 | 
252 |     % Output Log
253 |     if verbose >= 2
254 |         fprintf('%10d %10d %15.5e %15.5e %15.5e\n',i,funEvals*funEvalMultiplier,t,f,sum(abs(g(working))));
255 |     end
256 |     
257 |     % Plot Progress
258 |     if (nargin == 6)
259 |         [stop, dispstr] = ckpntFun(x);
260 |         fprintf('\b    %s\n', dispstr);
261 |         if stop, fprintf('Early termination condition(s) reached\n'); break; end
262 |     end
263 | 
264 |     % Check Optimality
265 |     if isempty(working)
266 |         if verbose >= 1
267 |             fprintf('All variables are at their bound and no further progress is possible\n');
268 |         end
269 |         break;
270 |     elseif norm(g(working)) <= optTol
271 |         if verbose >=1
272 |             fprintf('All working variables satisfy optimality condition\n');
273 |         end
274 |         break;
275 |     end
276 | 
277 |     % Check for lack of progress
278 |     if sum(abs(t*d)) < optTol
279 |         if verbose >= 1
280 |             fprintf('Step size below optTol\n');
281 |         end
282 |         break;
283 |     end
284 | 
285 |     if abs(f-f_old) < optTol
286 |         if verbose >= 1
287 |             fprintf('Function value changing by less than optTol\n');
288 |         end
289 |         break;
290 |     end
291 | 
292 |     if funEvals*funEvalMultiplier > maxIter
293 |         if verbose >= 1
294 |             fprintf('Function Evaluations exceeds maxIter\n');
295 |         end
296 |         break;
297 |     end
298 | 
299 |     % If necessary, compute Hessian
300 |     if secondOrder && lineSearchIters > 1
301 |         [f_new,g_new,H] = funObj(x);
302 |     end
303 | 
304 |     i = i + 1;
305 | end
306 | end
307 | 
308 | function [x] = projectBounds(x,LB,UB)
309 | x(x < LB) = LB(x < LB);
310 | x(x > UB) = UB(x > UB);
311 | end


--------------------------------------------------------------------------------
/backend/src/minConf/myProcessOptions.m:
--------------------------------------------------------------------------------
 1 | function [varargout] = myProcessOptions(options,varargin)
 2 | % Similar to processOptions, but case insensitive and
 3 | %   using a struct instead of a variable length list
 4 | 
 5 | options = toUpper(options);
 6 | 
 7 | for i = 1:2:length(varargin)
 8 |     if isfield(options,upper(varargin{i}))
 9 |         v = getfield(options,upper(varargin{i}));
10 |         if isempty(v)
11 |             varargout{(i+1)/2}=varargin{i+1};
12 |         else
13 |             varargout{(i+1)/2}=v;
14 |         end
15 |     else
16 |         varargout{(i+1)/2}=varargin{i+1};
17 |     end
18 | end
19 | 
20 | end
21 | 
22 | function [o] = toUpper(o)
23 | if ~isempty(o)
24 |     fn = fieldnames(o);
25 |     for i = 1:length(fn)
26 |         o = setfield(o,upper(fn{i}),getfield(o,fn{i}));
27 |     end
28 | end
29 | end


--------------------------------------------------------------------------------
/backend/src/minConf/polyinterp.m:
--------------------------------------------------------------------------------
  1 | function [minPos,fmin] = polyinterp(points,doPlot,xminBound,xmaxBound)
  2 | % function [minPos] = polyinterp(points,doPlot,xminBound,xmaxBound)
  3 | %
  4 | %   Minimum of interpolating polynomial based on function and derivative
  5 | %   values
  6 | %
  7 | %   In can also be used for extrapolation if {xmin,xmax} are outside
  8 | %   the domain of the points.
  9 | %
 10 | %   Input:
 11 | %       points(pointNum,[x f g])
 12 | %       doPlot: set to 1 to plot, default: 0
 13 | %       xmin: min value that brackets minimum (default: min of points)
 14 | %       xmax: max value that brackets maximum (default: max of points)
 15 | %
 16 | %   set f or g to sqrt(-1) if they are not known
 17 | %   the order of the polynomial is the number of known f and g values minus 1
 18 | 
 19 | if nargin < 2
 20 |     doPlot = 0;
 21 | end
 22 | 
 23 | nPoints = size(points,1);
 24 | order = sum(sum((imag(points(:,2:3))==0)))-1;
 25 | 
 26 | % Code for most common case:
 27 | %   - cubic interpolation of 2 points
 28 | %       w/ function and derivative values for both
 29 | %   - no xminBound/xmaxBound
 30 | 
 31 | if nPoints == 2 && order ==3 && nargin <= 2 && doPlot == 0
 32 |     % Solution in this case (where x2 is the farthest point):
 33 |     %    d1 = g1 + g2 - 3*(f1-f2)/(x1-x2);
 34 |     %    d2 = sqrt(d1^2 - g1*g2);
 35 |     %    minPos = x2 - (x2 - x1)*((g2 + d2 - d1)/(g2 - g1 + 2*d2));
 36 |     %    t_new = min(max(minPos,x1),x2);
 37 |     [minVal minPos] = min(points(:,1));
 38 |     notMinPos = -minPos+3;
 39 |     d1 = points(minPos,3) + points(notMinPos,3) - 3*(points(minPos,2)-points(notMinPos,2))/(points(minPos,1)-points(notMinPos,1));
 40 |     d2 = sqrt(d1^2 - points(minPos,3)*points(notMinPos,3));
 41 |     if isreal(d2)
 42 |         t = points(notMinPos,1) - (points(notMinPos,1) - points(minPos,1))*((points(notMinPos,3) + d2 - d1)/(points(notMinPos,3) - points(minPos,3) + 2*d2));
 43 |         minPos = min(max(t,points(minPos,1)),points(notMinPos,1));
 44 |     else
 45 |         minPos = mean(points(:,1));
 46 |     end
 47 |     return;
 48 | end
 49 | 
 50 | xmin = min(points(:,1));
 51 | xmax = max(points(:,1));
 52 | 
 53 | % Compute Bounds of Interpolation Area
 54 | if nargin < 3
 55 |     xminBound = xmin;
 56 | end
 57 | if nargin < 4
 58 |     xmaxBound = xmax;
 59 | end
 60 | 
 61 | % Constraints Based on available Function Values
 62 | A = zeros(0,order+1);
 63 | b = zeros(0,1);
 64 | for i = 1:nPoints
 65 |     if imag(points(i,2))==0
 66 |         constraint = zeros(1,order+1);
 67 |         for j = order:-1:0
 68 |             constraint(order-j+1) = points(i,1)^j;
 69 |         end
 70 |         A = [A;constraint];
 71 |         b = [b;points(i,2)];
 72 |     end
 73 | end
 74 | 
 75 | % Constraints based on available Derivatives
 76 | for i = 1:nPoints
 77 |     if isreal(points(i,3))
 78 |         constraint = zeros(1,order+1);
 79 |         for j = 1:order
 80 |             constraint(j) = (order-j+1)*points(i,1)^(order-j);
 81 |         end
 82 |         A = [A;constraint];
 83 |         b = [b;points(i,3)];
 84 |     end
 85 | end
 86 | 
 87 | % Find interpolating polynomial
 88 | params = A\b;
 89 | 
 90 | % Compute Critical Points
 91 | dParams = zeros(order,1);
 92 | for i = 1:length(params)-1
 93 |     dParams(i) = params(i)*(order-i+1);
 94 | end
 95 | 
 96 | if any(isinf(dParams))
 97 |     cp = [xminBound;xmaxBound;points(:,1)].';
 98 | else
 99 |     cp = [xminBound;xmaxBound;points(:,1);roots(dParams)].';
100 | end
101 | 
102 | % Test Critical Points
103 | fmin = inf;
104 | minPos = (xminBound+xmaxBound)/2; % Default to Bisection if no critical points valid
105 | for xCP = cp
106 |     if imag(xCP)==0 && xCP >= xminBound && xCP <= xmaxBound
107 |         fCP = polyval(params,xCP);
108 |         if imag(fCP)==0 && fCP < fmin
109 |             minPos = real(xCP);
110 |             fmin = real(fCP);
111 |         end
112 |     end
113 | end
114 | % Plot Situation
115 | if doPlot
116 |     figure(1); clf; hold on;
117 | 
118 |     % Plot Points
119 |     plot(points(:,1),points(:,2),'b*');
120 | 
121 |     % Plot Derivatives
122 |     for i = 1:nPoints
123 |         if isreal(points(i,3))
124 |             m = points(i,3);
125 |             b = points(i,2) - m*points(i,1);
126 |             plot([points(i,1)-.05 points(i,1)+.05],...
127 |                 [(points(i,1)-.05)*m+b (points(i,1)+.05)*m+b],'c.-');
128 |         end
129 |     end
130 | 
131 |     % Plot Function
132 |     x = min(xmin,xminBound)-.1:(max(xmax,xmaxBound)+.1-min(xmin,xminBound)-.1)/100:max(xmax,xmaxBound)+.1;
133 |     size(x)
134 |     for i = 1:length(x)
135 |         f(i) = polyval(params,x(i));
136 |     end
137 |     plot(x,f,'y');
138 |     axis([x(1)-.1 x(end)+.1 min(f)-.1 max(f)+.1]);
139 | 
140 |     % Plot Minimum
141 |     plot(minPos,fmin,'g+');
142 |     if doPlot == 1
143 |         pause(1);
144 |     end
145 | end


--------------------------------------------------------------------------------
/backend/src/ostrichinator_lite.m:
--------------------------------------------------------------------------------
  1 | function [xmin, xsal, flag, xhist, fhist, chist] = ostrichinator_lite(net, img, tclass, allwgpu, maxiter, faststp, stpthrs)
  2 | 
  3 | % ----------
  4 | % Copyright (c) 2015, Chuan-Yung Tsai
  5 | % All rights reserved.
  6 | % 
  7 | % Redistribution and use in source and binary forms, with or without
  8 | % modification, are permitted provided that the following conditions are met:
  9 | % 
 10 | % 1. Redistributions of source code must retain the above copyright notice, this
 11 | %    list of conditions and the following disclaimer. 
 12 | % 2. Redistributions in binary form must reproduce the above copyright notice,
 13 | %    this list of conditions and the following disclaimer in the documentation
 14 | %    and/or other materials provided with the distribution.
 15 | %
 16 | % THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 17 | % ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 18 | % WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 19 | % DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
 20 | % ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 21 | % (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 22 | % LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
 23 | % ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 24 | % (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 25 | % SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 26 | % 
 27 | % The views and conclusions contained in the software and documentation are those
 28 | % of the authors and should not be interpreted as representing official policies, 
 29 | % either expressed or implied, of the FreeBSD Project.
 30 | % ----------
 31 | 
 32 | % PARAMETERS
 33 | 
 34 | dispimg = 0;
 35 | erlystp = 1;
 36 | disthrs = 5;
 37 | bfgshis = 10;
 38 | enbalnc = 1;
 39 | fulhist = 0;
 40 | 
 41 | if dispimg, clf; subplot(2,2,1); image(img/255); end
 42 | if faststp, stpthrs = 1.0; end
 43 | 
 44 | stpcond = @(x)all(x >= stpthrs);
 45 | rndmult = 16;
 46 | 
 47 | % SHARED VARS
 48 | 
 49 | fevalnum = 0;
 50 | res = []; sal = [];
 51 | xhist = []; fhist = []; chist = [];
 52 | 
 53 | % SETUP NETWORKS & TARGET FUNCTION
 54 | 
 55 | for n = 1:numel(net)
 56 |     net{n}.layers(end) = []; 
 57 |     if allwgpu, net{n} = vl_simplenn_move(net{n}, 'gpu'); end
 58 | end
 59 | 
 60 | xshp = [227 227 3];
 61 | tfuno = @(x)vl_simplenn_opt(net, reshape(x,xshp), tclass, false); % original
 62 | if faststp && enbalnc, tfunn = @(x)vl_simplenn_opt(net, reshape(x,xshp), tclass, numel(net) > 1); % normalized
 63 | else tfunn = tfuno; end
 64 | 
 65 | img = img - 128;
 66 | 
 67 | % EARLY TERM.
 68 | 
 69 | tfuno(img);
 70 | [iprogress, iclass] = cal_progress(res, tclass); chist = [chist {iclass(:)}];
 71 | if fulhist, xhist = [xhist {img}]; fhist = [fhist {res}]; end
 72 | 
 73 | flag = stpcond(iprogress);
 74 | if (flag || (maxiter==1)), xmin = img + 128; xsal = sal; return; end
 75 | 
 76 | % RUN SOLVER
 77 | 
 78 | caldis = @(x)norm(x(:)-img(:))/sqrt(prod(xshp));
 79 | 
 80 | lb = -128; ub = 127;
 81 | clip = @(x)min(max(x,lb),ub);
 82 | xini = img; xini = clip(xini(:));
 83 | 
 84 | lb = repmat(lb, size(xini));
 85 | ub = repmat(ub, size(xini));
 86 | 
 87 | rng(0);
 88 | fstphse = 1;
 89 | rderfix = 0;
 90 | rdclean = 0;
 91 | 
 92 | while (fevalnum < maxiter)
 93 |     if ~rderfix
 94 |         if fstphse
 95 |             fprintf('-------------------- First Phase (OPT) --------------------\n');
 96 |             
 97 |             options.maxIter = maxiter - fevalnum; options.corrections = bfgshis;
 98 |             [xmin, ~, fevaltmp] = minConf_TMP(tfunn,xini,lb,ub,options,@plot_progress);
 99 |             fevalnum = fevalnum + fevaltmp;
100 |             
101 |             if stpcond(cal_progress(res, tclass))
102 |                 if rdclean, fstphse = 0; if (caldis(xmin)<disthrs), break; end
103 |                 else rderfix = 1; end
104 |             else
105 |                 xini = img + rndmult*randn(xshp); xini = clip(xini(:));
106 |             end
107 |         else
108 |             fprintf('-------------------- Second Phase (SHR) --------------------\n');
109 |             pdis = norm(xmin - img(:)) / sqrt(2); % SHRINKING
110 |             pfun = @(x)clip(img(:) + min(pdis,norm(x-img(:)))*(x-img(:))/max(norm(x-img(:)),eps));
111 |             xini = pfun(xmin);
112 |             
113 |             options.maxIter = maxiter - fevalnum; options.corrections = bfgshis;
114 |             [xmintmp, ~, fevaltmp] = minConf_PQN(tfunn,xini,pfun,options,@plot_progress);
115 |             fevalnum = fevalnum + fevaltmp;
116 |             
117 |             if stpcond(cal_progress(res, tclass))
118 |                 if rdclean, xmin = xmintmp; if (caldis(xmin)<disthrs), break; end
119 |                 else rderfix = 1; end
120 |             else
121 |                 fprintf('Cannot improve more\n'); break;
122 |             end
123 |         end
124 |     else
125 |         fprintf('-------------------- Fixing Rounding Issue --------------------\n');
126 |         
127 |         rderfix = 0;
128 |         
129 |         if fstphse, xini = xmin;
130 |         else xini = xmintmp; end
131 |         
132 |         pdis = norm(xini - img(:)); % MAINTAINING
133 |         pfun = @(x)clip(img(:) + min(pdis,norm(x-img(:)))*(x-img(:))/max(norm(x-img(:)),eps));
134 |         
135 |         options.maxIter = maxiter - fevalnum; options.corrections = bfgshis;
136 |         [xmintmp, ~, fevaltmp] = minConf_PQN(tfuno,xini,pfun,options,@plot_progress);
137 |         fevalnum = fevalnum + fevaltmp;
138 |         
139 |         if rdclean
140 |             fstphse = 0;
141 |             xmin = xmintmp;
142 |         else
143 |             if fstphse, xini = img + rndmult*randn(xshp); xini = clip(xini(:));
144 |             else fprintf('Cannot fix rounding issue in 2nd phase\n'); break; end
145 |         end
146 |     end
147 |     
148 |     fprintf('-------------------- FunEvals: %s --------------------\n', num2str(fevalnum));
149 | end
150 | 
151 | % FINAL CHECK & OUTPUT
152 | 
153 | xmin = clip(round(xmin));
154 | fprintf('Final distortion: %s\n', num2str(caldis(xmin),'%.2e'));
155 | 
156 | tfuno(xmin);
157 | [fprogress, fclass] = cal_progress(res, tclass); chist = [chist {fclass(:)}];
158 | if fulhist, xhist = [xhist {xmin}]; fhist = [fhist {res}]; end
159 | flag = stpcond(fprogress);
160 | 
161 | xmin = reshape(xmin + 128, xshp);
162 | xsal = sal;
163 | 
164 | % -------------------------------------------------------------------------
165 |     
166 |     function y = ii(x,i), y = ((1:numel(x)) == i)'; end
167 |     function y = mi(x), [~,y] = max(x(:)); end
168 | 
169 |     function [f, g] = vl_simplenn_opt(net, x, ch, nrml)
170 | 
171 |         if ~allwgpu, x = single(x); mcast = @(x)single(x);
172 |         else x = gpuArray(single(x)); mcast = @(x)gpuArray(single(x)); end
173 |         
174 |         f = 0;
175 |         if ~isempty(ch), g = zeros(size(x)); else g = []; end
176 |         
177 |         [res, sal] = deal(cell(size(net)));
178 |         
179 |         for i = 1:numel(net)
180 |             if ~isempty(ch)
181 |                 fch = ch{i}(1);
182 |                 if ~nrml
183 |                     tgt = @(x)x(fch); der = @(x)mcast(ii(x,fch));
184 |                 else
185 |                     tgt = @(x)x(fch)-max(x(:)); der = @(x)mcast(ii(x,fch)-ii(x,mi(x)));
186 |                 end
187 |             else
188 |                 fch = 1;
189 |                 tgt = @(x)x(fch); der = [];
190 |             end
191 |             
192 |             res{i} = vl_simplenn_fast(net{i}, x, der);
193 |             
194 |             sal{i} = double(gather(res{i}(1).dzdx));
195 |             res{i} = double(gather(res{i}(end).x(:)));
196 |             
197 |             f = f + tgt(res{i});
198 |             g = g + sum(sal{i}, 4);
199 |         end
200 |         
201 |         f = -f; 
202 |         g = -g(:);
203 |     end
204 |     
205 |     function [stop, dispstr] = plot_progress(x)
206 |         
207 |         if dispimg
208 |             dff = reshape(x,xshp)-img; dff = dff-min(dff(:)); dff = dff/max(dff(:));
209 |             
210 |             subplot(2,2,2); image(dff);
211 |             subplot(2,2,3); image((reshape(x,xshp)+128)/255);
212 |             subplot(2,2,4); imagesc(sum(abs(sum(cat(4,sal{:}),4)),3));
213 | 
214 |             drawnow;
215 |         end
216 |         
217 |         stop = false;
218 |         rdclean = 0;
219 |         
220 |         [cprogress, cclass, pscores] = cal_progress(res, tclass); chist = [chist {cclass(:)}];        
221 |         if fulhist, xhist = [xhist {x}]; fhist = [fhist {res}]; end
222 |             
223 |         % STOP CONDITIONING
224 |         if erlystp
225 |             if stpcond(cprogress)
226 |                 restmp = res;
227 |                 % CHECK RESULTS AFTER ROUNDING, NO BACKWARD PASS HERE
228 | 		vl_simplenn_opt(net, clip(round(reshape(x,xshp))), [], 0);
229 |                 fevalnum = fevalnum + 0.5;
230 |                 stop = stpcond(cal_progress(res, tclass));
231 |                 
232 |                 res = restmp;
233 |                 if stop, rdclean = 1; end
234 |             end
235 |         end
236 |         
237 |         dispstr = num2str(cprogress, '%.2e/');
238 |         dispstr = ['Progress: ' strtrim(dispstr(1:end-1)) ' Distortion: ' num2str(caldis(x),'%.2e')];
239 |         if erlystp && stpcond(cprogress) && ~rdclean, dispstr = [dispstr ' Rounding Issue!']; end
240 |     end
241 | 
242 |     function [progress, cc, pscores] = cal_progress(scores, tc)
243 |               
244 |         [progress, cc, pscores] = deal(zeros(1, numel(scores)));
245 |         
246 |         for i = 1:numel(scores)
247 |             [~,cc(i)] = max(scores{i});
248 |             pscores(i) = scores{i}(tc{i}(1));
249 |             
250 |             if faststp
251 | 		scores{i} = exp(scores{i} - max(scores{i}));
252 |                 progress(i) = scores{i}(tc{i}(1));
253 |             else
254 |                 % SOFTMAX
255 |                 scores{i} = exp(scores{i} - max(scores{i}));
256 |                 scores{i} = scores{i} ./ sum(scores{i});
257 | 
258 |                 progress(i) = scores{i}(tc{i}(1));
259 |             end
260 |         end
261 |     end
262 | 
263 | end
264 | 
265 | 


--------------------------------------------------------------------------------
/backend/src/vl_simplenn_fast.m:
--------------------------------------------------------------------------------
  1 | function res = vl_simplenn_fast(net, x, dzdy, res, varargin)
  2 | 
  3 | % Copyright (C) 2014 Andrea Vedaldi.
  4 | % All rights reserved.
  5 | %
  6 | % This file is part of the VLFeat library and is made available under
  7 | % the terms of the BSD license (see the COPYING file).
  8 | 
  9 | opts.conserveMemory = true ;
 10 | opts.sync = false ;
 11 | opts.disableDropout = false ;
 12 | opts.freezeDropout = false ;
 13 | 
 14 | n = numel(net.layers) ;
 15 | 
 16 | if (nargin <= 2) || isempty(dzdy)
 17 |   doder = false ;
 18 | else
 19 |   doder = true ;
 20 | end
 21 | 
 22 | gpuMode = isa(x, 'gpuArray') ;
 23 | 
 24 | if nargin <= 3 || isempty(res)
 25 |   res = struct(...
 26 |     'x', cell(1,n+1), ...
 27 |     'dzdx', cell(1,n+1)) ;
 28 | end
 29 | res(1).x = x ;
 30 | 
 31 | for i=1:n
 32 |   l = net.layers{i} ;
 33 |   switch l.type
 34 |     case 'conv'
 35 |       res(i+1).x = vl_nnconv(res(i).x, l.filters, l.biases, 'pad', l.pad, 'stride', l.stride) ;
 36 |     case 'pool'
 37 |       res(i+1).x = vl_nnpool(res(i).x, l.pool, 'pad', l.pad, 'stride', l.stride, 'method', l.method) ;
 38 |     case 'normalize'
 39 |       res(i+1).x = vl_nnnormalize(res(i).x, l.param) ;
 40 |     case 'softmax'
 41 |       res(i+1).x = vl_nnsoftmax(res(i).x) ;
 42 |     case 'loss'
 43 |       res(i+1).x = vl_nnloss(res(i).x, l.class) ;
 44 |     case 'softmaxloss'
 45 |       res(i+1).x = vl_nnsoftmaxloss(res(i).x, l.class) ;
 46 |     case 'relu'
 47 |       res(i+1).x = vl_nnrelu(res(i).x) ;
 48 |     case 'noffset'
 49 |       res(i+1).x = vl_nnnoffset(res(i).x, l.param) ;
 50 |     case 'dropout'
 51 |       if opts.disableDropout
 52 |         res(i+1).x = res(i).x ;
 53 |       elseif opts.freezeDropout
 54 |         [res(i+1).x, res(i+1).aux] = vl_nndropout(res(i).x, 'rate', l.rate, 'mask', res(i+1).aux) ;
 55 |       else
 56 |         [res(i+1).x, res(i+1).aux] = vl_nndropout(res(i).x, 'rate', l.rate) ;
 57 |       end
 58 |     case 'custom'
 59 |       res(i+1) = l.forward(l, res(i), res(i+1)) ;
 60 |     otherwise
 61 |       error('Unknown layer type %s', l.type) ;
 62 |   end
 63 |   if opts.conserveMemory & ~doder %% & i < numel(net.layers) - 1
 64 |     % TODO: forget unnecesary intermediate computations even when
 65 |     % derivatives are required
 66 |     res(i).x = [] ;
 67 |   end
 68 |   if gpuMode & opts.sync
 69 |     % This should make things slower, but on MATLAB 2014a it is necessary
 70 |     % for any decent performance.
 71 |     wait(gpuDevice) ;
 72 |   end
 73 | end
 74 | 
 75 | if doder
 76 |   if isa(dzdy, 'function_handle')
 77 |     res(n+1).dzdx = dzdy(res(end).x(:));
 78 |     res(n+1).dzdx = reshape(res(n+1).dzdx, [1 1 size(res(n+1).dzdx)]) ;
 79 |   else
 80 |     res(n+1).dzdx = dzdy ;
 81 |   end
 82 |   for i=n:-1:1
 83 |     l = net.layers{i} ;
 84 |     
 85 |     if (size(res(i+1).dzdx,4) > size(res(i).x,4))
 86 |         res(i).x = repmat(res(i).x(:,:,:,1), [1 1 1 size(res(i+1).dzdx,4)]);
 87 |     end
 88 |     
 89 |     switch l.type
 90 |       case 'conv'
 91 |         [res(i).dzdx, res(i).dzdw{1}, res(i).dzdw{2}] = ...
 92 |             vl_nnconv(res(i).x, l.filters, l.biases, ...
 93 |                       res(i+1).dzdx, ...
 94 |                       'pad', l.pad, 'stride', l.stride, 'NoDerFilters', 'NoDerBiases') ;
 95 |       case 'pool'
 96 |         res(i).dzdx = vl_nnpool(res(i).x, l.pool, res(i+1).dzdx, ...
 97 |           'pad', l.pad, 'stride', l.stride, 'method', l.method) ;
 98 |       case 'normalize'
 99 |         res(i).dzdx = vl_nnnormalize(res(i).x, l.param, res(i+1).dzdx) ;
100 |       case 'softmax'
101 |         res(i).dzdx = vl_nnsoftmax(res(i).x, res(i+1).dzdx) ;
102 |       case 'loss'
103 |         res(i).dzdx = vl_nnloss(res(i).x, l.class, res(i+1).dzdx) ;
104 |       case 'softmaxloss'
105 |         res(i).dzdx = vl_nnsoftmaxloss(res(i).x, l.class, res(i+1).dzdx) ;
106 |       case 'relu'
107 |         res(i).dzdx = vl_nnrelu(res(i).x, res(i+1).dzdx) ;
108 |       case 'noffset'
109 |         res(i).dzdx = vl_nnnoffset(res(i).x, l.param, res(i+1).dzdx) ;
110 |       case 'dropout'
111 |         if opts.disableDropout
112 |           res(i).dzdx = res(i+1).dzdx ;
113 |         else
114 |           res(i).dzdx = vl_nndropout(res(i).x, res(i+1).dzdx, 'mask', res(i+1).aux) ;
115 |         end
116 |       case 'custom'
117 |         res(i) = l.backward(l, res(i), res(i+1)) ;
118 |     end
119 |     if opts.conserveMemory
120 |       res(i+1).dzdx = [] ;
121 |       res(i).x = [];
122 |     end
123 |     if gpuMode & opts.sync
124 |       wait(gpuDevice) ;
125 |     end
126 |   end
127 | end
128 | 


--------------------------------------------------------------------------------
/backend/synset_words.txt:
--------------------------------------------------------------------------------
   1 | n01440764 tench, Tinca tinca
   2 | n01443537 goldfish, Carassius auratus
   3 | n01484850 great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias
   4 | n01491361 tiger shark, Galeocerdo cuvieri
   5 | n01494475 hammerhead, hammerhead shark
   6 | n01496331 electric ray, crampfish, numbfish, torpedo
   7 | n01498041 stingray
   8 | n01514668 cock
   9 | n01514859 hen
  10 | n01518878 ostrich, Struthio camelus
  11 | n01530575 brambling, Fringilla montifringilla
  12 | n01531178 goldfinch, Carduelis carduelis
  13 | n01532829 house finch, linnet, Carpodacus mexicanus
  14 | n01534433 junco, snowbird
  15 | n01537544 indigo bunting, indigo finch, indigo bird, Passerina cyanea
  16 | n01558993 robin, American robin, Turdus migratorius
  17 | n01560419 bulbul
  18 | n01580077 jay
  19 | n01582220 magpie
  20 | n01592084 chickadee
  21 | n01601694 water ouzel, dipper
  22 | n01608432 kite
  23 | n01614925 bald eagle, American eagle, Haliaeetus leucocephalus
  24 | n01616318 vulture
  25 | n01622779 great grey owl, great gray owl, Strix nebulosa
  26 | n01629819 European fire salamander, Salamandra salamandra
  27 | n01630670 common newt, Triturus vulgaris
  28 | n01631663 eft
  29 | n01632458 spotted salamander, Ambystoma maculatum
  30 | n01632777 axolotl, mud puppy, Ambystoma mexicanum
  31 | n01641577 bullfrog, Rana catesbeiana
  32 | n01644373 tree frog, tree-frog
  33 | n01644900 tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui
  34 | n01664065 loggerhead, loggerhead turtle, Caretta caretta
  35 | n01665541 leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea
  36 | n01667114 mud turtle
  37 | n01667778 terrapin
  38 | n01669191 box turtle, box tortoise
  39 | n01675722 banded gecko
  40 | n01677366 common iguana, iguana, Iguana iguana
  41 | n01682714 American chameleon, anole, Anolis carolinensis
  42 | n01685808 whiptail, whiptail lizard
  43 | n01687978 agama
  44 | n01688243 frilled lizard, Chlamydosaurus kingi
  45 | n01689811 alligator lizard
  46 | n01692333 Gila monster, Heloderma suspectum
  47 | n01693334 green lizard, Lacerta viridis
  48 | n01694178 African chameleon, Chamaeleo chamaeleon
  49 | n01695060 Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis
  50 | n01697457 African crocodile, Nile crocodile, Crocodylus niloticus
  51 | n01698640 American alligator, Alligator mississipiensis
  52 | n01704323 triceratops
  53 | n01728572 thunder snake, worm snake, Carphophis amoenus
  54 | n01728920 ringneck snake, ring-necked snake, ring snake
  55 | n01729322 hognose snake, puff adder, sand viper
  56 | n01729977 green snake, grass snake
  57 | n01734418 king snake, kingsnake
  58 | n01735189 garter snake, grass snake
  59 | n01737021 water snake
  60 | n01739381 vine snake
  61 | n01740131 night snake, Hypsiglena torquata
  62 | n01742172 boa constrictor, Constrictor constrictor
  63 | n01744401 rock python, rock snake, Python sebae
  64 | n01748264 Indian cobra, Naja naja
  65 | n01749939 green mamba
  66 | n01751748 sea snake
  67 | n01753488 horned viper, cerastes, sand viper, horned asp, Cerastes cornutus
  68 | n01755581 diamondback, diamondback rattlesnake, Crotalus adamanteus
  69 | n01756291 sidewinder, horned rattlesnake, Crotalus cerastes
  70 | n01768244 trilobite
  71 | n01770081 harvestman, daddy longlegs, Phalangium opilio
  72 | n01770393 scorpion
  73 | n01773157 black and gold garden spider, Argiope aurantia
  74 | n01773549 barn spider, Araneus cavaticus
  75 | n01773797 garden spider, Aranea diademata
  76 | n01774384 black widow, Latrodectus mactans
  77 | n01774750 tarantula
  78 | n01775062 wolf spider, hunting spider
  79 | n01776313 tick
  80 | n01784675 centipede
  81 | n01795545 black grouse
  82 | n01796340 ptarmigan
  83 | n01797886 ruffed grouse, partridge, Bonasa umbellus
  84 | n01798484 prairie chicken, prairie grouse, prairie fowl
  85 | n01806143 peacock
  86 | n01806567 quail
  87 | n01807496 partridge
  88 | n01817953 African grey, African gray, Psittacus erithacus
  89 | n01818515 macaw
  90 | n01819313 sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita
  91 | n01820546 lorikeet
  92 | n01824575 coucal
  93 | n01828970 bee eater
  94 | n01829413 hornbill
  95 | n01833805 hummingbird
  96 | n01843065 jacamar
  97 | n01843383 toucan
  98 | n01847000 drake
  99 | n01855032 red-breasted merganser, Mergus serrator
 100 | n01855672 goose
 101 | n01860187 black swan, Cygnus atratus
 102 | n01871265 tusker
 103 | n01872401 echidna, spiny anteater, anteater
 104 | n01873310 platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus
 105 | n01877812 wallaby, brush kangaroo
 106 | n01882714 koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus
 107 | n01883070 wombat
 108 | n01910747 jellyfish
 109 | n01914609 sea anemone, anemone
 110 | n01917289 brain coral
 111 | n01924916 flatworm, platyhelminth
 112 | n01930112 nematode, nematode worm, roundworm
 113 | n01943899 conch
 114 | n01944390 snail
 115 | n01945685 slug
 116 | n01950731 sea slug, nudibranch
 117 | n01955084 chiton, coat-of-mail shell, sea cradle, polyplacophore
 118 | n01968897 chambered nautilus, pearly nautilus, nautilus
 119 | n01978287 Dungeness crab, Cancer magister
 120 | n01978455 rock crab, Cancer irroratus
 121 | n01980166 fiddler crab
 122 | n01981276 king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica
 123 | n01983481 American lobster, Northern lobster, Maine lobster, Homarus americanus
 124 | n01984695 spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish
 125 | n01985128 crayfish, crawfish, crawdad, crawdaddy
 126 | n01986214 hermit crab
 127 | n01990800 isopod
 128 | n02002556 white stork, Ciconia ciconia
 129 | n02002724 black stork, Ciconia nigra
 130 | n02006656 spoonbill
 131 | n02007558 flamingo
 132 | n02009229 little blue heron, Egretta caerulea
 133 | n02009912 American egret, great white heron, Egretta albus
 134 | n02011460 bittern
 135 | n02012849 crane
 136 | n02013706 limpkin, Aramus pictus
 137 | n02017213 European gallinule, Porphyrio porphyrio
 138 | n02018207 American coot, marsh hen, mud hen, water hen, Fulica americana
 139 | n02018795 bustard
 140 | n02025239 ruddy turnstone, Arenaria interpres
 141 | n02027492 red-backed sandpiper, dunlin, Erolia alpina
 142 | n02028035 redshank, Tringa totanus
 143 | n02033041 dowitcher
 144 | n02037110 oystercatcher, oyster catcher
 145 | n02051845 pelican
 146 | n02056570 king penguin, Aptenodytes patagonica
 147 | n02058221 albatross, mollymawk
 148 | n02066245 grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus
 149 | n02071294 killer whale, killer, orca, grampus, sea wolf, Orcinus orca
 150 | n02074367 dugong, Dugong dugon
 151 | n02077923 sea lion
 152 | n02085620 Chihuahua
 153 | n02085782 Japanese spaniel
 154 | n02085936 Maltese dog, Maltese terrier, Maltese
 155 | n02086079 Pekinese, Pekingese, Peke
 156 | n02086240 Shih-Tzu
 157 | n02086646 Blenheim spaniel
 158 | n02086910 papillon
 159 | n02087046 toy terrier
 160 | n02087394 Rhodesian ridgeback
 161 | n02088094 Afghan hound, Afghan
 162 | n02088238 basset, basset hound
 163 | n02088364 beagle
 164 | n02088466 bloodhound, sleuthhound
 165 | n02088632 bluetick
 166 | n02089078 black-and-tan coonhound
 167 | n02089867 Walker hound, Walker foxhound
 168 | n02089973 English foxhound
 169 | n02090379 redbone
 170 | n02090622 borzoi, Russian wolfhound
 171 | n02090721 Irish wolfhound
 172 | n02091032 Italian greyhound
 173 | n02091134 whippet
 174 | n02091244 Ibizan hound, Ibizan Podenco
 175 | n02091467 Norwegian elkhound, elkhound
 176 | n02091635 otterhound, otter hound
 177 | n02091831 Saluki, gazelle hound
 178 | n02092002 Scottish deerhound, deerhound
 179 | n02092339 Weimaraner
 180 | n02093256 Staffordshire bullterrier, Staffordshire bull terrier
 181 | n02093428 American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier
 182 | n02093647 Bedlington terrier
 183 | n02093754 Border terrier
 184 | n02093859 Kerry blue terrier
 185 | n02093991 Irish terrier
 186 | n02094114 Norfolk terrier
 187 | n02094258 Norwich terrier
 188 | n02094433 Yorkshire terrier
 189 | n02095314 wire-haired fox terrier
 190 | n02095570 Lakeland terrier
 191 | n02095889 Sealyham terrier, Sealyham
 192 | n02096051 Airedale, Airedale terrier
 193 | n02096177 cairn, cairn terrier
 194 | n02096294 Australian terrier
 195 | n02096437 Dandie Dinmont, Dandie Dinmont terrier
 196 | n02096585 Boston bull, Boston terrier
 197 | n02097047 miniature schnauzer
 198 | n02097130 giant schnauzer
 199 | n02097209 standard schnauzer
 200 | n02097298 Scotch terrier, Scottish terrier, Scottie
 201 | n02097474 Tibetan terrier, chrysanthemum dog
 202 | n02097658 silky terrier, Sydney silky
 203 | n02098105 soft-coated wheaten terrier
 204 | n02098286 West Highland white terrier
 205 | n02098413 Lhasa, Lhasa apso
 206 | n02099267 flat-coated retriever
 207 | n02099429 curly-coated retriever
 208 | n02099601 golden retriever
 209 | n02099712 Labrador retriever
 210 | n02099849 Chesapeake Bay retriever
 211 | n02100236 German short-haired pointer
 212 | n02100583 vizsla, Hungarian pointer
 213 | n02100735 English setter
 214 | n02100877 Irish setter, red setter
 215 | n02101006 Gordon setter
 216 | n02101388 Brittany spaniel
 217 | n02101556 clumber, clumber spaniel
 218 | n02102040 English springer, English springer spaniel
 219 | n02102177 Welsh springer spaniel
 220 | n02102318 cocker spaniel, English cocker spaniel, cocker
 221 | n02102480 Sussex spaniel
 222 | n02102973 Irish water spaniel
 223 | n02104029 kuvasz
 224 | n02104365 schipperke
 225 | n02105056 groenendael
 226 | n02105162 malinois
 227 | n02105251 briard
 228 | n02105412 kelpie
 229 | n02105505 komondor
 230 | n02105641 Old English sheepdog, bobtail
 231 | n02105855 Shetland sheepdog, Shetland sheep dog, Shetland
 232 | n02106030 collie
 233 | n02106166 Border collie
 234 | n02106382 Bouvier des Flandres, Bouviers des Flandres
 235 | n02106550 Rottweiler
 236 | n02106662 German shepherd, German shepherd dog, German police dog, alsatian
 237 | n02107142 Doberman, Doberman pinscher
 238 | n02107312 miniature pinscher
 239 | n02107574 Greater Swiss Mountain dog
 240 | n02107683 Bernese mountain dog
 241 | n02107908 Appenzeller
 242 | n02108000 EntleBucher
 243 | n02108089 boxer
 244 | n02108422 bull mastiff
 245 | n02108551 Tibetan mastiff
 246 | n02108915 French bulldog
 247 | n02109047 Great Dane
 248 | n02109525 Saint Bernard, St Bernard
 249 | n02109961 Eskimo dog, husky
 250 | n02110063 malamute, malemute, Alaskan malamute
 251 | n02110185 Siberian husky
 252 | n02110341 dalmatian, coach dog, carriage dog
 253 | n02110627 affenpinscher, monkey pinscher, monkey dog
 254 | n02110806 basenji
 255 | n02110958 pug, pug-dog
 256 | n02111129 Leonberg
 257 | n02111277 Newfoundland, Newfoundland dog
 258 | n02111500 Great Pyrenees
 259 | n02111889 Samoyed, Samoyede
 260 | n02112018 Pomeranian
 261 | n02112137 chow, chow chow
 262 | n02112350 keeshond
 263 | n02112706 Brabancon griffon
 264 | n02113023 Pembroke, Pembroke Welsh corgi
 265 | n02113186 Cardigan, Cardigan Welsh corgi
 266 | n02113624 toy poodle
 267 | n02113712 miniature poodle
 268 | n02113799 standard poodle
 269 | n02113978 Mexican hairless
 270 | n02114367 timber wolf, grey wolf, gray wolf, Canis lupus
 271 | n02114548 white wolf, Arctic wolf, Canis lupus tundrarum
 272 | n02114712 red wolf, maned wolf, Canis rufus, Canis niger
 273 | n02114855 coyote, prairie wolf, brush wolf, Canis latrans
 274 | n02115641 dingo, warrigal, warragal, Canis dingo
 275 | n02115913 dhole, Cuon alpinus
 276 | n02116738 African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus
 277 | n02117135 hyena, hyaena
 278 | n02119022 red fox, Vulpes vulpes
 279 | n02119789 kit fox, Vulpes macrotis
 280 | n02120079 Arctic fox, white fox, Alopex lagopus
 281 | n02120505 grey fox, gray fox, Urocyon cinereoargenteus
 282 | n02123045 tabby, tabby cat
 283 | n02123159 tiger cat
 284 | n02123394 Persian cat
 285 | n02123597 Siamese cat, Siamese
 286 | n02124075 Egyptian cat
 287 | n02125311 cougar, puma, catamount, mountain lion, painter, panther, Felis concolor
 288 | n02127052 lynx, catamount
 289 | n02128385 leopard, Panthera pardus
 290 | n02128757 snow leopard, ounce, Panthera uncia
 291 | n02128925 jaguar, panther, Panthera onca, Felis onca
 292 | n02129165 lion, king of beasts, Panthera leo
 293 | n02129604 tiger, Panthera tigris
 294 | n02130308 cheetah, chetah, Acinonyx jubatus
 295 | n02132136 brown bear, bruin, Ursus arctos
 296 | n02133161 American black bear, black bear, Ursus americanus, Euarctos americanus
 297 | n02134084 ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus
 298 | n02134418 sloth bear, Melursus ursinus, Ursus ursinus
 299 | n02137549 mongoose
 300 | n02138441 meerkat, mierkat
 301 | n02165105 tiger beetle
 302 | n02165456 ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle
 303 | n02167151 ground beetle, carabid beetle
 304 | n02168699 long-horned beetle, longicorn, longicorn beetle
 305 | n02169497 leaf beetle, chrysomelid
 306 | n02172182 dung beetle
 307 | n02174001 rhinoceros beetle
 308 | n02177972 weevil
 309 | n02190166 fly
 310 | n02206856 bee
 311 | n02219486 ant, emmet, pismire
 312 | n02226429 grasshopper, hopper
 313 | n02229544 cricket
 314 | n02231487 walking stick, walkingstick, stick insect
 315 | n02233338 cockroach, roach
 316 | n02236044 mantis, mantid
 317 | n02256656 cicada, cicala
 318 | n02259212 leafhopper
 319 | n02264363 lacewing, lacewing fly
 320 | n02268443 dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk
 321 | n02268853 damselfly
 322 | n02276258 admiral
 323 | n02277742 ringlet, ringlet butterfly
 324 | n02279972 monarch, monarch butterfly, milkweed butterfly, Danaus plexippus
 325 | n02280649 cabbage butterfly
 326 | n02281406 sulphur butterfly, sulfur butterfly
 327 | n02281787 lycaenid, lycaenid butterfly
 328 | n02317335 starfish, sea star
 329 | n02319095 sea urchin
 330 | n02321529 sea cucumber, holothurian
 331 | n02325366 wood rabbit, cottontail, cottontail rabbit
 332 | n02326432 hare
 333 | n02328150 Angora, Angora rabbit
 334 | n02342885 hamster
 335 | n02346627 porcupine, hedgehog
 336 | n02356798 fox squirrel, eastern fox squirrel, Sciurus niger
 337 | n02361337 marmot
 338 | n02363005 beaver
 339 | n02364673 guinea pig, Cavia cobaya
 340 | n02389026 sorrel
 341 | n02391049 zebra
 342 | n02395406 hog, pig, grunter, squealer, Sus scrofa
 343 | n02396427 wild boar, boar, Sus scrofa
 344 | n02397096 warthog
 345 | n02398521 hippopotamus, hippo, river horse, Hippopotamus amphibius
 346 | n02403003 ox
 347 | n02408429 water buffalo, water ox, Asiatic buffalo, Bubalus bubalis
 348 | n02410509 bison
 349 | n02412080 ram, tup
 350 | n02415577 bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis
 351 | n02417914 ibex, Capra ibex
 352 | n02422106 hartebeest
 353 | n02422699 impala, Aepyceros melampus
 354 | n02423022 gazelle
 355 | n02437312 Arabian camel, dromedary, Camelus dromedarius
 356 | n02437616 llama
 357 | n02441942 weasel
 358 | n02442845 mink
 359 | n02443114 polecat, fitch, foulmart, foumart, Mustela putorius
 360 | n02443484 black-footed ferret, ferret, Mustela nigripes
 361 | n02444819 otter
 362 | n02445715 skunk, polecat, wood pussy
 363 | n02447366 badger
 364 | n02454379 armadillo
 365 | n02457408 three-toed sloth, ai, Bradypus tridactylus
 366 | n02480495 orangutan, orang, orangutang, Pongo pygmaeus
 367 | n02480855 gorilla, Gorilla gorilla
 368 | n02481823 chimpanzee, chimp, Pan troglodytes
 369 | n02483362 gibbon, Hylobates lar
 370 | n02483708 siamang, Hylobates syndactylus, Symphalangus syndactylus
 371 | n02484975 guenon, guenon monkey
 372 | n02486261 patas, hussar monkey, Erythrocebus patas
 373 | n02486410 baboon
 374 | n02487347 macaque
 375 | n02488291 langur
 376 | n02488702 colobus, colobus monkey
 377 | n02489166 proboscis monkey, Nasalis larvatus
 378 | n02490219 marmoset
 379 | n02492035 capuchin, ringtail, Cebus capucinus
 380 | n02492660 howler monkey, howler
 381 | n02493509 titi, titi monkey
 382 | n02493793 spider monkey, Ateles geoffroyi
 383 | n02494079 squirrel monkey, Saimiri sciureus
 384 | n02497673 Madagascar cat, ring-tailed lemur, Lemur catta
 385 | n02500267 indri, indris, Indri indri, Indri brevicaudatus
 386 | n02504013 Indian elephant, Elephas maximus
 387 | n02504458 African elephant, Loxodonta africana
 388 | n02509815 lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens
 389 | n02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca
 390 | n02514041 barracouta, snoek
 391 | n02526121 eel
 392 | n02536864 coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch
 393 | n02606052 rock beauty, Holocanthus tricolor
 394 | n02607072 anemone fish
 395 | n02640242 sturgeon
 396 | n02641379 gar, garfish, garpike, billfish, Lepisosteus osseus
 397 | n02643566 lionfish
 398 | n02655020 puffer, pufferfish, blowfish, globefish
 399 | n02666196 abacus
 400 | n02667093 abaya
 401 | n02669723 academic gown, academic robe, judge's robe
 402 | n02672831 accordion, piano accordion, squeeze box
 403 | n02676566 acoustic guitar
 404 | n02687172 aircraft carrier, carrier, flattop, attack aircraft carrier
 405 | n02690373 airliner
 406 | n02692877 airship, dirigible
 407 | n02699494 altar
 408 | n02701002 ambulance
 409 | n02704792 amphibian, amphibious vehicle
 410 | n02708093 analog clock
 411 | n02727426 apiary, bee house
 412 | n02730930 apron
 413 | n02747177 ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin
 414 | n02749479 assault rifle, assault gun
 415 | n02769748 backpack, back pack, knapsack, packsack, rucksack, haversack
 416 | n02776631 bakery, bakeshop, bakehouse
 417 | n02777292 balance beam, beam
 418 | n02782093 balloon
 419 | n02783161 ballpoint, ballpoint pen, ballpen, Biro
 420 | n02786058 Band Aid
 421 | n02787622 banjo
 422 | n02788148 bannister, banister, balustrade, balusters, handrail
 423 | n02790996 barbell
 424 | n02791124 barber chair
 425 | n02791270 barbershop
 426 | n02793495 barn
 427 | n02794156 barometer
 428 | n02795169 barrel, cask
 429 | n02797295 barrow, garden cart, lawn cart, wheelbarrow
 430 | n02799071 baseball
 431 | n02802426 basketball
 432 | n02804414 bassinet
 433 | n02804610 bassoon
 434 | n02807133 bathing cap, swimming cap
 435 | n02808304 bath towel
 436 | n02808440 bathtub, bathing tub, bath, tub
 437 | n02814533 beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon
 438 | n02814860 beacon, lighthouse, beacon light, pharos
 439 | n02815834 beaker
 440 | n02817516 bearskin, busby, shako
 441 | n02823428 beer bottle
 442 | n02823750 beer glass
 443 | n02825657 bell cote, bell cot
 444 | n02834397 bib
 445 | n02835271 bicycle-built-for-two, tandem bicycle, tandem
 446 | n02837789 bikini, two-piece
 447 | n02840245 binder, ring-binder
 448 | n02841315 binoculars, field glasses, opera glasses
 449 | n02843684 birdhouse
 450 | n02859443 boathouse
 451 | n02860847 bobsled, bobsleigh, bob
 452 | n02865351 bolo tie, bolo, bola tie, bola
 453 | n02869837 bonnet, poke bonnet
 454 | n02870880 bookcase
 455 | n02871525 bookshop, bookstore, bookstall
 456 | n02877765 bottlecap
 457 | n02879718 bow
 458 | n02883205 bow tie, bow-tie, bowtie
 459 | n02892201 brass, memorial tablet, plaque
 460 | n02892767 brassiere, bra, bandeau
 461 | n02894605 breakwater, groin, groyne, mole, bulwark, seawall, jetty
 462 | n02895154 breastplate, aegis, egis
 463 | n02906734 broom
 464 | n02909870 bucket, pail
 465 | n02910353 buckle
 466 | n02916936 bulletproof vest
 467 | n02917067 bullet train, bullet
 468 | n02927161 butcher shop, meat market
 469 | n02930766 cab, hack, taxi, taxicab
 470 | n02939185 caldron, cauldron
 471 | n02948072 candle, taper, wax light
 472 | n02950826 cannon
 473 | n02951358 canoe
 474 | n02951585 can opener, tin opener
 475 | n02963159 cardigan
 476 | n02965783 car mirror
 477 | n02966193 carousel, carrousel, merry-go-round, roundabout, whirligig
 478 | n02966687 carpenter's kit, tool kit
 479 | n02971356 carton
 480 | n02974003 car wheel
 481 | n02977058 cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM
 482 | n02978881 cassette
 483 | n02979186 cassette player
 484 | n02980441 castle
 485 | n02981792 catamaran
 486 | n02988304 CD player
 487 | n02992211 cello, violoncello
 488 | n02992529 cellular telephone, cellular phone, cellphone, cell, mobile phone
 489 | n02999410 chain
 490 | n03000134 chainlink fence
 491 | n03000247 chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour
 492 | n03000684 chain saw, chainsaw
 493 | n03014705 chest
 494 | n03016953 chiffonier, commode
 495 | n03017168 chime, bell, gong
 496 | n03018349 china cabinet, china closet
 497 | n03026506 Christmas stocking
 498 | n03028079 church, church building
 499 | n03032252 cinema, movie theater, movie theatre, movie house, picture palace
 500 | n03041632 cleaver, meat cleaver, chopper
 501 | n03042490 cliff dwelling
 502 | n03045698 cloak
 503 | n03047690 clog, geta, patten, sabot
 504 | n03062245 cocktail shaker
 505 | n03063599 coffee mug
 506 | n03063689 coffeepot
 507 | n03065424 coil, spiral, volute, whorl, helix
 508 | n03075370 combination lock
 509 | n03085013 computer keyboard, keypad
 510 | n03089624 confectionery, confectionary, candy store
 511 | n03095699 container ship, containership, container vessel
 512 | n03100240 convertible
 513 | n03109150 corkscrew, bottle screw
 514 | n03110669 cornet, horn, trumpet, trump
 515 | n03124043 cowboy boot
 516 | n03124170 cowboy hat, ten-gallon hat
 517 | n03125729 cradle
 518 | n03126707 crane
 519 | n03127747 crash helmet
 520 | n03127925 crate
 521 | n03131574 crib, cot
 522 | n03133878 Crock Pot
 523 | n03134739 croquet ball
 524 | n03141823 crutch
 525 | n03146219 cuirass
 526 | n03160309 dam, dike, dyke
 527 | n03179701 desk
 528 | n03180011 desktop computer
 529 | n03187595 dial telephone, dial phone
 530 | n03188531 diaper, nappy, napkin
 531 | n03196217 digital clock
 532 | n03197337 digital watch
 533 | n03201208 dining table, board
 534 | n03207743 dishrag, dishcloth
 535 | n03207941 dishwasher, dish washer, dishwashing machine
 536 | n03208938 disk brake, disc brake
 537 | n03216828 dock, dockage, docking facility
 538 | n03218198 dogsled, dog sled, dog sleigh
 539 | n03220513 dome
 540 | n03223299 doormat, welcome mat
 541 | n03240683 drilling platform, offshore rig
 542 | n03249569 drum, membranophone, tympan
 543 | n03250847 drumstick
 544 | n03255030 dumbbell
 545 | n03259280 Dutch oven
 546 | n03271574 electric fan, blower
 547 | n03272010 electric guitar
 548 | n03272562 electric locomotive
 549 | n03290653 entertainment center
 550 | n03291819 envelope
 551 | n03297495 espresso maker
 552 | n03314780 face powder
 553 | n03325584 feather boa, boa
 554 | n03337140 file, file cabinet, filing cabinet
 555 | n03344393 fireboat
 556 | n03345487 fire engine, fire truck
 557 | n03347037 fire screen, fireguard
 558 | n03355925 flagpole, flagstaff
 559 | n03372029 flute, transverse flute
 560 | n03376595 folding chair
 561 | n03379051 football helmet
 562 | n03384352 forklift
 563 | n03388043 fountain
 564 | n03388183 fountain pen
 565 | n03388549 four-poster
 566 | n03393912 freight car
 567 | n03394916 French horn, horn
 568 | n03400231 frying pan, frypan, skillet
 569 | n03404251 fur coat
 570 | n03417042 garbage truck, dustcart
 571 | n03424325 gasmask, respirator, gas helmet
 572 | n03425413 gas pump, gasoline pump, petrol pump, island dispenser
 573 | n03443371 goblet
 574 | n03444034 go-kart
 575 | n03445777 golf ball
 576 | n03445924 golfcart, golf cart
 577 | n03447447 gondola
 578 | n03447721 gong, tam-tam
 579 | n03450230 gown
 580 | n03452741 grand piano, grand
 581 | n03457902 greenhouse, nursery, glasshouse
 582 | n03459775 grille, radiator grille
 583 | n03461385 grocery store, grocery, food market, market
 584 | n03467068 guillotine
 585 | n03476684 hair slide
 586 | n03476991 hair spray
 587 | n03478589 half track
 588 | n03481172 hammer
 589 | n03482405 hamper
 590 | n03483316 hand blower, blow dryer, blow drier, hair dryer, hair drier
 591 | n03485407 hand-held computer, hand-held microcomputer
 592 | n03485794 handkerchief, hankie, hanky, hankey
 593 | n03492542 hard disc, hard disk, fixed disk
 594 | n03494278 harmonica, mouth organ, harp, mouth harp
 595 | n03495258 harp
 596 | n03496892 harvester, reaper
 597 | n03498962 hatchet
 598 | n03527444 holster
 599 | n03529860 home theater, home theatre
 600 | n03530642 honeycomb
 601 | n03532672 hook, claw
 602 | n03534580 hoopskirt, crinoline
 603 | n03535780 horizontal bar, high bar
 604 | n03538406 horse cart, horse-cart
 605 | n03544143 hourglass
 606 | n03584254 iPod
 607 | n03584829 iron, smoothing iron
 608 | n03590841 jack-o'-lantern
 609 | n03594734 jean, blue jean, denim
 610 | n03594945 jeep, landrover
 611 | n03595614 jersey, T-shirt, tee shirt
 612 | n03598930 jigsaw puzzle
 613 | n03599486 jinrikisha, ricksha, rickshaw
 614 | n03602883 joystick
 615 | n03617480 kimono
 616 | n03623198 knee pad
 617 | n03627232 knot
 618 | n03630383 lab coat, laboratory coat
 619 | n03633091 ladle
 620 | n03637318 lampshade, lamp shade
 621 | n03642806 laptop, laptop computer
 622 | n03649909 lawn mower, mower
 623 | n03657121 lens cap, lens cover
 624 | n03658185 letter opener, paper knife, paperknife
 625 | n03661043 library
 626 | n03662601 lifeboat
 627 | n03666591 lighter, light, igniter, ignitor
 628 | n03670208 limousine, limo
 629 | n03673027 liner, ocean liner
 630 | n03676483 lipstick, lip rouge
 631 | n03680355 Loafer
 632 | n03690938 lotion
 633 | n03691459 loudspeaker, speaker, speaker unit, loudspeaker system, speaker system
 634 | n03692522 loupe, jeweler's loupe
 635 | n03697007 lumbermill, sawmill
 636 | n03706229 magnetic compass
 637 | n03709823 mailbag, postbag
 638 | n03710193 mailbox, letter box
 639 | n03710637 maillot
 640 | n03710721 maillot, tank suit
 641 | n03717622 manhole cover
 642 | n03720891 maraca
 643 | n03721384 marimba, xylophone
 644 | n03724870 mask
 645 | n03729826 matchstick
 646 | n03733131 maypole
 647 | n03733281 maze, labyrinth
 648 | n03733805 measuring cup
 649 | n03742115 medicine chest, medicine cabinet
 650 | n03743016 megalith, megalithic structure
 651 | n03759954 microphone, mike
 652 | n03761084 microwave, microwave oven
 653 | n03763968 military uniform
 654 | n03764736 milk can
 655 | n03769881 minibus
 656 | n03770439 miniskirt, mini
 657 | n03770679 minivan
 658 | n03773504 missile
 659 | n03775071 mitten
 660 | n03775546 mixing bowl
 661 | n03776460 mobile home, manufactured home
 662 | n03777568 Model T
 663 | n03777754 modem
 664 | n03781244 monastery
 665 | n03782006 monitor
 666 | n03785016 moped
 667 | n03786901 mortar
 668 | n03787032 mortarboard
 669 | n03788195 mosque
 670 | n03788365 mosquito net
 671 | n03791053 motor scooter, scooter
 672 | n03792782 mountain bike, all-terrain bike, off-roader
 673 | n03792972 mountain tent
 674 | n03793489 mouse, computer mouse
 675 | n03794056 mousetrap
 676 | n03796401 moving van
 677 | n03803284 muzzle
 678 | n03804744 nail
 679 | n03814639 neck brace
 680 | n03814906 necklace
 681 | n03825788 nipple
 682 | n03832673 notebook, notebook computer
 683 | n03837869 obelisk
 684 | n03838899 oboe, hautboy, hautbois
 685 | n03840681 ocarina, sweet potato
 686 | n03841143 odometer, hodometer, mileometer, milometer
 687 | n03843555 oil filter
 688 | n03854065 organ, pipe organ
 689 | n03857828 oscilloscope, scope, cathode-ray oscilloscope, CRO
 690 | n03866082 overskirt
 691 | n03868242 oxcart
 692 | n03868863 oxygen mask
 693 | n03871628 packet
 694 | n03873416 paddle, boat paddle
 695 | n03874293 paddlewheel, paddle wheel
 696 | n03874599 padlock
 697 | n03876231 paintbrush
 698 | n03877472 pajama, pyjama, pj's, jammies
 699 | n03877845 palace
 700 | n03884397 panpipe, pandean pipe, syrinx
 701 | n03887697 paper towel
 702 | n03888257 parachute, chute
 703 | n03888605 parallel bars, bars
 704 | n03891251 park bench
 705 | n03891332 parking meter
 706 | n03895866 passenger car, coach, carriage
 707 | n03899768 patio, terrace
 708 | n03902125 pay-phone, pay-station
 709 | n03903868 pedestal, plinth, footstall
 710 | n03908618 pencil box, pencil case
 711 | n03908714 pencil sharpener
 712 | n03916031 perfume, essence
 713 | n03920288 Petri dish
 714 | n03924679 photocopier
 715 | n03929660 pick, plectrum, plectron
 716 | n03929855 pickelhaube
 717 | n03930313 picket fence, paling
 718 | n03930630 pickup, pickup truck
 719 | n03933933 pier
 720 | n03935335 piggy bank, penny bank
 721 | n03937543 pill bottle
 722 | n03938244 pillow
 723 | n03942813 ping-pong ball
 724 | n03944341 pinwheel
 725 | n03947888 pirate, pirate ship
 726 | n03950228 pitcher, ewer
 727 | n03954731 plane, carpenter's plane, woodworking plane
 728 | n03956157 planetarium
 729 | n03958227 plastic bag
 730 | n03961711 plate rack
 731 | n03967562 plow, plough
 732 | n03970156 plunger, plumber's helper
 733 | n03976467 Polaroid camera, Polaroid Land camera
 734 | n03976657 pole
 735 | n03977966 police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria
 736 | n03980874 poncho
 737 | n03982430 pool table, billiard table, snooker table
 738 | n03983396 pop bottle, soda bottle
 739 | n03991062 pot, flowerpot
 740 | n03992509 potter's wheel
 741 | n03995372 power drill
 742 | n03998194 prayer rug, prayer mat
 743 | n04004767 printer
 744 | n04005630 prison, prison house
 745 | n04008634 projectile, missile
 746 | n04009552 projector
 747 | n04019541 puck, hockey puck
 748 | n04023962 punching bag, punch bag, punching ball, punchball
 749 | n04026417 purse
 750 | n04033901 quill, quill pen
 751 | n04033995 quilt, comforter, comfort, puff
 752 | n04037443 racer, race car, racing car
 753 | n04039381 racket, racquet
 754 | n04040759 radiator
 755 | n04041544 radio, wireless
 756 | n04044716 radio telescope, radio reflector
 757 | n04049303 rain barrel
 758 | n04065272 recreational vehicle, RV, R.V.
 759 | n04067472 reel
 760 | n04069434 reflex camera
 761 | n04070727 refrigerator, icebox
 762 | n04074963 remote control, remote
 763 | n04081281 restaurant, eating house, eating place, eatery
 764 | n04086273 revolver, six-gun, six-shooter
 765 | n04090263 rifle
 766 | n04099969 rocking chair, rocker
 767 | n04111531 rotisserie
 768 | n04116512 rubber eraser, rubber, pencil eraser
 769 | n04118538 rugby ball
 770 | n04118776 rule, ruler
 771 | n04120489 running shoe
 772 | n04125021 safe
 773 | n04127249 safety pin
 774 | n04131690 saltshaker, salt shaker
 775 | n04133789 sandal
 776 | n04136333 sarong
 777 | n04141076 sax, saxophone
 778 | n04141327 scabbard
 779 | n04141975 scale, weighing machine
 780 | n04146614 school bus
 781 | n04147183 schooner
 782 | n04149813 scoreboard
 783 | n04152593 screen, CRT screen
 784 | n04153751 screw
 785 | n04154565 screwdriver
 786 | n04162706 seat belt, seatbelt
 787 | n04179913 sewing machine
 788 | n04192698 shield, buckler
 789 | n04200800 shoe shop, shoe-shop, shoe store
 790 | n04201297 shoji
 791 | n04204238 shopping basket
 792 | n04204347 shopping cart
 793 | n04208210 shovel
 794 | n04209133 shower cap
 795 | n04209239 shower curtain
 796 | n04228054 ski
 797 | n04229816 ski mask
 798 | n04235860 sleeping bag
 799 | n04238763 slide rule, slipstick
 800 | n04239074 sliding door
 801 | n04243546 slot, one-armed bandit
 802 | n04251144 snorkel
 803 | n04252077 snowmobile
 804 | n04252225 snowplow, snowplough
 805 | n04254120 soap dispenser
 806 | n04254680 soccer ball
 807 | n04254777 sock
 808 | n04258138 solar dish, solar collector, solar furnace
 809 | n04259630 sombrero
 810 | n04263257 soup bowl
 811 | n04264628 space bar
 812 | n04265275 space heater
 813 | n04266014 space shuttle
 814 | n04270147 spatula
 815 | n04273569 speedboat
 816 | n04275548 spider web, spider's web
 817 | n04277352 spindle
 818 | n04285008 sports car, sport car
 819 | n04286575 spotlight, spot
 820 | n04296562 stage
 821 | n04310018 steam locomotive
 822 | n04311004 steel arch bridge
 823 | n04311174 steel drum
 824 | n04317175 stethoscope
 825 | n04325704 stole
 826 | n04326547 stone wall
 827 | n04328186 stopwatch, stop watch
 828 | n04330267 stove
 829 | n04332243 strainer
 830 | n04335435 streetcar, tram, tramcar, trolley, trolley car
 831 | n04336792 stretcher
 832 | n04344873 studio couch, day bed
 833 | n04346328 stupa, tope
 834 | n04347754 submarine, pigboat, sub, U-boat
 835 | n04350905 suit, suit of clothes
 836 | n04355338 sundial
 837 | n04355933 sunglass
 838 | n04356056 sunglasses, dark glasses, shades
 839 | n04357314 sunscreen, sunblock, sun blocker
 840 | n04366367 suspension bridge
 841 | n04367480 swab, swob, mop
 842 | n04370456 sweatshirt
 843 | n04371430 swimming trunks, bathing trunks
 844 | n04371774 swing
 845 | n04372370 switch, electric switch, electrical switch
 846 | n04376876 syringe
 847 | n04380533 table lamp
 848 | n04389033 tank, army tank, armored combat vehicle, armoured combat vehicle
 849 | n04392985 tape player
 850 | n04398044 teapot
 851 | n04399382 teddy, teddy bear
 852 | n04404412 television, television system
 853 | n04409515 tennis ball
 854 | n04417672 thatch, thatched roof
 855 | n04418357 theater curtain, theatre curtain
 856 | n04423845 thimble
 857 | n04428191 thresher, thrasher, threshing machine
 858 | n04429376 throne
 859 | n04435653 tile roof
 860 | n04442312 toaster
 861 | n04443257 tobacco shop, tobacconist shop, tobacconist
 862 | n04447861 toilet seat
 863 | n04456115 torch
 864 | n04458633 totem pole
 865 | n04461696 tow truck, tow car, wrecker
 866 | n04462240 toyshop
 867 | n04465501 tractor
 868 | n04467665 trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi
 869 | n04476259 tray
 870 | n04479046 trench coat
 871 | n04482393 tricycle, trike, velocipede
 872 | n04483307 trimaran
 873 | n04485082 tripod
 874 | n04486054 triumphal arch
 875 | n04487081 trolleybus, trolley coach, trackless trolley
 876 | n04487394 trombone
 877 | n04493381 tub, vat
 878 | n04501370 turnstile
 879 | n04505470 typewriter keyboard
 880 | n04507155 umbrella
 881 | n04509417 unicycle, monocycle
 882 | n04515003 upright, upright piano
 883 | n04517823 vacuum, vacuum cleaner
 884 | n04522168 vase
 885 | n04523525 vault
 886 | n04525038 velvet
 887 | n04525305 vending machine
 888 | n04532106 vestment
 889 | n04532670 viaduct
 890 | n04536866 violin, fiddle
 891 | n04540053 volleyball
 892 | n04542943 waffle iron
 893 | n04548280 wall clock
 894 | n04548362 wallet, billfold, notecase, pocketbook
 895 | n04550184 wardrobe, closet, press
 896 | n04552348 warplane, military plane
 897 | n04553703 washbasin, handbasin, washbowl, lavabo, wash-hand basin
 898 | n04554684 washer, automatic washer, washing machine
 899 | n04557648 water bottle
 900 | n04560804 water jug
 901 | n04562935 water tower
 902 | n04579145 whiskey jug
 903 | n04579432 whistle
 904 | n04584207 wig
 905 | n04589890 window screen
 906 | n04590129 window shade
 907 | n04591157 Windsor tie
 908 | n04591713 wine bottle
 909 | n04592741 wing
 910 | n04596742 wok
 911 | n04597913 wooden spoon
 912 | n04599235 wool, woolen, woollen
 913 | n04604644 worm fence, snake fence, snake-rail fence, Virginia fence
 914 | n04606251 wreck
 915 | n04612504 yawl
 916 | n04613696 yurt
 917 | n06359193 web site, website, internet site, site
 918 | n06596364 comic book
 919 | n06785654 crossword puzzle, crossword
 920 | n06794110 street sign
 921 | n06874185 traffic light, traffic signal, stoplight
 922 | n07248320 book jacket, dust cover, dust jacket, dust wrapper
 923 | n07565083 menu
 924 | n07579787 plate
 925 | n07583066 guacamole
 926 | n07584110 consomme
 927 | n07590611 hot pot, hotpot
 928 | n07613480 trifle
 929 | n07614500 ice cream, icecream
 930 | n07615774 ice lolly, lolly, lollipop, popsicle
 931 | n07684084 French loaf
 932 | n07693725 bagel, beigel
 933 | n07695742 pretzel
 934 | n07697313 cheeseburger
 935 | n07697537 hotdog, hot dog, red hot
 936 | n07711569 mashed potato
 937 | n07714571 head cabbage
 938 | n07714990 broccoli
 939 | n07715103 cauliflower
 940 | n07716358 zucchini, courgette
 941 | n07716906 spaghetti squash
 942 | n07717410 acorn squash
 943 | n07717556 butternut squash
 944 | n07718472 cucumber, cuke
 945 | n07718747 artichoke, globe artichoke
 946 | n07720875 bell pepper
 947 | n07730033 cardoon
 948 | n07734744 mushroom
 949 | n07742313 Granny Smith
 950 | n07745940 strawberry
 951 | n07747607 orange
 952 | n07749582 lemon
 953 | n07753113 fig
 954 | n07753275 pineapple, ananas
 955 | n07753592 banana
 956 | n07754684 jackfruit, jak, jack
 957 | n07760859 custard apple
 958 | n07768694 pomegranate
 959 | n07802026 hay
 960 | n07831146 carbonara
 961 | n07836838 chocolate sauce, chocolate syrup
 962 | n07860988 dough
 963 | n07871810 meat loaf, meatloaf
 964 | n07873807 pizza, pizza pie
 965 | n07875152 potpie
 966 | n07880968 burrito
 967 | n07892512 red wine
 968 | n07920052 espresso
 969 | n07930864 cup
 970 | n07932039 eggnog
 971 | n09193705 alp
 972 | n09229709 bubble
 973 | n09246464 cliff, drop, drop-off
 974 | n09256479 coral reef
 975 | n09288635 geyser
 976 | n09332890 lakeside, lakeshore
 977 | n09399592 promontory, headland, head, foreland
 978 | n09421951 sandbar, sand bar
 979 | n09428293 seashore, coast, seacoast, sea-coast
 980 | n09468604 valley, vale
 981 | n09472597 volcano
 982 | n09835506 ballplayer, baseball player
 983 | n10148035 groom, bridegroom
 984 | n10565667 scuba diver
 985 | n11879895 rapeseed
 986 | n11939491 daisy
 987 | n12057211 yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum
 988 | n12144580 corn
 989 | n12267677 acorn
 990 | n12620546 hip, rose hip, rosehip
 991 | n12768682 buckeye, horse chestnut, conker
 992 | n12985857 coral fungus
 993 | n12998815 agaric
 994 | n13037406 gyromitra
 995 | n13040303 stinkhorn, carrion fungus
 996 | n13044778 earthstar
 997 | n13052670 hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa
 998 | n13054560 bolete
 999 | n13133613 ear, spike, capitulum
1000 | n15075141 toilet tissue, toilet paper, bathroom tissue
1001 | 


--------------------------------------------------------------------------------
/frontend.py:
--------------------------------------------------------------------------------
  1 | import os
  2 | import pandas
  3 | 
  4 | from uuid import UUID, uuid4
  5 | import urllib2, io
  6 | from PIL import Image as PILImage
  7 | import numpy
  8 | 
  9 | from flask import Flask, request, session, redirect, url_for, render_template, flash
 10 | from flask_wtf import Form, RecaptchaField
 11 | from wtforms import SelectMultipleField, widgets, SelectField
 12 | from flask_wtf.file import FileField
 13 | from wtforms.fields.html5 import URLField
 14 | from wtforms.validators import DataRequired, url
 15 | from keys import SECRET_KEY, RECAPTCHA_PUBLIC_KEY, RECAPTCHA_PRIVATE_KEY
 16 | 
 17 | from gevent.wsgi import WSGIServer
 18 | from gevent import monkey; monkey.patch_all()
 19 | #from flask.ext.compress import Compress
 20 | 
 21 | from backend.run import run_backend
 22 | 
 23 | # Comment out for running locally
 24 | import redis; client = redis.Redis(host="localhost", port=6379, db=1)
 25 | import backend.info
 26 | from keys import UPLOAD_PATH
 27 | # ----------
 28 | 
 29 | app = Flask(__name__)
 30 | app.config.from_object(__name__)
 31 | app.config['MAX_CONTENT_LENGTH'] = 2 * 1024 * 1024
 32 | #app.config['COMPRESS_LEVEL'] = 1
 33 | #Compress(app)
 34 | 
 35 | networks = [('1','Berkeley CaffeNet (ImageNet challenge 2012 winning level) [<a href="http://arxiv.org/abs/1408.5093">1</a>]'), 
 36 |             ('2','Oxford CNN-S (ImageNet challenge 2013 winning level) [<a href="http://arxiv.org/abs/1405.3531">2</a>]'), 
 37 |             ('3','Oxford VeryDeep-19 (ImageNet challenge 2014 winning level) [<a href="http://arxiv.org/abs/1409.1556">3</a>]')]
 38 | 
 39 | with open('backend/synset_words.txt') as f:
 40 | 	labels = pandas.DataFrame([
 41 | 		{'synset_id': l.strip().split(' ')[0], 'name': ' '.join(l.strip().split(' ')[1:]).split(',')[0]} for l in f.readlines() ])
 42 | 
 43 | labels = labels['name'].values
 44 | labels = [(str(i+1), '('+str(i+1).zfill(4)+') '+labels[i]) for i in xrange(labels.size)]
 45 | 
 46 | class MultiCheckboxField(SelectMultipleField):
 47 | 
 48 | 	widget = widgets.ListWidget(html_tag='ul', prefix_label=False)
 49 | 	option_widget = widgets.CheckboxInput()
 50 | 
 51 | class TaskForm(Form):
 52 | 
 53 | 	tasks = ['Random Noise', 'Uploading Image', 'Image at URL', 'Random Image']
 54 | 	network_selection = MultiCheckboxField(choices=networks, default=['1'], validators=[DataRequired()])
 55 | 	label_selection = SelectField(choices=labels, default=10, validators=[DataRequired()])
 56 | 	image_file = FileField()
 57 | 	image_url = URLField()
 58 | 	recaptcha = RecaptchaField()
 59 | 
 60 | def valid_uuid(uuid):
 61 | 
 62 | 	try: val = UUID(uuid, version=4)
 63 | 	except ValueError: return False
 64 | 	return val.hex == uuid
 65 | 
 66 | def add_label(idx_str):
 67 | 
 68 | 	return '<abbr title="{1}">{0}</abbr>'.format(idx_str.zfill(4), labels[int(idx_str)-1][1][7:])
 69 | 
 70 | def get_srv_load():
 71 | 	
 72 | 	srv_load = 0
 73 | 	# Comment out for running locally
 74 | 	def get_worker_num():
 75 | 		#reload(backend.info)
 76 | 		return backend.info.WORKER_NUM
 77 | 	
 78 | 	try: srv_load = 100 * client.llen("celery") / get_worker_num()
 79 | 	except: srv_load = 100
 80 | 	# ----------
 81 | 	return srv_load
 82 | 
 83 | @app.route('/')
 84 | def index():
 85 | 
 86 | 	req_taskid = request.args.get('taskid', '')
 87 | 	if valid_uuid(req_taskid) or (req_taskid.find('example') == 0):
 88 | 		session['taskid'] = req_taskid
 89 | 		return redirect(url_for('index'))
 90 | 	
 91 | 	form = TaskForm()
 92 | 	session.permanent = True
 93 | 	
 94 | 	advmode = ''; 
 95 | 	task_info = {'taskid':'','taskpar':'','results':'','ori_class':'','new_class':''}	
 96 | 	
 97 | 	if 'advmode' in session: advmode = session['advmode']
 98 | 	if 'taskid'  in session: task_info['taskid'] = session['taskid']
 99 | 	
100 | 	if task_info['taskid']:
101 | 
102 | 		if os.path.isfile('backend/log/' + task_info['taskid'] + '.txt'):
103 | 			progress = os.popen('tail -n4 backend/log/' + task_info['taskid'] + '.txt').read().split('\n')
104 | 			task_info['taskpar'] = os.popen('head -n1 backend/log/' + task_info['taskid'] + '.txt').read().replace('\n','')
105 | 		else:
106 | 			progress = []
107 | 		
108 | 		# RENDER RESULTS
109 | 		if len(progress) == 4 and progress[-1] == 'DONE':
110 | 			if (progress[-2] == '1'):
111 | 				task_info['results'] = 'Your task finished successfully.'
112 | 				task_info['ori_class'] = '(' + ', '.join([add_label(i) for i in progress[0].split()]) + ')'
113 | 				task_info['new_class'] = '(' + ', '.join([add_label(i) for i in progress[1].split()]) + ')'
114 | 			elif (progress[-2] == '0'):
115 | 				task_info['results'] = 'Your task didn\'t finish in the time limit, and here are the best results we got.'
116 | 				task_info['ori_class'] = '(' + ', '.join([add_label(i) for i in progress[0].split()]) + ')'
117 | 				task_info['new_class'] = '(' + ', '.join([add_label(i) for i in progress[1].split()]) + ')'
118 | 			else: #'-1', error
119 | 				task_info['results'] = 'Something went wrong and we will look at it. You can come back later and resubmit your task, thanks!'
120 | 	
121 | 	srv_load = '%.2f' % get_srv_load()
122 | 	return render_template("index.html", form=form, advmode=advmode, task_info=task_info, srv_load=srv_load)
123 | 
124 | @app.route('/run_task', methods=['POST'])
125 | def run_task():
126 | 
127 | 	form = TaskForm()
128 | 
129 | 	if not form.validate_on_submit():
130 | 		flash('Incorrect Form!')
131 | 		return redirect(url_for('index'))
132 | 
133 | 	try:
134 | 		if request.form['start'] == form.tasks[0]:
135 | 			run_image = numpy.random.randn(227,227,3)
136 | 			run_image = (run_image*16/256 + 0.5).clip(0,1)
137 | 			run_image = PILImage.fromarray((run_image*255).astype('uint8'))
138 | 		elif request.form['start'] == form.tasks[1]:
139 | 			run_image = PILImage.open(form.image_file.data)
140 | 		elif request.form['start'] == form.tasks[2]:
141 | 			run_image = urllib2.urlopen(form.image_url.data, timeout=2) # TIMEOUT FOR SAFETY
142 | 			if int(run_image.info().getheaders("Content-Length")[0]) > app.config['MAX_CONTENT_LENGTH']:
143 | 				flash('URL File Size over Limit!')
144 | 				raise Exception
145 | 			run_image = PILImage.open(io.BytesIO(run_image.read()))
146 | 		elif request.form['start'] == form.tasks[3]:
147 | 			run_image = urllib2.urlopen('http://lorempixel.com/227/227/', timeout=2) # TIMEOUT FOR SAFETY
148 | 			run_image = PILImage.open(io.BytesIO(run_image.read()))
149 | 		else:
150 | 			raise Exception
151 | 	except:
152 | 			flash('Image Loading Error!')
153 | 			return redirect(url_for('index'))
154 | 
155 | 	try:
156 | 		run_image = run_image.convert('RGB').resize((227,227),PILImage.ANTIALIAS)
157 | 		taskid = str(uuid4()).replace('-','')
158 | 		run_image.filename = taskid + '.png'
159 | 		#run_image.save('static/' + run_image.filename) # MOVED LATER
160 | 		session['taskid'] = taskid
161 | 	except:
162 | 		flash('Image Processing Error!')
163 | 		return redirect(url_for('index'))
164 | 	
165 | 	try:
166 | 		if (get_srv_load() >= 100.0): session.pop('taskid', None); raise Exception
167 | 
168 | 		with open('backend/log/' + taskid + '.txt', 'w') as f:
169 | 			f.write('Algorithm {0} and Class {1}\n'.format('['+', '.join(form.network_selection.data)+']', labels[int(form.label_selection.data) - 1][1]))
170 | 				
171 | 		network_selection = [str(int(str(i+1) in form.network_selection.data)) for i in xrange(len(networks))]
172 | 		run_image.save('static/' + run_image.filename) # SAVE WHEN SCHEDULED
173 | 		
174 | 		# RUN NONBLOCKING TASK
175 | 		# Comment out for running locally
176 | 		run_backend.delay([session['taskid']] + network_selection + [str(form.label_selection.data)])
177 | 		# Uncomment for running locally
178 | 		# run_backend([session['taskid']] + network_selection + [str(form.label_selection.data)])
179 | 	except:
180 | 		flash('Task Initialization/Scheduling Error!')
181 | 		return redirect(url_for('index'))
182 | 
183 | 	return redirect(url_for('index'))
184 | 
185 | @app.route('/del_task')
186 | def del_task():
187 | 	
188 | 	session.pop('taskid', None)
189 | 	return redirect(url_for('index'))
190 | 
191 | 
192 | # Comment out for running locally
193 | @app.route('/{0}'.format(UPLOAD_PATH), methods=['POST'])
194 | def upload_result():
195 | 	
196 | 	data = request.files['data']
197 | 	if os.path.splitext(data.filename)[1] == '.png':
198 | 		data.save('static/' + data.filename)
199 | 	elif os.path.splitext(data.filename)[1] == '.txt':
200 | 		with open('backend/log/' + data.filename, 'a') as logfile: logfile.write(data.stream.read())
201 | 	return ''
202 | # ----------
203 | 
204 | @app.route('/adv/') 
205 | def adv_mode(): 
206 | 	
207 | 	if 'advmode' in session: session.pop('advmode', None)
208 | 	else: session['advmode'] = '1'
209 | 	
210 | 	return redirect(url_for('index'))
211 | 
212 | if __name__ == "__main__":
213 | 	http_server = WSGIServer(('0.0.0.0',8080), app); http_server.serve_forever()
214 | 	
215 | 


--------------------------------------------------------------------------------
/static/.gitignore:
--------------------------------------------------------------------------------
1 | *
2 | !example*
3 | !.gitignore
4 | 


--------------------------------------------------------------------------------
/templates/index.html:
--------------------------------------------------------------------------------
  1 | <!DOCTYPE html>
  2 | <html lang="en">
  3 | 	<head>
  4 | 		<title> Ostrichinator Demo </title>
  5 | 		
  6 | 		<meta charset="utf-8">
  7 | 		<meta name="viewport" content="width=device-width, initial-scale=1">
  8 | 		{% if task_info['taskid'] and not task_info['results'] %} <meta http-equiv="refresh" content="60" /> {% endif %}
  9 | 		
 10 | 		<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
 11 | 		<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap-theme.min.css">
 12 | 		<style type='text/css'> 
 13 | 			.flash {background:#F2DEDE; padding:0.5em; display:inline-block}
 14 | 			.borderless tbody tr td, .borderless thead tr th {border:none}
 15 | 			.table-nonfluid {width:auto}
 16 | 			#spinner {text-align:middle; width:227px; height:227px; position:relative}
 17 | 			#wrapper {background-color:#ffffff; opacity:0.6; filter:alpha(opacity=60)}
 18 | 			ul {list-style:none; padding-left:0}
 19 | 		</style>
 20 | 		
 21 | 		<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
 22 | 		<script src="http://fgnass.github.io/spin.js/spin.min.js"></script>
 23 | 	</head>
 24 | 
 25 | 	<body>
 26 | 	<div class="container">
 27 | 		<h1><a href="http://coxlab.github.io/ostrichinator/">Are Deep Learning Algorithms Easily Hackable?</a></h1>
 28 | 		
 29 | 		{% for message in get_flashed_messages() %}
 30 | 			<div class="flash"> {{message}} </div>
 31 | 		{% endfor %}
 32 | 		
 33 | 		<form method="POST" action="{{url_for('run_task')}}" enctype="multipart/form-data">
 34 | 			{{form.hidden_tag()}}
 35 | 			<h2> Step 1 <small>Verify you're a human visitor</small> </h2>
 36 | 			{{form.recaptcha}}
 37 | 
 38 | 			<h2> Step 2 <small>Specify your hacking task</small> </h2>
 39 | 			<p> Target deep learning algorithm{% if advmode %}(s) &ndash; <i>select multiple to break them at once</i>{% endif %} {{form.network_selection}} </p>
 40 | 			<p> And you want the image to be classified as {{form.label_selection}} (see <a href="http://www.googledrive.com/host/0B8LpM_21I0tYY0dWMVRscVV0Vzg" target="_blank">here</a> for detailed definitions, or just <a onclick="randomize()">randomize</a> it.) </p>
 41 | 			<script> function randomize(){document.getElementById('label_selection').selectedIndex=Math.floor(1000*Math.random());}</script>
 42 | 
 43 | 			<h2> Step 3 <small>Pick an image and try hacking it</small> </h2>
 44 | 			<!-- <p> <input type="submit" name="start" value="{{form.tasks[0]}}" style="width:130px" class="btn btn-primary"> </p> -->
 45 | 			<p> <input type="submit" name="start" value="{{form.tasks[3]}}" style="width:130px" class="btn btn-primary">
 46 | 			<p> <input type="submit" name="start" value="{{form.tasks[2]}}" style="width:130px" class="btn btn-primary"> &nbsp;{{form.image_url}} </p>
 47 | 			<p> <input type="submit" name="start" value="{{form.tasks[1]}}" style="width:130px" class="btn btn-primary"> &nbsp;<input type="text" name="subfile" id="subfile"> &nbsp;<a class="btn btn-default" onclick="$('#image_file').click();"> Browse </a></p>
 48 | 				<div style="position:absolute; left:-999em"> {{form.image_file}} </div>
 49 | 				<script> $('#image_file').change(function(){$('#subfile').val($(this).val().split('\\').pop().split('/').pop());}); </script> 
 50 | 			<p><i> Random images are hosted by lorempixel.com. Images need to be smaller than 2MB. </i></p>
 51 | 			
 52 | 			{% if task_info['taskid'] %}
 53 | 				<h2> Results <small>Task targeting {{task_info['taskpar']}} (<a href={{url_for('index',taskid='%s' % task_info['taskid'])}}>link</a>)</small> </h2>
 54 | 				<div class="table-responsive"> <table class="table borderless table-nonfluid">
 55 | 					<tr>
 56 | 						{% if task_info['results'] %} 
 57 | 							<td align="center"> <img class="img-rounded" src={{url_for('static',filename='%s.png' % task_info['taskid'])}}>  </td>						
 58 | 							<td align="center"> <img class="img-rounded" src={{url_for('static',filename='%s-out.png' % task_info['taskid'])}}> </td>
 59 | 							<td align="center"> <img class="img-rounded" src={{url_for('static',filename='%s-dff.png' % task_info['taskid'])}}> </td>
 60 | 							<td align="center"> <img class="img-rounded" src={{url_for('static',filename='%s-sal.png' % task_info['taskid'])}}> </td>
 61 | 						{% else %}
 62 | 							<td align="center"> <div id="spinner"><div id="wrapper"><img class="img-rounded" src={{url_for('static',filename='%s.png' % task_info['taskid'])}}></div></div> </td>
 63 | 							
 64 | 							<script>
 65 | 								$(document).ready(function () {
 66 | 									var opts = {lines:10, length:45, width:10, radius:45, corners:1,  rotate:0, direction:1, color:'#000', speed:0.25, trail:75, shadow:true, hwaccel:false, className:'spin', zIndex:2e9, top:'50%', left:'50%'};
 67 | 									var target = document.getElementById('spinner');
 68 | 									var spinner = new Spinner(opts).spin(target);});
 69 | 							</script>
 70 | 						{% endif %}
 71 | 					</tr>
 72 | 					<tr>
 73 | 						{% if task_info['results'] %}
 74 | 							<td align="center"> <b>Initial Image</b> <br> {% if task_info['ori_class'] %} Classified as {{task_info['ori_class']|safe}}. {% endif %} </td>
 75 | 							<td align="center"> <b>Hacked Image</b> <br> {% if task_info['new_class'] %} Classified as {{task_info['new_class']|safe}}. {% endif %} </td>
 76 | 							<td align="center"> <b>Difference</b> <br> Normalized to full dynamic range. </td>
 77 | 							<td align="center"> <b>Heat Map</b> <br> Where the algorithm{% if advmode %}(s){% endif %} got fooled. </td>
 78 | 						{% else %}
 79 | 							<td align="center"> <b>Hacking in Progress</b> </td>
 80 | 						{% endif %}
 81 | 					</tr>
 82 | 				</table> </div>
 83 | 				 
 84 | 				{% if task_info['results'] %} 
 85 | 					<p><b> {{task_info['results']}} </b></p>
 86 | 					<p><i> Please take this <a href={{"https://docs.google.com/forms/d/1TtosZR4G9LWwFgOc61fsrTFrpLeDJWenxnJzayxmeT4/viewform?entry.924271448=%s" % task_info['taskid']}} target="_blank">survey</a> to help us better characterize deep learning algorithms and the way they agree or disagree with human vision. </i></p>
 87 | 				{% else %}
 88 | 					<p> Wait a bit and the page will reload itself to show the hacking results (depends on the difficulty of your task but usually less than 5 minutes). </p>
 89 | 				{% endif %}
 90 | 			{% endif %}
 91 | 		</form>
 92 | 
 93 | 		{% if task_info['taskid'] and not task_info['results'] %}
 94 | 			<p class="text-muted"><i> ONLY in case this is running too long (e.g. over 30 minutes, meaning something may have gone wrong), you are suggested to <a href="{{url_for('del_task')}}" class="text-danger">delete</a> task and resubmit it. </i></p>
 95 | 			
 96 | 			<script> $(document).ready(function() {$('input[name="start"]').attr('disabled','disabled');}); </script>
 97 | 		{% else %}
 98 | 			<script> $(document).ready(function() {$('input[name="start"]').removeAttr('disabled');}); </script>
 99 | 		{% endif %}
100 | 				
101 | 		{% if not advmode %}
102 | 			<!--You got it. Navigate to /adv/ and have some more fun.-->
103 | 			<script>
104 | 				var disableMultiSelection = function(group){
105 | 					return function() {var isChecked= $(this).prop("checked"); $(group).prop("checked", false); $(this).prop("checked", isChecked);} };
106 | 
107 | 				$("[id^=network_selection-]").click(disableMultiSelection("[id^=network_selection-]"));
108 | 			</script>
109 | 		{% endif %}
110 | 		
111 | 		<hr>
112 | 		This project is created by Chuan-Yung (Kenyon) Tsai, based on <a href="http://www.vlfeat.org/matconvnet/">MatConvNet</a> and <a href="http://www.cs.ubc.ca/~schmidtm/Software/minConf.html">minConf</a>.
113 | 		Server load is {{srv_load}}% now.
114 | 	</div>
115 | 	</body>
116 | </html>
117 | 
118 | 


--------------------------------------------------------------------------------