├── LICENSE ├── README.md └── active_select_demo.m /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Zongwei Zhou 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Paper 2 | This repository provides the official Matlab demo for the following paper: 3 | 4 | Fine-tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally
5 | [Zongwei Zhou](https://www.zongweiz.com/)1, Jae Shin1, Lei Zhang1, Suryakanth Gurudu2, [Michael B. Gotway](https://www.mayoclinic.org/biographies/gotway-michael-b-m-d/bio-20055566)2, and [Jianming Liang](https://chs.asu.edu/jianming-liang)1
6 | 1 Arizona State University, 2 Mayo Clinic
7 | The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
8 | [paper](http://openaccess.thecvf.com/content_cvpr_2017/papers/Zhou_Fine-Tuning_Convolutional_Neural_CVPR_2017_paper.pdf) | [code](https://github.com/MrGiovanni/Active-Learning) | [slides](https://docs.wixstatic.com/ugd/deaea1_d4ad03012be2408191885c697af60401.pdf) | [poster](https://docs.wixstatic.com/ugd/deaea1_c21a4361487244efba9da9d84feddc72.pdf) 9 | 10 | Active, Continual Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts
11 | [Zongwei Zhou](https://www.zongweiz.com/)1, Jae Shin1, Suryakanth Gurudu2, [Michael B. Gotway](https://www.mayoclinic.org/biographies/gotway-michael-b-m-d/bio-20055566)2, and [Jianming Liang](https://chs.asu.edu/jianming-liang)1
12 | 1 Arizona State University, 2 Mayo Clinic
13 | Medical Image Analysis (MedIA)
14 | [paper](https://arxiv.org/pdf/1802.00912.pdf) | [code](https://github.com/MrGiovanni/Active-Learning) | [slides](https://docs.wixstatic.com/ugd/deaea1_d4ad03012be2408191885c697af60401.pdf) 15 | 16 | ## Citation 17 | If you use this code for your research, please cite our [paper](http://openaccess.thecvf.com/content_cvpr_2017/papers/Zhou_Fine-Tuning_Convolutional_Neural_CVPR_2017_paper.pdf): 18 | ``` 19 | @inproceedings{zhou2017fine, 20 | title={Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally}, 21 | author={Zhou, Zongwei and Shin, Jae and Zhang, Lei and Gurudu, Suryakanth and Gotway, Michael and Liang, Jianming}, 22 | booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition}, 23 | pages={7340--7351}, 24 | year={2017}, 25 | url = {http://openaccess.thecvf.com/content_cvpr_2017/papers/Zhou_Fine-Tuning_Convolutional_Neural_CVPR_2017_paper.pdf} 26 | } 27 | 28 | @article{zhou2018active, 29 | title={Active, Continual Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts}, 30 | author={Zhou, Zongwei and Shin, Jae Y and Gurudu, Suryakanth R and Gotway, Michael B and Liang, Jianming}, 31 | journal={arXiv preprint arXiv:1802.00912}, 32 | year={2018} 33 | } 34 | 35 | @phdthesis{zhou2021towards, 36 | title={Towards Annotation-Efficient Deep Learning for Computer-Aided Diagnosis}, 37 | author={Zhou, Zongwei}, 38 | year={2021}, 39 | school={Arizona State University} 40 | } 41 | ``` 42 | 43 | ## Acknowledgement 44 | This research has been supported partially by ASU and Mayo Clinic through a Seed Grant and an Innovation Grant, and partially by NIH under Award Number R01HL128785. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH. This is a patent-pending technology. 45 | -------------------------------------------------------------------------------- /active_select_demo.m: -------------------------------------------------------------------------------- 1 | % Zhou, Zongwei, et al. "Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally." 2 | % IEEE conference on computer vision and pattern recognition, Hawaii. 2017. 3 | 4 | % Author: [Zongwei Zhou](http://www.zongweiz.com) 5 | % Email: zongweiz@asu.edu 6 | % Last modified: Jan.26.2017 7 | 8 | %% Table 1. examples of seven different patterns 9 | % Relationships among seven prediction patterns and six AIFT methods in active candidate selection. We assume that 10 | % a candidate has 11 patches, and their probabilities predicted by the current CNN are listed in Column 2. AIFT Entropyα, 11 | % Diversityα, and (Entropy+Diversity)α operate on the top or bottom α percent of the candidate’s patches based on the majority 12 | % prediction as described in Sec. 3.3. In this illustration, we choose α to be 1/4, meaning that the selection criterion (Eq. 3) is 13 | % computed based on 3 patches within each candidate. The first choice of each method is highlighted in yellow and the second 14 | % choice is in light yellow. 15 | 16 | function Alg_Analysis() 17 | 18 | close all; clear; clc; 19 | 20 | 21 | % A, B, C, D, E, F represents outputs from CNN after Softmax Layer belong to one candidate. 22 | A = [0.4 0.4 0.4 0.5 0.5 0.5 0.5 0.5 0.5 0.6 0.6]; 23 | B = [0.0 0.1 0.2 0.3 0.4 0.4 0.6 0.7 0.8 1.0 1.0]; 24 | C = [0.0 0.0 0.0 0.1 0.1 0.9 0.9 1.0 1.0 1.0 1.0]; 25 | D = [1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.9 0.9 0.9 0.9]; 26 | E = [0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.1 0.1 0.1]; 27 | F = [0.0 0.0 0.1 0.1 0.1 0.1 0.2 0.2 0.3 0.9 1.0]; 28 | G = [0.0 0.1 0.7 0.8 0.8 0.9 0.9 0.9 0.9 1.0 1.0]; 29 | 30 | %% Produce values in Table 1 by row pattern A 31 | disp('Table 1. pattern A'); 32 | disp(['Entropy_A = ', num2str(roundn(compute_matrix(A, 1, 0, 1), -2))]); 33 | disp(['Entropy^1/4_A = ', num2str(roundn(compute_matrix(A, 1, 0, 0.25), -2))]); 34 | disp(['Diversity_A = ', num2str(roundn(compute_matrix(A, 0, 1, 1), -2))]); 35 | disp(['Diversity^1/4_A = ', num2str(roundn(compute_matrix(A, 0, 1, 0.25), -2))]); 36 | disp(['(Entropy+Diversity)_A = ', num2str(roundn(compute_matrix(A, 1, 1, 1), -2))]); 37 | disp(['(Entropy+Diversity)^1/4_A = ', num2str(roundn(compute_matrix(A, 1, 1, 0.25), -2))]); 38 | disp(' '); 39 | 40 | %% Produce values in Table 1 by row pattern B 41 | disp('Table 1. pattern B'); 42 | disp(['Entropy_B = ', num2str(roundn(compute_matrix(B, 1, 0, 1), -2))]); 43 | disp(['Entropy^1/4_B = ', num2str(roundn(compute_matrix(B, 1, 0, 0.25), -2))]); 44 | disp(['Diversity_B = ', num2str(roundn(compute_matrix(B, 0, 1, 1), -2))]); 45 | disp(['Diversity^1/4_B = ', num2str(roundn(compute_matrix(B, 0, 1, 0.25), -2))]); 46 | disp(['(Entropy+Diversity)_B = ', num2str(roundn(compute_matrix(B, 1, 1, 1), -2))]); 47 | disp(['(Entropy+Diversity)^1/4_B = ', num2str(roundn(compute_matrix(B, 1, 1, 0.25), -2))]); 48 | disp(' '); 49 | 50 | %% Produce values in Table 1 by row pattern C 51 | disp('Table 1. pattern C'); 52 | disp(['Entropy_C = ', num2str(roundn(compute_matrix(C, 1, 0, 1), -2))]); 53 | disp(['Entropy^1/4_C = ', num2str(roundn(compute_matrix(C, 1, 0, 0.25), -2))]); 54 | disp(['Diversity_C = ', num2str(roundn(compute_matrix(C, 0, 1, 1), -2))]); 55 | disp(['Diversity^1/4_C = ', num2str(roundn(compute_matrix(C, 0, 1, 0.25), -2))]); 56 | disp(['(Entropy+Diversity)_C = ', num2str(roundn(compute_matrix(C, 1, 1, 1), -2))]); 57 | disp(['(Entropy+Diversity)^1/4_C = ', num2str(roundn(compute_matrix(C, 1, 1, 0.25), -2))]); 58 | disp(' '); 59 | 60 | %% Produce values in Table 1 by row pattern D 61 | disp('Table 1. pattern D'); 62 | disp(['Entropy_D = ', num2str(roundn(compute_matrix(D, 1, 0, 1), -2))]); 63 | disp(['Entropy^1/4_D = ', num2str(roundn(compute_matrix(D, 1, 0, 0.25), -2))]); 64 | disp(['Diversity_D = ', num2str(roundn(compute_matrix(D, 0, 1, 1), -2))]); 65 | disp(['Diversity^1/4_D = ', num2str(roundn(compute_matrix(D, 0, 1, 0.25), -2))]); 66 | disp(['(Entropy+Diversity)_D = ', num2str(roundn(compute_matrix(D, 1, 1, 1), -2))]); 67 | disp(['(Entropy+Diversity)^1/4_D = ', num2str(roundn(compute_matrix(D, 1, 1, 0.25), -2))]); 68 | disp(' '); 69 | 70 | %% Produce values in Table 1 by row pattern E 71 | disp('Table 1. pattern E'); 72 | disp(['Entropy_E = ', num2str(roundn(compute_matrix(E, 1, 0, 1), -2))]); 73 | disp(['Entropy^1/4_E = ', num2str(roundn(compute_matrix(E, 1, 0, 0.25), -2))]); 74 | disp(['Diversity_E = ', num2str(roundn(compute_matrix(E, 0, 1, 1), -2))]); 75 | disp(['Diversity^1/4_E = ', num2str(roundn(compute_matrix(E, 0, 1, 0.25), -2))]); 76 | disp(['(Entropy+Diversity)_E = ', num2str(roundn(compute_matrix(E, 1, 1, 1), -2))]); 77 | disp(['(Entropy+Diversity)^1/4_E = ', num2str(roundn(compute_matrix(E, 1, 1, 0.25), -2))]); 78 | disp(' '); 79 | 80 | %% Produce values in Table 1 by row pattern F 81 | disp('Table 1. pattern F'); 82 | disp(['Entropy_F = ', num2str(roundn(compute_matrix(F, 1, 0, 1), -2))]); 83 | disp(['Entropy^1/4_F = ', num2str(roundn(compute_matrix(F, 1, 0, 0.25), -2))]); 84 | disp(['Diversity_F = ', num2str(roundn(compute_matrix(F, 0, 1, 1), -2))]); 85 | disp(['Diversity^1/4_F = ', num2str(roundn(compute_matrix(F, 0, 1, 0.25), -2))]); 86 | disp(['(Entropy+Diversity)_F = ', num2str(roundn(compute_matrix(F, 1, 1, 1), -2))]); 87 | disp(['(Entropy+Diversity)^1/4_F = ', num2str(roundn(compute_matrix(F, 1, 1, 0.25), -2))]); 88 | disp(' '); 89 | 90 | %% Produce values in Table 1 by row pattern G 91 | disp('Table 1. pattern G'); 92 | disp(['Entropy_G = ', num2str(roundn(compute_matrix(G, 1, 0, 1), -2))]); 93 | disp(['Entropy^1/4_G = ', num2str(roundn(compute_matrix(G, 1, 0, 0.25), -2))]); 94 | disp(['Diversity_G = ', num2str(roundn(compute_matrix(G, 0, 1, 1), -2))]); 95 | disp(['Diversity^1/4_G = ', num2str(roundn(compute_matrix(G, 0, 1, 0.25), -2))]); 96 | disp(['(Entropy+Diversity)_G = ', num2str(roundn(compute_matrix(G, 1, 1, 1), -2))]); 97 | disp(['(Entropy+Diversity)^1/4_G = ', num2str(roundn(compute_matrix(G, 1, 1, 0.25), -2))]); 98 | disp(' '); 99 | 100 | 101 | % %% Visualize the diversity equation 102 | % x = 0:0.01:1; 103 | % y = 0:0.01:1; 104 | % [x,y] = meshgrid(x,y); 105 | % div = (x-y).*log(x./(y+eps))+(y-x).*log((1-x)./(1-y+eps)); 106 | % mesh(x,y,div) 107 | 108 | end 109 | 110 | %% Compute matrix R (Eq. 3 in the paper) 111 | function [S] = compute_matrix(p, lamda_1, lamda_2, alpha) 112 | 113 | % lamda_1 and lamda_2 are trade-offs between entropy and diversity 114 | % lamda_1: turn on/off entropy (1 or 0) 115 | % lamda_2: turn on/off diversity (1 or 0) 116 | 117 | 118 | % Handling noisy labels via majority selection (see Sec 3.3) 119 | top = round(length(p) * alpha); 120 | p = Majority(p,top); 121 | 122 | % Initialize the R matrix 123 | R = zeros(length(p)); 124 | for i = 1:length(p) 125 | for j = 1:i 126 | if i == j % entropy computed for diagonal of R 127 | R(i,i) = - lamda_1 * (p(i)*log(p(i)+eps)+(1-p(i))*log(1-p(i)+eps)); 128 | else % diversity computed for rest of R 129 | R(i,j) = lamda_2 * ((p(i)-p(j)) * log((p(i)+eps)/(p(j)+eps)) + ... 130 | (1-p(i)-1+p(j)) * log((1-p(i)+eps)/(1-p(j)+eps))); 131 | R(j,i) = R(i,j); 132 | end 133 | end 134 | end 135 | 136 | % Compute active selecting value from matrix R for this candidate 137 | % by numerical sum of R (see Alg 1 line 11) 138 | S = sum(R(:)); 139 | 140 | 141 | end 142 | 143 | 144 | %% Sec 3.3. Handling noisy labels via majority selection 145 | % eq. 4, calculate the mean and sort based on dominate predictions. 146 | function [p] = Majority(p,top) 147 | 148 | m = mean(p); 149 | if m > 0.5 150 | p = sort(p,'descend'); 151 | else 152 | p = sort(p,'ascend'); 153 | end 154 | p = p(1:top); 155 | 156 | end --------------------------------------------------------------------------------