├── .gitignore ├── 1_preprocessor ├── README.md ├── __init__.py ├── augmentation.py ├── config.py ├── crop_RBC.py ├── file_function.py └── main.py ├── 2_classifier ├── README.md ├── __init__.py ├── config.py ├── inference.py ├── main.py ├── networks │ ├── __init__.py │ └── resnet.py └── scripts │ ├── inference │ ├── resnet.sh │ └── vggnet.sh │ └── train │ ├── alexnet.sh │ ├── densenet.sh │ ├── resnet.sh │ └── vggnet.sh ├── 3_detector ├── README.md ├── __init__.py ├── baseline_prediction.py ├── config.py ├── detect_object.py ├── draw_bbox.py ├── grad_cam.py ├── inference_bbox.py ├── launch_model.py ├── misc_functions.py ├── networks │ ├── __init__.py │ └── resnet.py └── scripts │ ├── detect.sh │ └── inference.sh ├── INSTALL.md ├── LICENSE ├── README.md ├── SERVER.md └── imgs ├── cat_test1.jpg ├── cat_test2.jpg ├── catdog_test1.jpg ├── catdog_test2.jpg ├── detection ├── after.png └── before.png ├── dog_test1.jpg ├── dog_test2.jpg ├── figure.png ├── gradcam.jpg ├── heatmap_out.png ├── input.png ├── output.png ├── prediction.png ├── pytorch.png └── woof_meow.jpg /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | 103 | # results 104 | */results/* 105 | 106 | # Checkpoints & logs 107 | *.t7 108 | *.csv 109 | */results*/* 110 | -------------------------------------------------------------------------------- /1_preprocessor/README.md: -------------------------------------------------------------------------------- 1 | Cell Preprocessor module 2 | ================================================================================================ 3 | Cell Image Preprocessor module of CellNet 4 | 5 | # Requirements 6 | - python 2.7 7 | - [OpenCV](http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_gui/py_image_display/py_image_display.html) 8 | 9 | # Input directory 10 | The input directory should be in the given format: 11 | ```bash 12 | 13 | [:folder] 14 | |-[:class 0] 15 | |-[:img 0] 16 | |-[:img 1] 17 | |-[:img 2] 18 | ... 19 | |-[:class 1] 20 | |-[:class 2] 21 | ... 22 | ... 23 | ... 24 | 25 | ``` 26 | 27 | # Menu Options 28 | If you run the program, you will meet a menu script that will help you through various processes. 29 | 30 | ```bash 31 | $ python main.py 32 | 33 | ################## [ Options ] ########################### 34 | # Mode 1 'print' : Print names of image data file 35 | # Mode 2 'read' : [original/aug] Read names data 36 | # Mode 3 'resize': [target_size] Resize & Orgnaize data 37 | # Mode 4 'split' : Create a train-validation split of data 38 | # Mode 5 'count' : Check the distribution of raw data 39 | # Mode 6 'check' : Check the distribution of train/val split 40 | # Mode 7 'aug' : Augment the training data sample 41 | # Mode 8 'exit' : Terminate the program 42 | ########################################################## 43 | 44 | Enter mode name : 45 | 46 | ``` 47 | 48 | If you enter the mode name in the given line, the code will run the function that has been typed. 49 | 50 | # Modules 51 | 52 | ## 1. print 53 | ```bash 54 | Enter mode name : print 55 | ``` 56 | This module will print all the the file names of image related file formats(".jpg", ".png") 57 | 58 | ## 2. read 59 | ```bash 60 | Enter mode name : read 61 | ``` 62 | This module will read all the images and print out the spacial dimension of image related files. 63 | 64 | ## 3. resize 65 | ```bash 66 | Enter mode name : resize 67 | ``` 68 | This module will save all the resized images into your given directory 69 | 70 | ## 4. split 71 | ```bash 72 | Enter mode name : split 73 | ``` 74 | This module will organize your input file directory into the following format. 75 | You should manually set how much validation sets you want in your val class in val_num from [config.py](./config.py). 76 | 77 | ```bash 78 | [:folder] 79 | |-train 80 | |-[:class 0] 81 | |-[:img 0] 82 | |-[:img 1] 83 | |-[:img 2] 84 | ... 85 | |-[:class 1] 86 | |-[:class 2] 87 | ... 88 | ... 89 | ... 90 | |-val 91 | |-[:class 0] 92 | |-[:img 0] 93 | |-[:img 1] 94 | |-[:img 2] 95 | ... 96 | |-[:class 1] 97 | |-[:class 2] 98 | ... 99 | ... 100 | ... 101 | 102 | ``` 103 | 104 | ## 5. count 105 | ```bash 106 | Enter mode name : count 107 | ``` 108 | This will count the number of images within each sub-categories in the data. 109 | An example for the file directory after running module 5 (count) is as below. 110 | ```bash 111 | $ Enter mode name : count 112 | 113 | | Cat_vs_Dog dataset : 114 | | cat 12500 115 | | dog 12500 116 | ``` 117 | 118 | ## 6. check 119 | ```bash 120 | Enter mode name : check 121 | ``` 122 | This will check how your train/validation split is consisted. 123 | An example for the file directory after running module 4 (split) is as below. 124 | ```bash 125 | $ Enter mode name : check 126 | 127 | | train set : 128 | | cat 100 129 | | dog 100 130 | | val set : 131 | | cat 100 132 | | dog 100 133 | 134 | ``` 135 | 136 | ## 7. augmentation 137 | ```bash 138 | Enter mode name : aug 139 | ``` 140 | This module will apply various image augmentations and enlarge your training set. 141 | The input should be the splitted directory after running module 4 (split) 142 | 143 | 144 | -------------------------------------------------------------------------------- /1_preprocessor/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/1_preprocessor/__init__.py -------------------------------------------------------------------------------- /1_preprocessor/augmentation.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2017 3 | # Github : https://github.com/meliketoy/cellnet.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Preprocessing Implementation 7 | # 8 | # Module : 1_preprocessor 9 | # Description : augmentation.py 10 | # The code for image augmentation. 11 | # *********************************************************** 12 | 13 | import cv2 14 | import os 15 | import numpy as np 16 | import random 17 | 18 | # Give random contrast to the image. 19 | def random_contrast(image, lower=0.2, upper=1.8, seed=None): 20 | contrast_factor = np.random.uniform(lower, upper) 21 | return (image-np.mean(image))*contrast_factor + np.mean(image) 22 | 23 | # Give random brightness emphasis to the image. 24 | def random_brightness(image, max_delta=63, seed=None): 25 | delta = np.random.randint(-max_delta, max_delta) 26 | return image-delta 27 | 28 | # Random Crop a portion of the image. 29 | def random_crop(image, dim): 30 | if len(image.shape): 31 | W, H, D = image.shape 32 | w, h, d = dim 33 | else: 34 | W, H = image.shape 35 | w, h = size 36 | 37 | left, top = np.random.randint(W-w+1), np.random.randint(H-h+1) 38 | return image[left:left+w, top:top+h] 39 | 40 | # Rotate the image. 41 | def rotation(image, num, mode='random'): 42 | if(mode == 'strict'): 43 | deg = 90 * num 44 | (h,w) = image.shape[:2] 45 | center = (w/2, h/2) 46 | 47 | M = cv2.getRotationMatrix2D(center, deg, 1.0) 48 | rotated = cv2.warpAffine(image, M, (w,h)) 49 | elif(mode == 'random'): 50 | deg = random.randrange(1, 360) 51 | (h,w) = image.shape[:2] 52 | center = (w/2, h/2) 53 | mean_val = [0.0, 0.0, 0.0] 54 | 55 | for channel in range(3): 56 | one_channel = image[:,:,channel] 57 | outer = np.append(one_channel[0,:-1], one_channel[-1,1:]) 58 | tmp = np.append(one_channel[:-1,-1], one_channel[1:,0]) 59 | outer = np.append(outer,tmp) 60 | outer = outer.flatten() 61 | mean_val[channel] = np.mean(outer[outer!=0]) # Fill the void area around the rotated image with the mean-value of the non-zero pixels 62 | 63 | M = cv2.getRotationMatrix2D(center, deg, 1.0) 64 | rotated = cv2.warpAffine(image, M, (w,h), borderMode=cv2.BORDER_CONSTANT, borderValue=mean_val) 65 | 66 | return rotated 67 | -------------------------------------------------------------------------------- /1_preprocessor/config.py: -------------------------------------------------------------------------------- 1 | ############# Configuration file ############# 2 | 3 | # Name of dataset 4 | name = 'Cat_vs_Dog' 5 | 6 | # Base directory for data formats 7 | data_base = '/mnt/datasets/' + name 8 | 9 | # Base directory for augmented data formats 10 | resize_base = '/home/bumsoo/Data/resized/' 11 | split_base = '/home/bumsoo/Data/split/' 12 | 13 | # Directory for data formats 14 | resize_dir = resize_base + name 15 | split_dir = split_base + name 16 | 17 | # Train augmentation 18 | rotate_mode = 'strict' 19 | 20 | # Validation split 21 | split = 'fix' # [ratio/fix] 22 | val_ratio = 0.2 23 | val_num = 100 24 | -------------------------------------------------------------------------------- /1_preprocessor/crop_RBC.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import sys 4 | import random 5 | import numpy as np 6 | import config as cf 7 | import file_function as ff 8 | import math 9 | 10 | def generate_padding_image(image, stepSize, windowSize): 11 | border_x = int(stepSize - ((image.shape[0]-windowSize)%stepSize)) 12 | border_y = int(stepSize - ((image.shape[1]-windowSize)%stepSize)) 13 | 14 | pad_image = cv2.copyMakeBorder(image, 0, border_x, 0, border_y, cv2.BORDER_CONSTANT, value=[255,255,255]) 15 | 16 | return pad_image 17 | 18 | def random_crop(image, dim): 19 | if(len(image.shape)): 20 | W, H, D = image.shape 21 | w, h, d = dim 22 | else: 23 | W, H = image.shape 24 | w, h = size 25 | 26 | left, top = np.random.randint(W-w+1), np.random.randint(H-h+1) 27 | 28 | return image[left:left+w, top:top+h] 29 | 30 | def change_img_dir(in_dir, out_dir): 31 | img_cnt = 0 32 | ff.check_and_mkdir(out_dir) 33 | 34 | for subdir, dirs, files in os.walk(in_dir): 35 | for f in files: 36 | if ff.is_image(f): 37 | file_path = os.path.join(subdir, f) 38 | img = cv2.imread(file_path) 39 | 40 | img_cnt += 1 41 | out_path = os.path.join(out_dir, "RBC_%d.png" %img_cnt) 42 | print("Saving %s ..." %out_path) 43 | cv2.imwrite(out_path, img) 44 | 45 | def save_random_crop(in_dir, out_dir): 46 | windowSize = random.randint(80, 100) 47 | img_cnt = 0 48 | subdir_cnt = 0 49 | ff.check_and_mkdir(out_dir) 50 | 51 | for subdir, dirs, files in os.walk(in_dir): 52 | for f in files: 53 | if ff.is_image(f): 54 | file_path = os.path.join(subdir, f) 55 | image = cv2.imread(file_path) 56 | 57 | print("Saving from %s ... " %f) 58 | subdir_cnt += 1 59 | 60 | for i in range(5): 61 | crop = random_crop(image, (windowSize, windowSize, 3)) 62 | img_cnt += 1 63 | save_dir = os.path.join(out_dir, "Crop_%d.png" %img_cnt) 64 | print("\tSaving %s ..." %save_dir) 65 | cv2.imwrite(save_dir, crop) 66 | 67 | print(subdir_cnt) 68 | 69 | def save_sliding_windows(in_dir, out_dir, stepSize, windowSize): 70 | img_cnt = 0 71 | ff.check_and_mkdir(out_dir) 72 | 73 | for subdir, dirs, files in os.walk(in_dir): 74 | for f in files: 75 | if ff.is_image(f): 76 | file_path = os.path.join(subdir, f) 77 | image = cv2.imread(file_path) 78 | windowSize = random.randint(80, 100) 79 | #print(image.shape) 80 | 81 | #image = generate_padding_image(image, stepSize, windowSize) 82 | #print(image.shape) 83 | 84 | for x in range(0, image.shape[0], stepSize): 85 | for y in range(0, image.shape[0], stepSize): 86 | if (x+windowSize <= image.shape[0] and y+windowSize <= image.shape[0]): 87 | save_img = image[x:x+windowSize, y:y+windowSize, :] 88 | 89 | if (save_img.shape[0] == save_img.shape[1]): 90 | img_cnt += 1 91 | save_dir = os.path.join(out_dir, "Crop_%d.png" %img_cnt) 92 | print("Saving %s ..." %save_dir) 93 | cv2.imwrite(save_dir, save_img) 94 | 95 | def pick_random_slides(in_dir, out_dir): 96 | ff.check_and_mkdir(out_dir) 97 | 98 | path, dirs, files = os.walk(in_dir).next() 99 | file_count = len(files) 100 | 101 | lst = random.sample(range(0, file_count), (1000-377+25)) 102 | 103 | for file_num in lst: 104 | file_path = os.path.join(in_dir, "Crop_%d.png" %file_num) 105 | img = cv2.imread(file_path) 106 | 107 | save_path = os.path.join(out_dir, "Crop_%d.png" %file_num) 108 | print(save_path) 109 | cv2.imwrite(save_path, img) 110 | 111 | if __name__ == "__main__": 112 | # for atypical sets 113 | #in_dir = "/home/bumsoo/Junhyun/atypical" 114 | #out_dir = "/home/bumsoo/Data/resized/RBC/RBC" 115 | 116 | # for crop 117 | in_dir = "/home/bumsoo/Junhyun/smudge"; out_dir = "/home/bumsoo/Data/resized/Cropped" #os.path.join(cf.resized_base, "RBC") 118 | 119 | # for random pick 120 | #in_dir = "/home/bumsoo/Data/resized/Cropped"; out_dir = "/home/bumsoo/Data/resized/RBC/RBC" 121 | 122 | #change_img_dir(in_dir, out_dir) 123 | #save_sliding_windows(in_dir, out_dir, stepSize=50, windowSize=100) 124 | save_random_crop(in_dir, out_dir) 125 | #pick_random_slides(in_dir, out_dir) 126 | -------------------------------------------------------------------------------- /1_preprocessor/file_function.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2017 3 | # Github : https://github.com/meliketoy/cellnet.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Preprocessing Implementation 7 | # 8 | # Module : 1_preprocessor 9 | # Description : file_function.py 10 | # The function codes for file management. 11 | # *********************************************************** 12 | 13 | import os 14 | import cv2 15 | import sys 16 | import csv 17 | import augmentation as aug 18 | import config as cf 19 | import numpy as np 20 | from operator import div 21 | 22 | # print all the name of images in the directory. 23 | def print_all_imgs(in_dir): 24 | for subdir, dirs, files in os.walk(in_dir): 25 | for f in files: 26 | file_path = subdir + os.sep + f 27 | if (is_image(f)): 28 | print(file_path) 29 | 30 | # check if the given file is an image format 31 | def is_image(f): 32 | return f.endswith(".png") or f.endswith(".jpg") 33 | 34 | # check if dir exists. If not, mkdir. 35 | def check_and_mkdir(in_dir): 36 | if not os.path.exists(in_dir): 37 | print("Creating "+in_dir+"...") 38 | os.makedirs(in_dir) 39 | 40 | # read and print all the image sizes of the dir. 41 | def read_all_imgs(in_dir): 42 | for subdir, dirs, files in os.walk(in_dir): 43 | for f in files: 44 | file_path = subdir + os.sep + f 45 | if (is_image(f)): 46 | img = cv2.imread(file_path) 47 | print('{:<100} {:>10}'.format(file_path, str(img.shape))) 48 | # print(file_path + ",img size = "+str(img.shape)) 49 | 50 | # resize the imgs from in_dir, and save with exact same hierarchy in the out_dir 51 | def resize_images(in_dir, out_dir, target_size): 52 | check_and_mkdir(out_dir) # sanity check for the target output directory 53 | 54 | for subdir, dirs, files in os.walk(in_dir): 55 | for f in files: 56 | file_path = subdir + os.sep + f 57 | if (is_image(f)): 58 | img = cv2.imread(file_path) 59 | resized_img = cv2.resize(img, (target_size, target_size), interpolation = cv2.INTER_CUBIC) 60 | class_dir = out_dir + os.sep + file_path.split("/")[-2] 61 | if len(file_path.split("/")) >= 7: 62 | out_dir = cf.split_dir 63 | class_dir = os.path.join(out_dir, file_path.split("/")[-3], file_path.split("/")[-2]) 64 | check_and_mkdir(class_dir) # sanity check for the target class directory 65 | 66 | file_name = class_dir + os.sep + file_path.split("/")[-1] 67 | print(file_name) 68 | cv2.imwrite(file_name, resized_img) 69 | 70 | def resize_and_contrast(in_dir, out_dir, target_size): 71 | check_and_mkdir(out_dir) 72 | clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8)) 73 | 74 | for subdir, dirs, files in os.walk(in_dir): 75 | for f in files: 76 | file_path = subdir + os.sep + f 77 | if (is_image(f)): 78 | img = cv2.imread(file_path, 0) 79 | resized_img = cv2.resize(img, (target_size, target_size), interpolation = cv2.INTER_CUBIC) 80 | class_dir = out_dir + os.sep + file_path.split("/")[-2] 81 | check_and_mkdir(class_dir) 82 | 83 | file_name = class_dir + os.sep + file_path.split("/")[-1] 84 | print(file_name) 85 | 86 | norm_image = cv2.normalize(resized_img, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F) * 256 87 | # norm_image = clahe.apply(resized_img) 88 | cv2.imwrite(file_name, norm_image) 89 | 90 | # count the direct one-step sub directories (which will represent the class name) 91 | def class_info(in_dir, mode): 92 | class_lst = [] 93 | 94 | for subdir, dirs, files in os.walk(in_dir): 95 | class_lst = dirs # the 'dirs' variable after the first os.walk loop will return the list of classes 96 | break 97 | 98 | if(mode == "len"): 99 | return (len(class_lst)) 100 | elif(mode == "list"): 101 | return (class_lst) 102 | 103 | # count the containing images of each classes 104 | def count_each_class(in_dir): 105 | class_lst, cnt_lst = class_info(in_dir, "list"), [] 106 | 107 | for class_dir in class_lst: 108 | class_count = 0 109 | for subdir, dirs, files in os.walk(in_dir + os.sep + class_dir): 110 | for f in files: 111 | file_path = subdir + os.sep + f 112 | if (is_image(f)): 113 | class_count += 1 114 | 115 | print("\t| {:<15} {:>5}".format(class_dir, class_count)) 116 | cnt_lst.append(class_count) 117 | 118 | return cnt_lst 119 | 120 | # return whether the current phase is 'train' or 'validation' 121 | def return_phase(num, val_num): 122 | if (num < val_num): 123 | return "val" + os.sep 124 | else: 125 | return "train" + os.sep 126 | 127 | # create a train-val sub-organized directory from the original class directory 128 | def create_train_val_split(in_dir, split_dir, split=cf.split): 129 | print("Saving train-val splitted images into %s" %(split_dir)) 130 | check_and_mkdir(split_dir) 131 | class_lst = class_info(in_dir, "list") 132 | 133 | for phase in ["train", "val"]: 134 | phase_dir = split_dir + os.sep + phase # The output directory will be "./split/[:file_dir]/[:phase]/[:class]" 135 | check_and_mkdir(phase_dir) 136 | 137 | for cls in class_lst: 138 | cls_dir = split_dir + os.sep + phase + os.sep + cls # Where to read the image from 139 | check_and_mkdir(cls_dir) 140 | 141 | # val_num = cf.val_num # temporary 142 | for subdir, dirs, files in os.walk(in_dir): 143 | val_num = int(len(files)*cf.val_ratio) if (split=='ratio') else cf.val_num 144 | cnt = 0 145 | for f in files: 146 | file_path = subdir + os.sep + f 147 | if(is_image(f)): 148 | img = cv2.imread(file_path) 149 | cv2.imwrite(split_dir + os.sep + return_phase(cnt, val_num) + subdir.split("/")[-1] + os.sep + f, img) 150 | cnt += 1 151 | 152 | return split_dir 153 | 154 | # get train-val information 155 | def get_split_info(split_dir): 156 | # Must be activated after the 'split' option. 157 | for phase in ["train", "val"]: 158 | print("| %s set : " %phase) 159 | count_each_class(split_dir + os.sep + phase) 160 | 161 | return split_dir 162 | 163 | # train data augmentation 164 | def aug_train(split_dir, mode): 165 | train_dir = split_dir + os.sep + "train" 166 | 167 | for subdir, dirs, files in os.walk(train_dir): 168 | for f in files: 169 | file_path = subdir + os.sep + f 170 | if (is_image(f)): 171 | print(file_path) 172 | name, ext = os.path.splitext(f) 173 | img = cv2.imread(file_path) 174 | for i in range(1,4): 175 | rot_dir = (subdir + os.sep + name + "_aug_"+str(i*90)+ext) 176 | if(mode == 'random'): 177 | cv2.imwrite(rot_dir, aug.rotation(img, 0, 'random')) 178 | elif(mode == 'strict'): 179 | cv2.imwrite(rot_dir, aug.rotation(img, i, 'strict')) 180 | else: 181 | print("The mode should be either random | strict") 182 | sys.exit(1) 183 | 184 | def train_mean(split_dir): 185 | train_dir = split_dir + os.sep + "train" 186 | train_img_num = 0 187 | train_mean_lst = [0.0, 0.0, 0.0] 188 | 189 | for subdir, dirs, files in os.walk(train_dir): 190 | for f in files: 191 | file_path = subdir + os.sep + f 192 | if (is_image(f)): 193 | img = cv2.imread(file_path) 194 | train_img_num += 1 195 | for channel in range(3): 196 | train_mean_lst[channel] += img[:,:,channel].mean() 197 | 198 | mean_map = map(div, train_mean_lst, [train_img_num, train_img_num, train_img_num]) 199 | return map(div, mean_map, [255.0, 255.0, 255.0]) 200 | 201 | def train_std(split_dir, train_mean): 202 | train_dir = split_dir + os.sep + "train" 203 | train_img_num = 0 204 | train_std_lst = [0.0, 0.0, 0.0] 205 | 206 | for subdir, dirs, files in os.walk(train_dir): 207 | for f in files: 208 | file_path = subdir + os.sep + f 209 | if (is_image(f)): 210 | img = cv2.imread(file_path) 211 | train_img_num += 1 212 | for channel in range(3): 213 | train_std_lst[channel] += img[:,:,channel].var() # per image var() 214 | 215 | std_map = map(div, train_std_lst, [train_img_num, train_img_num, train_img_num]) 216 | std_map = np.sqrt(std_map) 217 | return map(div, std_map, [255.0, 255.0, 255.0]) 218 | -------------------------------------------------------------------------------- /1_preprocessor/main.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2017 3 | # Github : https://github.com/meliketoy/cellnet.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Preprocessing Implementation 7 | # 8 | # Module : 1_preprocessor 9 | # Description : main.py 10 | # The main code for data mangement. 11 | # *********************************************************** 12 | 13 | import cv2 14 | import os 15 | import sys 16 | import file_function as ff 17 | import config as cf 18 | 19 | def print_menu(): 20 | print("\nSelect a mode by its name or number.\nType [exit] to terminate.\n") 21 | print("################## [ Options ] ###########################") 22 | print("# Mode 1 'print' : Print names of image data file") 23 | print("# Mode 2 'read' : [original/resized] Read names data") 24 | print("# Mode 3 'resize' : [target_size] Resize & Orgnaize data") 25 | print("# Mode 4 'split' : Create a train-validation split of data") 26 | print("# Mode 5 'count' : Check the distribution of raw data") 27 | print("# Mode 6 'check' : Check the distribution of train/val split") 28 | print("# Mode 7 'aug' : Augment the training data samples") 29 | print("# Mode 8 'meanstd': Return the meanstd value of the training set") 30 | print("# Mode 9 'test' : Preprocess the test data samples") 31 | print("##########################################################") 32 | 33 | 34 | if __name__ == "__main__": 35 | while(1): 36 | print_menu() 37 | mode = raw_input('\nEnter mode name : ') 38 | 39 | ############################################## 40 | # @ Module 1 : Print names of image data file 41 | if (mode == 'print' or mode == '1'): 42 | ff.print_all_imgs(cf.data_base) 43 | 44 | ############################################# 45 | # @ Module 2 : Read all images 46 | elif (mode == 'read' or mode == '2'): 47 | path = raw_input('Enter [original/resized] : ') 48 | if (not path in ['original', 'resized']): 49 | print("[Error] : Please define the mode between [original/resized].") 50 | else: 51 | if(path == 'original'): 52 | ff.read_all_imgs(cf.data_base) 53 | elif(path == 'resized'): 54 | ff.read_all_imgs(cf.resize_dir) 55 | 56 | ############################################# 57 | # @ Module 3 : Resize and check images 58 | elif (mode == 'resize' or mode == '3'): 59 | ff.check_and_mkdir(cf.resize_base) 60 | target_size = int(raw_input('Enter size : ')) 61 | ff.resize_images(cf.data_base, cf.resize_dir, target_size) 62 | # ff.resize_and_contrast(cf.data_base, cf.resize_dir, target_size) 63 | 64 | ############################################# 65 | # @ Module 4 : Train-Validation split 66 | elif (mode == 'split' or mode == '4'): 67 | ff.check_and_mkdir(cf.split_base) 68 | split_dir = ff.create_train_val_split(cf.resize_dir, cf.split_dir) 69 | print("Train-Validation split directory = " + cf.split_dir) 70 | 71 | ############################################ 72 | # @ Module 5 : Check the dataset 73 | elif (mode == 'count' or mode == '5'): 74 | print("| " + cf.resize_dir.split("/")[-1] + " dataset : ") 75 | ff.count_each_class(cf.resize_dir) 76 | elif (mode == 'check' or mode == '6'): 77 | ff.get_split_info(cf.split_dir) 78 | 79 | ############################################ 80 | # @ Module 6 : Training data augmentation 81 | elif (mode == 'aug' or mode == '7'): 82 | #if (len(sys.argv) < 3): 83 | # print("[Error] : Please define size in the second arguement.") 84 | #else: 85 | ff.aug_train(cf.split_dir, cf.rotate_mode)#sys.argv[2]) 86 | 87 | ############################################# 88 | # @ Module 7 : Retrieve Training data meanstd 89 | elif (mode == 'meanstd' or mode == '8'): 90 | mean = ff.train_mean(cf.split_dir) 91 | std = ff.train_std(cf.split_dir, mean) 92 | 93 | print("mean = " + str(mean)) 94 | print("std = " + str(std)) 95 | 96 | ############################################# 97 | # @ Module 8 : Preprocess test data 98 | elif (mode == 'test' or mode == '9'): 99 | # [TO DO] : Implement Test Preprocessor 100 | print("[TO DO] : Implement Test Preprocessor") 101 | 102 | ############################################# 103 | elif (mode == 'exit'): 104 | print("\nGood Bye!\n") 105 | sys.exit(0) 106 | else: 107 | print("[Error] : Wrong input in 'mode name', please enter again.") 108 | ############################################# 109 | -------------------------------------------------------------------------------- /2_classifier/README.md: -------------------------------------------------------------------------------- 1 | Cell Classifier module 2 | ================================================================================================ 3 | Cell classifier module of CellNet 4 | 5 | ## Fine-Tuning 6 | In practice, very few people train an entire Convolutional Network from scratch (with random initialization), because it is relatively rare to have a dataset of sufficient size. Instead, it is common to pretrain a ConvNet on a very large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the ConvNet either as an initialization or a fixed feature extractor for the task of interest. 7 | 8 | Futher explanations can be found [here](http://cs231n.github.io/transfer-learning/). 9 | 10 | ## Basic Setups 11 | Open [config.py](./config.py), and edit the lines below to your data directory. 12 | 13 | ```bash 14 | data_base = [:dir to your original dataset] 15 | aug_base = [:dir to your actually trained dataset] 16 | ``` 17 | 18 | For training, your data file system should be in the following hierarchy. 19 | Organizing codes for your data into the given requirements will be provided [here](https://github.com/meliketoy/image-preprocessing) 20 | 21 | ```bash 22 | [:data file name] 23 | 24 | |-train 25 | |-[:class 0] 26 | |-[:class 1] 27 | |-[:class 2] 28 | ... 29 | |-[:class n] 30 | |-val 31 | |-[:class 0] 32 | |-[:class 1] 33 | |-[:class 2] 34 | ... 35 | |-[:class n] 36 | ``` 37 | 38 | ## How to run 39 | After you have cloned the repository, you can train the dataset by running the script below. 40 | 41 | You can set the dimension of the additional layer in [config.py](./config.py) 42 | 43 | The resetClassifier option will automatically detect the number of classes in your data folder and reset the last classifier layer to the according number. 44 | 45 | ```bash 46 | # zero-base training 47 | python main.py --lr [:lr] --depth [:depth] --resetClassifier 48 | 49 | # fine-tuning 50 | python main.py --finetune --lr [:lr] --depth [:depth] 51 | 52 | # fine-tuning with additional linear layers 53 | python main.py --finetune --addlayer --lr [:lr] --depth [:depth] 54 | ``` 55 | 56 | ## Train various networks 57 | 58 | Supporting networks 59 | - AlexNet 60 | - VGGNet 61 | - ResNet 62 | - DenseNet 63 | 64 | Please modify the [scripts](./train) and run the line below. 65 | 66 | ```bash 67 | $ ./train/[:network].sh 68 | 69 | # For example, if you want to pretrain alexnet, run the code below. 70 | $ ./train/alexnet.sh 71 | ``` 72 | 73 | ## Test (Inference) various networks 74 | 75 | For testing out your fine-tuned model on alexnet, VGG(11, 13, 16, 19), ResNet(18, 34, 50, 101, 152), 76 | 77 | First, set your data directory as test_dir in [config.py](./config.py). 78 | 79 | Please modify the [scripts](./test) and run the line below. 80 | 81 | ```bash 82 | 83 | $ ./test/[:network].sh 84 | 85 | ``` 86 | For example, if you have trained ResNet with 50 layers, first modify the [resnet test script](./test/resnet.sh) 87 | 88 | ```bash 89 | $ vi ./test/resnet.sh 90 | 91 | python main.py \ 92 | --net_type resnet \ 93 | --depth 50 94 | --testOnly 95 | 96 | $ ./test/resnet.sh 97 | 98 | ``` 99 | -------------------------------------------------------------------------------- /2_classifier/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/2_classifier/__init__.py -------------------------------------------------------------------------------- /2_classifier/config.py: -------------------------------------------------------------------------------- 1 | #################### Configuration File #################### 2 | 3 | # Base directory for data formats 4 | name = 'Cat_vs_Dog' 5 | 6 | data_dir = '/home/mnt/datasets/' 7 | aug_dir = '/home/bumsoo/Data/split/' 8 | 9 | # Databases for each formats 10 | data_base = data_dir + name 11 | aug_base = aug_dir + name 12 | test_dir = aug_dir + name + '/val' 13 | 14 | # model option 15 | batch_size = 16 16 | num_epochs = 40 17 | lr_decay_epoch=20 18 | feature_size = 500 19 | 20 | # Global meanstd 21 | mean = [0.42352142932368259, 0.46167925008138017, 0.49023161345837163] 22 | std = [0.22595048333178538, 0.22503028985594206, 0.23220585942785971] 23 | -------------------------------------------------------------------------------- /2_classifier/inference.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2018 3 | # Github : https://github.com/meliketoy/gradcam.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Fine tuning Implementation 7 | # 8 | # Description : inference.py 9 | # The main code for inference test phase of trained model. 10 | # *********************************************************** 11 | 12 | from __future__ import print_function, division 13 | 14 | import torch 15 | import torch.nn as nn 16 | import torch.optim as optim 17 | import torch.backends.cudnn as cudnn 18 | import numpy as np 19 | import config as cf 20 | import torchvision 21 | import time 22 | import copy 23 | import os 24 | import sys 25 | import argparse 26 | import csv 27 | import operator 28 | 29 | from torchvision import datasets, models, transforms 30 | from networks import * 31 | from torch.autograd import Variable 32 | from PIL import Image 33 | 34 | parser = argparse.ArgumentParser(description='Pytorch Cell Classifier Training') 35 | parser.add_argument('--net_type', default='resnet', type=str, help='model') 36 | parser.add_argument('--depth', default=50, type=int, help='depth of model') 37 | args = parser.parse_args() 38 | 39 | # Phase 1 : Data Upload 40 | print('\n[Phase 1] : Data Preperation') 41 | 42 | data_dir = cf.test_dir 43 | trainset_dir = cf.data_base.split("/")[-1] + os.sep 44 | print("| Preparing %s dataset..." %(cf.test_dir.split("/")[-1])) 45 | 46 | use_gpu = torch.cuda.is_available() 47 | 48 | dsets = datasets.ImageFolder(data_dir, None) 49 | 50 | # Set the classes of labels 51 | H = datasets.ImageFolder(os.path.join(cf.aug_base, 'train')) 52 | dset_classes = H.classes 53 | 54 | print("| Inferencing for %d classes" %len(dset_classes)) 55 | 56 | # Phase 2 : Model setup 57 | print('\n[Phase 2] : Model setup') 58 | 59 | def getNetwork(args): 60 | if (args.net_type == 'alexnet'): 61 | file_name = 'alexnet' 62 | elif (args.net_type == 'vggnet'): 63 | file_name = 'vgg-%s' %(args.depth) 64 | elif (args.net_type == 'densenet'): 65 | file_name = 'densenet-%s' %(args.depth) 66 | elif (args.net_type == 'resnet'): 67 | file_name = 'resnet-%s' %(args.depth) 68 | else: 69 | print('[Error]: Network should be either [alexnet / vggnet / resnet]') 70 | sys.exit(1) 71 | 72 | return file_name 73 | 74 | def softmax(x): 75 | return np.exp(x) / np.sum(np.exp(x), axis=0) 76 | 77 | print("| Loading checkpoint model for inference phase...") 78 | assert os.path.isdir('checkpoint'), '[Error]: No checkpoint directory found!' 79 | assert os.path.isdir('checkpoint/'+trainset_dir), '[Error]: No model has been trained on the dataset!' 80 | file_name = getNetwork(args) 81 | checkpoint = torch.load('./checkpoint/'+trainset_dir+file_name+'.t7') 82 | model = checkpoint['model'] 83 | 84 | if use_gpu: 85 | model.cuda() 86 | cudnn.benchmark = True 87 | 88 | model.eval() 89 | 90 | sample_input = Variable(torch.randn(1,3,224,224), volatile=True) 91 | if use_gpu: 92 | sample_input = sample_input.cuda() 93 | 94 | print("\n[Phase 3] : Score Inference") 95 | 96 | def is_image(f): 97 | return f.endswith(".png") or f.endswith(".jpg") 98 | 99 | test_transform = transforms.Compose([ 100 | transforms.Scale(224), 101 | transforms.CenterCrop(224), 102 | transforms.ToTensor(), 103 | transforms.Normalize(cf.mean, cf.std) 104 | ]) 105 | 106 | if not os.path.isdir('result'): 107 | os.mkdir('result') 108 | 109 | output_file = "./result/"+cf.test_dir.split("/")[-1]+"_inference.csv" 110 | 111 | with open(output_file, 'wb') as csvfile: 112 | fields = ['file_name', 'prediction'] 113 | writer = csv.DictWriter(csvfile, fieldnames=fields) 114 | for subdir, dirs, files in os.walk(data_dir): 115 | cor = 0 # number of correct answers 116 | tot = 0 117 | for f in files: 118 | file_path = subdir + os.sep + f 119 | if (is_image(f)): 120 | image = Image.open(file_path)#.convert('RGB') 121 | if test_transform is not None: 122 | image = test_transform(image) 123 | inputs = image 124 | inputs = Variable(inputs, volatile=True) 125 | if use_gpu: 126 | inputs = inputs.cuda() 127 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) # add batch dim in the front 128 | 129 | outputs = model(inputs) 130 | softmax_res = softmax(outputs.data.cpu().numpy()[0]) 131 | index, score = max(enumerate(softmax_res), key=operator.itemgetter(1)) 132 | # sorted_lst = sorted(zip(softmax_res, dset_classes), reverse=True)[:3] # Get Top-3 Results 133 | 134 | if not (dset_classes[index] in file_path.split("/")[-1]): 135 | print(file_path + "," + str(dset_classes[index]) + ": " + str(score)) 136 | """ 137 | if (file_path.split("/")[-2] != dset_classes[index]): 138 | print(file_path + "\t" + str(dset_classes[index]) + "\t" + str(score)) # print wrong answers. 139 | else: 140 | cor += 1 141 | """ 142 | 143 | writer.writerow({'file_name': file_path, 'prediction':dset_classes[index]}); tot += 1 144 | -------------------------------------------------------------------------------- /2_classifier/main.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2018 3 | # Github : https://github.com/meliketoy/gradcam.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Fine tuning Implementation 7 | # 8 | # Description : main.py 9 | # The main code for training classification networks. 10 | # *********************************************************** 11 | 12 | from __future__ import print_function, division 13 | 14 | import torch 15 | import torch.nn as nn 16 | import torch.optim as optim 17 | import torch.backends.cudnn as cudnn 18 | import numpy as np 19 | import config as cf 20 | import torchvision 21 | import time 22 | import copy 23 | import os 24 | import sys 25 | import argparse 26 | import csv 27 | 28 | from torchvision import datasets, models, transforms 29 | from networks import * 30 | from torch.autograd import Variable 31 | 32 | parser = argparse.ArgumentParser(description='PyTorch Digital Mammography Training') 33 | parser.add_argument('--lr', default=1e-3, type=float, help='learning rate') 34 | parser.add_argument('--net_type', default='resnet', type=str, help='model') 35 | parser.add_argument('--depth', default=50, type=int, help='depth of model') 36 | parser.add_argument('--weight_decay', default=5e-4, type=float, help='weight decay') 37 | parser.add_argument('--finetune', '-f', action='store_true', help='Fine tune pretrained model') 38 | parser.add_argument('--addlayer','-a',action='store_true', help='Add additional layer in fine-tuning') 39 | parser.add_argument('--resetClassifier', '-r', action='store_true', help='Reset classifier') 40 | parser.add_argument('--testOnly', '-t', action='store_true', help='Test mode with the saved model') 41 | args = parser.parse_args() 42 | 43 | # Phase 1 : Data Upload 44 | print('\n[Phase 1] : Data Preperation') 45 | 46 | data_transforms = { 47 | 'train': transforms.Compose([ 48 | #transforms.Scale(236), 49 | transforms.RandomSizedCrop(224), 50 | transforms.RandomHorizontalFlip(), 51 | transforms.ToTensor(), 52 | transforms.Normalize(cf.mean, cf.std) 53 | ]), 54 | 'val': transforms.Compose([ 55 | transforms.Scale(224), 56 | transforms.CenterCrop(224), 57 | transforms.ToTensor(), 58 | transforms.Normalize(cf.mean, cf.std) 59 | ]), 60 | } 61 | 62 | data_dir = cf.aug_base 63 | dataset_dir = cf.data_base.split("/")[-1] + os.sep 64 | print("| Preparing model trained on %s dataset..." %(cf.data_base.split("/")[-1])) 65 | dsets = { 66 | x : datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) 67 | for x in ['train', 'val'] 68 | } 69 | dset_loaders = { 70 | x : torch.utils.data.DataLoader(dsets[x], batch_size = cf.batch_size, shuffle=(x=='train'), num_workers=4) 71 | for x in ['train', 'val'] 72 | } 73 | 74 | dset_sizes = {x: len(dsets[x]) for x in ['train', 'val']} 75 | dset_classes = dsets['train'].classes 76 | 77 | use_gpu = torch.cuda.is_available() 78 | 79 | # Phase 2 : Model setup 80 | print('\n[Phase 2] : Model setup') 81 | 82 | def getNetwork(args): 83 | if (args.net_type == 'alexnet'): 84 | net = models.alexnet(pretrained=args.finetune) 85 | file_name = 'alexnet' 86 | elif (args.net_type == 'vggnet'): 87 | if(args.depth == 11): 88 | net = models.vgg11(pretrained=args.finetune) 89 | elif(args.depth == 13): 90 | net = models.vgg13(pretrained=args.finetune) 91 | elif(args.depth == 16): 92 | net = models.vgg16(pretrained=args.finetune) 93 | elif(args.depth == 19): 94 | net = models.vgg19(pretrained=args.finetune) 95 | else: 96 | print('Error : VGGnet should have depth of either [11, 13, 16, 19]') 97 | sys.exit(1) 98 | file_name = 'vgg-%s' %(args.depth) 99 | elif (args.net_type == 'resnet'): 100 | net = resnet(args.finetune, args.depth) 101 | 102 | file_name = 'resnet-%s' %(args.depth) 103 | else: 104 | print('Error : Network should be either [alexnet / vggnet / resnet / densenet]') 105 | sys.exit(1) 106 | 107 | return net, file_name 108 | 109 | def softmax(x): 110 | return np.exp(x) / np.sum(np.exp(x), axis=0) 111 | 112 | # Test only option 113 | if (args.testOnly): 114 | print("| Loading checkpoint model for test phase...") 115 | assert os.path.isdir('checkpoint'), 'Error: No checkpoint directory found!' 116 | _, file_name = getNetwork(args) 117 | print('| Loading '+file_name+".t7...") 118 | checkpoint = torch.load('./checkpoint/'+dataset_dir+'/'+file_name+'.t7') 119 | model = checkpoint['model'] 120 | 121 | if use_gpu: 122 | model.cuda() 123 | model = torch.nn.DataParallel(model, device_ids=range(torch.cuda.device_count())) 124 | cudnn.benchmark = True 125 | 126 | model.eval() 127 | test_loss = 0 128 | correct = 0 129 | total = 0 130 | 131 | testsets = datasets.ImageFolder(cf.test_dir, data_transforms['val']) 132 | 133 | testloader = torch.utils.data.DataLoader( 134 | testsets, 135 | batch_size = 1, 136 | shuffle = False, 137 | num_workers=1 138 | ) 139 | 140 | print("\n[Phase 3 : Inference on %s]" %cf.test_dir) 141 | for batch_idx, (inputs, targets) in enumerate(testloader): 142 | if use_gpu: 143 | inputs, targets = inputs.cuda(), targets.cuda() 144 | inputs, targets = Variable(inputs, volatile=True), Variable(targets) 145 | outputs = model(inputs) 146 | 147 | print(outputs.data.cpu().numpy()[0]) 148 | file_name = 'densenet-%s' %(args.depth) 149 | softmax_res = softmax(outputs.data.cpu().numpy()[0]) 150 | 151 | _, predicted = torch.max(outputs.data, 1) 152 | total += targets.size(0) 153 | correct += predicted.eq(targets.data).cpu().sum() 154 | 155 | acc = 100.*correct/total 156 | print("| Test Result\tAcc@1 %.2f%%" %(acc)) 157 | 158 | sys.exit(0) 159 | 160 | # Training model 161 | def train_model(model, criterion, optimizer, lr_scheduler, num_epochs=cf.num_epochs): 162 | global dataset_dir 163 | since = time.time() 164 | 165 | best_model, best_acc = model, 0.0 166 | 167 | print('\n[Phase 3] : Training Model') 168 | print('| Training Epochs = %d' %num_epochs) 169 | print('| Initial Learning Rate = %f' %args.lr) 170 | print('| Optimizer = SGD') 171 | output_file = "./logs/"+args.net_type+".csv" 172 | 173 | with open(output_file, 'wb') as csvfile: 174 | fields = ['epoch', 'train_acc', 'val_acc'] 175 | writer = csv.DictWriter(csvfile, fieldnames=fields) 176 | for epoch in range(num_epochs): 177 | train_acc = 0 178 | val_acc = 0 179 | for phase in ['train', 'val']: 180 | 181 | if phase == 'train': 182 | optimizer, lr = lr_scheduler(optimizer, epoch) 183 | print('\n=> Training Epoch #%d, LR=%f' %(epoch+1, lr)) 184 | model.train(True) 185 | else: 186 | model.train(False) 187 | model.eval() 188 | 189 | running_loss, running_corrects, tot = 0.0, 0, 0 190 | 191 | for batch_idx, (inputs, labels) in enumerate(dset_loaders[phase]): 192 | if use_gpu: 193 | inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda()) 194 | else: 195 | inputs, labels = Variable(inputs), Variable(labels) 196 | 197 | optimizer.zero_grad() 198 | 199 | # Forward Propagation 200 | outputs = model(inputs) 201 | _, preds = torch.max(outputs.data, 1) 202 | loss = criterion(outputs, labels) 203 | 204 | # Backward Propagation 205 | if phase == 'train': 206 | loss.backward() 207 | optimizer.step() 208 | 209 | # Statistics 210 | running_loss += loss.data[0] 211 | running_corrects += preds.eq(labels.data).cpu().sum() 212 | tot += labels.size(0) 213 | 214 | if (phase == 'train'): 215 | sys.stdout.write('\r') 216 | sys.stdout.write('| Epoch [%2d/%2d] Iter [%3d/%3d]\t\tLoss %.4f\tAcc %.2f%%' 217 | %(epoch+1, num_epochs, batch_idx+1, 218 | (len(dsets[phase])//cf.batch_size)+1, loss.data[0], 100.*running_corrects/tot)) 219 | sys.stdout.flush() 220 | sys.stdout.write('\r') 221 | 222 | epoch_loss = running_loss / dset_sizes[phase] 223 | epoch_acc = running_corrects / dset_sizes[phase] 224 | 225 | if (phase == 'train'): 226 | train_acc = epoch_acc 227 | 228 | if (phase == 'val'): 229 | print('\n| Validation Epoch #%d\t\t\tLoss %.4f\tAcc %.2f%%' 230 | %(epoch+1, loss.data[0], 100.*epoch_acc)) 231 | 232 | if epoch_acc > best_acc : 233 | print('| Saving Best model...\t\t\tTop1 %.2f%%' %(100.*epoch_acc)) 234 | best_acc = epoch_acc 235 | best_model = copy.deepcopy(model) 236 | state = { 237 | 'model': best_model, 238 | 'acc': epoch_acc, 239 | 'epoch':epoch, 240 | } 241 | if not os.path.isdir('checkpoint'): 242 | os.mkdir('checkpoint') 243 | save_point = './checkpoint/'+dataset_dir 244 | if not os.path.isdir(save_point): 245 | os.mkdir(save_point) 246 | torch.save(state, save_point+file_name+'.t7') 247 | 248 | val_acc = epoch_acc 249 | 250 | writer.writerow({'epoch': epoch+1, 'train_acc': train_acc, 'val_acc': val_acc}) 251 | 252 | csvfile.close() 253 | time_elapsed = time.time() - since 254 | print('\nTraining completed in\t{:.0f} min {:.0f} sec'. format(time_elapsed // 60, time_elapsed % 60)) 255 | print('Best validation Acc\t{:.2f}%'.format(best_acc*100)) 256 | 257 | return best_model 258 | 259 | def exp_lr_scheduler(optimizer, epoch, init_lr=args.lr, weight_decay=args.weight_decay, lr_decay_epoch=cf.lr_decay_epoch): 260 | lr = init_lr * (0.5**(epoch // lr_decay_epoch)) 261 | 262 | for param_group in optimizer.param_groups: 263 | param_group['lr'] = lr 264 | param_group['weight_decay'] = weight_decay 265 | 266 | return optimizer, lr 267 | 268 | model_ft, file_name = getNetwork(args) 269 | 270 | if(args.resetClassifier): 271 | print('| Reset final classifier...') 272 | if(args.addlayer): 273 | print('| Add features of size %d' %cf.feature_size) 274 | num_ftrs = model_ft.fc.in_features 275 | feature_model = list(model_ft.fc.children()) 276 | feature_model.append(nn.Linear(num_ftrs, cf.feature_size)) 277 | feature_model.append(nn.BatchNorm1d(cf.feature_size)) 278 | feature_model.append(nn.ReLU(inplace=True)) 279 | feature_model.append(nn.Linear(cf.feature_size, len(dset_classes))) 280 | model_ft.fc = nn.Sequential(*feature_model) 281 | else: 282 | if(args.net_type == 'alexnet' or args.net_type == 'vggnet'): 283 | num_ftrs = model_ft.classifier[6].in_features 284 | feature_model = list(model_ft.classifier.children()) 285 | feature_model.pop() 286 | feature_model.append(nn.Linear(num_ftrs, len(dset_classes))) 287 | model_ft.classifier = nn.Sequential(*feature_model) 288 | elif(args.net_type == 'resnet'): 289 | num_ftrs = model_ft.fc.in_features 290 | model_ft.fc = nn.Linear(num_ftrs, len(dset_classes)) 291 | 292 | if use_gpu: 293 | model_ft = model_ft.cuda() 294 | model_ft = torch.nn.DataParallel(model_ft, device_ids=range(torch.cuda.device_count())) 295 | cudnn.benchmark = True 296 | 297 | if __name__ == "__main__": 298 | criterion = nn.CrossEntropyLoss() 299 | optimizer_ft = optim.SGD(model_ft.parameters(), lr=args.lr, momentum=0.9, weight_decay=args.weight_decay) 300 | model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=cf.num_epochs) 301 | -------------------------------------------------------------------------------- /2_classifier/networks/__init__.py: -------------------------------------------------------------------------------- 1 | from .resnet import * 2 | -------------------------------------------------------------------------------- /2_classifier/networks/resnet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import math 3 | import torch.utils.model_zoo as model_zoo 4 | 5 | __all__ = ['ResNet', 'resnet'] 6 | 7 | 8 | model_urls = { 9 | 'resnet18': 'http://download.pytorch.org/models/resnet18-5c106cde.pth', 10 | 'resnet34': 'http://download.pytorch.org/models/resnet34-333f7ec4.pth', 11 | 'resnet50': 'http://download.pytorch.org/models/resnet50-19c8e357.pth', 12 | 'resnet101': 'http://download.pytorch.org/models/resnet101-5d3b4d8f.pth', 13 | 'resnet152': 'http://download.pytorch.org/models/resnet152-b121ed2d.pth', 14 | } 15 | 16 | 17 | def conv3x3(in_planes, out_planes, stride=1): 18 | "3x3 convolution with padding" 19 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, 20 | padding=1, bias=False) 21 | 22 | def cfg(depth): 23 | depth_lst = [18, 34, 50, 101, 152] 24 | assert (depth in depth_lst), "Error : ResNet depth should be either 18, 34, 50, 101, 152" 25 | cf_dict = { 26 | '18' : (BasicBlock, [2,2, 2,2]), 27 | '34' : (BasicBlock, [3,4, 6,3]), 28 | '50' : (Bottleneck, [3,4, 6,3]), 29 | '101': (Bottleneck, [3,4,23,3]), 30 | '152': (Bottleneck, [3,8,36,3]), 31 | } 32 | 33 | return cf_dict[str(depth)] 34 | 35 | 36 | class BasicBlock(nn.Module): 37 | expansion = 1 38 | 39 | def __init__(self, inplanes, planes, stride=1, downsample=None): 40 | super(BasicBlock, self).__init__() 41 | self.conv1 = conv3x3(inplanes, planes, stride) 42 | self.bn1 = nn.BatchNorm2d(planes) 43 | self.relu = nn.ReLU(inplace=True) 44 | self.conv2 = conv3x3(planes, planes) 45 | self.bn2 = nn.BatchNorm2d(planes) 46 | self.downsample = downsample 47 | self.stride = stride 48 | 49 | def forward(self, x): 50 | residual = x 51 | 52 | out = self.conv1(x) 53 | out = self.bn1(out) 54 | out = self.relu(out) 55 | 56 | out = self.conv2(out) 57 | out = self.bn2(out) 58 | 59 | if self.downsample is not None: 60 | residual = self.downsample(x) 61 | 62 | out += residual 63 | out = self.relu(out) 64 | 65 | return out 66 | 67 | 68 | class Bottleneck(nn.Module): 69 | expansion = 4 70 | 71 | def __init__(self, inplanes, planes, stride=1, downsample=None): 72 | super(Bottleneck, self).__init__() 73 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False) 74 | self.bn1 = nn.BatchNorm2d(planes) 75 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, 76 | padding=1, bias=False) 77 | self.bn2 = nn.BatchNorm2d(planes) 78 | self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False) 79 | self.bn3 = nn.BatchNorm2d(planes * 4) 80 | self.relu = nn.ReLU(inplace=True) 81 | self.downsample = downsample 82 | self.stride = stride 83 | 84 | def forward(self, x): 85 | residual = x 86 | 87 | out = self.conv1(x) 88 | out = self.bn1(out) 89 | out = self.relu(out) 90 | 91 | out = self.conv2(out) 92 | out = self.bn2(out) 93 | out = self.relu(out) 94 | 95 | out = self.conv3(out) 96 | out = self.bn3(out) 97 | 98 | if self.downsample is not None: 99 | residual = self.downsample(x) 100 | 101 | out += residual 102 | out = self.relu(out) 103 | 104 | return out 105 | 106 | class ResNet(nn.Module): 107 | 108 | def __init__(self, block, layers, num_classes=1000): 109 | self.inplanes = 64 110 | super(ResNet, self).__init__() 111 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, 112 | bias=False) 113 | self.bn1 = nn.BatchNorm2d(64) 114 | self.relu = nn.ReLU(inplace=True) 115 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 116 | self.layer1 = self._make_layer(block, 64, layers[0]) 117 | self.layer2 = self._make_layer(block, 128, layers[1], stride=2) 118 | self.layer3 = self._make_layer(block, 256, layers[2], stride=2) 119 | self.layer4 = self._make_layer(block, 512, layers[3], stride=2) 120 | self.avgpool = nn.AvgPool2d(7) 121 | self.fc = nn.Linear(512 * block.expansion, num_classes) 122 | 123 | for m in self.modules(): 124 | if isinstance(m, nn.Conv2d): 125 | n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels 126 | m.weight.data.normal_(0, math.sqrt(2. / n)) 127 | elif isinstance(m, nn.BatchNorm2d): 128 | m.weight.data.fill_(1) 129 | m.bias.data.zero_() 130 | 131 | def _make_layer(self, block, planes, blocks, stride=1): 132 | downsample = None 133 | if stride != 1 or self.inplanes != planes * block.expansion: 134 | downsample = nn.Sequential( 135 | nn.Conv2d(self.inplanes, planes * block.expansion, 136 | kernel_size=1, stride=stride, bias=False), 137 | nn.BatchNorm2d(planes * block.expansion), 138 | ) 139 | 140 | layers = [] 141 | layers.append(block(self.inplanes, planes, stride, downsample)) 142 | self.inplanes = planes * block.expansion 143 | for i in range(1, blocks): 144 | layers.append(block(self.inplanes, planes)) 145 | 146 | return nn.Sequential(*layers) 147 | 148 | def forward(self, x): 149 | x = self.conv1(x) 150 | x = self.bn1(x) 151 | x = self.relu(x) 152 | x = self.maxpool(x) 153 | 154 | x = self.layer1(x) 155 | x = self.layer2(x) 156 | x = self.layer3(x) 157 | x = self.layer4(x) 158 | 159 | x = self.avgpool(x) 160 | x = x.view(x.size(0), -1) 161 | x = self.fc(x) 162 | 163 | return x 164 | 165 | def resnet(pretrained=False, depth=18, **kwargs): 166 | """Constructs ResNet models for various depths 167 | Args: 168 | pretrained (bool): If True, returns a model pre-trained on ImageNet 169 | depth (int) : Integer input of either 18, 34, 50, 101, 152 170 | """ 171 | block, num_blocks = cfg(depth) 172 | model = ResNet(block, num_blocks, **kwargs) 173 | if (pretrained): 174 | print("| Downloading ImageNet fine-tuned ResNet-%d..." %depth) 175 | model.load_state_dict(model_zoo.load_url(model_urls['resnet%d' %depth])) 176 | return model 177 | -------------------------------------------------------------------------------- /2_classifier/scripts/inference/resnet.sh: -------------------------------------------------------------------------------- 1 | python inference.py \ 2 | --net_type resnet \ 3 | --depth 50 \ 4 | -------------------------------------------------------------------------------- /2_classifier/scripts/inference/vggnet.sh: -------------------------------------------------------------------------------- 1 | python inference.py \ 2 | --net_type vggnet \ 3 | --depth 19 \ 4 | -------------------------------------------------------------------------------- /2_classifier/scripts/train/alexnet.sh: -------------------------------------------------------------------------------- 1 | python main.py \ 2 | --lr 1e-3 \ 3 | --weight_decay 1e-4 \ 4 | --net_type alexnet \ 5 | --resetClassifier \ 6 | --finetune 7 | -------------------------------------------------------------------------------- /2_classifier/scripts/train/densenet.sh: -------------------------------------------------------------------------------- 1 | python main.py \ 2 | --lr 1e-3 \ 3 | --weight_decay 5e-4 \ 4 | --net_type densenet \ 5 | --depth 121 \ 6 | --resetClassifier \ 7 | --finetune 8 | -------------------------------------------------------------------------------- /2_classifier/scripts/train/resnet.sh: -------------------------------------------------------------------------------- 1 | python main.py \ 2 | --lr 1e-3 \ 3 | --weight_decay 5e-4 \ 4 | --net_type resnet \ 5 | --depth 50 \ 6 | --resetClassifier \ 7 | --finetune 8 | #--testOnly 9 | -------------------------------------------------------------------------------- /2_classifier/scripts/train/vggnet.sh: -------------------------------------------------------------------------------- 1 | python main.py \ 2 | --lr 1e-3 \ 3 | --weight_decay 5e-4 \ 4 | --net_type vggnet \ 5 | --depth 16 \ 6 | --resetClassifier \ 7 | --finetune 8 | -------------------------------------------------------------------------------- /3_detector/README.md: -------------------------------------------------------------------------------- 1 | Grad-CAM module 2 | ================================================================================================ 3 | Grad-CAM pytorch implementation of [original paper](http://openaccess.thecvf.com/content_ICCV_2017/papers/Selvaraju_Grad-CAM_Visual_Explanations_ICCV_2017_paper.pdf). 4 | 5 | ## Basic Setups 6 | Open [config.py](./config.py), and edit the lines below to your data directory. 7 | 8 | ```bash 9 | name = [:The name of your dataset that you trained on module 3 (classifier)] 10 | data_base = [:dir to your original dataset] 11 | aug_base = [:dir to your actually trained dataset] 12 | ``` 13 | 14 | For training, your data file system should be in the following hierarchy. 15 | Organizing codes for your data into the given requirements will be provided in the [preprocessor module](../1_preprocessor) 16 | 17 | ```bash 18 | [:data file name] 19 | 20 | |-train 21 | |-[:class 0] 22 | |-[:class 1] 23 | |-[:class 2] 24 | ... 25 | |-[:class n] 26 | |-val 27 | |-[:class 0] 28 | |-[:class 1] 29 | |-[:class 2] 30 | ... 31 | |-[:class n] 32 | ``` 33 | 34 | ## How to run 35 | After you have cloned the repository, you can train the dataset by running the script below. 36 | 37 | You can set the dimension of the additional layer in [config.py](./config.py) 38 | 39 | ```bash 40 | # grad-cam exploits 41 | python launch_model --net_type [:net_type] --depth [:depth] 42 | 43 | # For example, for the resnet-50 model I've trained, type 44 | python launch_model --net_type resnet --depth 50 45 | ``` 46 | 47 | ## Test out various networks 48 | Before testing out the networks, make sure that you have a trained weight obtained in the checkpoint file of the [classifier module](../2_classifier) 49 | 50 | Supporting networks 51 | - AlexNet [:TODO] 52 | - VGGNet [:TODO] 53 | - ResNet 54 | 55 | ## Results 56 | 57 | - Original Image 58 | 59 | ![alt-text-1](../imgs/catdog_test1.jpg "cat") ![alt-text-2](../imgs/catdog_test1.jpg "dog") 60 | 61 | - Grad-CAM Image 62 | 63 | ### Attention for cat 64 | ![alt-text-1](../imgs/cat_test1.jpg "cat") ![alt-text-2](../imgs/cat_test2.jpg "dog") 65 | 66 | ### Attention for dog 67 | ![alt-text-1](../imgs/dog_test2.jpg "cat") ![alt-text-2](../imgs/dog_test2.jpg "dog") 68 | -------------------------------------------------------------------------------- /3_detector/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/3_detector/__init__.py -------------------------------------------------------------------------------- /3_detector/baseline_prediction.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import sys 4 | import numpy as np 5 | 6 | import torch 7 | import torch.nn as nn 8 | import torch.backends.cudnn as cudnn 9 | import torchvision 10 | import argparse 11 | import config as cf 12 | import operator 13 | import csv 14 | 15 | from torchvision import datasets, models, transforms 16 | from networks import * 17 | from torch.autograd import Variable 18 | from PIL import Image 19 | 20 | parser = argparse.ArgumentParser(description='Baseline') 21 | parser.add_argument('--net_type', default='resnet', type=str, help='model') 22 | parser.add_argument('--depth', default=50, type=str, help='depth of model') 23 | args = parser.parse_args() 24 | 25 | # Phase 1 : Model Upload 26 | print('\n[Test Phase] : Model Weight Upload') 27 | use_gpu = torch.cuda.is_available() 28 | 29 | # upload labels 30 | data_dir = cf.aug_dir+'Only_WBC' 31 | trainset_dir = 'Only_WBC/' 32 | dsets = datasets.ImageFolder(data_dir, None) 33 | 34 | H = datasets.ImageFolder(os.path.join(data_dir, 'train')) 35 | dset_classes = H.classes 36 | 37 | def softmax(x): 38 | return np.exp(x) / np.sum(np.exp(x), axis=0) 39 | 40 | def getNetwork(args): 41 | if (args.net_type == 'alexnet'): 42 | file_name = 'alexnet' 43 | elif (args.net_type == 'vggnet'): 44 | file_name = 'vgg-%s' %(args.depth) 45 | elif (args.net_type == 'resnet'): 46 | file_name = 'resnet-%s' %(args.depth) 47 | else: 48 | print('[Error]: Network should be either [alexnet / vgget / resnet]') 49 | sys.exit(1) 50 | 51 | return file_name 52 | 53 | # uploading the model 54 | print("| Loading checkpoint model for crop inference...") 55 | assert os.path.isdir('../3_classifier/checkpoint'),'[Error]: No checkpoint directory found!' 56 | assert os.path.isdir('../3_classifier/checkpoint/'+trainset_dir),'[Error]: There is no model weight to upload!' 57 | file_name = getNetwork(args) 58 | checkpoint = torch.load('../3_classifier/checkpoint/'+trainset_dir+file_name+'.t7') 59 | model = checkpoint['model'] 60 | 61 | if use_gpu: 62 | model.cuda() 63 | cudnn.benchmark = True 64 | 65 | model.eval() 66 | 67 | sample_input = Variable(torch.randn(1,3,224,224), volatile=False) 68 | if use_gpu: 69 | sampe_input = sample_input.cuda() 70 | 71 | test_transform = transforms.Compose([ 72 | transforms.Scale(224), 73 | transforms.CenterCrop(224), 74 | transforms.ToTensor(), 75 | transforms.Normalize(cf.mean, cf.std) 76 | ]) 77 | 78 | def check_and_mkdir(in_dir): 79 | if not os.path.exists(in_dir): 80 | os.makedirs(in_dir) 81 | 82 | check_and_mkdir('results/baseline/') 83 | background_root = '/home/bumsoo/Data/test/CT_20/' 84 | 85 | for thresh in [200, 1]: 86 | print("| Baseline with Threshold : %d" %thresh) 87 | check_and_mkdir('results/baseline/%d' %thresh) 88 | for test_num in range(1, 27+1): 89 | print("\t| Inferencing TEST%d..." %test_num) 90 | baseline_dir = '/home/bumsoo/Data/baseline_info/%d_TEST%d.csv' %(thresh, test_num) 91 | 92 | with open(baseline_dir, 'r') as csvfile: 93 | reader = csv.reader(csvfile) 94 | 95 | check_and_mkdir('results/baseline/%d/TEST%d/' %(thresh, test_num)) 96 | with open('results/baseline/%d/TEST%d/TEST%d.csv' %(thresh, test_num, test_num), 'w') as wrfile: 97 | fieldnames = ['prediction', 'x', 'y', 'w', 'h'] 98 | writer = csv.DictWriter(wrfile, fieldnames=fieldnames) 99 | 100 | original_img = cv2.imread(background_root + 'TEST%d.png' %test_num) 101 | 102 | for row in reader: 103 | x,y,w,h = map(int, row) 104 | crop = original_img[y:y+h, x:x+w] 105 | crop = cv2.cvtColor(crop, cv2.COLOR_BGR2RGB) 106 | 107 | if test_transform is not None: 108 | img = test_transform(Image.fromarray(crop, mode='RGB')) 109 | 110 | inputs = img 111 | inputs = Variable(inputs, volatile=True) 112 | 113 | if use_gpu : 114 | inputs = inputs.cuda() 115 | 116 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) 117 | 118 | outputs = model(inputs) 119 | softmax_res = softmax(outputs.data.cpu().numpy()[0]) 120 | index, score = max(enumerate(softmax_res), key=operator.itemgetter(1)) 121 | 122 | pred = dset_classes[index] 123 | 124 | writer.writerow({ 125 | 'prediction': pred, 126 | 'x': x, 127 | 'y': y, 128 | 'w': w, 129 | 'h': h 130 | }) 131 | 132 | 133 | -------------------------------------------------------------------------------- /3_detector/config.py: -------------------------------------------------------------------------------- 1 | #################### Configuration File #################### 2 | 3 | # Base directory for data formats 4 | name = 'Cat_vs_Dog' 5 | 6 | # Specific directories 7 | data_dir = '/home/mnt/datasets/' 8 | aug_dir = '/home/bumsoo/Data/split/' 9 | 10 | data_base = data_dir + name 11 | aug_base = aug_dir + name 12 | test_dir = '/home/bumsoo/Data/test/' + name 13 | 14 | # model directory 15 | model_dir = '../2_classifier/checkpoints' 16 | 17 | # model option 18 | batch_size = 16 19 | num_epochs = 100 20 | lr_decay_epoch=20 21 | feature_size = 500 22 | 23 | # Global meanstd 24 | mean = [0.42352142932368259, 0.46167925008138017, 0.49023161345837163] 25 | std = [0.22595048333178538, 0.22503028985594206, 0.23220585942785971] 26 | -------------------------------------------------------------------------------- /3_detector/detect_object.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2017 3 | # Github : https://github.com/meliketoy/cellnet.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Grad CAM Implementation 7 | # 8 | # Description : detect_cell.py 9 | # The main code for grad-CAM image localization. 10 | # *********************************************************** 11 | 12 | from __future__ import print_function, division 13 | 14 | import cv2 15 | import torch 16 | import torch.nn as nn 17 | import torch.optim as optim 18 | import torch.backends.cudnn as cudnn 19 | import numpy as np 20 | import config as cf 21 | import torchvision 22 | import time 23 | import copy 24 | import os 25 | import sys 26 | import argparse 27 | import csv 28 | import operator 29 | 30 | from grad_cam import * 31 | from time import sleep 32 | from torchvision import datasets, models, transforms 33 | from networks import * 34 | from torch.autograd import Variable 35 | from PIL import Image 36 | from misc_functions import save_class_activation_on_image 37 | from grad_cam import BackPropagation, GradCAM, GuidedBackPropagation 38 | 39 | parser = argparse.ArgumentParser(description='Pytorch Cell Classification weight upload') 40 | parser.add_argument('--net_type', default='resnet', type=str, help='model') 41 | parser.add_argument('--depth', default=50, type=int, help='depth of model') 42 | parser.add_argument('--subtype', default=None, type=str, help='Type to find') 43 | args = parser.parse_args() 44 | 45 | # Phase 1 : Model Upload 46 | print('\n[Phase 1] : Model Weight Upload') 47 | use_gpu = torch.cuda.is_available() 48 | 49 | # upload labels 50 | data_dir = cf.aug_base 51 | trainset_dir = cf.data_base.split("/")[-1]+os.sep 52 | 53 | dsets = datasets.ImageFolder(data_dir, None) 54 | H = datasets.ImageFolder(os.path.join(cf.aug_base, 'train')) 55 | dset_classes = H.classes 56 | 57 | def softmax(x): 58 | return np.exp(x) / np.sum(np.exp(x), axis=0) 59 | 60 | def getNetwork(args): 61 | if (args.net_type == 'alexnet'): 62 | file_name = 'alexnet' 63 | elif (args.net_type == 'vggnet'): 64 | file_name = 'vgg-%s' %(args.depth) 65 | elif (args.net_type == 'resnet'): 66 | file_name = 'resnet-%s' %(args.depth) 67 | elif (args.net_type == 'densenet'): 68 | file_name = 'densenet-%s' %(args.depth) 69 | else: 70 | print('[Error]: Network should be either [alexnet / vgget / resnet / densenet]') 71 | sys.exit(1) 72 | 73 | return file_name 74 | 75 | def random_crop(image, dim): 76 | if len(image.shape): 77 | W, H, D = image.shape 78 | w, h, d = dim 79 | else: 80 | W, H = image.shape 81 | w, h = size 82 | 83 | left, top = np.random.randint(W-w+1), np.random.randint(H-h+1) 84 | return image[left:left+w, top:top+h], left, top 85 | 86 | def return_class_idx(class_name): 87 | global dset_classes 88 | 89 | for i,j in enumerate(dset_classes): 90 | if class_name == j: 91 | return i 92 | 93 | print(class_name + " is not an appropriate class to search.") 94 | sys.exit(1) # Wrong class name input 95 | 96 | def generate_sliding_windows(image, stepSize, windowSize): 97 | list_windows = [] 98 | cnt = 0 99 | 100 | for x in xrange(0, image.size[0], stepSize): 101 | for y in range(0, image.size[1], stepSize): 102 | if(x+windowSize <= image.size[0] and y+windowSize <= image.size[1]): 103 | list_windows.append(image.crop((x,y,x+windowSize,y+windowSize))) 104 | elif (x+windowSize > image.size[0] and y+windowSize > image.size[1]) : 105 | list_windows.append(image.crop((image.size[0]-windowSize,image.size[1]-windowSize,image.size[0],image.size[1]))) 106 | elif (x+windowSize > image.size[0]): 107 | list_windows.append(image.crop((image.size[0]-windowSize,y,image.size[0],y+windowSize))) 108 | elif (y+windowSize > image.size[1]): 109 | list_windows.append(image.crop((x,image.size[1]-windowSize,x+windowSize,image.size[1]))) 110 | 111 | return list_windows 112 | 113 | def generate_padding_image(image, mode='cv2'): 114 | if (mode == 'cv2'): 115 | border_x = int((args.stepSize - ((image.shape[0]-args.windowSize)%args.stepSize))) 116 | border_y = int((args.stepSize - ((image.shape[1]-args.windowSize)%args.stepSize))) 117 | pad_image = cv2.copyMakeBorder(image, 0, border_x, 0, border_y, cv2.BORDER_CONSTANT, value=[255,255,255]) 118 | elif (mode == 'PIL'): 119 | border_x = args.stepSize - ((image.size[0]-args.windowSize)%args.stepSize) 120 | border_y = args.stepSize - ((image.size[1]-args.windowSize)%args.stepSize) 121 | pad_image = Image.new("RGB", (image.size[0]+border_x, image.size[1]+border_y), color=255) 122 | pad_image.paste(image, (0, 0)) 123 | 124 | return pad_image 125 | 126 | def check_and_mkdir(in_dir): 127 | if not os.path.exists(in_dir): 128 | print("Creating "+in_dir+"...") 129 | os.makedirs(in_dir) 130 | 131 | if __name__ == "__main__": 132 | # uploading the model 133 | print("| Loading checkpoint model for grad-CAM...") 134 | assert os.path.isdir('../2_classifier/checkpoint'),'[Error]: No checkpoint directory found!' 135 | assert os.path.isdir('../2_classifier/checkpoint/'+trainset_dir),'[Error]: There is no model weight to upload!' 136 | file_name = getNetwork(args) 137 | checkpoint = torch.load('../2_classifier/checkpoint/'+trainset_dir+file_name+'.t7') 138 | model = checkpoint['model'] 139 | 140 | if use_gpu: 141 | model.cuda() 142 | cudnn.benchmark = True 143 | 144 | model.eval() 145 | 146 | sample_input = Variable(torch.randn(1,3,224,224), volatile=False) 147 | if use_gpu: 148 | sampe_input = sample_input.cuda() 149 | 150 | def is_image(f): 151 | return f.endswith(".png") or f.endswith(".jpg") 152 | 153 | test_transform = transforms.Compose([ 154 | transforms.Scale(224), 155 | transforms.CenterCrop(224), 156 | transforms.ToTensor(), 157 | transforms.Normalize(cf.mean, cf.std) 158 | ]) 159 | 160 | #@ Code for extracting a grad-CAM region for a given class 161 | gcam = GradCAM(model._modules.items()[0][1], cuda=use_gpu) 162 | 163 | print("\n[Phase 2] : Gradient Detection") 164 | if args.subtype != None: 165 | search_id = return_class_idx(args.subtype) 166 | 167 | if not (args.subtype in dset_classes): 168 | print("The given subtype does not exists!") 169 | args.subtype = None 170 | 171 | if args.subtype == None: 172 | print("| Checking Activated Regions for the Top-1 Class") 173 | else: 174 | print("| Checking Activated Regions for " + dset_classes[search_id] + "...") 175 | 176 | for subdir, dirs, files in os.walk(cf.test_dir): 177 | for f in files: 178 | file_path = os.path.join(subdir, f) 179 | if (is_image(f)): 180 | image = Image.open(file_path) 181 | #original = cv2.imread(file_path) 182 | if test_transform is not None: 183 | image = test_transform(image) 184 | inputs = image 185 | inputs = Variable(inputs, volatile=False) 186 | if use_gpu: 187 | inputs = inputs.cuda() 188 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) 189 | 190 | probs, idx = gcam.forward(inputs) 191 | 192 | if (args.subtype == None): 193 | comp_idx = idx[0] 194 | item_id = 0 195 | else: 196 | comp_idx = search_id 197 | item_id = (np.where(idx.cpu().numpy() == (search_id)))[0][0] 198 | 199 | gcam.backward(idx=comp_idx) 200 | output = gcam.generate(target_layer = 'layer4.2') 201 | 202 | heatmap = output 203 | original = inputs.data.cpu().numpy() 204 | print(original) 205 | original = np.transpose(original, (0,2,3,1))[0] 206 | original = original * cf.std + cf.mean 207 | original = np.uint8(original * 255.0) 208 | #original = cv2.cvtColor(original, cv2.COLOR_RGB2BGR) 209 | #original = cv2.resize(original, (224, 224)) 210 | mask = np.uint8(heatmap * 255.0) 211 | 212 | check_and_mkdir("./results/heatmaps") 213 | check_and_mkdir("./results/masks") 214 | 215 | save_dir = "./results/heatmaps/"+f 216 | mask_dir = "./results/masks/"+f 217 | 218 | print(save_dir) 219 | print(mask_dir) 220 | 221 | print(original.shape) 222 | 223 | gcam.save(save_dir, heatmap, original) 224 | cv2.imwrite("./results/"+f, original) 225 | cv2.imwrite(mask_dir, mask) 226 | -------------------------------------------------------------------------------- /3_detector/draw_bbox.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import sys 4 | import csv 5 | import numpy as np 6 | import config as cf 7 | 8 | def check_and_mkdir(in_dir): 9 | if not os.path.exists(in_dir): 10 | print("Creating "+in_dir+"...") 11 | os.makedirs(in_dir) 12 | 13 | if __name__ == "__main__": 14 | check_and_mkdir('./results/bbox/') 15 | 16 | for file_number in range(1, (27+1)): 17 | print("| Predicting Bounding Box for TEST%d..." %file_number) 18 | original_img = cv2.imread("/home/bumsoo/Data/test/MICCAI_img/TEST%d.png" %file_number) 19 | mask_img = cv2.imread('./results/masks/TEST%d.png' %file_number) 20 | 21 | ret, threshed_img = cv2.threshold(cv2.cvtColor(mask_img, cv2.COLOR_BGR2GRAY), 100, 255, cv2.THRESH_BINARY) 22 | kernel = np.ones((3,3), np.uint8) 23 | closing = cv2.morphologyEx(threshed_img, cv2.MORPH_CLOSE, kernel, iterations=4) 24 | 25 | _, contours, _ = cv2.findContours(closing, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) 26 | 27 | count = 0 28 | 29 | # Predictions (GREEN) 30 | for cnt in contours: 31 | area = cv2.contourArea(cnt) 32 | 33 | if (area > 30**2): 34 | # ellipse 35 | #ellipse = cv2.fitEllipse(cnt) 36 | #cv2.ellipse(original_img, ellipse, (0,255,0), 2) 37 | 38 | # bounding box 39 | x, y, w, h = cv2.boundingRect(cnt) 40 | cv2.rectangle(original_img, (x,y), (x+w, y+h), (0,255,0), 2) 41 | 42 | # Ground truth (RED) 43 | with open(cf.test_dir+'%d/TEST%d.csv' %(file_number, file_number)) as csvfile: 44 | reader = csv.reader(csvfile) 45 | for row in reader: 46 | x, y, w, h = map(int, row[1:]) 47 | 48 | cv2.rectangle(original_img, (x,y), (x+w, y+h), (0, 0, 255), 2) 49 | 50 | cv2.imwrite('./results/bbox/TEST%d.png' %file_number, original_img) 51 | -------------------------------------------------------------------------------- /3_detector/grad_cam.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | # 4 | # Author: Kazuto Nakashima 5 | # URL: http://kazuto1011.github.io 6 | # Created: 2017-05-26 7 | 8 | from __future__ import print_function 9 | 10 | from collections import OrderedDict 11 | 12 | import cv2 13 | import numpy as np 14 | import torch 15 | import torch.nn as nn 16 | from torch.autograd import Variable 17 | from torch.nn import functional as F 18 | 19 | class PropagationBase(object): 20 | 21 | def __init__(self, model, cuda=False): 22 | self.model = model 23 | self.model.eval() 24 | if cuda: 25 | self.model.cuda() 26 | self.cuda = cuda 27 | self.all_fmaps = OrderedDict() 28 | self.all_grads = OrderedDict() 29 | self._set_hook_func() 30 | self.image = None 31 | 32 | def _set_hook_func(self): 33 | raise NotImplementedError 34 | 35 | def _encode_one_hot(self, idx): 36 | one_hot = torch.FloatTensor(1, self.preds.size()[-1]).zero_() 37 | one_hot[0][idx] = 1.0 38 | return one_hot.cuda() if self.cuda else one_hot 39 | 40 | def forward(self, image): 41 | self.image = image 42 | self.preds = self.model.forward(self.image) 43 | self.probs = F.softmax(self.preds, dim=0)[0] 44 | self.prob, self.idx = self.probs.data.sort(0, True) 45 | return self.prob, self.idx 46 | 47 | def backward(self, idx): 48 | self.model.zero_grad() 49 | one_hot = self._encode_one_hot(idx) 50 | self.preds.backward(gradient=one_hot, retain_graph=True) 51 | 52 | 53 | class GradCAM(PropagationBase): 54 | 55 | def _set_hook_func(self): 56 | 57 | def func_f(module, input, output): 58 | self.all_fmaps[id(module)] = output.data.cpu() 59 | 60 | def func_b(module, grad_in, grad_out): 61 | self.all_grads[id(module)] = grad_out[0].cpu() 62 | 63 | for module in self.model.named_modules(): 64 | module[1].register_forward_hook(func_f) 65 | module[1].register_backward_hook(func_b) 66 | 67 | def _find(self, outputs, target_layer): 68 | for key, value in outputs.items(): 69 | for module in self.model.named_modules(): 70 | if id(module[1]) == key: 71 | if module[0] == target_layer: 72 | return value 73 | raise ValueError('Invalid layer name: {}'.format(target_layer)) 74 | 75 | def _normalize(self, grads): 76 | l2_norm = torch.sqrt(torch.mean(torch.pow(grads, 2))) + 1e-5 77 | return grads / l2_norm.data[0] 78 | 79 | def _compute_grad_weights(self, grads): 80 | grads = self._normalize(grads) 81 | self.map_size = grads.size()[2:] 82 | return nn.AvgPool2d(self.map_size)(grads) 83 | 84 | def generate(self, target_layer): 85 | fmaps = self._find(self.all_fmaps, target_layer) 86 | grads = self._find(self.all_grads, target_layer) 87 | weights = self._compute_grad_weights(grads) 88 | 89 | gcam = torch.FloatTensor(self.map_size).zero_() 90 | for fmap, weight in zip(fmaps[0], weights[0]): 91 | res = fmap * weight.data.expand_as(fmap) 92 | gcam += fmap * weight.data.expand_as(fmap) 93 | gcam = F.relu(Variable(gcam)) 94 | 95 | gcam = gcam.data.cpu().numpy() 96 | gcam -= gcam.min() 97 | if(gcam.max() != 0): 98 | gcam /= gcam.max() 99 | gcam = cv2.resize(gcam, (self.image.size(3), self.image.size(2))) 100 | 101 | return gcam 102 | 103 | def save(self, filename, gcam, raw_image): 104 | gcam = cv2.applyColorMap(np.uint8(gcam * 255.0), cv2.COLORMAP_JET) 105 | gcam = gcam.astype(np.float) + raw_image.astype(np.float) 106 | if(gcam.max() != 0): 107 | gcam = gcam / gcam.max() * 255.0 108 | cv2.imwrite(filename, np.uint8(gcam)) 109 | 110 | 111 | class BackPropagation(PropagationBase): 112 | def _find(self, outputs, target_layer): 113 | for key, value in outputs.items(): 114 | for module in self.model.named_modules(): 115 | if id(module[1]) == key: 116 | if module[0] == target_layer: 117 | return value 118 | raise ValueError('Invalid layer name: {}'.format(target_layer)) 119 | 120 | def _set_hook_func(self): 121 | 122 | def func_b(module, grad_in, grad_out): 123 | self.all_grads[id(module)] = grad_in[0].cpu() 124 | 125 | for module in self.model.named_modules(): 126 | module[1].register_backward_hook(func_b) 127 | 128 | def generate(self, target_layer): 129 | grads = self._find(self.all_grads, target_layer) 130 | gradients_as_arr = grads.data[0].numpy()[0] 131 | return gradients_as_arr 132 | 133 | def save(self, filename, data): 134 | abs_max = np.maximum(-1 * data.min(), data.max()) 135 | data = data / abs_max * 127.0 + 127.0 136 | cv2.imwrite(filename, np.uint8(data)) 137 | 138 | 139 | class GuidedBackPropagation(BackPropagation): 140 | 141 | def _set_hook_func(self): 142 | 143 | def func_b(module, grad_in, grad_out): 144 | self.all_grads[id(module)] = grad_in[0].cpu() 145 | 146 | # Cut off negative gradients 147 | if isinstance(module, nn.ReLU): 148 | return (torch.clamp(grad_in[0], min=0.0),) 149 | 150 | for module in self.model.named_modules(): 151 | module[1].register_backward_hook(func_b) 152 | -------------------------------------------------------------------------------- /3_detector/inference_bbox.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import sys 4 | import numpy as np 5 | 6 | import torch 7 | import torch.nn as nn 8 | import torch.backends.cudnn as cudnn 9 | import torchvision 10 | import argparse 11 | import config as cf 12 | import operator 13 | import csv 14 | 15 | from torchvision import datasets, models, transforms 16 | from networks import * 17 | from torch.autograd import Variable 18 | from PIL import Image 19 | 20 | parser = argparse.ArgumentParser(description='Pytorch Cell Classification weight upload') 21 | parser.add_argument('--net_type', default='resnet', type=str, help='model') 22 | parser.add_argument('--depth', default=50, type=int, help='depth of model') 23 | args = parser.parse_args() 24 | 25 | # Phase 1 : Model Upload 26 | print('\n[Test Phase] : Model Weight Upload') 27 | use_gpu = torch.cuda.is_available() 28 | 29 | # upload labels 30 | data_dir = cf.aug_base 31 | trainset_dir = cf.data_base.split("/")[-1]+os.sep 32 | dsets = datasets.ImageFolder(data_dir, None) 33 | 34 | H = datasets.ImageFolder(os.path.join(cf.aug_base, 'train')) 35 | dset_classes = H.classes 36 | 37 | def softmax(x): 38 | return np.exp(x) / np.sum(np.exp(x), axis=0) 39 | 40 | def getNetwork(args): 41 | if (args.net_type == 'alexnet'): 42 | file_name = 'alexnet' 43 | elif (args.net_type == 'vggnet'): 44 | file_name = 'vgg-%s' %(args.depth) 45 | elif (args.net_type == 'resnet'): 46 | file_name = 'resnet-%s' %(args.depth) 47 | else: 48 | print('[Error]: Network should be either [alexnet / vgget / resnet]') 49 | sys.exit(1) 50 | 51 | return file_name 52 | 53 | # uploading the model 54 | print("| Loading checkpoint model for crop inference...") 55 | assert os.path.isdir('../3_classifier/checkpoint'),'[Error]: No checkpoint directory found!' 56 | assert os.path.isdir('../3_classifier/checkpoint/'+trainset_dir),'[Error]: There is no model weight to upload!' 57 | file_name = getNetwork(args) 58 | checkpoint = torch.load('../3_classifier/checkpoint/'+trainset_dir+file_name+'.t7') 59 | model = checkpoint['model'] 60 | 61 | if use_gpu: 62 | model.cuda() 63 | cudnn.benchmark = True 64 | 65 | model.eval() 66 | 67 | sample_input = Variable(torch.randn(1,3,224,224), volatile=False) 68 | if use_gpu: 69 | sampe_input = sample_input.cuda() 70 | 71 | test_transform = transforms.Compose([ 72 | transforms.Scale(224), 73 | transforms.CenterCrop(224), 74 | transforms.ToTensor(), 75 | transforms.Normalize(cf.mean, cf.std) 76 | ]) 77 | 78 | def check_and_mkdir(in_dir): 79 | if not os.path.exists(in_dir): 80 | os.makedirs(in_dir) 81 | 82 | for file_number in range(args.start, args.finish+1): 83 | print("| Predicting Box Inference for TEST%d..." %file_number) 84 | original_img = cv2.imread('/home/bumsoo/Data/test/MICCAI_img/TEST%d.png' %file_number) 85 | mask_img = cv2.imread('./results/masks/TEST%d.png' %file_number) 86 | ret, threshed_img = cv2.threshold(cv2.cvtColor(mask_img, cv2.COLOR_BGR2GRAY), 100, 255, cv2.THRESH_BINARY) 87 | kernel = np.ones((3,3), np.uint8) 88 | closing = cv2.morphologyEx(threshed_img, cv2.MORPH_CLOSE, kernel, iterations=4) 89 | 90 | _, contours, _ = cv2.findContours(closing, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) 91 | 92 | count = 0 93 | 94 | with open('results/inferenced/TEST%d/TEST%d.csv' %(file_number, file_number), 'w') as csvfile: 95 | fieldnames = ['prediction', 'x', 'y', 'w', 'h'] 96 | writer = csv.DictWriter(csvfile, fieldnames=fieldnames) 97 | for cnt in contours: 98 | area = cv2.contourArea(cnt) 99 | #print(area) 100 | 101 | if (area > 30**2): 102 | x, y, w, h = cv2.boundingRect(cnt) 103 | crop = original_img[y:y+h, x:x+w] 104 | crop = cv2.cvtColor(crop, cv2.COLOR_BGR2RGB) # Swap the image into RGB input 105 | 106 | if test_transform is not None: 107 | img = test_transform(Image.fromarray(crop, mode='RGB')) 108 | 109 | inputs = img 110 | inputs = Variable(inputs, volatile=True) 111 | 112 | if use_gpu: 113 | inputs = inputs.cuda() 114 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) 115 | 116 | outputs = model(inputs) 117 | softmax_res = softmax(outputs.data.cpu().numpy()[0]) 118 | index, score = max(enumerate(softmax_res), key=operator.itemgetter(1)) 119 | 120 | count += 1 121 | if ('RBC' in dset_classes[index]): 122 | print("\tRBC_%d : %f" %(count, score)) 123 | else: 124 | if ("Neutrophil" in dset_classes[index] and score < 0.9): 125 | if h1_transform is not None: 126 | img = h1_transform(Image.fromarray(crop, mode='RGB')) 127 | inputs = img 128 | inputs = Variable(inputs, volatile=True) 129 | 130 | if use_gpu: 131 | inputs = inputs.cuda() 132 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) 133 | 134 | H1_outputs = model_NH(inputs) 135 | hr_softmax = softmax(H1_outputs.data.cpu().numpy()[0]) 136 | idx, sc = max(enumerate(hr_softmax), key=operator.itemgetter(1)) 137 | 138 | answ = H1_classes[idx] 139 | elif "Lymphocyte" in dset_classes[index] and score < 0.9: 140 | if h2_transform is not None: 141 | img = h2_transform(Image.fromarray(crop, mode='RGB')) 142 | inputs = img 143 | inputs = Variable(inputs, volatile=True) 144 | 145 | if use_gpu: 146 | inputs = inputs.cuda() 147 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) 148 | 149 | H2_outputs = model_NH(inputs) 150 | hr_softmax = softmax(H2_outputs.data.cpu().numpy()[0]) 151 | idx, sc = max(enumerate(hr_softmax), key=operator.itemgetter(1)) 152 | 153 | answ = H2_classes[idx] 154 | else: 155 | answ = dset_classes[index] 156 | 157 | crop = cv2.cvtColor(crop, cv2.COLOR_RGB2BGR) 158 | # cv2.imwrite("./results/cropped/TEST%d/%s_%d.png" %(file_number, H1_classes[index], count), crop) 159 | writer.writerow({ 160 | 'prediction': answ, 161 | 'x': x, 162 | 'y': y, 163 | 'w': w, 164 | 'h': h 165 | }) 166 | print("\t%s_%d : %f" %(answ, count, score)) 167 | #print('\t%s_%d : %f' %(dset_classes[index], count, score)) 168 | -------------------------------------------------------------------------------- /3_detector/launch_model.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2017 3 | # Github : https://github.com/meliketoy/cellnet.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Grad CAM Implementation 7 | # 8 | # Description : launch_model.py 9 | # The main code for grad-CAM image localization. 10 | # *********************************************************** 11 | 12 | from __future__ import print_function, division 13 | 14 | import cv2 15 | import torch 16 | import torch.nn as nn 17 | import torch.optim as optim 18 | import torch.backends.cudnn as cudnn 19 | import numpy as np 20 | import config as cf 21 | import torchvision 22 | import time 23 | import copy 24 | import os 25 | import sys 26 | import argparse 27 | import csv 28 | import operator 29 | from grad_cam import * 30 | 31 | from torchvision import datasets, models, transforms 32 | from networks import * 33 | from torch.autograd import Variable 34 | from PIL import Image 35 | from misc_functions import save_class_activation_on_image 36 | from grad_cam import BackPropagation, GradCAM, GuidedBackPropagation 37 | 38 | parser = argparse.ArgumentParser(description='Pytorch Cell Classification weight upload') 39 | parser.add_argument('--net_type', default='resnet', type=str, help='model') 40 | parser.add_argument('--depth', default=50, type=int, help='depth of model') 41 | args = parser.parse_args() 42 | 43 | # Phase 1 : Model Upload 44 | print('\n[Phase 1] : Model Weight Upload') 45 | use_gpu = torch.cuda.is_available() 46 | 47 | # upload labels 48 | data_dir = cf.test_base 49 | trainset_dir = cf.data_base.split("/")[-1]+os.sep 50 | 51 | dsets = datasets.ImageFolder(data_dir, None) 52 | H = datasets.ImageFolder(os.path.join(cf.aug_base, 'train')) 53 | dset_classes = H.classes 54 | 55 | def softmax(x): 56 | return np.exp(x) / np.sum(np.exp(x), axis=0) 57 | 58 | def getNetwork(args): 59 | if (args.net_type == 'alexnet'): 60 | file_name = 'alexnet' 61 | elif (args.net_type == 'vggnet'): 62 | file_name = 'vgg-%s' %(args.depth) 63 | elif (args.net_type == 'resnet'): 64 | file_name = 'resnet-%s' %(args.depth) 65 | else: 66 | print('[Error]: Network should be either [alexnet / vgget / resnet]') 67 | sys.exit(1) 68 | 69 | return file_name 70 | 71 | def random_crop(image, dim): 72 | if len(image.shape): 73 | W, H, D = image.shape 74 | w, h, d = dim 75 | else: 76 | W, H = image.shape 77 | w, h = size 78 | 79 | left, top = np.random.randint(W-w+1), np.random.randint(H-h+1) 80 | return image[left:left+w, top:top+h], left, top 81 | 82 | # uploading the model 83 | print("| Loading checkpoint model for grad-CAM...") 84 | assert os.path.isdir('../3_classifier/checkpoint'),'[Error]: No checkpoint directory found!' 85 | assert os.path.isdir('../3_classifier/checkpoint/'+trainset_dir),'[Error]: There is no model weight to upload!' 86 | file_name = getNetwork(args) 87 | checkpoint = torch.load('../3_classifier/checkpoint/'+trainset_dir+file_name+'.t7') 88 | model = checkpoint['model'] 89 | 90 | if use_gpu: 91 | model.cuda() 92 | cudnn.benchmark = True 93 | 94 | model.eval() 95 | 96 | sample_input = Variable(torch.randn(1,3,224,224), volatile=False) 97 | if use_gpu: 98 | sampe_input = sample_input.cuda() 99 | 100 | def is_image(f): 101 | return f.endswith(".png") or f.endswith(".jpg") 102 | 103 | test_transform = transforms.Compose([ 104 | transforms.Scale(224), 105 | transforms.CenterCrop(224), 106 | transforms.ToTensor(), 107 | transforms.Normalize(cf.mean, cf.std) 108 | ]) 109 | 110 | """ 111 | #@ Code for inference test 112 | 113 | img = Image.open(cf.image_path) 114 | if test_transform is not None: 115 | img = test_transform(img) 116 | inputs = img 117 | inputs = Variable(inputs, volatile=False, requires_grad=True) 118 | 119 | if use_gpu: 120 | inputs = inputs.cuda() 121 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) 122 | 123 | outputs = model(inputs) 124 | softmax_res = softmax(outputs.data.cpu().numpy()[0]) 125 | 126 | index,score = max(enumerate(softmax_res), key=operator.itemgetter(1)) 127 | 128 | print('| Uploading %s' %(cf.image_path.split("/")[-1])) 129 | print('| prediction = ' + dset_classes[index]) 130 | """ 131 | 132 | #@ Code for extracting a grad-CAM region for a given class 133 | gcam = GradCAM(model._modules.items()[0][1], cuda=use_gpu)#model=model._modules.items()[0][1], cuda=use_gpu) 134 | gbp = GuidedBackPropagation(model=model._modules.items()[0][1], cuda=use_gpu) 135 | 136 | #print(dset_classes) 137 | WBC_id = 16 # Neutrophil Segmented 138 | print("Checking Activated Regions for " + dset_classes[WBC_id] + "...") 139 | 140 | for i in range(14): 141 | file_name = './cell_data/2_%s_Neutrophil Segmented.png' %(str(i)) 142 | print("Opening "+file_name+"...") 143 | 144 | original_image = cv2.imread(file_name) 145 | resize_ratio = 224./min(original_image.shape[0:2]) 146 | resized = cv2.resize(original_image, (0,0), fx=resize_ratio, fy=resize_ratio) 147 | cropped, left, top = random_crop(resized, (224, 224, 3)) 148 | print(cropped.size) 149 | if test_transform is not None: 150 | img = test_transform(Image.fromarray(cropped, mode='RGB')) 151 | 152 | # center_cropped = original_image[16:240, 16:240, :] 153 | # expand the image based on the short side 154 | 155 | inputs = img 156 | inputs = Variable(inputs, requires_grad=True) 157 | 158 | if use_gpu: 159 | inputs = inputs.cuda() 160 | inputs = inputs.view(1, inputs.size(0), inputs.size(1), inputs.size(2)) 161 | 162 | probs, idx = gcam.forward(inputs) 163 | #probs, idx = gbp.forward(inputs) 164 | 165 | # Grad-CAM 166 | gcam.backward(idx=WBC_id) 167 | output = gcam.generate(target_layer='layer4.2') 168 | 169 | # Guided Back Propagation 170 | #gbp.backward(idx=WBC_id) 171 | #feature = gbp.generate(target_layer='conv1') 172 | 173 | # Guided Grad-CAM 174 | #output = np.multiply(feature, region) 175 | 176 | gcam.save('./results/%s.png' %str(i), output, cropped) 177 | cv2.imwrite('./results/map%s.png' %str(i), cropped) 178 | 179 | for j in range(3): 180 | print('\t{:5f}\t{}\n'.format(probs[j], dset_classes[idx[j]])) 181 | 182 | """ 183 | @ Code for extracting the Top-3 Results for each image 184 | topk = 3 185 | 186 | for i in range(0, topk): 187 | gcam.backward(idx=idx[i]) 188 | output = gcam.generate(target_layer='layer4.2') 189 | 190 | gcam.save('./results/{}_gcam.png'.format(dset_classes[idx[i]]), output, center_cropped) 191 | print('\t{:.5f}\t{}'.format(probs[i], dset_classes[idx[i]])) 192 | """ 193 | -------------------------------------------------------------------------------- /3_detector/misc_functions.py: -------------------------------------------------------------------------------- 1 | # ************************************************************ 2 | # Author : Bumsoo Kim, 2017 3 | # Github : https://github.com/meliketoy/cellnet.pytorch 4 | # 5 | # Korea University, Data-Mining Lab 6 | # Deep Convolutional Network Grad CAM Implementation 7 | # 8 | # Description : misc_function.py 9 | # The main code for grad-CAM image localization. 10 | # *********************************************************** 11 | import os 12 | import cv2 13 | import copy 14 | import numpy as np 15 | 16 | import torch 17 | from torch.autograd import Variable 18 | from torchvision import models 19 | 20 | def preprocess_image(cv2im, resize_im=True): 21 | mean = [0.485, 0.456, 0.406] 22 | std = [0.229, 0.224, 0.225] 23 | 24 | if resize_im: 25 | cv2im = cv2.resize(cv2im, (224, 224)) 26 | im_as_arr = np.float32(cv2im) 27 | im_as_arr = np.ascontiguousarray(im_as_arr[..., ::-1]) 28 | im_as_arr = im_as_arr.transpose(2,0,1) 29 | 30 | for channel, _ in enumerate(im_as_arr): 31 | im_as_arr[channel] /= 255 32 | im_as_arr[channel] -= mean[channel] 33 | im_as_arr[channel] /= std[channel] 34 | 35 | im_as_ten = torch.from_numpy(im_as_arr).float() 36 | im_as_ten.unsqueeze_(0) 37 | 38 | im_as_var = Variable(im_as_ten, requires_grad = True) 39 | 40 | return im_as_var 41 | 42 | def save_gradient_images(gradient, file_name): 43 | """ 44 | @ func: 45 | exports the original gradient image. 46 | 47 | @ args: 48 | gradient : Numpy array of the gradient with shape (3, 224, 224) 49 | file_name : File name to be exported 50 | """ 51 | 52 | gradient -= gradient.min() 53 | gradient /= gradient.max() 54 | gradient = np.uint8(gradient * 255).transpose(1, 2, 0) 55 | path_to_file = os.path.join('./results', file_name+'.jpg') 56 | 57 | # Convert RGB to GBR 58 | gradient = gradient[..., ::-1] 59 | cv2.imwrite(path_to_file, gradient) 60 | 61 | def save_class_activation_on_image(org_img, activation_map, file_name): 62 | """ 63 | @ func: 64 | Saves CAM(Class Activation Map) on the original image. 65 | 66 | @ args: 67 | org_img (PIL img): Original image 68 | activation_map : Numpy array of the activation map in grayscale (0~255) 69 | file_name : String for the file name of the exported image. 70 | """ 71 | 72 | # Grayscale activation map 73 | path_to_file = os.path.join('./results', file_name+'_Cam_Grayscale.jpg') 74 | cv2.imwrite(path_to_file, activation_map) 75 | 76 | # Heatmap of activation map 77 | activation_heatmap = cv2.applyColorMap(activation_map, cv2.COLORMAP_JET) 78 | path_to_file = os.path.join('./results', file_name+'_Cam_Heatmap.jpg') 79 | cv2.imwrite(path_to_file, activation_heatmap) 80 | 81 | # Heatmap on picture 82 | org_img = cv2.resize(org_img, (224,224)) 83 | img_with_heatmap = np.float32(activation_heatmap) + np.float32(org_img) 84 | img_with_heatmap = img_with_heatmap / np.max(img_with_heatmap) 85 | path_to_file = os.path.join('./results', file_name+'_Cam_On_Image.jpg') 86 | cv2.imwrite(path_to_file, np.uint8(255*img_with_heatmap)) 87 | 88 | def recreate_image(im_var): 89 | """ 90 | @ func: 91 | Recreated image from a torch variable. 92 | 93 | @ args: 94 | im_var : Image to recreate 95 | 96 | @ return: 97 | recreated_im : Numpy array of the recreated img 98 | """ 99 | 100 | reverse_mean = map(lambda x : -x, cf.mean) 101 | reverse_std = map(lambda x : 1/x, cf.std) 102 | 103 | recreated_im = copy.copy(im_as_var.data.numpy()[0]) 104 | 105 | for c in range(3): 106 | recreated_im[c] /= reverse_std[c] 107 | recreated_im[c] -= reverse_mean[c] 108 | 109 | recreated_im[recreated_im > 1] = 1 110 | recreated_im[recreated_im < 0] = 0 111 | recreated_im = np.round(recreated_im * 255) 112 | 113 | recreated_im = np.uint8(recreated_im).transpose(1, 2, 0) 114 | 115 | # Convert RBG to GBR 116 | recreated_im = recreated_im[..., ::-1] 117 | return recreated_im 118 | 119 | def get_positive_negative_saliency(gradient): # What is saliency? 120 | """ 121 | @ func: 122 | Generates positive and negative saliency maps based on the gradient 123 | 124 | @ args: 125 | gradient : Numpy array of the gradients to visualize 126 | 127 | @ returns: 128 | Positive and Negative Saliency Maps of the gradient 129 | """ 130 | 131 | pos_saliency = (np.maximum(0, gradient) / gradient.max()) 132 | neg_saliency = (np.maximum(0, -gradient) / -gradient.min()) 133 | 134 | return pos_saliency, neg_saliency 135 | 136 | -------------------------------------------------------------------------------- /3_detector/networks/__init__.py: -------------------------------------------------------------------------------- 1 | from .resnet import * 2 | -------------------------------------------------------------------------------- /3_detector/networks/resnet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import math 3 | import torch.utils.model_zoo as model_zoo 4 | 5 | __all__ = ['ResNet', 'resnet'] 6 | 7 | 8 | model_urls = { 9 | 'resnet18': 'http://download.pytorch.org/models/resnet18-5c106cde.pth', 10 | 'resnet34': 'http://download.pytorch.org/models/resnet34-333f7ec4.pth', 11 | 'resnet50': 'http://download.pytorch.org/models/resnet50-19c8e357.pth', 12 | 'resnet101': 'http://download.pytorch.org/models/resnet101-5d3b4d8f.pth', 13 | 'resnet152': 'http://download.pytorch.org/models/resnet152-b121ed2d.pth', 14 | } 15 | 16 | 17 | def conv3x3(in_planes, out_planes, stride=1): 18 | "3x3 convolution with padding" 19 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, 20 | padding=1, bias=False) 21 | 22 | def cfg(depth): 23 | depth_lst = [18, 34, 50, 101, 152] 24 | assert (depth in depth_lst), "Error : ResNet depth should be either 18, 34, 50, 101, 152" 25 | cf_dict = { 26 | '18' : (BasicBlock, [2,2, 2,2]), 27 | '34' : (BasicBlock, [3,4, 6,3]), 28 | '50' : (Bottleneck, [3,4, 6,3]), 29 | '101': (Bottleneck, [3,4,23,3]), 30 | '152': (Bottleneck, [3,8,36,3]), 31 | } 32 | 33 | return cf_dict[str(depth)] 34 | 35 | 36 | class BasicBlock(nn.Module): 37 | expansion = 1 38 | 39 | def __init__(self, inplanes, planes, stride=1, downsample=None): 40 | super(BasicBlock, self).__init__() 41 | self.conv1 = conv3x3(inplanes, planes, stride) 42 | self.bn1 = nn.BatchNorm2d(planes) 43 | self.relu = nn.ReLU(inplace=True) 44 | self.conv2 = conv3x3(planes, planes) 45 | self.bn2 = nn.BatchNorm2d(planes) 46 | self.downsample = downsample 47 | self.stride = stride 48 | 49 | def forward(self, x): 50 | residual = x 51 | 52 | out = self.conv1(x) 53 | out = self.bn1(out) 54 | out = self.relu(out) 55 | 56 | out = self.conv2(out) 57 | out = self.bn2(out) 58 | 59 | if self.downsample is not None: 60 | residual = self.downsample(x) 61 | 62 | out += residual 63 | out = self.relu(out) 64 | 65 | return out 66 | 67 | 68 | class Bottleneck(nn.Module): 69 | expansion = 4 70 | 71 | def __init__(self, inplanes, planes, stride=1, downsample=None): 72 | super(Bottleneck, self).__init__() 73 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False) 74 | self.bn1 = nn.BatchNorm2d(planes) 75 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, 76 | padding=1, bias=False) 77 | self.bn2 = nn.BatchNorm2d(planes) 78 | self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False) 79 | self.bn3 = nn.BatchNorm2d(planes * 4) 80 | self.relu = nn.ReLU(inplace=True) 81 | self.downsample = downsample 82 | self.stride = stride 83 | 84 | def forward(self, x): 85 | residual = x 86 | 87 | out = self.conv1(x) 88 | out = self.bn1(out) 89 | out = self.relu(out) 90 | 91 | out = self.conv2(out) 92 | out = self.bn2(out) 93 | out = self.relu(out) 94 | 95 | out = self.conv3(out) 96 | out = self.bn3(out) 97 | 98 | if self.downsample is not None: 99 | residual = self.downsample(x) 100 | 101 | out += residual 102 | out = self.relu(out) 103 | 104 | return out 105 | 106 | class ResNet(nn.Module): 107 | 108 | def __init__(self, block, layers, num_classes=1000): 109 | self.inplanes = 64 110 | super(ResNet, self).__init__() 111 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, 112 | bias=False) 113 | self.bn1 = nn.BatchNorm2d(64) 114 | self.relu = nn.ReLU(inplace=True) 115 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 116 | self.layer1 = self._make_layer(block, 64, layers[0]) 117 | self.layer2 = self._make_layer(block, 128, layers[1], stride=2) 118 | self.layer3 = self._make_layer(block, 256, layers[2], stride=2) 119 | self.layer4 = self._make_layer(block, 512, layers[3], stride=2) 120 | self.avgpool = nn.AvgPool2d(7) 121 | self.fc = nn.Linear(512 * block.expansion, num_classes) 122 | 123 | for m in self.modules(): 124 | if isinstance(m, nn.Conv2d): 125 | n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels 126 | m.weight.data.normal_(0, math.sqrt(2. / n)) 127 | elif isinstance(m, nn.BatchNorm2d): 128 | m.weight.data.fill_(1) 129 | m.bias.data.zero_() 130 | 131 | def _make_layer(self, block, planes, blocks, stride=1): 132 | downsample = None 133 | if stride != 1 or self.inplanes != planes * block.expansion: 134 | downsample = nn.Sequential( 135 | nn.Conv2d(self.inplanes, planes * block.expansion, 136 | kernel_size=1, stride=stride, bias=False), 137 | nn.BatchNorm2d(planes * block.expansion), 138 | ) 139 | 140 | layers = [] 141 | layers.append(block(self.inplanes, planes, stride, downsample)) 142 | self.inplanes = planes * block.expansion 143 | for i in range(1, blocks): 144 | layers.append(block(self.inplanes, planes)) 145 | 146 | return nn.Sequential(*layers) 147 | 148 | def forward(self, x): 149 | x = self.conv1(x) 150 | x = self.bn1(x) 151 | x = self.relu(x) 152 | x = self.maxpool(x) 153 | 154 | x = self.layer1(x) 155 | x = self.layer2(x) 156 | x = self.layer3(x) 157 | x = self.layer4(x) 158 | 159 | x = self.avgpool(x) 160 | x = x.view(x.size(0), -1) 161 | x = self.fc(x) 162 | 163 | return x 164 | 165 | def resnet(pretrained=False, depth=18, **kwargs): 166 | """Constructs ResNet models for various depths 167 | Args: 168 | pretrained (bool): If True, returns a model pre-trained on ImageNet 169 | depth (int) : Integer input of either 18, 34, 50, 101, 152 170 | """ 171 | block, num_blocks = cfg(depth) 172 | model = ResNet(block, num_blocks, **kwargs) 173 | if (pretrained): 174 | print("| Downloading ImageNet fine-tuned ResNet-%d..." %depth) 175 | model.load_state_dict(model_zoo.load_url(model_urls['resnet%d' %depth])) 176 | return model 177 | -------------------------------------------------------------------------------- /3_detector/scripts/detect.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | python detect_object.py \ 4 | --net_type resnet \ 5 | --depth 50 \ 6 | --subtype dog 7 | -------------------------------------------------------------------------------- /3_detector/scripts/inference.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | #rm -rf ./results/cropped/* 4 | 5 | python inference_bbox.py \ 6 | --net_type resnet \ 7 | --depth 50 \ 8 | --start 1 \ 9 | --finish 27 \ 10 | -------------------------------------------------------------------------------- /INSTALL.md: -------------------------------------------------------------------------------- 1 | # INSTALLATION GUIDE 2 | This is the Installation guide for the overall repository. 3 | 4 | ## Install NVIDIA-driver 5 | GIGABYTE based BIOS setting 6 | - First, DO NOT PLUG IN your GPU until the driver is set up. 7 | - Internal Graphic : Auto > Enable 8 | - Display Priority : PCIe > Internal 9 | 10 | NVIDIA driver download .run file : [click here](http://www.nvidia.co.kr/Download/index.aspx) 11 | 12 | If you click download from the above site, you will get a .run file format for installing drivers. 13 | 14 | ### 1. Stop display manager 15 | 16 | Before you run the .run file, you first need to stop your Xserver display manager. 17 | 18 | Press [Ctrl] + [Alt] + [F1], enter the script below 19 | 20 | ```bash 21 | $ service --status-all | grep dm 22 | 23 | (Result) [+] [:dm] 24 | ``` 25 | 26 | The part described as [:dm] is your display manager. 27 | 28 | Substitute the [:dm] part below with the result of the script above. 29 | 30 | ```bash 31 | $ sudo service [:dm] stop 32 | 33 | (Result) * Stopping Light Display Manager [:dm] 34 | ``` 35 | 36 | ### 2. Run the nvidia-driver installer 37 | 38 | Run the code below. Press 'Yes' for every option they ask. 39 | 40 | ```bash 41 | $ sh /NVIDIA-Linux_x86_64-375.20.run 42 | ``` 43 | 44 | After you have successfully installed, you shall see the same results when typing the code below. 45 | 46 | ```bash 47 | $ nvidia-smi 48 | ``` 49 | 50 | ```bash 51 | +-----------------------------------------------------------------------------+ 52 | | NVIDIA-SMI 375.20 Driver Version: 375.20 | 53 | |-------------------------------+----------------------+----------------------+ 54 | | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 55 | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 56 | |===============================+======================+======================| 57 | | 0 TITAN X (Pascal) Off | 0000:4B:00.0 Off | N/A | 58 | | 67% 86C P2 249W / 250W | 5026MiB / 12221MiB | 82% Default | 59 | +-------------------------------+----------------------+----------------------+ 60 | | 1 TITAN X (Pascal) Off | 0000:4C:00.0 On | N/A | 61 | | 88% 90C P2 225W / 250W | 3842MiB / 12213MiB | 78% Default | 62 | +-------------------------------+----------------------+----------------------+ 63 | 64 | +-----------------------------------------------------------------------------+ 65 | | Processes: GPU Memory | 66 | | GPU PID Type Process name Usage | 67 | |=============================================================================| 68 | | No running processes found | 69 | +-----------------------------------------------------------------------------+ 70 | 71 | ``` 72 | 73 | ### 3. Reboot the system 74 | 75 | ```bash 76 | $ sudo reboot 77 | ``` 78 | 79 | ## Install CUDA toolkit 80 | 81 | You can skip the step above and automatically install the driver within CUDA installation. 82 | 83 | Check the details below. 84 | 85 | ### * Download the CUDA download .run file from the CUDA download page 86 | 87 | CUDA download page : [click here](https://developer.nvidia.com/cuda-downloads) 88 | 89 | Before executing the file, stop the display manager by following the description above. 90 | 91 | ```bash 92 | $ sudo sh /cuda_8.0.44_linux.run 93 | ``` 94 | 95 | ### * Link your CUDA in .bashrc 96 | 97 | ```bash 98 | $ sudo apt-get install vim 99 | 100 | $ git clone https://github.com/amix/vimrc.git ~/.vim_runtime 101 | 102 | # Awesome version 103 | $ sh ~/.vim_runtime/install_awesome_vimrc.sh 104 | 105 | # Basic version 106 | $ sh ~/.vim_runtime/install_basic_vimrc.sh 107 | ``` 108 | 109 | Open your ~/.bashrc file. 110 | 111 | ```bash 112 | vi ~/.bashrc 113 | 114 | # Press Shift + G, Add the lines on the bottom 115 | 116 | export PATH=$PATH:/usr/local/cuda/bin 117 | export LD_LIBRARY_PATH=/usr/local/cuda/lib64 118 | export CUDA_HOME=/usr/local/cuda 119 | ``` 120 | 121 | To check if the CUDA toolkit is successfully installed, type the line below. 122 | 123 | ```bash 124 | $ nvcc --version 125 | 126 | * (Result) 127 | nvcc: NVIDIA (R) Cuda compiler driver 128 | Copyright (c) 2005-2016 NVIDIA Corporation 129 | Built on Sun_Sep__4_22:14:01_CDT_2016 130 | Cuda compilation tools, release 8.0, V8.0.44 131 | ``` 132 | 133 | ## Install cuDNN library kit 134 | 135 | cuDNN download page : [click here](https://developer.nvidia.com/rdp/cudnn-download) 136 | 137 | (Membership is required, just sign in!) 138 | 139 | Download the newest cuDNN v5.1. 140 | 141 | ```bash 142 | $ cd 143 | $ tar -zxvf ./cudnn-8.0-linux-x64-v5.1.tgz 144 | $ sudo cp cuda/include/cudnn.h /usr/local/cuda/include/ 145 | $ sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64/ 146 | $ sudo chmod a+r /usr/local/cuda/include/cudnn.h 147 | $ sudo chmod a+r /usr/local/cuda/lib64/libcudnn* 148 | ``` 149 | 150 | ## Install Pytorch for python 2.7 151 | Clone the repository in the pytroch install page below, or type the following command lines. 152 | 153 | ```bash 154 | pip install http://download.pytorch.org/whl/cu80/torch-0.1.12.post2-cp27-none-linux_x86_64.whl 155 | pip install torchvision 156 | ``` 157 | 158 | Pytorch install page : [click here](https://pytorch.org) 159 | 160 | ```bash 161 | $ git clone https://github.com/torch/distro.git ~/torch --recursive 162 | $ cd ~/torch; bash install-deps; 163 | $ ./install.sh 164 | $ source ~/.bashrc 165 | ``` 166 | 167 | ## Install basic dependencies 168 | For basic image augmentation, the following must be installed within your environment. 169 | - [OpenCV]() 170 | - [PIL Image]() 171 | 172 | ## Now, Enjoy! 173 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Bumsoo Kim 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | 3 | ------------------------------------------------------------------------------------- 4 | 5 | This is the repository for Pytorch Implementation of 6 | "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization". 7 | If you have any issues regarding this repository, please contact meliketoy@gmail.com. 8 | 9 | You can see the original paper [here](https://arxiv.org/pdf/1610.02391.pdf) 10 | 11 | ## Modules 12 | - [#1 Preprocessor](./1_preprocessor) 13 | - [#2 Classifier](./2_classifier) 14 | - [#3 Detector](./3_detector) 15 | 16 | ## Requirements 17 | See the [installation instruction](INSTALL.md) for a step-by-step installation guide. 18 | See the [server instruction](SERVER.md) for server settup. 19 | - Install [cuda-8.0](https://developer.nvidia.com/cuda-downloads) 20 | - Install [cudnn v5.1](https://developer.nvidia.com/cudnn) 21 | - Download [Pytorch for python-2.7](https://pytorch.org) and clone the repository. 22 | - Enjoy! 23 | 24 | ```bash 25 | pip install http://download.pytorch.org/whl/cu80/torch-0.1.12.post2-cp27-none-linux_x86_64.whl 26 | pip install torchvision 27 | git clone https://github.com/meliketoy/gradcam.pytorch 28 | ``` 29 | 30 | ## Grad-CAM 31 | 32 | "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization". 33 | 34 | ![alt_tag](./imgs/gradcam.jpg) 35 | 36 | In this repo, we will be training and testing the model with a very simple, [cat vs dog](https://www.kaggle.com/c/dogs-vs-cats) dataset. 37 | You can view and download the dataset yourself by clicking the link above. 38 | 39 |

40 | 41 | Implementation on your own private data only requires modifications in the directory within the configuration files inside each modules. 42 | 43 | ### STEP 1 : Data preperation 44 | You can prepare your data with the [preprocessing module](./1_preprocessor). 45 | In the [configuration file](./1_preprocessor/config.py), set the directory to the directory containing the training data. 46 | 47 | As we are fine-tuning the model, we will only be taking a small portion of the original training set. 48 | 49 | ```bash 50 | $ cd ./1_preprocessor 51 | $ python main 52 | 53 | > Enter mode name : split # This will make a train-validation split in your 'split_dir' in config.py 54 | > Enter mode name : check # This will print out the distribution of your split. 55 | > Enter mode name : meanstd # This will print out the meanstd value of your train set. 56 | ``` 57 | 58 | Copy the value of meanstd in the third line, and paste it in the configurations of each [module 3](./2_classifier/config.py) and [module 4](./3_detector/config.py). View the [README-preprocessor](./1_preprocessor/README.md) for further instructions. 59 | 60 | ### STEP 2 : Classification 61 | Then, in the [classifier module](./2_classifier), run the line below 62 | ```bash 63 | $ ./scripts/train/resnet 64 | ``` 65 | 66 | This will fine-tune a pre-trained resnet-50 model on your dataset. 67 | To train your network on different models & layers, view the [scripts](./2_classifier/scripts). See [README-classifier](./2_classifier/README.md) for further instructions. 68 | 69 | ### STEP 3 : Detection 70 | After you have trained your model, there will be a model saved in the [checkpoint directory](./2_classifier/checkpoints). 71 | The files in directory will be automatically updated in the detector module, searched by the directory name of your training set. 72 | 73 | In the [configuration of module 4](./3_detector/config.py), match the 'name' variable identical to the 'name' you used in your classification training data directory name. 74 | 75 | The heatmap generation for each of the test data can be done by running, 76 | ```bash 77 | $ ./scripts/detect.sh 78 | ``` 79 | 80 | This will generate a heatmap which will look like 81 | 82 | ### Attention for cat 83 | ![alt-text-1](./imgs/cat_test1.jpg "cat1") ![alt-text-2](./imgs/cat_test2.jpg "cat2") 84 | 85 | ### Attention for dog 86 | ![alt-text-1](./imgs/dog_test1.jpg "dog1") ![alt-text-2](./imgs/dog_test2.jpg "dog2") 87 | 88 | See [README-detector](./3_detector/README.md) for further instructions. 89 | 90 | ### FUTURE WORKS : Semi-supervised Object Detection 91 | This strategy could be used as a method to perform semi-supervised detection, a detection learning when only given the classification label and not any local annotations. 92 | 93 | Implementation on luekocyte detection(which I submitted a paper on) will look like, 94 | ![alt_tag](./imgs/prediction.png) 95 | 96 | If you want to change the model configuration, see the [script](./3_detector/scripts/inference.sh) or the [configuration file](./3_detector/config.py) 97 | -------------------------------------------------------------------------------- /SERVER.md: -------------------------------------------------------------------------------- 1 | # SERVER MANAGEMENT 2 | This is the management guide for server installation. 3 | 4 | ## Welcome message 5 | Install figlet 6 | ```bash 7 | $ sudo apt-get install figlet 8 | ``` 9 | 10 | ```bash 11 | $ sudo vi /etc/bash.bashrc 12 | 13 | # Press [Shift] + [G] and write the code on the bottom 14 | clear 15 | printf "Welcome to Ubuntu 16.04.5 LTS (GNU/Linux-Mint-18 x86_64)\n" 16 | printf "This is the server for the cellnet implementation.\n\n" 17 | printf " * Documentation: https://github.com/meliketoy/cellnet.pytorch\n\n" 18 | printf "##############################################################\n" 19 | figlet -f slant "Bumsoo Kim" 20 | printf "\n\n" 21 | printf " Data Mining & Information System Lab\n" 22 | printf " GPU Computing machine : bumsoo@163.152.163.10\n\n" 23 | printf " Administrator : Bumsoo Kim\n" 24 | printf " Please read the document\n" 25 | printf " https://github.com/meliketoy/wide-residual-network/README.md\n" 26 | printf "##############################################################\n\n" 27 | ``` 28 | 29 | ## Remote Server control 30 | 31 | ### 1. SCP call 32 | 33 | ```bash 34 | 35 | # Upload your local file to server 36 | $ scp -P 22 /file [:username]@[:server_ip]: 37 | 38 | # Download the server file to your local 39 | $ scp -P 22 [:username]@[:server_ip]:/file 40 | 41 | # Upload your local directory to server 42 | $ scp -P 22 -r /file [:username]@[:server_ip]: 43 | 44 | # Download the server file to your local 45 | $ scp -P 22 -r [:username]@[:server_ip]:/file 46 | 47 | ``` 48 | 49 | ### 2. Save sessions by name 50 | ```bash 51 | $ sudo vi ~/.bashrc 52 | 53 | # Press [Shift] + [G] and enter the function on the bottom. 54 | 55 | function server_func() { 56 | echo -n "[Enter the name of server]: " 57 | read server_name 58 | 59 | # echo "Logging in to server $server_name ..." 60 | if [ $server_name == [:servername] ]; then 61 | ssh [:usr]@[:ip].[:ip].[:ip].[:ip] 62 | fi 63 | } 64 | alias dmis_remote=server_func 65 | ``` 66 | 67 | ## Github control 68 | 69 | ```bash 70 | $ sudo vi ~/.netrc 71 | 72 | machine github.com 73 | login [:username] 74 | password [:password] 75 | ``` 76 | 77 | ### Jupyter notebook configuration 78 | 79 | For jupyter notebook configuration, type in the command line below. 80 | ```bash 81 | $ jupyter notebook --generate-config 82 | 83 | * Result : 84 | Writing default config to: /.jupyter/jupyter_notebook_config.py 85 | 86 | $ vi ~/.jupyter/jupyter_notebook_config.py 87 | ``` 88 | 89 | presh [Esc], then enter /ip to find the ip configuration. You will find the line below 90 | ``` bash 91 | ## The IP address the notebook server will listen on. 92 | #c.NotebookApp.ip = 'localhost' 93 | ``` 94 | 95 | Erase the '#' and change it into ... 96 | ```bash 97 | c.NotebookApp.ip = '163.152.163.112' # the ip address for your server 98 | ``` 99 | 100 | presh [Esc], then enter /port to find the port number. You will find the line below 101 | ```bash 102 | ## The port the notebook server will listen on. 103 | #c.NotebookApp.port = 8888 104 | ``` 105 | 106 | Erase the '#' and enter whatever port number you want 107 | ```bash 108 | c.NotebookApp.port = 9999 109 | 110 | ``` 111 | 112 | Now, Enjoy! 113 | -------------------------------------------------------------------------------- /imgs/cat_test1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/cat_test1.jpg -------------------------------------------------------------------------------- /imgs/cat_test2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/cat_test2.jpg -------------------------------------------------------------------------------- /imgs/catdog_test1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/catdog_test1.jpg -------------------------------------------------------------------------------- /imgs/catdog_test2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/catdog_test2.jpg -------------------------------------------------------------------------------- /imgs/detection/after.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/detection/after.png -------------------------------------------------------------------------------- /imgs/detection/before.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/detection/before.png -------------------------------------------------------------------------------- /imgs/dog_test1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/dog_test1.jpg -------------------------------------------------------------------------------- /imgs/dog_test2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/dog_test2.jpg -------------------------------------------------------------------------------- /imgs/figure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/figure.png -------------------------------------------------------------------------------- /imgs/gradcam.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/gradcam.jpg -------------------------------------------------------------------------------- /imgs/heatmap_out.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/heatmap_out.png -------------------------------------------------------------------------------- /imgs/input.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/input.png -------------------------------------------------------------------------------- /imgs/output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/output.png -------------------------------------------------------------------------------- /imgs/prediction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/prediction.png -------------------------------------------------------------------------------- /imgs/pytorch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/pytorch.png -------------------------------------------------------------------------------- /imgs/woof_meow.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bmsookim/gradcam.pytorch/7b71c55b1debf633918793bd7db5ac4533647eb0/imgs/woof_meow.jpg --------------------------------------------------------------------------------