├── .gitignore
├── README.md
├── data
├── Makefile
├── README.md
├── credentials.txt.template
└── lena.png
├── docs
├── ft-dl-loss.png
├── ft-dlw-loss.png
├── fttl-keras-nov2016.pptx
├── tl-dl1-loss.png
├── tl-dl2-loss.png
├── vgg16-ft.dia
├── vgg16-ft.png
├── vgg16-original.dia
├── vgg16-original.png
├── vgg16-tl.dia
└── vgg16-tl.png
└── src
├── augment-images.ipynb
├── augment-images.py
├── caffe2keras-rebuild.ipynb
├── caffe2keras-rebuild.py
├── caffe2keras-save.py
├── confusion-to-heatmap.py
├── examine-trained-model.ipynb
├── ft-dl-train.py
├── ft-dlw-train.py
├── fttlutils.py
├── image-convolutions.ipynb
├── make-sample.py
├── preprocess-images.py
├── sample-images.py
├── tl-dl-aug-train.py
├── tl-dl1-train.py
├── tl-dl2-train.py
├── tl-lr-aug-train.py
├── tl-lr-train.py
└── vectorize-images.py
/.gitignore:
--------------------------------------------------------------------------------
1 | *pyc
2 | data/*
3 | .*
4 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Transfer Learning and Fine Tuning for Cross Domain Image Classification with Keras
2 |
3 | Supporting code for my talk at [Accel.AI](http://accel.ai/) Demystifying Deep Learning and AI event on November 19-20 2016 at Oakland CA.
4 |
5 | Slides are [here](http://www.slideshare.net/sujitpal/transfer-learning-and-fine-tuning-for-cross-domain-image-classification-with-keras)
6 |
7 | ## Abstract:
8 |
9 | I describe how a Deep Convolutional Network (DCNN) trained on the ImageNet dataset can be used to classify images in a completely different domain. The intuition that the training process teaches the DCNN to extract good features from images is explored with visualizations. Transfer Learning freezes the bottom layers of the DCNN to extract image vectors from a training set in a different domain, which can then be used to train a new classifier for this domain. Fine tuning involves training the pre-trained network further for the target domain. Both approaches are demonstrated using a VGG-16 network pre-trained on ImageNet to classify medical images into 5 categories. Code examples are provided using Keras.
10 |
11 | ## Dataset
12 |
13 | Dataset used comes from [Diabetic Retinopathy Detection competition](https://www.kaggle.com/c/diabetic-retinopathy-detection) on Kaggle. Dataset is a set of 35,126 of digital color fundus photographs of the retina. The code here uses a sample of 1,000 images sampled from this dataset, 200 per each of the 5 Diabetes Retinopathy images (No DR, Mild DR, Moderate DR, Severe DR and Proliferative DR). See the [data/README.md](data/README.md) for details.
14 |
15 | ## VGG-16 Model
16 |
17 |
18 |
19 | ## Results
20 |
21 | ### Transfer Learning
22 |
23 |
24 |
25 | ### Transfer Learning + Logistic Regression
26 |
27 | Result (based on Cohen's Kappa score) places this entry at position 79-80 on public leaderboard (as of Nov 9 2016).
28 |
29 | Accuracy: 0.36333, Cohen's Kappa Score: 0.51096
30 |
31 | Confusion Matrix:
32 | [[15 19 17 9 0]
33 | [20 24 10 5 1]
34 | [12 13 13 12 10]
35 | [ 7 4 11 24 14]
36 | [ 4 2 11 10 33]]
37 |
38 | Classification Report:
39 | precision recall f1-score support
40 |
41 | 0 0.26 0.25 0.25 60
42 | 1 0.39 0.40 0.39 60
43 | 2 0.21 0.22 0.21 60
44 | 3 0.40 0.40 0.40 60
45 | 4 0.57 0.55 0.56 60
46 |
47 | avg / total 0.36 0.36 0.36 300
48 |
49 | ### Transfer Learning + 1 layer MLP
50 |
51 | Result (based on Cohen's Kappa score) places this entry at position 25-26 on public leaderboard (as of Nov 9 2016).
52 |
53 |
54 |
55 | Final Model (DL#1)
56 |
57 | Accuracy: 0.66667, Cohen's Kappa Score: 0.74558
58 |
59 | Confusion Matrix:
60 | [[40 6 8 0 0]
61 | [ 5 46 12 2 0]
62 | [ 7 5 46 4 2]
63 | [ 5 5 8 31 11]
64 | [ 2 3 6 9 37]]
65 |
66 | Classification Report:
67 | precision recall f1-score support
68 |
69 | 1 0.68 0.74 0.71 54
70 | 2 0.71 0.71 0.71 65
71 | 3 0.57 0.72 0.64 64
72 | 4 0.67 0.52 0.58 60
73 | 5 0.74 0.65 0.69 57
74 |
75 | avg / total 0.67 0.67 0.67 300
76 |
77 |
78 | ### Transfer Learning + 2 layer MLP
79 |
80 | Results in lower performance than 1 layer MLP. Other tuning was to increase batch size, use Adadelta optimizer with lower learning rate. Network is slower because of additional layer and slightly worse in performance than 1 layer.
81 |
82 |
83 |
84 | Final Model (DL#2)
85 |
86 | Accuracy: 0.63333, Cohen's Kappa Score: 0.70822
87 |
88 | Confusion Matrix:
89 | [[36 11 4 1 2]
90 | [13 45 4 1 2]
91 | [ 5 15 35 5 4]
92 | [ 5 8 6 31 10]
93 | [ 4 1 3 6 43]]
94 |
95 | Classification Report:
96 | precision recall f1-score support
97 |
98 | 1 0.57 0.67 0.62 54
99 | 2 0.56 0.69 0.62 65
100 | 3 0.67 0.55 0.60 64
101 | 4 0.70 0.52 0.60 60
102 | 5 0.70 0.75 0.73 57
103 |
104 | avg / total 0.64 0.63 0.63 300
105 |
106 |
107 | ### Fine Tuning
108 |
109 |
110 |
111 | ### Fine Tuning with Random Weights for FC
112 |
113 | Result (based on Cohen's Kappa score) places this entry at position 26-27 on public leaderboard (as of Nov 9 2016).
114 |
115 |
116 |
117 | Final Model (FT#1)
118 |
119 | Accuracy: 0.61667, Cohen's Kappa Score: 0.74487
120 |
121 | Confusion Matrix:
122 | [[32 7 10 4 1]
123 | [13 37 11 4 0]
124 | [ 7 3 45 6 3]
125 | [ 1 4 8 32 15]
126 | [ 0 3 3 12 39]]
127 |
128 | Classification Report:
129 | precision recall f1-score support
130 |
131 | 0 0.60 0.59 0.60 54
132 | 1 0.69 0.57 0.62 65
133 | 2 0.58 0.70 0.64 64
134 | 3 0.55 0.53 0.54 60
135 | 4 0.67 0.68 0.68 57
136 |
137 | avg / total 0.62 0.62 0.62 300
138 |
139 |
140 | ### Fine Tuning with Learned Weights for FC
141 |
142 | Result (based on Cohen's Kappa score) places this entry at position 32-33 on public leaderboard (as of Nov 9 2016).
143 |
144 |
145 |
146 | Final Model (FT#2)
147 |
148 | Accuracy: 0.63000, Cohen's Kappa Score: 0.72214
149 |
150 | Confusion Matrix:
151 | [[34 11 6 2 1]
152 | [ 5 49 8 3 0]
153 | [ 6 12 40 5 1]
154 | [ 3 9 10 30 8]
155 | [ 1 3 7 10 36]]
156 |
157 | Classification Report:
158 | precision recall f1-score support
159 |
160 | 0 0.69 0.63 0.66 54
161 | 1 0.58 0.75 0.66 65
162 | 2 0.56 0.62 0.59 64
163 | 3 0.60 0.50 0.55 60
164 | 4 0.78 0.63 0.70 57
165 |
166 | avg / total 0.64 0.63 0.63 300
167 |
168 |
169 | ## References
170 |
171 | * [How to use pre-trained models with Keras](https://keras.io/getting-started/faq/#how-can-i-use-pre-trained-models-in-keras) - Keras documentation by Francois Chollet.
172 | * [Keras Applications (pretrained Deep Learning Models)](https://keras.io/applications/) - Keras documentation by Francois Chollet.
173 | * [Keras Blog: Building Powerful Image Classification Models using very little data](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html) - blog post by Francois Chollet, creator of Keras.
174 | * [KDNuggets: Recycling Deep Learning Representations with Transfer Learning](http://www.kdnuggets.com/2015/08/recycling-deep-learning-representations-transfer-ml.html) - blog post by Zachary Chase Lipton.
175 | * [Deep Neural Networks in Azure: Transfer Learning and Fine Tuning](https://info.microsoft.com/CO-AAIoT-WBNR-FY17-09Sep-27-Deep-Neural-Networks-in-Azure-Transfer-Learning-and-Fine-tuning-253624_Registration.html) - webinar presented by Anusua Trivedi of Microsoft.
176 | * [The Neural Network Zoo](http://www.asimovinstitute.org/neural-network-zoo/) - blog post by Fjodor Van Veen of The Asimov Institute.
177 |
178 |
--------------------------------------------------------------------------------
/data/Makefile:
--------------------------------------------------------------------------------
1 | # Requires presence of credentials.txt file containing login/password in the following format:
2 | # UserName=my_username&Password=my_password
3 |
4 | COMPETITION=diabetic-retinopathy-detection
5 |
6 | all: download_files
7 |
8 | session.cookie: credentials.txt
9 | curl -o /dev/null -c session.cookie https://www.kaggle.com/account/login
10 | curl -o /dev/null -c session.cookie -b session.cookie -L -d @credentials.txt https://www.kaggle.com/account/login
11 |
12 | files.txt: session.cookie
13 | curl -c session.cookie -b session.cookie -L http://www.kaggle.com/c/$(COMPETITION)/data | \
14 | grep -o \"[^\"]*\/download[^\"]*\" | sed -e 's/"//g' -e 's/^/http:\/\/www.kaggle.com/' > files.txt
15 |
16 | download_files: files.txt session.cookie
17 | mkdir -p files
18 | cd files && xargs -n 1 curl -b ../session.cookie -L -O < ../files.txt
19 |
20 | .PHONY: clean
21 |
22 | clean:
23 | rm session.cookie files.txt files/*.zip
24 |
--------------------------------------------------------------------------------
/data/README.md:
--------------------------------------------------------------------------------
1 | # Data for this project
2 |
3 | Data for this project comes from the [Kaggle Diabetic Retinopathy Detection Competition](https://www.kaggle.com/c/diabetic-retinopathy-detection). We use a random sample of 1,000 images from the 35,126 images in this dataset.
4 |
5 | The following Makefile will create the directory structure under this directory. Unfortunately, the curl commands download HTML for the train.00[1-5].zip files. You will need to download them off the site via the browser.
6 |
7 | You will need to make the credentials.txt file using the provided template file. Replace the values with your Kaggle user name and password. Run the following commands.
8 |
9 | make
10 | cd files
11 |
12 | Next manually download trainLabels.csv.zip and sampleSubmission.csv.zip onto the HTML versions the make command downloaded. Also manually download the train.zip.00[1-5] files over the HTML versions downloaded.
13 |
14 | unzip -a trainLabels.csv.zip
15 | rm trainLabels.csv.zip
16 | unzip -a sampleSubmission.csv.zip
17 | rm sampleSubmission.csv.zip
18 | cd files
19 | rm test.zip*
20 | 7za x train.zip.001
21 |
22 | The last command will write 35126 images into the train directory under the current directory (files).
23 |
24 | You can now build the sample from the dataset.
25 |
26 | cd ..
27 | ../../src/make-sample.py
28 |
29 | This will generate a shell script sampleImages.sh in current directory.
30 |
31 | cd files
32 | mkdir sample
33 | cd sample
34 | mkdir 0 1 2 3 4
35 | cd ../..
36 | bash sampleImages.sh
37 |
38 | This will copy 1000 images, 200 each into each of the 5 category directories.
39 |
40 | Finally create a directory to hold the models.
41 |
42 | cd .. # you are in files now
43 | mkdir models
44 |
45 |
--------------------------------------------------------------------------------
/data/credentials.txt.template:
--------------------------------------------------------------------------------
1 | Username=your_kaggle_username&Password=your_kaggle_password
2 |
--------------------------------------------------------------------------------
/data/lena.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/data/lena.png
--------------------------------------------------------------------------------
/docs/ft-dl-loss.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/ft-dl-loss.png
--------------------------------------------------------------------------------
/docs/ft-dlw-loss.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/ft-dlw-loss.png
--------------------------------------------------------------------------------
/docs/fttl-keras-nov2016.pptx:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/fttl-keras-nov2016.pptx
--------------------------------------------------------------------------------
/docs/tl-dl1-loss.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/tl-dl1-loss.png
--------------------------------------------------------------------------------
/docs/tl-dl2-loss.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/tl-dl2-loss.png
--------------------------------------------------------------------------------
/docs/vgg16-ft.dia:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/vgg16-ft.dia
--------------------------------------------------------------------------------
/docs/vgg16-ft.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/vgg16-ft.png
--------------------------------------------------------------------------------
/docs/vgg16-original.dia:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/vgg16-original.dia
--------------------------------------------------------------------------------
/docs/vgg16-original.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/vgg16-original.png
--------------------------------------------------------------------------------
/docs/vgg16-tl.dia:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/vgg16-tl.dia
--------------------------------------------------------------------------------
/docs/vgg16-tl.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sujitpal/fttl-with-keras/330c5571f68958f27aaa56f80b14babc2afd3a45/docs/vgg16-tl.png
--------------------------------------------------------------------------------
/src/augment-images.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from keras.applications.vgg16 import VGG16
4 | from keras.applications.vgg16 import preprocess_input
5 | from keras.models import Model
6 | from keras.optimizers import SGD
7 | from keras.preprocessing import image
8 | from sklearn.cross_validation import StratifiedShuffleSplit
9 | import numpy as np
10 | import os
11 |
12 | def get_next_image_loc(imgdir):
13 | for root, dirs, files in os.walk(imgdir):
14 | for name in files:
15 | path = os.path.join(root, name).split(os.path.sep)[::-1]
16 | yield (path[1], path[0])
17 |
18 |
19 | def write_vectors(model, X, y, tag, data_dir, batch_size):
20 | fXout = open(os.path.join(data_dir,
21 | "images-500-{:s}-X.txt".format(tag)), "wb")
22 | fyout = open(os.path.join(data_dir,
23 | "images-500-{:s}-y.txt".format(tag)), "wb")
24 | num_written = 0
25 | for i in range(0, X.shape[0], batch_size):
26 | Xbatch = X[i:i + batch_size]
27 | ybatch = y[i:i + batch_size]
28 | vecs = model.predict(Xbatch)
29 | for vec, label in zip(vecs, ybatch):
30 | vec = vec.flatten()
31 | vec_str = ",".join(["{:.5f}".format(v) for v in vec.tolist()])
32 | fXout.write("{:s}\n".format(vec_str))
33 | fyout.write("{:d}\n".format(label))
34 | if num_written % 100 == 0:
35 | print("\twrote {:d} {:s} records".format(num_written, tag))
36 | num_written += 1
37 | print("\twrote {:d} {:s} records, COMPLETE".format(num_written, tag))
38 | fXout.close()
39 | fyout.close()
40 |
41 |
42 | ########################## main ##########################
43 |
44 | DATA_DIR = "../data"
45 | IMAGE_DIR = os.path.join(DATA_DIR, "images-500")
46 | IMAGE_WIDTH = 224
47 | BATCH_SIZE = 10
48 | NUM_TO_AUGMENT = 10
49 |
50 | np.random.seed(42)
51 |
52 | # load images and labels from images directory
53 | print("Loading images and labels from images directory...")
54 | xs, ys = [], []
55 | for label, image_file in get_next_image_loc(IMAGE_DIR):
56 | ys.append(int(label))
57 | img = image.load_img(os.path.join(IMAGE_DIR, label, image_file),
58 | target_size=(IMAGE_WIDTH, IMAGE_WIDTH))
59 | img4d = image.img_to_array(img)
60 | img4d = np.expand_dims(img4d, axis=0)
61 | img4d = preprocess_input(img4d)
62 | xs.append(img4d[0])
63 | X = np.array(xs)
64 | y = np.array(ys)
65 |
66 | # using regular train_test_split results in classes not being represented
67 | print("Initial split into train/val/test...")
68 | splitter = StratifiedShuffleSplit(y, n_iter=1, test_size=0.3,
69 | random_state=42)
70 | for train, test in splitter:
71 | Xtrain, Xtest, ytrain, ytest = X[train], X[test], y[train], y[test]
72 | break
73 | print(Xtrain.shape, Xtest.shape, ytrain.shape, ytest.shape)
74 |
75 | # instantiate ImageDataGenerator to create approximately 10 images for
76 | # each input training image
77 | print("Augmenting training set images...")
78 | datagen = image.ImageDataGenerator(
79 | featurewise_center=True,
80 | featurewise_std_normalization=True,
81 | rotation_range=20,
82 | width_shift_range=0.2,
83 | height_shift_range=0.2,
84 | shear_range=0.2,
85 | horizontal_flip=True)
86 |
87 | xtas, ytas = [], []
88 | for i in range(Xtrain.shape[0]):
89 | num_aug = 0
90 | x = Xtrain[i][np.newaxis]
91 | datagen.fit(x)
92 | for x_aug in datagen.flow(x, batch_size=1):
93 | if num_aug >= NUM_TO_AUGMENT:
94 | break
95 | xtas.append(x_aug[0])
96 | ytas.append(ytrain[i])
97 | num_aug += 1
98 |
99 | Xtrain = np.array(xtas)
100 | ytrain = np.array(ytas)
101 |
102 | print(Xtrain.shape, Xtest.shape, ytrain.shape, ytest.shape)
103 |
104 | # Instantiate VGG16 model and remove bottleneck
105 | print("Instantiating VGG16 model and removing top layers...")
106 | vgg16_model = VGG16(weights="imagenet", include_top=True)
107 | model = Model(input=vgg16_model.input,
108 | output=vgg16_model.get_layer("block5_pool").output)
109 | sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
110 | model.compile(optimizer=sgd, loss="categorical_crossentropy")
111 |
112 | # Read each of train, validation and test vectors out to named files
113 | print("Writing vectors to files...")
114 | write_vectors(model, Xtrain, ytrain, "train", DATA_DIR, BATCH_SIZE)
115 | write_vectors(model, Xtest, ytest, "test", DATA_DIR, BATCH_SIZE)
116 |
--------------------------------------------------------------------------------
/src/caffe2keras-rebuild.py:
--------------------------------------------------------------------------------
1 | from __future__ import division, print_function
2 | from keras import backend as K
3 | from keras.engine.topology import Layer, InputSpec
4 | from keras.layers import Input
5 | from keras.layers.core import Activation, Dense, Flatten
6 | from keras.layers.convolutional import Convolution2D, ZeroPadding2D
7 | from keras.layers.normalization import BatchNormalization
8 | from keras.layers.pooling import MaxPooling2D
9 | from keras.models import Model, load_model
10 | from scipy.misc import imresize
11 | import matplotlib.pyplot as plt
12 | import numpy as np
13 | import os
14 | import re
15 |
16 |
17 | class LocalResponseNormalization(Layer):
18 |
19 | def __init__(self, n=5, alpha=0.0005, beta=0.75, k=2, **kwargs):
20 | self.n = n
21 | self.alpha = alpha
22 | self.beta = beta
23 | self.k = k
24 | super(LocalResponseNormalization, self).__init__(**kwargs)
25 |
26 | def build(self, input_shape):
27 | self.shape = input_shape
28 | super(LocalResponseNormalization, self).build(input_shape)
29 |
30 | def call(self, x, mask=None):
31 | if K.image_dim_ordering == "th":
32 | _, f, r, c = self.shape
33 | else:
34 | _, r, c, f = self.shape
35 | half_n = self.n // 2
36 | squared = K.square(x)
37 | pooled = K.pool2d(squared, (half_n, half_n), strides=(1, 1),
38 | border_mode="same", pool_mode="avg")
39 | if K.image_dim_ordering == "th":
40 | summed = K.sum(pooled, axis=1, keepdims=True)
41 | averaged = (self.alpha / self.n) * K.repeat_elements(summed, f, axis=1)
42 | else:
43 | summed = K.sum(pooled, axis=3, keepdims=True)
44 | averaged = (self.alpha / self.n) * K.repeat_elements(summed, f, axis=3)
45 | denom = K.pow(self.k + averaged, self.beta)
46 | return x / denom
47 |
48 | def get_output_shape_for(self, input_shape):
49 | return input_shape
50 |
51 | def transform_conv_weight(W):
52 | # for non FC layers, do this because Keras/Theano does convolution vs
53 | # Caffe correlation
54 | for i in range(W.shape[0]):
55 | for j in range(W.shape[1]):
56 | W[i, j] = np.rot90(W[i, j], 2)
57 | return W
58 |
59 | def transform_fc_weight(W):
60 | return W.T
61 |
62 | def preprocess_image(img, resize_wh, mean_image):
63 | # resize
64 | img4d = imresize(img, (resize_wh, resize_wh))
65 | img4d = img4d.astype("float32")
66 | # BGR -> RGB
67 | img4d = img4d[:, :, ::-1]
68 | # swap axes to theano mode
69 | img4d = np.transpose(img4d, (2, 0, 1))
70 | # add batch dimension
71 | img4d = np.expand_dims(img4d, axis=0)
72 | # subtract mean image
73 | img4d -= mean_image
74 | # clip to uint
75 | img4d = np.clip(img4d, 0, 255).astype("uint8")
76 | return img4d
77 |
78 | DATA_DIR = "../data/vgg-cnn"
79 | CAT_IMAGE = os.path.join(DATA_DIR, "cat.jpg")
80 | MEAN_IMAGE = os.path.join(DATA_DIR, "mean_image.npy")
81 | CAFFE_WEIGHTS_DIR = os.path.join(DATA_DIR, "saved-weights")
82 | LABEL_FILE = os.path.join(DATA_DIR, "caffe2keras-labels.txt")
83 | KERAS_MODEL_FILE = os.path.join(DATA_DIR, "vggcnn-keras.h5")
84 | RESIZE_WH = 224
85 |
86 | # caffe model layers (reference)
87 | CAFFE_LAYER_NAMES = [
88 | "data",
89 | "conv1", "norm1", "pool1",
90 | "conv2", "pool2",
91 | "conv3",
92 | "conv4",
93 | "conv5", "pool5",
94 | "fc6",
95 | "fc7",
96 | "prob"
97 | ]
98 | CAFFE_LAYER_SHAPES = {
99 | "data" : (10, 3, 224, 224),
100 | "conv1": (10, 96, 109, 109),
101 | "norm1": (10, 96, 109, 109),
102 | "pool1": (10, 96, 37, 37),
103 | "conv2": (10, 256, 33, 33),
104 | "pool2": (10, 256, 17, 17),
105 | "conv3": (10, 512, 17, 17),
106 | "conv4": (10, 512, 17, 17),
107 | "conv5": (10, 512, 17, 17),
108 | "pool5": (10, 512, 6, 6),
109 | "fc6" : (10, 4096),
110 | "fc7" : (10, 4096),
111 | "fc8" : (10, 1000),
112 | "prob" : (10, 1000)
113 | }
114 |
115 | print("caffe:")
116 | for layer_name in CAFFE_LAYER_NAMES:
117 | print(layer_name, CAFFE_LAYER_SHAPES[layer_name])
118 |
119 | # data (10, 3, 224, 224)
120 | # conv1 (10, 96, 109, 109)
121 | # norm1 (10, 96, 109, 109)
122 | # pool1 (10, 96, 37, 37)
123 | # conv2 (10, 256, 33, 33)
124 | # pool2 (10, 256, 17, 17)
125 | # conv3 (10, 512, 17, 17)
126 | # conv4 (10, 512, 17, 17)
127 | # conv5 (10, 512, 17, 17)
128 | # pool5 (10, 512, 6, 6)
129 | # fc6 (10, 4096)
130 | # fc7 (10, 4096)
131 | # prob (10, 1000)
132 |
133 | # set theano dimension ordering
134 | # NOTE: results match Caffe using Theano backend only
135 | K.set_image_dim_ordering("th")
136 |
137 | # load weights
138 | W_conv1 = transform_conv_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_conv1.npy")))
139 | b_conv1 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_conv1.npy"))
140 |
141 | W_conv2 = transform_conv_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_conv2.npy")))
142 | b_conv2 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_conv2.npy"))
143 |
144 | W_conv3 = transform_conv_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_conv3.npy")))
145 | b_conv3 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_conv3.npy"))
146 |
147 | W_conv4 = transform_conv_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_conv4.npy")))
148 | b_conv4 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_conv4.npy"))
149 |
150 | W_conv5 = transform_conv_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_conv5.npy")))
151 | b_conv5 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_conv5.npy"))
152 |
153 | W_fc6 = transform_fc_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_fc6.npy")))
154 | b_fc6 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_fc6.npy"))
155 |
156 | W_fc7 = transform_fc_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_fc7.npy")))
157 | b_fc7 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_fc7.npy"))
158 |
159 | W_fc8 = transform_fc_weight(np.load(os.path.join(CAFFE_WEIGHTS_DIR, "W_fc8.npy")))
160 | b_fc8 = np.load(os.path.join(CAFFE_WEIGHTS_DIR, "b_fc8.npy"))
161 |
162 | # define network
163 | data = Input(shape=(3, 224, 224), name="DATA")
164 |
165 | conv1 = Convolution2D(96, 7, 7, subsample=(2, 2),
166 | weights=(W_conv1, b_conv1))(data)
167 | conv1 = Activation("relu", name="CONV1")(conv1)
168 |
169 | norm1 = LocalResponseNormalization(name="NORM1")(conv1)
170 |
171 | pool1 = ZeroPadding2D(padding=(0, 2, 0, 2))(norm1)
172 | pool1 = MaxPooling2D(pool_size=(3, 3), strides=(3, 3), name="POOL1")(pool1)
173 |
174 | conv2 = Convolution2D(256, 5, 5, weights=(W_conv2, b_conv2))(pool1)
175 | conv2 = Activation("relu", name="CONV2")(conv2)
176 |
177 | pool2 = ZeroPadding2D(padding=(0, 1, 0, 1))(conv2)
178 | pool2 = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name="POOL2")(pool2)
179 |
180 | conv3 = ZeroPadding2D(padding=(0, 2, 0, 2))(pool2)
181 | conv3 = Convolution2D(512, 3, 3, weights=(W_conv3, b_conv3))(conv3)
182 | conv3 = Activation("relu", name="CONV3")(conv3)
183 |
184 | conv4 = ZeroPadding2D(padding=(0, 2, 0, 2))(conv3)
185 | conv4 = Convolution2D(512, 3, 3, weights=(W_conv4, b_conv4))(conv4)
186 | conv4 = Activation("relu", name="CONV4")(conv4)
187 |
188 | conv5 = ZeroPadding2D(padding=(0, 2, 0, 2))(conv4)
189 | conv5 = Convolution2D(512, 3, 3, weights=(W_conv5, b_conv5))(conv5)
190 | conv5 = Activation("relu", name="CONV5")(conv5)
191 |
192 | pool5 = ZeroPadding2D(padding=(0, 1, 0, 1))(conv5)
193 | pool5 = MaxPooling2D(pool_size=(3, 3), strides=(3, 3), name="POOL5")(pool5)
194 |
195 | fc6 = Flatten()(pool5)
196 | fc6 = Dense(4096, weights=(W_fc6, b_fc6))(fc6)
197 | fc6 = Activation("relu", name="FC6")(fc6)
198 |
199 | fc7 = Dense(4096, weights=(W_fc7, b_fc7))(fc6)
200 | fc7 = Activation("relu", name="FC7")(fc7)
201 |
202 | fc8 = Dense(1000, weights=(W_fc8, b_fc8), name="FC8")(fc7)
203 | prob = Activation("softmax", name="PROB")(fc8)
204 |
205 | model = Model(input=[data], output=[prob])
206 |
207 | model.compile(optimizer="adam", loss="categorical_crossentropy")
208 |
209 | print("keras")
210 | for layer in model.layers:
211 | print(layer.name, layer.output_shape)
212 |
213 | # DATA (None, 3, 224, 224)
214 | # convolution2d_1 (None, 96, 109, 109)
215 | # CONV1 (None, 96, 109, 109)
216 | # NORM1 (None, 96, 109, 109)
217 | # zeropadding2d_1 (None, 96, 111, 111)
218 | # POOL1 (None, 96, 37, 37)
219 | # convolution2d_2 (None, 256, 33, 33)
220 | # CONV2 (None, 256, 33, 33)
221 | # zeropadding2d_2 (None, 256, 34, 34)
222 | # POOL2 (None, 256, 17, 17)
223 | # zeropadding2d_3 (None, 256, 19, 19)
224 | # convolution2d_3 (None, 512, 17, 17)
225 | # CONV3 (None, 512, 17, 17)
226 | # zeropadding2d_4 (None, 512, 19, 19)
227 | # convolution2d_4 (None, 512, 17, 17)
228 | # CONV4 (None, 512, 17, 17)
229 | # zeropadding2d_5 (None, 512, 19, 19)
230 | # convolution2d_5 (None, 512, 17, 17)
231 | # CONV5 (None, 512, 17, 17)
232 | # zeropadding2d_6 (None, 512, 18, 18)
233 | # POOL5 (None, 512, 6, 6)
234 | # flatten_1 (None, 18432)
235 | # dense_1 (None, 4096)
236 | # FC6 (None, 4096)
237 | # dense_2 (None, 4096)
238 | # FC7 (None, 4096)
239 | # FC8 (None, 1000)
240 | # PROB (None, 1000)
241 |
242 | # prediction
243 | id2label = {}
244 | flabel = open(LABEL_FILE, "rb")
245 | for line in flabel:
246 | lid, lname = line.strip().split("\t")
247 | id2label[int(lid)] = lname
248 | flabel.close()
249 |
250 | mean_image = np.load(MEAN_IMAGE)
251 | image = plt.imread(CAT_IMAGE)
252 | img4d = preprocess_image(image, RESIZE_WH, mean_image)
253 |
254 | print(image.shape, mean_image.shape, img4d.shape)
255 |
256 | preds = model.predict(img4d)[0]
257 | print(np.argmax(preds))
258 | # 281
259 |
260 | top_preds = np.argsort(preds)[::-1][0:10]
261 | # array([281, 285, 282, 277, 287, 284, 283, 263, 387, 892])
262 |
263 | pred_probas = [(x, id2label[x], preds[x]) for x in top_preds]
264 | print(pred_probas)
265 |
266 | print("Saving model...")
267 | model.save(KERAS_MODEL_FILE)
268 |
269 |
--------------------------------------------------------------------------------
/src/caffe2keras-save.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | import caffe
4 | import numpy as np
5 | import os
6 |
7 | CAFFE_HOME="/home/ubuntu/mnt/caffe"
8 |
9 | MODEL_DIR = os.path.join(CAFFE_HOME, "models", "vgg_cnn_s")
10 | MODEL_PROTO = os.path.join(MODEL_DIR, "deploy.prototxt")
11 | MODEL_WEIGHTS = os.path.join(MODEL_DIR, "VGG_CNN_S.caffemodel")
12 | MEAN_IMAGE = os.path.join(MODEL_DIR, "VGG_mean.binaryproto")
13 |
14 | OUTPUT_DIR = "../../data/vgg-cnn-weights"
15 |
16 | caffe.set_mode_cpu()
17 | net = caffe.Net(MODEL_PROTO, MODEL_WEIGHTS, caffe.TEST)
18 |
19 | for k, v in net.params.items():
20 | print(k, v[0].data.shape, v[1].data.shape)
21 | np.save(os.path.join(OUTPUT_DIR, "W_{:s}.npy".format(k)), v[0].data)
22 | np.save(os.path.join(OUTPUT_DIR, "b_{:s}.npy".format(k)), v[1].data)
23 |
24 | # layer W.shape b.shape
25 | #conv1 (96, 3, 7, 7) (96,)
26 | #conv2 (256, 96, 5, 5) (256,)
27 | #conv3 (512, 256, 3, 3) (512,)
28 | #conv4 (512, 512, 3, 3) (512,)
29 | #conv5 (512, 512, 3, 3) (512,)
30 | #fc6 (4096, 18432) (4096,)
31 | #fc7 (4096, 4096) (4096,)
32 | #fc8 (1000, 4096) (1000,)
33 |
34 | blob = caffe.proto.caffe_pb2.BlobProto()
35 | with open(MEAN_IMAGE, 'rb') as fmean:
36 | mean_data = fmean.read()
37 | blob.ParseFromString(mean_data)
38 | mu = np.array(caffe.io.blobproto_to_array(blob))
39 | print("Mean image:", mu.shape)
40 | np.save(os.path.join(OUTPUT_DIR, "mean_image.npy"), mu)
41 |
42 | #Mean image: (1, 3, 224, 224)
43 |
44 | for layer_name, blob in net.blobs.iteritems():
45 | print(layer_name, blob.data.shape)
46 |
47 | #data (10, 3, 224, 224)
48 | #conv1 (10, 96, 109, 109)
49 | #norm1 (10, 96, 109, 109)
50 | #pool1 (10, 96, 37, 37)
51 | #conv2 (10, 256, 33, 33)
52 | #pool2 (10, 256, 17, 17)
53 | #conv3 (10, 512, 17, 17)
54 | #conv4 (10, 512, 17, 17)
55 | #conv5 (10, 512, 17, 17)
56 | #pool5 (10, 512, 6, 6)
57 | #fc6 (10, 4096)
58 | #fc7 (10, 4096)
59 | #fc8 (10, 1000)
60 | #prob (10, 1000)
--------------------------------------------------------------------------------
/src/confusion-to-heatmap.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | import numpy as np
3 | import matplotlib.pyplot as plt
4 |
5 | conf_arr = [[33,2,0,0,0,0,0,0,0,1,3],
6 | [3,31,0,0,0,0,0,0,0,0,0],
7 | [0,4,41,0,0,0,0,0,0,0,1],
8 | [0,1,0,30,0,6,0,0,0,0,1],
9 | [0,0,0,0,38,10,0,0,0,0,0],
10 | [0,0,0,3,1,39,0,0,0,0,4],
11 | [0,2,2,0,4,1,31,0,0,0,2],
12 | [0,1,0,0,0,0,0,36,0,2,0],
13 | [0,0,0,0,0,0,1,5,37,5,1],
14 | [3,0,0,0,0,0,0,0,0,39,0],
15 | [0,0,0,0,0,0,0,0,0,0,38]]
16 |
17 | norm_conf = []
18 | for i in conf_arr:
19 | a = 0
20 | tmp_arr = []
21 | a = sum(i, 0)
22 | for j in i:
23 | tmp_arr.append(float(j)/float(a))
24 | norm_conf.append(tmp_arr)
25 |
26 | fig = plt.figure()
27 | plt.clf()
28 | ax = fig.add_subplot(111)
29 | ax.set_aspect(1)
30 | res = ax.imshow(np.array(norm_conf), cmap=plt.cm.jet,
31 | interpolation='nearest')
32 |
33 | width, height = conf_arr.shape
34 |
35 | for x in xrange(width):
36 | for y in xrange(height):
37 | ax.annotate(str(conf_arr[x][y]), xy=(y, x),
38 | horizontalalignment='center',
39 | verticalalignment='center')
40 |
41 | cb = fig.colorbar(res)
42 | alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
43 | plt.xticks(range(width), alphabet[:width])
44 | plt.yticks(range(height), alphabet[:height])
45 | plt.savefig('confusion_matrix.png', format='png')
46 |
--------------------------------------------------------------------------------
/src/ft-dl-train.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from keras.applications.vgg16 import VGG16
4 | from keras.applications.vgg16 import preprocess_input
5 | from keras.callbacks import ModelCheckpoint
6 | from keras.layers import Dense, Dropout, Reshape
7 | from keras.optimizers import SGD
8 | from keras.models import Model, load_model
9 | from keras.preprocessing import image
10 | from keras.utils import np_utils
11 | import numpy as np
12 | import os
13 |
14 | import fttlutils
15 |
16 | ################################# main #################################
17 |
18 | DATA_DIR = "../data/files"
19 | MODEL_DIR = os.path.join(DATA_DIR, "models")
20 | IMAGE_DIR = os.path.join(DATA_DIR, "sample")
21 | BATCH_SIZE = 32
22 | NUM_EPOCHS = 20
23 | IMAGE_WIDTH = 224
24 |
25 | np.random.seed(42)
26 |
27 | # data
28 | ys, fs = [], []
29 | flabels = open(os.path.join(DATA_DIR, "images-y.txt"), "rb")
30 | for line in flabels:
31 | ys.append(int(line.strip()))
32 | flabels.close()
33 | ffilenames = open(os.path.join(DATA_DIR, "images-f.txt"), "rb")
34 | for line in ffilenames:
35 | fs.append(line.strip())
36 | ffilenames.close()
37 | xs = []
38 | for y, f in zip(ys, fs):
39 | img = image.load_img(os.path.join(IMAGE_DIR, str(y), f),
40 | target_size=(IMAGE_WIDTH, IMAGE_WIDTH))
41 | img4d = image.img_to_array(img)
42 | img4d = np.expand_dims(img4d, axis=0)
43 | img4d = preprocess_input(img4d)
44 | xs.append(img4d[0])
45 |
46 | X = np.array(xs)
47 | y = np.array(ys)
48 | Y = np_utils.to_categorical(y, nb_classes=5)
49 |
50 | Xtrain, Xtest, Ytrain, Ytest = fttlutils.train_test_split(
51 | X, Y, test_size=0.3, random_state=42)
52 | print(Xtrain.shape, Xtest.shape, Ytrain.shape, Ytest.shape)
53 |
54 | # build model
55 |
56 | # (1) instantiate VGG16 and remove top layers
57 | vgg16_model = VGG16(weights="imagenet", include_top=True)
58 | # visualize layers
59 | #print("VGG16 model layers")
60 | #for i, layer in enumerate(vgg16_model.layers):
61 | # print(i, layer.name, layer.output_shape)
62 | #
63 | #(0, 'input_6', (None, 224, 224, 3))
64 | #(1, 'block1_conv1', (None, 224, 224, 64))
65 | #(2, 'block1_conv2', (None, 224, 224, 64))
66 | #(3, 'block1_pool', (None, 112, 112, 64))
67 | #(4, 'block2_conv1', (None, 112, 112, 128))
68 | #(5, 'block2_conv2', (None, 112, 112, 128))
69 | #(6, 'block2_pool', (None, 56, 56, 128))
70 | #(7, 'block3_conv1', (None, 56, 56, 256))
71 | #(8, 'block3_conv2', (None, 56, 56, 256))
72 | #(9, 'block3_conv3', (None, 56, 56, 256))
73 | #(10, 'block3_pool', (None, 28, 28, 256))
74 | #(11, 'block4_conv1', (None, 28, 28, 512))
75 | #(12, 'block4_conv2', (None, 28, 28, 512))
76 | #(13, 'block4_conv3', (None, 28, 28, 512))
77 | #(14, 'block4_pool', (None, 14, 14, 512))
78 | #(15, 'block5_conv1', (None, 14, 14, 512))
79 | #(16, 'block5_conv2', (None, 14, 14, 512))
80 | #(17, 'block5_conv3', (None, 14, 14, 512))
81 | #(18, 'block5_pool', (None, 7, 7, 512))
82 | #(19, 'flatten', (None, 25088))
83 | #(20, 'fc1', (None, 4096))
84 | #(21, 'fc2', (None, 4096))
85 | #(22, 'predictions', (None, 1000))
86 |
87 | # (2) remove the top layer
88 | base_model = Model(input=vgg16_model.input,
89 | output=vgg16_model.get_layer("block5_pool").output)
90 |
91 | # (3) attach a new top layer
92 | base_out = base_model.output
93 | base_out = Reshape((25088,))(base_out)
94 | top_fc1 = Dense(256, activation="relu")(base_out)
95 | top_fc1 = Dropout(0.5)(top_fc1)
96 | # output layer: (None, 5)
97 | top_preds = Dense(5, activation="softmax")(top_fc1)
98 |
99 | # (4) freeze weights until the last but one convolution layer (block4_pool)
100 | for layer in base_model.layers[0:14]:
101 | layer.trainable = False
102 |
103 | # (5) create new hybrid model
104 | model = Model(input=base_model.input, output=top_preds)
105 |
106 | # (6) compile and train the model
107 | sgd = SGD(lr=1e-4, momentum=0.9)
108 | model.compile(optimizer=sgd, loss="categorical_crossentropy",
109 | metrics=["accuracy"])
110 |
111 | best_model = os.path.join(MODEL_DIR, "ft-dl-model-best.h5")
112 | checkpoint = ModelCheckpoint(filepath=best_model, verbose=1,
113 | save_best_only=True)
114 | history = model.fit([Xtrain], [Ytrain], nb_epoch=NUM_EPOCHS,
115 | batch_size=BATCH_SIZE, validation_split=0.1,
116 | callbacks=[checkpoint])
117 | fttlutils.plot_loss(history)
118 |
119 | # evaluate final model
120 | Ytest_ = model.predict(Xtest)
121 | ytest = np_utils.categorical_probas_to_classes(Ytest)
122 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
123 | fttlutils.print_stats(ytest, ytest_, "Final Model (FT#1)")
124 | model.save(os.path.join(MODEL_DIR, "ft-dl-model-final.h5"))
125 |
126 | # load best model and evaluate
127 | model = load_model(os.path.join(MODEL_DIR, "ft-dl-model-best.h5"))
128 | model.compile(optimizer=sgd, loss="categorical_crossentropy",
129 | metrics=["accuracy"])
130 | Ytest_ = model.predict(Xtest)
131 | ytest = np_utils.categorical_probas_to_classes(Ytest)
132 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
133 | fttlutils.print_stats(ytest, ytest_, "Best Model (FT#1)")
134 |
--------------------------------------------------------------------------------
/src/ft-dlw-train.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from keras.applications.vgg16 import VGG16
4 | from keras.applications.vgg16 import preprocess_input
5 | from keras.callbacks import ModelCheckpoint
6 | from keras.layers import Dense, Dropout, Reshape
7 | from keras.optimizers import SGD
8 | from keras.models import Model, load_model
9 | from keras.preprocessing import image
10 | from keras.utils import np_utils
11 | import numpy as np
12 | import os
13 |
14 | import fttlutils
15 |
16 | ################################# main #################################
17 |
18 | DATA_DIR = "../data/files"
19 | MODEL_DIR = os.path.join(DATA_DIR, "models")
20 | IMAGE_DIR = os.path.join(DATA_DIR, "sample")
21 | BATCH_SIZE = 32
22 | NUM_EPOCHS = 20
23 | IMAGE_WIDTH = 224
24 |
25 | np.random.seed(42)
26 |
27 | # data
28 | ys, fs = [], []
29 | flabels = open(os.path.join(DATA_DIR, "images-y.txt"), "rb")
30 | for line in flabels:
31 | ys.append(int(line.strip()))
32 | flabels.close()
33 | ffilenames = open(os.path.join(DATA_DIR, "images-f.txt"), "rb")
34 | for line in ffilenames:
35 | fs.append(line.strip())
36 | ffilenames.close()
37 | xs = []
38 | for y, f in zip(ys, fs):
39 | img = image.load_img(os.path.join(IMAGE_DIR, str(y), f),
40 | target_size=(IMAGE_WIDTH, IMAGE_WIDTH))
41 | img4d = image.img_to_array(img)
42 | img4d = np.expand_dims(img4d, axis=0)
43 | img4d = preprocess_input(img4d)
44 | xs.append(img4d[0])
45 |
46 | X = np.array(xs)
47 | y = np.array(ys)
48 | Y = np_utils.to_categorical(y, nb_classes=5)
49 |
50 | Xtrain, Xtest, Ytrain, Ytest = fttlutils.train_test_split(
51 | X, Y, test_size=0.3, random_state=42)
52 | print(Xtrain.shape, Xtest.shape, Ytrain.shape, Ytest.shape)
53 |
54 | # build model
55 |
56 | # (1) instantiate VGG16 and remove top layers
57 | vgg16_model = VGG16(weights="imagenet", include_top=True)
58 | # visualize layers
59 | #print("VGG16 model layers")
60 | #for i, layer in enumerate(vgg16_model.layers):
61 | # print(i, layer.name, layer.output_shape)
62 | #
63 | #(0, 'input_6', (None, 224, 224, 3))
64 | #(1, 'block1_conv1', (None, 224, 224, 64))
65 | #(2, 'block1_conv2', (None, 224, 224, 64))
66 | #(3, 'block1_pool', (None, 112, 112, 64))
67 | #(4, 'block2_conv1', (None, 112, 112, 128))
68 | #(5, 'block2_conv2', (None, 112, 112, 128))
69 | #(6, 'block2_pool', (None, 56, 56, 128))
70 | #(7, 'block3_conv1', (None, 56, 56, 256))
71 | #(8, 'block3_conv2', (None, 56, 56, 256))
72 | #(9, 'block3_conv3', (None, 56, 56, 256))
73 | #(10, 'block3_pool', (None, 28, 28, 256))
74 | #(11, 'block4_conv1', (None, 28, 28, 512))
75 | #(12, 'block4_conv2', (None, 28, 28, 512))
76 | #(13, 'block4_conv3', (None, 28, 28, 512))
77 | #(14, 'block4_pool', (None, 14, 14, 512))
78 | #(15, 'block5_conv1', (None, 14, 14, 512))
79 | #(16, 'block5_conv2', (None, 14, 14, 512))
80 | #(17, 'block5_conv3', (None, 14, 14, 512))
81 | #(18, 'block5_pool', (None, 7, 7, 512))
82 | #(19, 'flatten', (None, 25088))
83 | #(20, 'fc1', (None, 4096))
84 | #(21, 'fc2', (None, 4096))
85 | #(22, 'predictions', (None, 1000))
86 |
87 | # (2) remove the top layer
88 | base_model = Model(input=vgg16_model.input,
89 | output=vgg16_model.get_layer("block5_pool").output)
90 |
91 | # (3) load trained model and attach to top
92 | base_out = base_model.output
93 | base_out = Reshape((25088,))(base_out)
94 | top_fc1 = Dense(256, activation="relu", name="dl1fc1")(base_out)
95 | top_fc1 = Dropout(0.5)(top_fc1)
96 | # output layer: (None, 5)
97 | top_preds = Dense(5, activation="softmax", name="dl1preds")(top_fc1)
98 |
99 | # (4) freeze weights until the last but one convolution layer (block4_pool)
100 | for layer in base_model.layers[0:14]:
101 | layer.trainable = False
102 |
103 | # (5) create new hybrid model
104 | model = Model(input=base_model.input, output=top_preds)
105 |
106 | # (6) load weights for the final layers
107 | model.load_weights(os.path.join(MODEL_DIR, "tl-dl1-model-final.h5"),
108 | by_name=True)
109 |
110 | # (7) compile and train the model
111 | sgd = SGD(lr=1e-4, momentum=0.9)
112 | model.compile(optimizer=sgd, loss="categorical_crossentropy",
113 | metrics=["accuracy"])
114 |
115 | best_model = os.path.join(MODEL_DIR, "ft-dlw-model-best.h5")
116 | checkpoint = ModelCheckpoint(filepath=best_model, verbose=1,
117 | save_best_only=True)
118 | history = model.fit([Xtrain], [Ytrain], nb_epoch=NUM_EPOCHS,
119 | batch_size=BATCH_SIZE, validation_split=0.1,
120 | callbacks=[checkpoint])
121 | fttlutils.plot_loss(history)
122 |
123 | # evaluate final model
124 | Ytest_ = model.predict(Xtest)
125 | ytest = np_utils.categorical_probas_to_classes(Ytest)
126 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
127 | fttlutils.print_stats(ytest, ytest_, "Final Model (FT#2)")
128 | model.save(os.path.join(MODEL_DIR, "ft-dlw-model-final.h5"))
129 |
130 | # load best model and evaluate
131 | model = load_model(os.path.join(MODEL_DIR, "ft-dlw-model-best.h5"))
132 | model.compile(optimizer=sgd, loss="categorical_crossentropy",
133 | metrics=["accuracy"])
134 | Ytest_ = model.predict(Xtest)
135 | ytest = np_utils.categorical_probas_to_classes(Ytest)
136 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
137 | fttlutils.print_stats(ytest, ytest_, "Best Model (FT#2)")
138 |
--------------------------------------------------------------------------------
/src/fttlutils.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from sklearn.model_selection import StratifiedShuffleSplit
4 | from sklearn.metrics import accuracy_score
5 | from sklearn.metrics import confusion_matrix
6 | from sklearn.metrics import classification_report
7 | from sklearn.metrics import cohen_kappa_score
8 | import matplotlib.pyplot as plt
9 |
10 | def train_test_split(X, Y, test_size, random_state):
11 | # using regular train_test_split results in classes not being represented
12 | splitter = StratifiedShuffleSplit(n_splits=1,
13 | test_size=test_size,
14 | random_state=random_state)
15 | for train, test in splitter.split(X, Y):
16 | Xtrain, Xtest, Ytrain, Ytest = X[train], X[test], Y[train], Y[test]
17 | break
18 | return Xtrain, Xtest, Ytrain, Ytest
19 |
20 | def plot_loss(history):
21 | # visualize training loss and accuracy
22 | plt.subplot(211)
23 | plt.title("Accuracy")
24 | plt.plot(history.history["acc"], color="r", label="Train")
25 | plt.plot(history.history["val_acc"], color="b", label="Validation")
26 | plt.legend(loc="best")
27 |
28 | plt.subplot(212)
29 | plt.title("Loss")
30 | plt.plot(history.history["loss"], color="r", label="Train")
31 | plt.plot(history.history["val_loss"], color="b", label="Validation")
32 | plt.legend(loc="best")
33 |
34 | plt.tight_layout()
35 | plt.show()
36 |
37 | def print_stats(ytest, ytest_, model_name):
38 | print(model_name)
39 | print("Accuracy: {:.5f}, Cohen's Kappa Score: {:.5f}".format(
40 | accuracy_score(ytest, ytest_),
41 | cohen_kappa_score(ytest, ytest_, weights="quadratic")))
42 | print("Confusion Matrix:")
43 | print(confusion_matrix(ytest, ytest_))
44 | print("Classification Report:")
45 | print(classification_report(ytest, ytest_))
46 |
47 |
--------------------------------------------------------------------------------
/src/make-sample.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | import numpy as np
4 | import os
5 |
6 | DOWNLOAD_DIR = "../data/files"
7 | SHELL_SCRIPT = os.path.join(DOWNLOAD_DIR, "scriptImages.sh")
8 | LABEL_FILE = os.path.join(DOWNLOAD_DIR, "trainLabels.csv")
9 |
10 | label2images = {}
11 | flab = open(LABEL_FILE, "rb")
12 | for line in flab:
13 | if line.startswith("image,"):
14 | continue
15 | image_name, label = line.strip().split(",")
16 | if label2images.has_key(label):
17 | label2images[label] += image_name
18 | else:
19 | label2images[label] = [image_name]
20 | flab.close()
21 |
22 | fsh = open(SHELL_SCRIPT, "wb")
23 | for label in label2images.keys():
24 | indices = np.arange(len(label2images[label]))
25 | print("label=", label, "len=", len(label2images[label]))
26 | sample_indices = np.random.choice(indices, size=200, replace=False)
27 | images = label2images[label]
28 | for ind in sample_indices:
29 | print("cp {:s}.jpeg sample/{:s}/".format(image_name, label))
30 | fsh.write("cp {:s}.jpeg sample/{:s}/\n".format(image_name, label))
31 |
32 | fsh.close()
33 |
--------------------------------------------------------------------------------
/src/preprocess-images.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | import cv2
4 | import numpy as np
5 | import matplotlib.pyplot as plt
6 | import os
7 |
8 |
9 | def plot_images(images):
10 | images = images[0:9]
11 | fig, axes = plt.subplots(3, 3)
12 | axes = np.ravel(axes)
13 | for i in range(len(images)):
14 | if len(images[i].shape) == 2:
15 | axes[i].imshow(images[i], cmap="gray")
16 | else:
17 | axes[i].imshow(images[i], interpolation="nearest")
18 | axes[i].set_xticks([])
19 | axes[i].set_yticks([])
20 | plt.xticks([])
21 | plt.yticks([])
22 | plt.tight_layout()
23 | plt.show()
24 |
25 |
26 | def get_next_image_loc(imgdir):
27 | for root, dirs, files in os.walk(imgdir):
28 | for name in files:
29 | path = os.path.join(root, name).split(os.path.sep)[::-1]
30 | yield (path[1], path[0])
31 |
32 |
33 | def compute_edges(image):
34 | image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
35 | image = cv2.GaussianBlur(image, (11, 11), 0)
36 | sobel_x = cv2.Sobel(image, cv2.CV_64F, 1, 0)
37 | sobel_x = np.uint8(np.absolute(sobel_x))
38 | sobel_y = cv2.Sobel(image, cv2.CV_64F, 0, 1)
39 | sobel_y = np.uint8(np.absolute(sobel_y))
40 | edged = cv2.bitwise_or(sobel_x, sobel_y)
41 | return edged
42 |
43 |
44 | def crop_image_to_edge(image, threshold=10, margin=0.2):
45 | edged = compute_edges(image)
46 | # find edge along center and crop
47 | mid_y = edged.shape[0] // 2
48 | notblack_x = np.where(edged[mid_y, :] >= threshold)[0]
49 | if notblack_x.shape[0] == 0:
50 | lb_x = 0
51 | ub_x = edged.shape[1]
52 | else:
53 | lb_x = notblack_x[0]
54 | ub_x = notblack_x[-1]
55 | if lb_x > margin * edged.shape[1]:
56 | lb_x = 0
57 | if (edged.shape[1] - ub_x) > margin * edged.shape[1]:
58 | ub_x = edged.shape[1]
59 | mid_x = edged.shape[1] // 2
60 | notblack_y = np.where(edged[:, mid_x] >= threshold)[0]
61 | if notblack_y.shape[0] == 0:
62 | lb_y = 0
63 | ub_y = edged.shape[0]
64 | else:
65 | lb_y = notblack_y[0]
66 | ub_y = notblack_y[-1]
67 | if lb_y > margin * edged.shape[0]:
68 | lb_y = 0
69 | if (edged.shape[0] - ub_y) > margin * edged.shape[0]:
70 | ub_y = edged.shape[0]
71 | cropped = image[lb_y:ub_y, lb_x:ub_x, :]
72 | return cropped
73 |
74 |
75 | def crop_image_to_aspect(image, tar=1.2):
76 | # load image
77 | image_bw = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
78 | # compute aspect ratio
79 | h, w = image_bw.shape[0], image_bw.shape[1]
80 | sar = h / w if h > w else w / h
81 | if sar < tar:
82 | return image
83 | else:
84 | k = 0.5 * (1.0 - (tar / sar))
85 | if h > w:
86 | lb = int(k * h)
87 | ub = h - lb
88 | cropped = image[lb:ub, :, :]
89 | else:
90 | lb = int(k * w)
91 | ub = w - lb
92 | cropped = image[:, lb:ub, :]
93 | return cropped
94 |
95 |
96 | def brighten_image_hsv(image, global_mean_v):
97 | image_hsv = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
98 | h, s, v = cv2.split(image_hsv)
99 | mean_v = int(np.mean(v))
100 | v = v - mean_v + global_mean_v
101 | image_hsv = cv2.merge((h, s, v))
102 | image_bright = cv2.cvtColor(image_hsv, cv2.COLOR_HSV2RGB)
103 | return image_bright
104 |
105 |
106 | def brighten_image_rgb(image, global_mean_rgb):
107 | r, g, b = cv2.split(image)
108 | m = np.array([np.mean(r), np.mean(g), np.mean(b)])
109 | brightened = image + global_mean_v - m
110 | return brightened
111 |
112 |
113 | ############################# main #############################
114 |
115 | DATA_DIR = "../data/files/sample"
116 | DATA_DIR2 = "../data/files/sample2"
117 |
118 | # random sample for printing
119 | sample_image_idxs = set(np.random.randint(0, high=1000, size=9).tolist())
120 | sample_images = []
121 |
122 | curr_idx = 0
123 | vs = []
124 | mean_rgbs = []
125 | for image_dir, image_name in get_next_image_loc(DATA_DIR):
126 | if curr_idx % 100 == 0:
127 | print("Reading {:d} images".format(curr_idx))
128 | image = cv2.imread(os.path.join(DATA_DIR, image_dir, image_name))
129 | if curr_idx in sample_image_idxs:
130 | sample_images.append(image)
131 | image_hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
132 | h, s, v = cv2.split(image_hsv)
133 | vs.append(np.mean(v))
134 | image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
135 | r, g, b = cv2.split(image_rgb)
136 | mean_rgbs.append(np.array([np.mean(r), np.mean(g), np.mean(b)]))
137 | curr_idx += 1
138 | print("Reading {:d} images, complete".format(curr_idx))
139 | global_mean_v = int(np.mean(np.array(vs)))
140 | global_mean_rgbs = np.mean(mean_rgbs, axis=0)
141 |
142 | # plot sample images at various steps
143 | sample_images_rgb = [cv2.cvtColor(simg, cv2.COLOR_BGR2RGB)
144 | for simg in sample_images]
145 | plot_images(sample_images_rgb)
146 | sample_cropped = [crop_image_to_aspect(simg) for simg in sample_images_rgb]
147 | sample_resized = [cv2.resize(simg, (int(1.2 * 224), 224))
148 | for simg in sample_cropped]
149 | plot_images(sample_resized)
150 | sample_brightened_hsv = [brighten_image_hsv(simg, global_mean_v)
151 | for simg in sample_resized]
152 | plot_images(sample_brightened_hsv)
153 | sample_brightened_rgb = [brighten_image_rgb(simg, global_mean_rgbs)
154 | for simg in sample_resized]
155 | plot_images(sample_brightened_rgb)
156 |
157 | # save all images to disk
158 | curr_idx = 0
159 | for image_dir, image_name in get_next_image_loc(DATA_DIR):
160 | if curr_idx % 100 == 0:
161 | print("Writing {:d} preprocessed images".format(curr_idx))
162 | image = cv2.imread(os.path.join(DATA_DIR, image_dir, image_name))
163 | cropped = crop_image_to_aspect(image)
164 | resized = cv2.resize(cropped, (int(1.2 * 224), 224))
165 | # brightened = brighten_image_hsv(resized, global_mean_v)
166 | brightened = brighten_image_rgb(resized, global_mean_rgbs)
167 | plt.imsave(os.path.join(DATA_DIR2, image_dir, image_name), brightened)
168 | curr_idx += 1
169 | print("Wrote {:d} images, complete".format(curr_idx))
170 |
--------------------------------------------------------------------------------
/src/sample-images.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | import matplotlib.pyplot as plt
4 | import numpy as np
5 | import os
6 |
7 | IMAGE_DIR = "../data/files/sample"
8 |
9 | ##################################
10 | # Presentation Goals slide
11 | ##################################
12 | def presentation_goals():
13 | files = os.listdir(IMAGE_DIR)
14 | fig, axes = plt.subplots(3, 3)
15 | axes = np.ravel(axes)
16 | for i in range(9):
17 | label = np.random.randint(4)
18 | files = os.listdir(os.path.join(IMAGE_DIR, str(label)))
19 | img_file = files[np.random.randint(len(files))]
20 | img = plt.imread(os.path.join(IMAGE_DIR, str(label), img_file))
21 | axes[i].imshow(img, interpolation="nearest")
22 | axes[i].set_xticks([])
23 | axes[i].set_yticks([])
24 | plt.xticks([])
25 | plt.yticks([])
26 | plt.tight_layout()
27 | plt.show()
28 |
29 | ##################################
30 | # What is DR slide
31 | ##################################
32 | def what_is_dr():
33 | plt.subplot(511)
34 | img = plt.imread(os.path.join(IMAGE_DIR, "0", "13363_left.jpeg"))
35 | plt.title("No DR")
36 | plt.imshow(img)
37 | plt.xticks([])
38 | plt.yticks([])
39 |
40 | plt.subplot(512)
41 | img = plt.imread(os.path.join(IMAGE_DIR, "1", "14664_left.jpeg"))
42 | plt.title("Mild DR")
43 | plt.imshow(img)
44 | plt.xticks([])
45 | plt.yticks([])
46 |
47 | plt.subplot(513)
48 | img = plt.imread(os.path.join(IMAGE_DIR, "2", "14323_left.jpeg"))
49 | plt.title("Moderate DR")
50 | plt.imshow(img)
51 | plt.xticks([])
52 | plt.yticks([])
53 |
54 | plt.subplot(514)
55 | img = plt.imread(os.path.join(IMAGE_DIR, "3", "12612_right.jpeg"))
56 | plt.title("Severe DR")
57 | plt.imshow(img)
58 | plt.xticks([])
59 | plt.yticks([])
60 |
61 | plt.subplot(515)
62 | plt.title("Proliferative DR")
63 | img = plt.imread(os.path.join(IMAGE_DIR, "4", "15376_left.jpeg"))
64 | plt.imshow(img)
65 | plt.xticks([])
66 | plt.yticks([])
67 |
68 | plt.tight_layout()
69 | plt.show()
70 |
71 | ######################## main ########################
72 | presentation_goals()
73 | #what_is_dr()
--------------------------------------------------------------------------------
/src/tl-dl-aug-train.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from keras.callbacks import ModelCheckpoint
4 | from keras.layers import Dense, Dropout, Input
5 | from keras.models import Model, load_model
6 | from keras.utils import np_utils
7 | from sklearn.metrics import *
8 | from sklearn.cross_validation import StratifiedShuffleSplit
9 | import numpy as np
10 | import matplotlib.pyplot as plt
11 | import os
12 |
13 | DATA_DIR = "../data"
14 | NUM_EPOCHS = 75
15 | BATCH_SIZE = 32
16 |
17 | # data
18 | print("Loading data...")
19 | Xtrain = np.loadtxt(os.path.join(DATA_DIR, "images-500-train-X.txt"),
20 | delimiter=",")
21 | ytrain = np.loadtxt(os.path.join(DATA_DIR, "images-500-train-y.txt"),
22 | delimiter=",", dtype=np.int)
23 | Ytrain = np_utils.to_categorical(ytrain - 1, nb_classes=5)
24 | print("\ttrain:", Xtrain.shape, Ytrain.shape)
25 |
26 | Xtest = np.loadtxt(os.path.join(DATA_DIR, "images-500-test-X.txt"),
27 | delimiter=",")
28 | ytest = np.loadtxt(os.path.join(DATA_DIR, "images-500-test-y.txt"),
29 | delimiter=",", dtype=np.int)
30 | Ytest = np_utils.to_categorical(ytest - 1, nb_classes=5)
31 | print("\ttest:", Xtest.shape, Ytest.shape)
32 |
33 | np.random.seed(42)
34 |
35 | # model
36 | # input: (None, 25088)
37 | imgvecs = Input(shape=(Xtrain.shape[1],), dtype="float32")
38 | # hidden layer: (None, 256)
39 | fc1 = Dense(256, activation="relu")(imgvecs)
40 | fc1_drop = Dropout(0.5)(fc1)
41 | # output layer: (None, 5)
42 | labels = Dense(5, activation="softmax")(fc1_drop)
43 |
44 | ## model 2
45 | ## input: (None, 25088)
46 | #imgvecs = Input(shape=(Xtrain.shape[1],), dtype="float32")
47 | ## hidden layer: (None, 2048)
48 | #fc1 = Dense(4096, activation="relu")(imgvecs)
49 | #fc1_drop = Dropout(0.5)(fc1)
50 | ## hidden layer: (None, 256)
51 | #fc2 = Dense(256, activation="relu")(fc1_drop)
52 | #fc2_drop = Dropout(0.5)(fc2)
53 | ## output layer: (None, 5)
54 | #labels = Dense(5, activation="softmax")(fc2_drop)
55 |
56 |
57 | model = Model(input=[imgvecs], output=[labels])
58 |
59 | model.compile(optimizer="adadelta", loss="categorical_crossentropy",
60 | metrics=["accuracy"])
61 |
62 | best_model = os.path.join(DATA_DIR, "dl-model-aug-best.h5")
63 | checkpoint = ModelCheckpoint(filepath=best_model, verbose=1,
64 | save_best_only=True)
65 | history = model.fit([Xtrain], [Ytrain], nb_epoch=NUM_EPOCHS,
66 | batch_size=BATCH_SIZE, validation_split=0.1,
67 | callbacks=[checkpoint])
68 |
69 | # visualize training loss and accuracy
70 | plt.subplot(211)
71 | plt.title("Accuracy")
72 | plt.plot(history.history["acc"], color="r", label="Train")
73 | plt.plot(history.history["val_acc"], color="b", label="Validation")
74 | plt.legend(loc="best")
75 |
76 | plt.subplot(212)
77 | plt.title("Loss")
78 | plt.plot(history.history["loss"], color="r", label="Train")
79 | plt.plot(history.history["val_loss"], color="b", label="Validation")
80 | plt.legend(loc="best")
81 |
82 | plt.tight_layout()
83 | plt.show()
84 |
85 | # evaluate final model
86 | Ytest_ = model.predict(Xtest)
87 | ytest = np_utils.categorical_probas_to_classes(Ytest) + 1
88 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_) + 1
89 |
90 | print("Final model")
91 | print("Accuracy: {:.5f}".format(accuracy_score(ytest_, ytest)))
92 | print("Confusion Matrix:")
93 | print(confusion_matrix(ytest_, ytest))
94 | print("Classification Report:")
95 | print(classification_report(ytest_, ytest))
96 |
97 | model.save(os.path.join(DATA_DIR, "dl-model-aug-final.h5"))
98 |
99 | # load best model and evaluate
100 |
101 | model = load_model(os.path.join(DATA_DIR, "dl-model-aug-best.h5"))
102 | model.compile(optimizer="rmsprop", loss="categorical_crossentropy",
103 | metrics=["accuracy"])
104 | Ytest_ = model.predict(Xtest)
105 | ytest = np_utils.categorical_probas_to_classes(Ytest) + 1
106 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_) + 1
107 |
108 | print("Best model")
109 | print("Accuracy: {:.5f}".format(accuracy_score(ytest_, ytest)))
110 | print("Confusion Matrix:")
111 | print(confusion_matrix(ytest_, ytest))
112 | print("Classification Report:")
113 | print(classification_report(ytest_, ytest))
114 |
115 |
--------------------------------------------------------------------------------
/src/tl-dl1-train.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from keras.callbacks import ModelCheckpoint
4 | from keras.layers import Dense, Dropout, Input
5 | from keras.layers.normalization import BatchNormalization
6 | from keras.models import Model, load_model
7 | from keras.utils import np_utils
8 | import numpy as np
9 | import os
10 |
11 | import fttlutils
12 |
13 | DATA_DIR = "../data/files"
14 | MODEL_DIR = os.path.join(DATA_DIR, "models")
15 | NUM_EPOCHS = 50
16 | BATCH_SIZE = 32
17 |
18 | # data
19 | X = np.loadtxt(os.path.join(DATA_DIR, "images-X.txt"), delimiter=",")
20 | y = np.loadtxt(os.path.join(DATA_DIR, "images-y.txt"), delimiter=",",
21 | dtype=np.int)
22 | Y = np_utils.to_categorical(y, nb_classes=5)
23 |
24 | np.random.seed(42)
25 |
26 | Xtrain, Xtest, Ytrain, Ytest = fttlutils.train_test_split(
27 | X, Y, test_size=0.3, random_state=42)
28 | print(Xtrain.shape, Xtest.shape, Ytrain.shape, Ytest.shape)
29 |
30 | # model
31 | # input: (None, 25088)
32 | imgvecs = Input(shape=(Xtrain.shape[1],), dtype="float32")
33 | # hidden layer: (None, 256)
34 | fc1 = Dense(256,
35 | activation="relu",
36 | init="he_uniform",
37 | name="dl1fc1")(imgvecs)
38 | fc1 = BatchNormalization()(fc1)
39 | fc1 = Dropout(0.5)(fc1)
40 | # output layer: (None, 5)
41 | predictions = Dense(5, activation="softmax", name="dl1preds")(fc1)
42 |
43 | model = Model(input=[imgvecs], output=[predictions])
44 |
45 | model.compile(optimizer="adadelta", loss="categorical_crossentropy",
46 | metrics=["accuracy"])
47 |
48 | best_model = os.path.join(MODEL_DIR, "tl-dl1-model-best.h5")
49 | checkpoint = ModelCheckpoint(filepath=best_model, verbose=1,
50 | save_best_only=True)
51 | history = model.fit([Xtrain], [Ytrain], nb_epoch=NUM_EPOCHS,
52 | batch_size=BATCH_SIZE, validation_split=0.1,
53 | callbacks=[checkpoint])
54 | fttlutils.plot_loss(history)
55 |
56 | # evaluate final model
57 | Ytest_ = model.predict(Xtest)
58 | ytest = np_utils.categorical_probas_to_classes(Ytest)
59 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
60 | fttlutils.print_stats(ytest, ytest_, "Final Model (DL#1)")
61 | model.save(os.path.join(MODEL_DIR, "tl-dl1-model-final.h5"))
62 |
63 | # load best model and evaluate
64 | model = load_model(os.path.join(MODEL_DIR, "tl-dl1-model-best.h5"))
65 | model.compile(optimizer="adadelta", loss="categorical_crossentropy",
66 | metrics=["accuracy"])
67 | Ytest_ = model.predict(Xtest)
68 | ytest = np_utils.categorical_probas_to_classes(Ytest)
69 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
70 | fttlutils.print_stats(ytest, ytest_, "Best Model (DL#1)")
71 |
--------------------------------------------------------------------------------
/src/tl-dl2-train.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from keras.callbacks import ModelCheckpoint
4 | from keras.layers import Dense, Dropout, Input
5 | from keras.models import Model, load_model
6 | from keras.optimizers import Adadelta
7 | from keras.utils import np_utils
8 | import numpy as np
9 | import os
10 |
11 | import fttlutils
12 |
13 | DATA_DIR = "../data/files"
14 | MODEL_DIR = os.path.join(DATA_DIR, "models")
15 | NUM_EPOCHS = 50
16 | BATCH_SIZE = 64
17 |
18 | # data
19 | X = np.loadtxt(os.path.join(DATA_DIR, "images-X.txt"), delimiter=",")
20 | y = np.loadtxt(os.path.join(DATA_DIR, "images-y.txt"), delimiter=",",
21 | dtype=np.int)
22 | Y = np_utils.to_categorical(y, nb_classes=5)
23 |
24 | np.random.seed(42)
25 |
26 | Xtrain, Xtest, Ytrain, Ytest = fttlutils.train_test_split(
27 | X, Y, test_size=0.3, random_state=42)
28 | print(Xtrain.shape, Xtest.shape, Ytrain.shape, Ytest.shape)
29 |
30 | # model 2
31 | # input: (None, 25088)
32 | imgvecs = Input(shape=(Xtrain.shape[1],), dtype="float32")
33 | # hidden layer: (None, 2048)
34 | fc1 = Dense(256, activation="relu")(imgvecs)
35 | fc1 = Dropout(0.5)(fc1)
36 | ## hidden layer: (None, 256)
37 | fc2 = Dense(128, activation="relu")(fc1)
38 | fc2 = Dropout(0.5)(fc2)
39 | # output layer: (None, 5)
40 | predictions = Dense(5, activation="softmax")(fc2)
41 |
42 | model = Model(input=[imgvecs], output=[predictions])
43 |
44 | optimizer = Adadelta(lr=0.1)
45 | model.compile(optimizer=optimizer, loss="categorical_crossentropy",
46 | metrics=["accuracy"])
47 |
48 | best_model = os.path.join(MODEL_DIR, "tl-dl2-model-best.h5")
49 | checkpoint = ModelCheckpoint(filepath=best_model, verbose=1,
50 | save_best_only=True)
51 | history = model.fit([Xtrain], [Ytrain], nb_epoch=NUM_EPOCHS,
52 | batch_size=BATCH_SIZE, validation_split=0.1,
53 | callbacks=[checkpoint])
54 | fttlutils.plot_loss(history)
55 |
56 | # evaluate final model
57 | Ytest_ = model.predict(Xtest)
58 | ytest = np_utils.categorical_probas_to_classes(Ytest)
59 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
60 | fttlutils.print_stats(ytest, ytest_, "Final Model (DL#2)")
61 | model.save(os.path.join(MODEL_DIR, "tl-dl2-model-final.h5"))
62 |
63 | # load best model and evaluate
64 |
65 | model = load_model(os.path.join(MODEL_DIR, "tl-dl2-model-best.h5"))
66 | model.compile(optimizer=optimizer, loss="categorical_crossentropy",
67 | metrics=["accuracy"])
68 | Ytest_ = model.predict(Xtest)
69 | ytest = np_utils.categorical_probas_to_classes(Ytest)
70 | ytest_ = np_utils.categorical_probas_to_classes(Ytest_)
71 | fttlutils.print_stats(ytest, ytest_, "Best Model (DL#2)")
72 |
73 |
--------------------------------------------------------------------------------
/src/tl-lr-aug-train.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from sklearn.cross_validation import train_test_split
4 | from sklearn.linear_model import LogisticRegression
5 | from sklearn.metrics import *
6 | import cPickle as pickle
7 | import numpy as np
8 | import os
9 |
10 | DATA_DIR = "../data"
11 |
12 | # data
13 | print("Loading data...")
14 | Xtrain = np.loadtxt(os.path.join(DATA_DIR, "images-500-train-X.txt"),
15 | delimiter=",")
16 | ytrain = np.loadtxt(os.path.join(DATA_DIR, "images-500-train-y.txt"),
17 | delimiter=",", dtype=np.int)
18 | print("\ttrain:", Xtrain.shape, ytrain.shape)
19 |
20 | Xtest = np.loadtxt(os.path.join(DATA_DIR, "images-500-test-X.txt"),
21 | delimiter=",")
22 | ytest = np.loadtxt(os.path.join(DATA_DIR, "images-500-test-y.txt"),
23 | delimiter=",", dtype=np.int)
24 | print("\ttest:", Xtest.shape, ytest.shape)
25 |
26 | np.random.seed(42)
27 |
28 | # model
29 | clf = LogisticRegression()
30 | clf.fit(Xtrain, ytrain)
31 |
32 | ytest_ = clf.predict(Xtest)
33 |
34 | print("Accuracy: {:.3f}".format(accuracy_score(ytest, ytest_)))
35 | print("Confusion Matrix:")
36 | print(confusion_matrix(ytest, ytest_))
37 | print("Classification Report:")
38 | print(classification_report(ytest, ytest_))
39 |
40 | with open(os.path.join(DATA_DIR, "lr-model-aug.pkl"), "wb") as fmodel:
41 | pickle.dump(clf, fmodel)
42 |
--------------------------------------------------------------------------------
/src/tl-lr-train.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from sklearn.linear_model import LogisticRegression
4 | import cPickle as pickle
5 | import numpy as np
6 | import os
7 |
8 | import fttlutils
9 |
10 | ##################### main ######################
11 |
12 | DATA_DIR = "../data/files"
13 | MODEL_DIR = os.path.join(DATA_DIR, "models")
14 |
15 | # data
16 | X = np.loadtxt(os.path.join(DATA_DIR, "images-X.txt"), delimiter=",")
17 | y = np.loadtxt(os.path.join(DATA_DIR, "images-y.txt"), delimiter=",",
18 | dtype=np.int)
19 |
20 | Xtrain, Xtest, ytrain, ytest = fttlutils.train_test_split(
21 | X, y, test_size=0.3, random_state=42)
22 | print(Xtrain.shape, Xtest.shape, ytrain.shape, ytest.shape)
23 |
24 | # model
25 | clf = LogisticRegression()
26 | clf.fit(Xtrain, ytrain)
27 |
28 | ytest_ = clf.predict(Xtest)
29 | fttlutils.print_stats(ytest, ytest_, "LR Model")
30 | with open(os.path.join(MODEL_DIR, "lr-model.pkl"), "wb") as fmodel:
31 | pickle.dump(clf, fmodel)
32 |
--------------------------------------------------------------------------------
/src/vectorize-images.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | from __future__ import division, print_function
3 | from keras.applications.vgg16 import VGG16
4 | from keras.applications.vgg16 import preprocess_input
5 | from keras.models import Model
6 | from keras.optimizers import SGD
7 | from keras.preprocessing import image
8 | import numpy as np
9 | import os
10 |
11 | def get_next_image_loc(imgdir):
12 | for root, dirs, files in os.walk(imgdir):
13 | for name in files:
14 | path = os.path.join(root, name).split(os.path.sep)[::-1]
15 | yield (path[1], path[0])
16 |
17 |
18 | def vectorize_batch(image_locs, image_dir, image_width, model,
19 | fvec_x, fvec_y, fvec_f):
20 | Xs, ys, fs = [], [], []
21 | for subdir, filename in image_locs:
22 | # preprocess image for loading into CNN
23 | img = image.load_img(os.path.join(image_dir, subdir, filename),
24 | target_size=(image_width, image_width))
25 | img4d = image.img_to_array(img)
26 | img4d = np.expand_dims(img4d, axis=0)
27 | img4d = preprocess_input(img4d)
28 | Xs.append(img4d[0])
29 | ys.append(int(subdir))
30 | fs.append(filename)
31 | X = np.array(Xs)
32 | vecs = model.predict(X)
33 | # output shape is (10, 7, 7, 512)
34 | for i in range(len(Xs)):
35 | vec = vecs[i].flatten()
36 | vec_str = ",".join(["{:.5f}".format(x) for x in vec.tolist()])
37 | fvec_x.write("{:s}\n".format(vec_str))
38 | fvec_y.write("{:d}\n".format(ys[i]))
39 | fvec_f.write("{:s}\n".format(fs[i]))
40 | return len(Xs)
41 |
42 |
43 | ############################ main ############################
44 |
45 | DATA_DIR = "../data/files"
46 | IMAGE_DIR = os.path.join(DATA_DIR, "sample")
47 | BATCH_SIZE = 10
48 |
49 | IMAGE_WIDTH = 224
50 |
51 | VEC_FILE_X = os.path.join(DATA_DIR, "images-X.txt")
52 | VEC_FILE_Y = os.path.join(DATA_DIR, "images-y.txt")
53 | VEC_FILE_F = os.path.join(DATA_DIR, "images-f.txt")
54 |
55 | # load VGG-16 model
56 | vgg16_model = VGG16(weights="imagenet", include_top=True)
57 | model = Model(input=vgg16_model.input,
58 | output=vgg16_model.get_layer("block5_pool").output)
59 |
60 | sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
61 | model.compile(optimizer=sgd, loss="categorical_crossentropy")
62 |
63 | fvec_x = open(VEC_FILE_X, "wb")
64 | fvec_y = open(VEC_FILE_Y, "wb")
65 | fvec_f = open(VEC_FILE_F, "wb")
66 |
67 | batch = []
68 | nbr_written = 0
69 | for image_loc in get_next_image_loc(IMAGE_DIR):
70 | batch.append(image_loc)
71 | if len(batch) == 10:
72 | nbr_written += vectorize_batch(batch, IMAGE_DIR, IMAGE_WIDTH, model,
73 | fvec_x, fvec_y, fvec_f)
74 | print("Vectors generated for {:d} images...".format(nbr_written))
75 | batch = []
76 | if len(batch) > 0:
77 | nbr_written += vectorize_batch(batch, IMAGE_DIR, IMAGE_WIDTH, model,
78 | fvec_x, fvec_y, fvec_f)
79 | print("Vectors generated for {:d} images, COMPLETE".format(nbr_written))
80 |
81 | fvec_x.close()
82 | fvec_y.close()
83 | fvec_f.close()
84 |
--------------------------------------------------------------------------------