├── data └── .gitkeep ├── .gitignore ├── vgg_faces_keras ├── ak.png ├── Aamir_Khan.jpg └── README.md ├── image_style_transfer └── ak.png ├── vgg_segmentation_keras ├── drawio_diagrams │ ├── fcn8s_diagram_drawio.jpg │ ├── fcn16s_diagram_drawio.jpg │ ├── fcn16s_diagram_drawio.xml │ └── fcn8s_diagram_drawio.xml ├── README.md ├── fcn_keras2.py └── utils.py ├── dogsandcats_keras ├── README.md ├── gpu_check.py ├── vgg16.py ├── vgg16bn.py ├── utils.py ├── dogsandcats.ipynb ├── dogsandcats_v2.ipynb ├── dogsandcats_v3.ipynb └── dogsandcats_v4.ipynb ├── LICENSE.txt ├── README.md ├── autoencoder_keras ├── vae_theory_mardown_only.ipynb └── finance_autoencoder.ipynb ├── timeserie └── utils_modified.py └── insurance_scikit └── metrics.py /data/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | **/.ipynb_checkpoints 2 | 3 | /data 4 | 5 | *.pyc 6 | *.jpg 7 | *.png 8 | *.jpeg 9 | -------------------------------------------------------------------------------- /vgg_faces_keras/ak.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mzaradzki/neuralnets/HEAD/vgg_faces_keras/ak.png -------------------------------------------------------------------------------- /image_style_transfer/ak.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mzaradzki/neuralnets/HEAD/image_style_transfer/ak.png -------------------------------------------------------------------------------- /vgg_faces_keras/Aamir_Khan.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mzaradzki/neuralnets/HEAD/vgg_faces_keras/Aamir_Khan.jpg -------------------------------------------------------------------------------- /vgg_segmentation_keras/drawio_diagrams/fcn8s_diagram_drawio.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mzaradzki/neuralnets/HEAD/vgg_segmentation_keras/drawio_diagrams/fcn8s_diagram_drawio.jpg -------------------------------------------------------------------------------- /vgg_segmentation_keras/drawio_diagrams/fcn16s_diagram_drawio.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mzaradzki/neuralnets/HEAD/vgg_segmentation_keras/drawio_diagrams/fcn16s_diagram_drawio.jpg -------------------------------------------------------------------------------- /dogsandcats_keras/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Dogs and Cats : Kaggle image recognition 3 | 4 | 5 | https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition 6 | 7 | 8 | This folder follows FastAI (Jeremy Howard) guidelines to build a Deep Convolution Network that classifies cats and dogs. 9 | 10 | The script uses Keras with Theano backend : see installation steps on main README of the repo. 11 | 12 | The script was run on AWS EC2 using P2 instance. 13 | 14 | Idea of FastAI MOOC : dont train a full DCN but instead load the configuration of the VGG one, then retrain the last layer. 15 | 16 | 17 | 18 | Images need to be organized as follow : 19 | 20 | data/dogsandcats/train/dog/ (.jpg here) 21 | 22 | data/dogsandcats/train/cat/ (.jpg here) 23 | 24 | data/dogsandcats/valid/dog/ (.jpg here) 25 | 26 | data/dogsandcats/valid/cat/ (.jpg here) 27 | 28 | data/dogsandcats/test/test/ (.jpg here) 29 | 30 | For convenience a sample copy of these images is mirrored in a dogsandcats_small folder. 31 | 32 | -------------------------------------------------------------------------------- /vgg_faces_keras/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | Implement the **VGG-Face** model using **Keras** a sequential model. 4 | 5 | My post on Medium explaining this : 6 | * https://medium.com/@m.zaradzki/face-recognition-with-keras-and-opencv-2baf2a83b799# 7 | 8 | References from the authors of the model: 9 | * http://www.robots.ox.ac.uk/~vgg/software/vgg_face/ 10 | * http://www.robots.ox.ac.uk/~vgg/publications/2015/Parkhi15/parkhi15.pdf 11 | 12 | The VGG page has the model weights in .mat format (MatConvNet) so we write a script to transpose them for Keras. 13 | 14 | As the VGG Face model works best on image centered on faces we use **OpenCV** to locate the faces in an image to crop the most relevant section. 15 | 16 | We use Keras Functional API to derive a face-feature-vector model based on the logit layer (second-last). This model can be used to check if two images are represent the same faces even if the faces are not part of the training sample. This is done using hte cosine similarity of the face feature vectors. -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 m.zaradzki 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /dogsandcats_keras/gpu_check.py: -------------------------------------------------------------------------------- 1 | from theano import function, config, shared, sandbox 2 | import theano.tensor as T 3 | import numpy 4 | import time 5 | 6 | 7 | # See this page : http://deeplearning.net/software/theano/tutorial/using_gpu.html 8 | # From console : 9 | # $ THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32 python gpu_check.py 10 | # $ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python gpu_check.py 11 | 12 | 13 | def test(): 14 | vlen = 10 * 30 * 768 # 10 x #cores x # threads per core 15 | iters = 1000 16 | 17 | rng = numpy.random.RandomState(22) 18 | x = shared(numpy.asarray(rng.rand(vlen), config.floatX)) 19 | f = function([], T.exp(x)) 20 | print(f.maker.fgraph.toposort()) 21 | t0 = time.time() 22 | for i in range(iters): 23 | r = f() 24 | t1 = time.time() 25 | print("Looping %d times took %f seconds" % (iters, t1 - t0)) 26 | print("Result is %s" % (r,)) 27 | if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]): 28 | print('Used the cpu') 29 | return False 30 | else: 31 | print('Used the gpu') 32 | return True 33 | 34 | 35 | if __name__ == "__main__": 36 | t = test() -------------------------------------------------------------------------------- /vgg_segmentation_keras/drawio_diagrams/fcn16s_diagram_drawio.xml: -------------------------------------------------------------------------------- 1 | 1ZpRb5swEIB/DY+bAsYOeVyzrttDtUpRte3RAQdQCWZgknS/fibYAWy3aVJCSiNV9tnY5vNxdz6wwHy9u8txFt3TgCSWMwl2FvhqOY4HPf6/EjzXAgjsWhDmcVCLWoJF/I8I4URIyzggRacjozRhcdYV+jRNic86MpzndNvttqJJd9YMh0QTLHyc6NJfccAicVsOauTfSRxGcmYbzeqWJfafwpyWqZjPcsBq/1c3r7EcS9xoEeGAblsicGuBeU4pq0vr3ZwkFVqJrb7u2wuth3XnJGVvucCpL9jgpBS3/iPNSiYWx54lkG0UM7LIsF/Vt3zPLXATsXXCazYvruIkmdOE5vveIMDEW/lcXrCcPpFWC/I9slzxFn2hYu0bkjOya4nEwu8IXROWP/MuotUVDIWO2ZLpttkxKERRa7PkZVjoSHgYuOHECwKVGRvQsM1v7HdC40ri+EZoAVoiiPqBNrseNNcAzRkFNBsp1LzhqEEDNTAKag7oUjvgGIAaMlBzR0ENTLrUDvUBqHkGanAc1KYKNTQctZlODY0Cmqt4UNfgDMCFoMlIr01tOg5q0ytS033o4vH+VGw9QIDe9Z43QyDxoCHII7pelnyWmxNDV0i8wDXpkOcsAeoroFBcow11fO6l8OkRxdjwOYqPPFiTIfDpocXo8CkW7FAfAt909PiA4jYBGBCfHqKNDZ9rK/hmA+LTY7XH3vARmwOcmvDN0BTgnvBBeD18MovXwqfRI2nwpcrA8Zqf4KKI/S40sovZ71b5Dy9PPsOqlvLlVE0TWWna1EjP84k50lt60IWTQ4vM3jmvwS9omfuka58YzkMiu4mAgwSdrKG+RUeCHynLSYJZvOnmGk37ImZ4oDFf8YvnahcqW1vfj7jKaSX/lIHUQNZTxqkZaOPw7cXPrW5Z1aF4eb2uGivCyavL0vp7nf68UK+gUdnDFrxNi/UA6HG3/43MFCA1wWJIFdgmRbRVjTnLGOiB0M+SjSV5jBQlGzR7rMdAp5tRo7GsWgJcRCSwjhz12vZNPhAfxL4h6aXVs+Wp9s2G5xm4c2yKHped7RmFN5S+sbPZjaM8bbvbLk5ud0cF0IdSAVs54EHw8VXAdTQVmNN0Q5OSxTTlDTcJ9Z/eaRwHSnEdT9xPL2QbXf3V2gOlSZyGvTnngY45CkMPvonhtA+GeoLsMSvwOusV4zAxjnrWNuQZL4ZRjxNPtun6KeTU84/JCBw//Rit+VHPIE/mbc9QO+NrOQK3Jz8wO88NnHrUgcqLYvmGpa+ji6vH3JaDElbFAfGGF0O23/xatKL7tTXqiv6WVDZ8KvZf3XzhHWwn2zWNcpQ5ZiSkeUwKOR5fXT1kdxou7kz+7kfkcur+QdT47BO7+t5AG+jscIZXm6+O6u7Nl13g9j8= -------------------------------------------------------------------------------- /vgg_segmentation_keras/drawio_diagrams/fcn8s_diagram_drawio.xml: -------------------------------------------------------------------------------- 1 | 1Zpbj6s2EIB/TV6r4AuQx256etqHo660WrV9dMAJaAmOjLPJ9tfXWczFHvbkRkg4Kx3ZYwPm88x4ZsIEz9f775Jtkh8i5tkETeP9BP82QSikof7/IPgoBTSgpWAl07gUeY3gJf2PG+HUSLdpzAtrohIiU+nGFkYiz3mkLBmTUuzsaUuR2U/dsBUHgpeIZVD6dxqrxLwW8hv5HzxdJdWTPX9WjixY9LaSYpub500QXn7+K4fXrLqXedEiYbHYtUT42wTPpRCqbK33c54d0FbYyut+/2K0XrfkuTrlAlRe8M6yrXn1P/PNVpnFqY8KyC5JFX/ZsOjQ3+k9n+CnRK0z3fN0c5lm2VxkQn7OxjHj4TLS8kJJ8cZbI34U8sVSj8CFmrW/c6n4viUyC//OxZor+aGnmFFiGBod8yqmu2bHqBElrc2qLmNGR1b1jRtOumFQdWPDANv8ybsSmlYSFHVCi/2FT/1+oM3uB410QEOjgOb5DrVwOGq0gxoeBTWEbWo1jgGo+R3UyCio4alNre4PQC3soEbHQS1wqPnDUZtBav4ooBHnBCUdhwG+EbQq0mtTC8ZBLbgjNXiGvrz+OBdbDxBoeD976wgkngECmYj1Yquf8nRm6Ep5GJMuHQrRAvt9BRTO0ehRiI/cCh+MKMaGDzlnZO1NhsAHQ4vR4XM8WN0fAl8wenzYOTYxHhAfDNHGho94Dr7ZgPhgrPbaGz7uaYBBF76ZH2DWEz5K74cPewAfoMfz+NdDBU73oowVRRrZ0Pg+Vf+02v/q9vQXeujlejmHoWnVacZKpFUtDnXEfmHEu2O/RUgJnf4MfiG2MuK2f1JMrng1zQQcPLaqhnCLjgQ/lUzyjKn03a41du2LecKzSPWKv8yrCXW2tnwfcxVqFf+cG7mBbOjcp2QA7qO3l320pm0OE4qv10vcWJFOf7osMD+05utGuYJGZestOE2L4Qn+11aNpeoZYGrRGbTsCU+f8+2/08q/9Aym/cxlqpfLpZHFrEh4PDmSz7QNm/jQsEn4UIYdIFvxa0M/17A9epllX2BMBAGVmIv8XWRblYpcDzxlInq70rIGSuyPlyuDGxkWgT8oPAuRpfkKkHvw4M5hGNKTGAZ9MIRlgddNwdabXjEOE+S5GYbvDYcRlgfGFiL7jh8dMkEjMLi4T4XOd/OEAX9HIDDJBwTGnid0hRNG9iDhBA7tSBG5UcCp4QRF9o2wd1o4cX6i4Cx4dixRcNZlf+xwdaJAYLj7utd/I3OHgVNw8WbwUPamHXpYC6/yBbDkcr4vqC2+cQBu1mCGWnnDBTlCCI0aP5ZRu5sJbPHk5J8c8Q795QjVq1+jAtCxn5lIfqrHpYeBs/lHFanalLYilbXHe+mNE1HSS1NL5/uhW9WMqPPFTfVTdV+uncJK5pwpvhIy5UUPynk7RXsQBSKV+zi7NuFWL90bXex3dLf5cLKc3nycir/9Dw== -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # neuralnets 2 | 3 | Experiments with **Theano**, **TensorFlow** and **Keras** 4 | 5 | 6 | ## Main sub-projects 7 | 8 | * **autoencoder_keras** : implements auto-encoder (de-noising, variational, mixture) 9 | 10 | * **dogsandcats_keras** : implements several models and training procedure for Kaggle "dogs and cats" competition 11 | 12 | * **vgg_faces_keras** : implements face identification using VGG model 13 | 14 | * **vgg_segmentation_keras** : implements pixel wise classification of images 15 | 16 | 17 | ## Setup instruction on AWS 18 | 19 | These scripts are run on AWS EC2 GPU "g2.2x" instance based on AMI (Ireland) : 20 | 21 | cs231n_caffe_torch7_keras_lasagne_v2 **ami-e8a1fe9b** 22 | 23 | At EC2 configuration time, to setup Jupyter web I follow this tutorial : 24 | 25 | http://efavdb.com/deep-learning-with-jupyter-on-aws/ 26 | 27 | To re-use the same folder across multiple EC2 launches I use AWS EFS : 28 | ``` 29 | ($ sudo apt-get update ?) 30 | $ sudo apt-get -y install nfs-common 31 | ($ sudo reboot ?) 32 | $ cd caffe 33 | $ mkdir neuralnets 34 | $ cd .. 35 | $ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).YOUR_EFS_HERE.efs.YOUR_ZONE_HERE.amazonaws.com:/ caffe/neuralnets 36 | ($ clone Git repo in neuralnets directory ?) 37 | ``` 38 | Note : the security group of the EFS folder and EC2 instace needs to be configured correctly : 39 | 40 | http://docs.aws.amazon.com/efs/latest/ug/accessing-fs-create-security-groups.html 41 | 42 | 43 | The EC2 AMI comes with Theano but TensorFlow needs to be installed : 44 | ``` 45 | ($ easy_install --upgrade pip ?) 46 | $ pip install tensorflow 47 | ``` 48 | 49 | WARNING : With this setup Theano makes use of the GPU but TensorFlow only runs on the CPU 50 | 51 | To run Theano script with GPU : 52 | ``` 53 | $ cd caffe/neuralnets/nb_theano 54 | $ THEANO_FLAGS='floatX=float32,device=gpu' python dA.py 55 | ``` 56 | 57 | To unmount the EFS folder before closing down the EC2 instance : 58 | ``` 59 | $ sudo umount caffe/neuralnets 60 | ``` 61 | -------------------------------------------------------------------------------- /vgg_segmentation_keras/README.md: -------------------------------------------------------------------------------- 1 | 2 | Implements and tests the **FCN-16s** and **FCN-8s** models for image segmentation using **Keras** deep-learning library. 3 | 4 | My post on Medium explaining the model architecture and its Keras implementation : 5 | * https://medium.com/@m.zaradzki/image-segmentation-with-neural-net-d5094d571b1e# 6 | 7 | 8 | References from the authors of the model: 9 | * title: **Fully Convolutional Networks for Semantic Segmentation** 10 | * authors: **Jonathan Long, Evan Shelhamer, Trevor Darrell** 11 | * link: http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf 12 | 13 | 14 | ### Extracts from the article relating to the model architecture 15 | 16 | **remark**: The model is derived from VGG16. 17 | 18 | **remark** : Deconvolution and conv-transpose are synonyms, they perform up-sampling. 19 | 20 | #### 4.1. From classifier to dense FCN 21 | 22 | We decapitate each net by discarding the final classifier layer [**code comment** : *this is why fc8 is not included*], and convert all fully connected layers to convolutions. 23 | 24 | We append a 1x1 convolution with channel dimension 21 [**code comment** : *layer named score_fr*] to predict scores for each of the PASCAL classes (including background) at each of the coarse output locations, followed by a deconvolution layer to bilinearly upsample the coarse outputs to pixel-dense outputs as described in Section 3.3. 25 | 26 | 27 | #### 4.2. Combining what and where 28 | We define a new fully convolutional net (FCN) for segmentation that combines layers of the feature hierarchy and 29 | refines the spatial precision of the output. 30 | While fully convolutionalized classifiers can be fine-tuned to segmentation as shown in 4.1, and even score highly on the standard metric, their output is dissatisfyingly coarse. 31 | The 32 pixel stride at the final prediction layer limits the scale of detail in the upsampled output. 32 | 33 | We address this by adding skips that combine the final prediction layer with lower layers with finer strides. 34 | This turns a line topology into a DAG [**code comment** : *this is why some latter stage layers have 2 inputs*], with edges that skip ahead from lower layers to higher ones. 35 | As they see fewer pixels, the finer scale predictions should need fewer layers, so it makes sense to make them from shallower net outputs. 36 | Combining fine layers and coarse layers lets the model make local predictions that respect global structure. 37 | 38 | We first divide the output stride in half by predicting from a 16 pixel stride layer. 39 | We add a 1x1 convolution layer on top of pool4 [**code comment** : *the score_pool4_filter layer*] to produce additional class predictions. 40 | We fuse this output with the predictions computed on top of conv7 (convolutionalized fc7) at stride 32 by adding a 2x upsampling layer and summing [**code comment** : *layer named sum*] both predictions [**code warning** : *requires first layer crop to insure the same size*]. 41 | 42 | Finally, the stride 16 predictions are upsampled back to the image [**code comment** : *layer named upsample_new*]. 43 | 44 | We call this net FCN-16s. -------------------------------------------------------------------------------- /dogsandcats_keras/vgg16.py: -------------------------------------------------------------------------------- 1 | from __future__ import division, print_function 2 | 3 | import os, json 4 | from glob import glob 5 | import numpy as np 6 | from scipy import misc, ndimage 7 | from scipy.ndimage.interpolation import zoom 8 | 9 | from keras.utils.data_utils import get_file 10 | from keras import backend as K 11 | from keras.layers.normalization import BatchNormalization 12 | from keras.utils.data_utils import get_file 13 | from keras.models import Sequential 14 | from keras.layers.core import Flatten, Dense, Dropout, Lambda 15 | from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D 16 | from keras.layers.pooling import GlobalAveragePooling2D 17 | from keras.optimizers import SGD, RMSprop, Adam 18 | from keras.preprocessing import image 19 | 20 | 21 | vgg_mean = np.array([123.68, 116.779, 103.939], dtype=np.float32).reshape((3,1,1)) 22 | def vgg_preprocess(x): 23 | x = x - vgg_mean 24 | return x[:, ::-1] # reverse axis rgb->bgr 25 | 26 | 27 | class Vgg16(): 28 | """The VGG 16 Imagenet model""" 29 | 30 | 31 | def __init__(self): 32 | self.FILE_PATH = 'http://www.platform.ai/models/' 33 | self.create() 34 | self.get_classes() 35 | 36 | 37 | def get_classes(self): 38 | fname = 'imagenet_class_index.json' 39 | fpath = get_file(fname, self.FILE_PATH+fname, cache_subdir='models') 40 | with open(fpath) as f: 41 | class_dict = json.load(f) 42 | self.classes = [class_dict[str(i)][1] for i in range(len(class_dict))] 43 | 44 | def predict(self, imgs, details=False): 45 | all_preds = self.model.predict(imgs) 46 | idxs = np.argmax(all_preds, axis=1) 47 | preds = [all_preds[i, idxs[i]] for i in range(len(idxs))] 48 | classes = [self.classes[idx] for idx in idxs] 49 | return np.array(preds), idxs, classes 50 | 51 | 52 | def ConvBlock(self, layers, filters): 53 | model = self.model 54 | for i in range(layers): 55 | model.add(ZeroPadding2D((1, 1))) 56 | model.add(Convolution2D(filters, 3, 3, activation='relu')) 57 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 58 | 59 | 60 | def FCBlock(self): 61 | model = self.model 62 | model.add(Dense(4096, activation='relu')) 63 | model.add(Dropout(0.5)) 64 | 65 | 66 | def create(self): 67 | model = self.model = Sequential() 68 | model.add(Lambda(vgg_preprocess, input_shape=(3,224,224))) 69 | 70 | self.ConvBlock(2, 64) 71 | self.ConvBlock(2, 128) 72 | self.ConvBlock(3, 256) 73 | self.ConvBlock(3, 512) 74 | self.ConvBlock(3, 512) 75 | 76 | model.add(Flatten()) 77 | self.FCBlock() 78 | self.FCBlock() 79 | model.add(Dense(1000, activation='softmax')) 80 | 81 | fname = 'vgg16.h5' 82 | model.load_weights(get_file(fname, self.FILE_PATH+fname, cache_subdir='models')) 83 | 84 | 85 | def get_batches(self, path, gen=image.ImageDataGenerator(), shuffle=True, batch_size=8, class_mode='categorical'): 86 | return gen.flow_from_directory(path, target_size=(224,224), 87 | class_mode=class_mode, shuffle=shuffle, batch_size=batch_size) 88 | 89 | 90 | def ft(self, num): 91 | model = self.model 92 | model.pop() 93 | for layer in model.layers: layer.trainable=False 94 | model.add(Dense(num, activation='softmax')) 95 | self.compile() 96 | 97 | def finetune(self, batches): 98 | model = self.model 99 | model.pop() 100 | for layer in model.layers: layer.trainable=False 101 | model.add(Dense(batches.nb_class, activation='softmax')) 102 | self.compile() 103 | 104 | 105 | def compile(self, lr=0.001): 106 | self.model.compile(optimizer=Adam(lr=lr), 107 | loss='categorical_crossentropy', metrics=['accuracy']) 108 | 109 | 110 | def fit_data(self, trn, labels, val, val_labels, nb_epoch=1, batch_size=64): 111 | self.model.fit(trn, labels, nb_epoch=nb_epoch, 112 | validation_data=(val, val_labels), batch_size=batch_size) 113 | 114 | 115 | def fit(self, batches, val_batches, nb_epoch=1): 116 | self.model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=nb_epoch, 117 | validation_data=val_batches, nb_val_samples=val_batches.nb_sample) 118 | 119 | 120 | def test(self, path, batch_size=8): 121 | test_batches = self.get_batches(path, shuffle=False, batch_size=batch_size, class_mode=None) 122 | return test_batches, self.model.predict_generator(test_batches, test_batches.nb_sample) 123 | 124 | -------------------------------------------------------------------------------- /dogsandcats_keras/vgg16bn.py: -------------------------------------------------------------------------------- 1 | from __future__ import division, print_function 2 | 3 | import os, json 4 | from glob import glob 5 | import numpy as np 6 | from scipy import misc, ndimage 7 | from scipy.ndimage.interpolation import zoom 8 | 9 | from keras.utils.data_utils import get_file 10 | from keras import backend as K 11 | from keras.layers.normalization import BatchNormalization 12 | from keras.utils.data_utils import get_file 13 | from keras.models import Sequential 14 | from keras.layers.core import Flatten, Dense, Dropout, Lambda 15 | from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D 16 | from keras.layers.pooling import GlobalAveragePooling2D 17 | from keras.optimizers import SGD, RMSprop, Adam 18 | from keras.preprocessing import image 19 | 20 | 21 | vgg_mean = np.array([123.68, 116.779, 103.939], dtype=np.float32).reshape((3,1,1)) 22 | def vgg_preprocess(x): 23 | x = x - vgg_mean 24 | return x[:, ::-1] # reverse axis rgb->bgr 25 | 26 | 27 | class Vgg16BN(): 28 | """The VGG 16 Imagenet model with Batch Normalization for the Dense Layers""" 29 | 30 | 31 | def __init__(self, size=(224,224), include_top=True): 32 | self.FILE_PATH = 'http://www.platform.ai/models/' 33 | self.create(size, include_top) 34 | self.get_classes() 35 | 36 | 37 | def get_classes(self): 38 | fname = 'imagenet_class_index.json' 39 | fpath = get_file(fname, self.FILE_PATH+fname, cache_subdir='models') 40 | with open(fpath) as f: 41 | class_dict = json.load(f) 42 | self.classes = [class_dict[str(i)][1] for i in range(len(class_dict))] 43 | 44 | def predict(self, imgs, details=False): 45 | all_preds = self.model.predict(imgs) 46 | idxs = np.argmax(all_preds, axis=1) 47 | preds = [all_preds[i, idxs[i]] for i in range(len(idxs))] 48 | classes = [self.classes[idx] for idx in idxs] 49 | return np.array(preds), idxs, classes 50 | 51 | 52 | def ConvBlock(self, layers, filters): 53 | model = self.model 54 | for i in range(layers): 55 | model.add(ZeroPadding2D((1, 1))) 56 | model.add(Convolution2D(filters, 3, 3, activation='relu')) 57 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 58 | 59 | 60 | def FCBlock(self): 61 | model = self.model 62 | model.add(Dense(4096, activation='relu')) 63 | model.add(BatchNormalization()) 64 | model.add(Dropout(0.5)) 65 | 66 | 67 | def create(self, size, include_top): 68 | if size != (224,224): 69 | include_top=False 70 | 71 | model = self.model = Sequential() 72 | model.add(Lambda(vgg_preprocess, input_shape=(3,)+size)) 73 | 74 | self.ConvBlock(2, 64) 75 | self.ConvBlock(2, 128) 76 | self.ConvBlock(3, 256) 77 | self.ConvBlock(3, 512) 78 | self.ConvBlock(3, 512) 79 | 80 | if not include_top: 81 | fname = 'vgg16_bn_conv.h5' 82 | model.load_weights(get_file(fname, self.FILE_PATH+fname, cache_subdir='models')) 83 | return 84 | 85 | model.add(Flatten()) 86 | self.FCBlock() 87 | self.FCBlock() 88 | model.add(Dense(1000, activation='softmax')) 89 | 90 | fname = 'vgg16_bn.h5' 91 | model.load_weights(get_file(fname, self.FILE_PATH+fname, cache_subdir='models')) 92 | 93 | 94 | def get_batches(self, path, gen=image.ImageDataGenerator(), shuffle=True, batch_size=8, class_mode='categorical'): 95 | return gen.flow_from_directory(path, target_size=(224,224), 96 | class_mode=class_mode, shuffle=shuffle, batch_size=batch_size) 97 | 98 | 99 | def ft(self, num): 100 | model = self.model 101 | model.pop() 102 | for layer in model.layers: layer.trainable=False 103 | model.add(Dense(num, activation='softmax')) 104 | self.compile() 105 | 106 | def finetune(self, batches): 107 | model = self.model 108 | model.pop() 109 | for layer in model.layers: layer.trainable=False 110 | model.add(Dense(batches.nb_class, activation='softmax')) 111 | self.compile() 112 | 113 | 114 | def compile(self, lr=0.001): 115 | self.model.compile(optimizer=Adam(lr=lr), 116 | loss='categorical_crossentropy', metrics=['accuracy']) 117 | 118 | 119 | def fit_data(self, trn, labels, val, val_labels, nb_epoch=1, batch_size=64): 120 | self.model.fit(trn, labels, nb_epoch=nb_epoch, 121 | validation_data=(val, val_labels), batch_size=batch_size) 122 | 123 | 124 | def fit(self, batches, val_batches, nb_epoch=1): 125 | self.model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=nb_epoch, 126 | validation_data=val_batches, nb_val_samples=val_batches.nb_sample) 127 | 128 | 129 | def test(self, path, batch_size=8): 130 | test_batches = self.get_batches(path, shuffle=False, batch_size=batch_size, class_mode=None) 131 | return test_batches, self.model.predict_generator(test_batches, test_batches.nb_sample) 132 | 133 | -------------------------------------------------------------------------------- /autoencoder_keras/vae_theory_mardown_only.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# From Oliver Durr" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Variational Autoencoder (VAE)\n", 15 | "A tutorial with code for a VAE as described in [Kingma and Welling, 2013](http://arxiv.org/abs/1312.6114). A talk with more details was given at the [DataLab Brown Bag Seminar](https://home.zhaw.ch/~dueo/bbs/files/vae.pdf).\n", 16 | "\n", 17 | "Much of the code was taken, from https://jmetzen.github.io/2015-11-27/vae.html. However, I tried to focus more on the mathematical understanding, not so much on design of the algorithm.\n", 18 | "\n", 19 | "### Some theoretical considerations \n", 20 | "\n", 21 | "#### Outline\n", 22 | "Situation: $x$ is from a high-dimensional space and $z$ is from a low-dimensional (latent) space, from which we like to reconstruct $p(x)$. \n", 23 | "\n", 24 | "We consider a parameterized model $p_\\theta(x|z)$ (with parameter $\\theta$), to construct x for a given value of $z$. We build this model: \n", 25 | "\n", 26 | "* $p_\\theta(x | z)$ with a neural network determening the parameters $\\mu, \\Sigma$ of a Gaussian (or as done here with a Bernoulli-Density). \n", 27 | "\n", 28 | "#### Inverting $p_\\theta(x | z)$\n", 29 | "\n", 30 | "The inversion is not possible, we therefore approximate $p(z|x)$ by $q_\\phi (z|x)$ again a combination of a NN determening the parameters of a Gaussian\n", 31 | "\n", 32 | "* $q_\\phi(z | x)$ with a neural network + Gaussian \n", 33 | "\n", 34 | "#### Training\n", 35 | "\n", 36 | "We train the network treating it as an autoencoder. \n", 37 | "\n", 38 | "#### Lower bound of the Log-Likelihood\n", 39 | "The likelihood cannot be determined analytically. Therefore, in a first step we derive a lower (variational) bound $L^{v}$ of the log likelihood, for a given image. Technically we assume a discrete latent space. For a continous case simply replace the sum by the appropriate integral over the respective densities. We replace the inaccessible conditional propability $p(z|x)$ with an approximation $q(z|x)$ for which we later use a neural network topped by a Gaussian.\n", 40 | "\n", 41 | "\\begin{align}\n", 42 | "L & = \\log\\left(p(x)\\right) &\\\\\n", 43 | " & = \\sum_z q(z|x) \\; \\log\\left(p(x)\\right) &\\text{multiplied with 1 }\\\\\n", 44 | " & = \\sum_z q(z|x) \\; \\log\\left(\\frac{p(z,x)}{p(z|x)}\\right) &\\\\\n", 45 | " & = \\sum_z q(z|x) \\; \\log\\left(\\frac{p(z,x)}{q(z|x)} \\frac{q(z|x)}{p(z|x)}\\right) &\\\\\n", 46 | " & = \\sum_z q(z|x) \\; \\log\\left(\\frac{p(z,x)}{q(z|x)}\\right) + \\sum_z q(z|x) \\; \\log\\left(\\frac{q(z|x)}{p(z|x)}\\right) &\\\\\n", 47 | " & = L^{\\tt{v}} + D_{\\tt{KL}} \\left( q(z|x) || p(z|x) \\right) &\\\\\n", 48 | " & \\ge L^{\\tt{v}} \\\\\n", 49 | "\\end{align}\n", 50 | "\n", 51 | "The KL-Divergence $D_{\\tt{KL}}$ is always positive, and the smaller the better $q(z|x)$ approximates $p(z|x)$\n" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "### Rewritting $L^\\tt{v}$\n", 59 | "We split $L^\\tt{v}$ into two parts.\n", 60 | "\n", 61 | "\\begin{align}\n", 62 | "L^{\\tt{v}} & = \\sum_z q(z|x) \\; \\log\\left(\\frac{p(z,x)}{q(z|x)}\\right) & \\text{with} \\;\\;p(z,x) = p(x|z) \\,p(z)\\\\\n", 63 | " & = \\sum_z q(z|x) \\; \\log\\left(\\frac{p(x|z) p(z)}{q(z|x)}\\right) &\\\\\n", 64 | " & = \\sum_z q(z|x) \\; \\log\\left(\\frac{p(z)}{q(z|x)}\\right) + \\sum_z q(z|x) \\; \\log\\left(p(x|z)\\right) &\\\\\n", 65 | " & = -D_{\\tt{KL}} \\left( q(z|x) || p(z) \\right) + \\mathbb{E}_{q(z|x)}\\left( \\log\\left(p(x|z)\\right)\\right) &\\text{putting in } x^{(i)} \\text{ for } x\\\\\n", 66 | " & = -D_{\\tt{KL}} \\left( q(z|x^{(i)}) || p(z) \\right) + \\mathbb{E}_{q(z|x^{(i)})}\\left( \\log\\left(p(x^{(i)}|z)\\right)\\right) &\\\\\n", 67 | "\\end{align}\n", 68 | "\n", 69 | "Approximating $\\mathbb{E}_{q(z|x^{(i)})}$ with sampling form the distribution $q(z|x^{(i)})$\n", 70 | "\n", 71 | "#### Sampling \n", 72 | "With $z^{(i,l)}$ $l = 1,2,\\ldots L$ sampled from $z^{(i,l)} \\thicksim q(z|x^{(i)})$\n", 73 | "\\begin{align}\n", 74 | "L^{\\tt{v}} & = -D_{\\tt{KL}} \\left( q(z|x^{(i)}) || p(z) \\right) \n", 75 | "+ \\mathbb{E}_{q(z|x^{(i)})}\\left( \\log\\left(p(x^{(i)}|z)\\right)\\right) &\\\\\n", 76 | "L^{\\tt{v}} & \\approx -D_{\\tt{KL}} \\left( q(z|x^{(i)}) || p(z) \\right) \n", 77 | "+ \\frac{1}{L} \\sum_{i=1}^L \\log\\left(p(x^{(i)}|z^{(i,l)})\\right) &\\\\\n", 78 | "\\end{align}\n", 79 | "\n", 80 | "#### Calculation of $D_{\\tt{KL}} \\left( q(z|x^{(i)}) || p(z) \\right)$\n", 81 | "TODO" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": { 88 | "collapsed": true 89 | }, 90 | "outputs": [], 91 | "source": [] 92 | } 93 | ], 94 | "metadata": { 95 | "kernelspec": { 96 | "display_name": "Python 2", 97 | "language": "python", 98 | "name": "python2" 99 | }, 100 | "language_info": { 101 | "codemirror_mode": { 102 | "name": "ipython", 103 | "version": 2 104 | }, 105 | "file_extension": ".py", 106 | "mimetype": "text/x-python", 107 | "name": "python", 108 | "nbconvert_exporter": "python", 109 | "pygments_lexer": "ipython2", 110 | "version": "2.7.11" 111 | } 112 | }, 113 | "nbformat": 4, 114 | "nbformat_minor": 0 115 | } 116 | -------------------------------------------------------------------------------- /timeserie/utils_modified.py: -------------------------------------------------------------------------------- 1 | from __future__ import division, print_function 2 | import math, os, json, sys, re 3 | import cPickle as pickle 4 | from glob import glob 5 | import numpy as np 6 | #from matplotlib import pyplot as plt 7 | from operator import itemgetter, attrgetter, methodcaller 8 | from collections import OrderedDict 9 | import itertools 10 | from itertools import chain 11 | 12 | #import pandas as pd 13 | #import PIL 14 | #from PIL import Image 15 | #from numpy.random import random, permutation, randn, normal, uniform, choice 16 | #from numpy import newaxis 17 | #import scipy 18 | #from scipy import misc, ndimage 19 | #from scipy.ndimage.interpolation import zoom 20 | #from scipy.ndimage import imread 21 | #from sklearn.metrics import confusion_matrix 22 | import bcolz 23 | #from sklearn.preprocessing import OneHotEncoder 24 | #from sklearn.manifold import TSNE 25 | 26 | #from IPython.lib.display import FileLink 27 | 28 | #import theano 29 | #from theano import shared, tensor as T 30 | #from theano.tensor.nnet import conv2d, nnet 31 | #from theano.tensor.signal import pool 32 | 33 | import keras 34 | #from keras import backend as K 35 | from keras.utils.data_utils import get_file 36 | #from keras.utils import np_utils 37 | from keras.utils.np_utils import to_categorical 38 | from keras.models import Sequential, Model 39 | #from keras.layers import Input, Embedding, Reshape, merge, LSTM, Bidirectional 40 | #from keras.layers import TimeDistributed, Activation, SimpleRNN, GRU 41 | #from keras.layers.core import Flatten, Dense, Dropout, Lambda 42 | #from keras.regularizers import l2, activity_l2, l1, activity_l1 43 | #from keras.layers.normalization import BatchNormalization 44 | #from keras.optimizers import SGD, RMSprop, Adam 45 | from keras.utils.layer_utils import layer_from_config 46 | #from keras.metrics import categorical_crossentropy, categorical_accuracy 47 | #from keras.layers.convolutional import * 48 | from keras.preprocessing import image, sequence 49 | #from keras.preprocessing.text import Tokenizer 50 | 51 | #np.set_printoptions(precision=4, linewidth=100) 52 | 53 | 54 | def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, batch_size=4, class_mode='categorical', 55 | target_size=(224,224)): 56 | return gen.flow_from_directory(dirname, target_size=target_size, 57 | class_mode=class_mode, shuffle=shuffle, batch_size=batch_size) 58 | 59 | 60 | def onehot(x): 61 | return to_categorical(x) 62 | 63 | 64 | def wrap_config(layer): 65 | return {'class_name': layer.__class__.__name__, 'config': layer.get_config()} 66 | 67 | 68 | def copy_layer(layer): 69 | return layer_from_config(wrap_config(layer)) 70 | 71 | 72 | def copy_layers(layers): 73 | return [copy_layer(layer) for layer in layers] 74 | 75 | 76 | def copy_weights(from_layers, to_layers): 77 | for from_layer,to_layer in zip(from_layers, to_layers): 78 | to_layer.set_weights(from_layer.get_weights()) 79 | 80 | 81 | def copy_model(m): 82 | res = Sequential(copy_layers(m.layers)) 83 | copy_weights(m.layers, res.layers) 84 | return res 85 | 86 | 87 | def insert_layer(model, new_layer, index): 88 | res = Sequential() 89 | for i,layer in enumerate(model.layers): 90 | if i==index: res.add(new_layer) 91 | copied = layer_from_config(wrap_config(layer)) 92 | res.add(copied) 93 | copied.set_weights(layer.get_weights()) 94 | return res 95 | 96 | ''' 97 | def adjust_dropout(weights, prev_p, new_p): 98 | scal = (1-prev_p)/(1-new_p) 99 | return [o*scal for o in weights] 100 | ''' 101 | 102 | def get_data(path, target_size=(224,224)): 103 | batches = get_batches(path, shuffle=False, batch_size=1, class_mode=None, target_size=target_size) 104 | return np.concatenate([batches.next() for i in range(batches.nb_sample)]) 105 | 106 | ''' 107 | def plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues): 108 | """ 109 | This function prints and plots the confusion matrix. 110 | Normalization can be applied by setting `normalize=True`. 111 | (This function is copied from the scikit docs.) 112 | """ 113 | plt.figure() 114 | plt.imshow(cm, interpolation='nearest', cmap=cmap) 115 | plt.title(title) 116 | plt.colorbar() 117 | tick_marks = np.arange(len(classes)) 118 | plt.xticks(tick_marks, classes, rotation=45) 119 | plt.yticks(tick_marks, classes) 120 | 121 | if normalize: 122 | cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] 123 | print(cm) 124 | thresh = cm.max() / 2. 125 | for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])): 126 | plt.text(j, i, cm[i, j], horizontalalignment="center", color="white" if cm[i, j] > thresh else "black") 127 | 128 | plt.tight_layout() 129 | plt.ylabel('True label') 130 | plt.xlabel('Predicted label') 131 | ''' 132 | 133 | def save_array(fname, arr): 134 | c = bcolz.carray(arr, rootdir=fname, mode='w') 135 | c.flush() 136 | 137 | 138 | def load_array(fname): 139 | return bcolz.open(fname)[:] 140 | 141 | 142 | def get_classes(path): 143 | batches = get_batches(path+'train', shuffle=False, batch_size=1) 144 | val_batches = get_batches(path+'valid', shuffle=False, batch_size=1) 145 | test_batches = get_batches(path+'test', shuffle=False, batch_size=1) 146 | return (val_batches.classes, batches.classes, onehot(val_batches.classes), onehot(batches.classes), 147 | val_batches.filenames, batches.filenames, test_batches.filenames) 148 | 149 | 150 | def split_at(model, layer_type): 151 | layers = model.layers 152 | layer_idx = [index for index,layer in enumerate(layers) 153 | if type(layer) is layer_type][-1] 154 | return layers[:layer_idx+1], layers[layer_idx+1:] 155 | -------------------------------------------------------------------------------- /vgg_segmentation_keras/fcn_keras2.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import numpy as np 3 | 4 | from keras.models import Sequential, Model 5 | from keras.layers import Input, Dropout, Permute, Add, add 6 | from keras.layers import Convolution2D, ZeroPadding2D, MaxPooling2D, Deconvolution2D, Cropping2D 7 | 8 | 9 | def convblock(cdim, nb, bits=3): 10 | L = [] 11 | 12 | for k in range(1, bits + 1): 13 | convname = 'conv' + str(nb) + '_' + str(k) 14 | if False: 15 | # first version I tried 16 | L.append(ZeroPadding2D((1, 1))) 17 | L.append(Convolution2D(cdim, kernel_size=(3, 3), activation='relu', name=convname)) 18 | else: 19 | L.append(Convolution2D(cdim, kernel_size=(3, 3), padding='same', activation='relu', name=convname)) 20 | 21 | L.append(MaxPooling2D(pool_size=(2, 2), strides=(2, 2))) 22 | 23 | return L 24 | 25 | 26 | def fcn32_blank(image_size=512): 27 | withDO = False # no effect during evaluation but usefull for fine-tuning 28 | 29 | if True: 30 | mdl = Sequential() 31 | 32 | # First layer is a dummy-permutation = Identity to specify input shape 33 | mdl.add(Permute((1, 2, 3), input_shape=(image_size, image_size, 3))) # WARNING : axis 0 is the sample dim 34 | 35 | for l in convblock(64, 1, bits=2): 36 | mdl.add(l) 37 | 38 | for l in convblock(128, 2, bits=2): 39 | mdl.add(l) 40 | 41 | for l in convblock(256, 3, bits=3): 42 | mdl.add(l) 43 | 44 | for l in convblock(512, 4, bits=3): 45 | mdl.add(l) 46 | 47 | for l in convblock(512, 5, bits=3): 48 | mdl.add(l) 49 | 50 | mdl.add(Convolution2D(4096, kernel_size=(7, 7), padding='same', activation='relu', name='fc6')) # WARNING border 51 | if withDO: 52 | mdl.add(Dropout(0.5)) 53 | mdl.add(Convolution2D(4096, kernel_size=(1, 1), padding='same', activation='relu', name='fc7')) # WARNING border 54 | if withDO: 55 | mdl.add(Dropout(0.5)) 56 | 57 | # WARNING : model decapitation i.e. remove the classifier step of VGG16 (usually named fc8) 58 | 59 | mdl.add(Convolution2D(21, kernel_size=(1, 1), padding='same', activation='relu', name='score_fr')) 60 | 61 | convsize = mdl.layers[-1].output_shape[2] 62 | deconv_output_size = (convsize - 1) * 2 + 4 # INFO: =34 when images are 512x512 63 | # WARNING : valid, same or full ? 64 | mdl.add(Deconvolution2D(21, kernel_size=(4, 4), strides=(2, 2), padding='valid', activation=None, name='score2')) 65 | 66 | extra_margin = deconv_output_size - convsize * 2 # INFO: =2 when images are 512x512 67 | assert (extra_margin > 0) 68 | assert (extra_margin % 2 == 0) 69 | # INFO : cropping as deconv gained pixels 70 | # print(extra_margin) 71 | c = ((0, extra_margin), (0, extra_margin)) 72 | # print(c) 73 | mdl.add(Cropping2D(cropping=c)) 74 | # print(mdl.summary()) 75 | 76 | return mdl 77 | 78 | else: 79 | # See following link for a version based on Keras functional API : 80 | # gist.github.com/EncodeTS/6bbe8cb8bebad7a672f0d872561782d9 81 | raise ValueError('not implemented') 82 | 83 | 84 | # WARNING : explanation about Deconvolution2D layer 85 | # http://stackoverflow.com/questions/39018767/deconvolution2d-layer-in-keras 86 | # the code example in the help (??Deconvolution2D) is very usefull too 87 | # ?? Deconvolution2D 88 | 89 | def fcn_32s_to_16s(fcn32model=None): 90 | if fcn32model is None: 91 | fcn32model = fcn32_blank() 92 | 93 | fcn32shape = fcn32model.layers[-1].output_shape 94 | assert (len(fcn32shape) == 4) 95 | assert (fcn32shape[0] is None) # batch axis 96 | assert (fcn32shape[3] == 21) # number of filters 97 | assert (fcn32shape[1] == fcn32shape[2]) # must be square 98 | 99 | fcn32size = fcn32shape[1] # INFO: =32 when images are 512x512 100 | 101 | if fcn32size != 32: 102 | print('WARNING : handling of image size different from 512x512 has not been tested') 103 | 104 | sp4 = Convolution2D(21, kernel_size=(1, 1), padding='same', activation=None, name='score_pool4') 105 | 106 | # INFO : to replicate MatConvNet.DAGN.Sum layer see documentation at : 107 | # https://keras.io/getting-started/sequential-model-guide/ 108 | summed = add(inputs=[sp4(fcn32model.layers[14].output), fcn32model.layers[-1].output]) 109 | 110 | # INFO : 111 | # final 16x16 upsampling of "summed" using deconv layer upsample_new (32, 32, 21, 21) 112 | # deconv setting is valid if (528-32)/16 + 1 = deconv_input_dim (= fcn32size) 113 | deconv_output_size = (fcn32size - 1) * 16 + 32 # INFO: =528 when images are 512x512 114 | upnew = Deconvolution2D(21, kernel_size=(32, 32), 115 | padding='valid', # WARNING : valid, same or full ? 116 | strides=(16, 16), 117 | activation=None, 118 | name='upsample_new') 119 | 120 | extra_margin = deconv_output_size - fcn32size * 16 # INFO: =16 when images are 512x512 121 | assert (extra_margin > 0) 122 | assert (extra_margin % 2 == 0) 123 | # print(extra_margin) 124 | # INFO : cropping as deconv gained pixels 125 | crop_margin = Cropping2D(cropping=((0, extra_margin), (0, extra_margin))) 126 | 127 | return Model(fcn32model.input, crop_margin(upnew(summed))) 128 | 129 | 130 | def prediction(kmodel, crpimg, transform=False): 131 | # INFO : crpimg should be a cropped image of the right dimension 132 | 133 | # transform=True seems more robust but I think the RGB channels are not in right order 134 | 135 | imarr = np.array(crpimg).astype(np.float32) 136 | 137 | if transform: 138 | imarr[:, :, 0] -= 129.1863 139 | imarr[:, :, 1] -= 104.7624 140 | imarr[:, :, 2] -= 93.5940 141 | # 142 | # WARNING : in this script (https://github.com/rcmalli/keras-vggface) colours are switched 143 | aux = copy.copy(imarr) 144 | imarr[:, :, 0] = aux[:, :, 2] 145 | imarr[:, :, 2] = aux[:, :, 0] 146 | 147 | # imarr[:,:,0] -= 129.1863 148 | # imarr[:,:,1] -= 104.7624 149 | # imarr[:,:,2] -= 93.5940 150 | 151 | # imarr = imarr.transpose((2, 0, 1)) 152 | imarr = np.expand_dims(imarr, axis=0) 153 | 154 | return kmodel.predict(imarr) 155 | 156 | 157 | if __name__ == "__main__": 158 | md = fcn32_blank() 159 | md = fcn_32s_to_16s(md) 160 | print(md.summary()) 161 | -------------------------------------------------------------------------------- /dogsandcats_keras/utils.py: -------------------------------------------------------------------------------- 1 | from __future__ import division,print_function 2 | import math, os, json, sys, re 3 | import cPickle as pickle 4 | from glob import glob 5 | import numpy as np 6 | from matplotlib import pyplot as plt 7 | from operator import itemgetter, attrgetter, methodcaller 8 | from collections import OrderedDict 9 | import itertools 10 | from itertools import chain 11 | 12 | import pandas as pd 13 | import PIL 14 | from PIL import Image 15 | from numpy.random import random, permutation, randn, normal, uniform, choice 16 | from numpy import newaxis 17 | import scipy 18 | from scipy import misc, ndimage 19 | from scipy.ndimage.interpolation import zoom 20 | from scipy.ndimage import imread 21 | from sklearn.metrics import confusion_matrix 22 | import bcolz 23 | from sklearn.preprocessing import OneHotEncoder 24 | from sklearn.manifold import TSNE 25 | 26 | from IPython.lib.display import FileLink 27 | 28 | import theano 29 | from theano import shared, tensor as T 30 | from theano.tensor.nnet import conv2d, nnet 31 | from theano.tensor.signal import pool 32 | 33 | import keras 34 | from keras import backend as K 35 | from keras.utils.data_utils import get_file 36 | from keras.utils import np_utils 37 | from keras.utils.np_utils import to_categorical 38 | from keras.models import Sequential, Model 39 | from keras.layers import Input, Embedding, Reshape, merge, LSTM, Bidirectional 40 | from keras.layers import TimeDistributed, Activation, SimpleRNN, GRU 41 | from keras.layers.core import Flatten, Dense, Dropout, Lambda 42 | from keras.regularizers import l2, activity_l2, l1, activity_l1 43 | from keras.layers.normalization import BatchNormalization 44 | from keras.optimizers import SGD, RMSprop, Adam 45 | from keras.utils.layer_utils import layer_from_config 46 | from keras.metrics import categorical_crossentropy, categorical_accuracy 47 | from keras.layers.convolutional import * 48 | from keras.preprocessing import image, sequence 49 | from keras.preprocessing.text import Tokenizer 50 | 51 | from vgg16 import * 52 | from vgg16bn import * 53 | np.set_printoptions(precision=4, linewidth=100) 54 | 55 | 56 | to_bw = np.array([0.299, 0.587, 0.114]) 57 | def gray(img): 58 | return np.rollaxis(img,0,3).dot(to_bw) 59 | def to_plot(img): 60 | return np.rollaxis(img, 0, 3).astype(np.uint8) 61 | def plot(img): 62 | plt.imshow(to_plot(img)) 63 | 64 | 65 | def floor(x): 66 | return int(math.floor(x)) 67 | def ceil(x): 68 | return int(math.ceil(x)) 69 | 70 | def plots(ims, figsize=(12,6), rows=1, interp=False, titles=None): 71 | if type(ims[0]) is np.ndarray: 72 | ims = np.array(ims).astype(np.uint8) 73 | if (ims.shape[-1] != 3): 74 | ims = ims.transpose((0,2,3,1)) 75 | f = plt.figure(figsize=figsize) 76 | for i in range(len(ims)): 77 | sp = f.add_subplot(rows, len(ims)//rows, i+1) 78 | if titles is not None: 79 | sp.set_title(titles[i], fontsize=18) 80 | plt.imshow(ims[i], interpolation=None if interp else 'none') 81 | 82 | 83 | def do_clip(arr, mx): 84 | clipped = np.clip(arr, (1-mx)/1, mx) 85 | return clipped/clipped.sum(axis=1)[:, np.newaxis] 86 | 87 | 88 | def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, batch_size=4, class_mode='categorical', 89 | target_size=(224,224)): 90 | return gen.flow_from_directory(dirname, target_size=target_size, 91 | class_mode=class_mode, shuffle=shuffle, batch_size=batch_size) 92 | 93 | 94 | def onehot(x): 95 | return to_categorical(x) 96 | 97 | 98 | def wrap_config(layer): 99 | return {'class_name': layer.__class__.__name__, 'config': layer.get_config()} 100 | 101 | 102 | def copy_layer(layer): return layer_from_config(wrap_config(layer)) 103 | 104 | 105 | def copy_layers(layers): return [copy_layer(layer) for layer in layers] 106 | 107 | 108 | def copy_weights(from_layers, to_layers): 109 | for from_layer,to_layer in zip(from_layers, to_layers): 110 | to_layer.set_weights(from_layer.get_weights()) 111 | 112 | 113 | def copy_model(m): 114 | res = Sequential(copy_layers(m.layers)) 115 | copy_weights(m.layers, res.layers) 116 | return res 117 | 118 | 119 | def insert_layer(model, new_layer, index): 120 | res = Sequential() 121 | for i,layer in enumerate(model.layers): 122 | if i==index: res.add(new_layer) 123 | copied = layer_from_config(wrap_config(layer)) 124 | res.add(copied) 125 | copied.set_weights(layer.get_weights()) 126 | return res 127 | 128 | 129 | def adjust_dropout(weights, prev_p, new_p): 130 | scal = (1-prev_p)/(1-new_p) 131 | return [o*scal for o in weights] 132 | 133 | 134 | def get_data(path, target_size=(224,224)): 135 | batches = get_batches(path, shuffle=False, batch_size=1, class_mode=None, target_size=target_size) 136 | return np.concatenate([batches.next() for i in range(batches.nb_sample)]) 137 | 138 | 139 | def plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues): 140 | """ 141 | This function prints and plots the confusion matrix. 142 | Normalization can be applied by setting `normalize=True`. 143 | (This function is copied from the scikit docs.) 144 | """ 145 | plt.figure() 146 | plt.imshow(cm, interpolation='nearest', cmap=cmap) 147 | plt.title(title) 148 | plt.colorbar() 149 | tick_marks = np.arange(len(classes)) 150 | plt.xticks(tick_marks, classes, rotation=45) 151 | plt.yticks(tick_marks, classes) 152 | 153 | if normalize: 154 | cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] 155 | print(cm) 156 | thresh = cm.max() / 2. 157 | for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])): 158 | plt.text(j, i, cm[i, j], horizontalalignment="center", color="white" if cm[i, j] > thresh else "black") 159 | 160 | plt.tight_layout() 161 | plt.ylabel('True label') 162 | plt.xlabel('Predicted label') 163 | 164 | 165 | def save_array(fname, arr): 166 | c=bcolz.carray(arr, rootdir=fname, mode='w') 167 | c.flush() 168 | 169 | 170 | def load_array(fname): 171 | return bcolz.open(fname)[:] 172 | 173 | 174 | def mk_size(img, r2c): 175 | r,c,_ = img.shape 176 | curr_r2c = r/c 177 | new_r, new_c = r,c 178 | if r2c>curr_r2c: 179 | new_r = floor(c*r2c) 180 | else: 181 | new_c = floor(r/r2c) 182 | arr = np.zeros((new_r, new_c, 3), dtype=np.float32) 183 | r2=(new_r-r)//2 184 | c2=(new_c-c)//2 185 | arr[floor(r2):floor(r2)+r,floor(c2):floor(c2)+c] = img 186 | return arr 187 | 188 | 189 | def mk_square(img): 190 | x,y,_ = img.shape 191 | maxs = max(img.shape[:2]) 192 | y2=(maxs-y)//2 193 | x2=(maxs-x)//2 194 | arr = np.zeros((maxs,maxs,3), dtype=np.float32) 195 | arr[floor(x2):floor(x2)+x,floor(y2):floor(y2)+y] = img 196 | return arr 197 | 198 | 199 | def vgg_ft(out_dim): 200 | vgg = Vgg16() 201 | vgg.ft(out_dim) 202 | model = vgg.model 203 | return model 204 | 205 | def vgg_ft_bn(out_dim): 206 | vgg = Vgg16BN() 207 | vgg.ft(out_dim) 208 | model = vgg.model 209 | return model 210 | 211 | 212 | def get_classes(path): 213 | batches = get_batches(path+'train', shuffle=False, batch_size=1) 214 | val_batches = get_batches(path+'valid', shuffle=False, batch_size=1) 215 | test_batches = get_batches(path+'test', shuffle=False, batch_size=1) 216 | return (val_batches.classes, batches.classes, onehot(val_batches.classes), onehot(batches.classes), 217 | val_batches.filenames, batches.filenames, test_batches.filenames) 218 | 219 | 220 | def split_at(model, layer_type): 221 | layers = model.layers 222 | layer_idx = [index for index,layer in enumerate(layers) 223 | if type(layer) is layer_type][-1] 224 | return layers[:layer_idx+1], layers[layer_idx+1:] 225 | 226 | 227 | class MixIterator(object): 228 | def __init__(self, iters): 229 | self.iters = iters 230 | self.multi = type(iters) is list 231 | if self.multi: 232 | self.N = sum([it[0].N for it in self.iters]) 233 | else: 234 | self.N = sum([it.N for it in self.iters]) 235 | 236 | def reset(self): 237 | for it in self.iters: it.reset() 238 | 239 | def __iter__(self): 240 | return self 241 | 242 | def next(self, *args, **kwargs): 243 | if self.multi: 244 | nexts = [[next(it) for it in o] for o in self.iters] 245 | n0s = np.concatenate([n[0] for n in o]) 246 | n1s = np.concatenate([n[1] for n in o]) 247 | return (n0, n1) 248 | else: 249 | nexts = [next(it) for it in self.iters] 250 | n0 = np.concatenate([n[0] for n in nexts]) 251 | n1 = np.concatenate([n[1] for n in nexts]) 252 | return (n0, n1) 253 | 254 | -------------------------------------------------------------------------------- /insurance_scikit/metrics.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python2.7 2 | 3 | import numpy as np 4 | 5 | 6 | def confusion_matrix(rater_a, rater_b, min_rating=None, max_rating=None): 7 | """ 8 | Returns the confusion matrix between rater's ratings 9 | """ 10 | assert(len(rater_a) == len(rater_b)) 11 | if min_rating is None: 12 | min_rating = min(rater_a + rater_b) 13 | if max_rating is None: 14 | max_rating = max(rater_a + rater_b) 15 | num_ratings = int(max_rating - min_rating + 1) 16 | conf_mat = [[0 for i in range(num_ratings)] 17 | for j in range(num_ratings)] 18 | for a, b in zip(rater_a, rater_b): 19 | conf_mat[a - min_rating][b - min_rating] += 1 20 | return conf_mat 21 | 22 | 23 | def histogram(ratings, min_rating=None, max_rating=None): 24 | """ 25 | Returns the counts of each type of rating that a rater made 26 | """ 27 | if min_rating is None: 28 | min_rating = min(ratings) 29 | if max_rating is None: 30 | max_rating = max(ratings) 31 | num_ratings = int(max_rating - min_rating + 1) 32 | hist_ratings = [0 for x in range(num_ratings)] 33 | for r in ratings: 34 | hist_ratings[r - min_rating] += 1 35 | return hist_ratings 36 | 37 | 38 | def quadratic_weighted_kappa(rater_a, rater_b, min_rating=None, max_rating=None): 39 | """ 40 | Calculates the quadratic weighted kappa 41 | quadratic_weighted_kappa calculates the quadratic weighted kappa 42 | value, which is a measure of inter-rater agreement between two raters 43 | that provide discrete numeric ratings. Potential values range from -1 44 | (representing complete disagreement) to 1 (representing complete 45 | agreement). A kappa value of 0 is expected if all agreement is due to 46 | chance. 47 | 48 | quadratic_weighted_kappa(rater_a, rater_b), where rater_a and rater_b 49 | each correspond to a list of integer ratings. These lists must have the 50 | same length. 51 | 52 | The ratings should be integers, and it is assumed that they contain 53 | the complete range of possible ratings. 54 | 55 | quadratic_weighted_kappa(X, min_rating, max_rating), where min_rating 56 | is the minimum possible rating, and max_rating is the maximum possible 57 | rating 58 | """ 59 | rater_a = np.array(rater_a, dtype=int) 60 | rater_b = np.array(rater_b, dtype=int) 61 | assert(len(rater_a) == len(rater_b)) 62 | if min_rating is None: 63 | min_rating = min(min(rater_a), min(rater_b)) 64 | if max_rating is None: 65 | max_rating = max(max(rater_a), max(rater_b)) 66 | conf_mat = confusion_matrix(rater_a, rater_b, 67 | min_rating, max_rating) 68 | num_ratings = len(conf_mat) 69 | num_scored_items = float(len(rater_a)) 70 | 71 | hist_rater_a = histogram(rater_a, min_rating, max_rating) 72 | hist_rater_b = histogram(rater_b, min_rating, max_rating) 73 | 74 | numerator = 0.0 75 | denominator = 0.0 76 | 77 | for i in range(num_ratings): 78 | for j in range(num_ratings): 79 | expected_count = (hist_rater_a[i] * hist_rater_b[j] 80 | / num_scored_items) 81 | d = pow(i - j, 2.0) / pow(num_ratings - 1, 2.0) 82 | numerator += d * conf_mat[i][j] / num_scored_items 83 | denominator += d * expected_count / num_scored_items 84 | 85 | return 1.0 - numerator / denominator 86 | 87 | 88 | def linear_weighted_kappa(rater_a, rater_b, min_rating=None, max_rating=None): 89 | """ 90 | Calculates the linear weighted kappa 91 | linear_weighted_kappa calculates the linear weighted kappa 92 | value, which is a measure of inter-rater agreement between two raters 93 | that provide discrete numeric ratings. Potential values range from -1 94 | (representing complete disagreement) to 1 (representing complete 95 | agreement). A kappa value of 0 is expected if all agreement is due to 96 | chance. 97 | 98 | linear_weighted_kappa(rater_a, rater_b), where rater_a and rater_b 99 | each correspond to a list of integer ratings. These lists must have the 100 | same length. 101 | 102 | The ratings should be integers, and it is assumed that they contain 103 | the complete range of possible ratings. 104 | 105 | linear_weighted_kappa(X, min_rating, max_rating), where min_rating 106 | is the minimum possible rating, and max_rating is the maximum possible 107 | rating 108 | """ 109 | assert(len(rater_a) == len(rater_b)) 110 | if min_rating is None: 111 | min_rating = min(rater_a + rater_b) 112 | if max_rating is None: 113 | max_rating = max(rater_a + rater_b) 114 | conf_mat = confusion_matrix(rater_a, rater_b, 115 | min_rating, max_rating) 116 | num_ratings = len(conf_mat) 117 | num_scored_items = float(len(rater_a)) 118 | 119 | hist_rater_a = histogram(rater_a, min_rating, max_rating) 120 | hist_rater_b = histogram(rater_b, min_rating, max_rating) 121 | 122 | numerator = 0.0 123 | denominator = 0.0 124 | 125 | for i in range(num_ratings): 126 | for j in range(num_ratings): 127 | expected_count = (hist_rater_a[i] * hist_rater_b[j] 128 | / num_scored_items) 129 | d = abs(i - j) / float(num_ratings - 1) 130 | numerator += d * conf_mat[i][j] / num_scored_items 131 | denominator += d * expected_count / num_scored_items 132 | 133 | return 1.0 - numerator / denominator 134 | 135 | 136 | def kappa(rater_a, rater_b, min_rating=None, max_rating=None): 137 | """ 138 | Calculates the kappa 139 | kappa calculates the kappa 140 | value, which is a measure of inter-rater agreement between two raters 141 | that provide discrete numeric ratings. Potential values range from -1 142 | (representing complete disagreement) to 1 (representing complete 143 | agreement). A kappa value of 0 is expected if all agreement is due to 144 | chance. 145 | 146 | kappa(rater_a, rater_b), where rater_a and rater_b 147 | each correspond to a list of integer ratings. These lists must have the 148 | same length. 149 | 150 | The ratings should be integers, and it is assumed that they contain 151 | the complete range of possible ratings. 152 | 153 | kappa(X, min_rating, max_rating), where min_rating 154 | is the minimum possible rating, and max_rating is the maximum possible 155 | rating 156 | """ 157 | assert(len(rater_a) == len(rater_b)) 158 | if min_rating is None: 159 | min_rating = min(rater_a + rater_b) 160 | if max_rating is None: 161 | max_rating = max(rater_a + rater_b) 162 | conf_mat = confusion_matrix(rater_a, rater_b, 163 | min_rating, max_rating) 164 | num_ratings = len(conf_mat) 165 | num_scored_items = float(len(rater_a)) 166 | 167 | hist_rater_a = histogram(rater_a, min_rating, max_rating) 168 | hist_rater_b = histogram(rater_b, min_rating, max_rating) 169 | 170 | numerator = 0.0 171 | denominator = 0.0 172 | 173 | for i in range(num_ratings): 174 | for j in range(num_ratings): 175 | expected_count = (hist_rater_a[i] * hist_rater_b[j] 176 | / num_scored_items) 177 | if i == j: 178 | d = 0.0 179 | else: 180 | d = 1.0 181 | numerator += d * conf_mat[i][j] / num_scored_items 182 | denominator += d * expected_count / num_scored_items 183 | 184 | return 1.0 - numerator / denominator 185 | 186 | 187 | def mean_quadratic_weighted_kappa(kappas, weights=None): 188 | """ 189 | Calculates the mean of the quadratic 190 | weighted kappas after applying Fisher's r-to-z transform, which is 191 | approximately a variance-stabilizing transformation. This 192 | transformation is undefined if one of the kappas is 1.0, so all kappa 193 | values are capped in the range (-0.999, 0.999). The reverse 194 | transformation is then applied before returning the result. 195 | 196 | mean_quadratic_weighted_kappa(kappas), where kappas is a vector of 197 | kappa values 198 | 199 | mean_quadratic_weighted_kappa(kappas, weights), where weights is a vector 200 | of weights that is the same size as kappas. Weights are applied in the 201 | z-space 202 | """ 203 | kappas = np.array(kappas, dtype=float) 204 | if weights is None: 205 | weights = np.ones(np.shape(kappas)) 206 | else: 207 | weights = weights / np.mean(weights) 208 | 209 | # ensure that kappas are in the range [-.999, .999] 210 | kappas = np.array([min(x, .999) for x in kappas]) 211 | kappas = np.array([max(x, -.999) for x in kappas]) 212 | 213 | z = 0.5 * np.log((1 + kappas) / (1 - kappas)) * weights 214 | z = np.mean(z) 215 | return (np.exp(2 * z) - 1) / (np.exp(2 * z) + 1) 216 | 217 | 218 | def weighted_mean_quadratic_weighted_kappa(solution, submission): 219 | predicted_score = submission[submission.columns[-1]].copy() 220 | predicted_score.name = "predicted_score" 221 | if predicted_score.index[0] == 0: 222 | predicted_score = predicted_score[:len(solution)] 223 | predicted_score.index = solution.index 224 | combined = solution.join(predicted_score, how="left") 225 | groups = combined.groupby(by="essay_set") 226 | kappas = [quadratic_weighted_kappa(group[1]["essay_score"], group[1]["predicted_score"]) for group in groups] 227 | weights = [group[1]["essay_weight"].irow(0) for group in groups] 228 | return mean_quadratic_weighted_kappa(kappas, weights=weights) 229 | 230 | -------------------------------------------------------------------------------- /vgg_segmentation_keras/utils.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import numpy as np 3 | 4 | from keras.models import Sequential, Model 5 | from keras.layers import Input, Dropout, Permute 6 | from keras.layers import Convolution2D, ZeroPadding2D, MaxPooling2D, Deconvolution2D, Cropping2D 7 | from keras.layers import merge 8 | 9 | 10 | def convblock(cdim, nb, bits=3): 11 | L = [] 12 | 13 | for k in range(1,bits+1): 14 | convname = 'conv'+str(nb)+'_'+str(k) 15 | if False: 16 | # first version I tried 17 | L.append( ZeroPadding2D((1, 1)) ) 18 | L.append( Convolution2D(cdim, 3, 3, activation='relu', name=convname) ) 19 | else: 20 | L.append( Convolution2D(cdim, 3, 3, border_mode='same', activation='relu', name=convname) ) 21 | 22 | L.append( MaxPooling2D((2, 2), strides=(2, 2)) ) 23 | 24 | return L 25 | 26 | 27 | def fcn32_blank(image_size=512): 28 | 29 | withDO = False # no effect during evaluation but usefull for fine-tuning 30 | 31 | if True: 32 | mdl = Sequential() 33 | 34 | # First layer is a dummy-permutation = Identity to specify input shape 35 | mdl.add( Permute((1,2,3), input_shape=(3,image_size,image_size)) ) # WARNING : axis 0 is the sample dim 36 | 37 | for l in convblock(64, 1, bits=2): 38 | mdl.add(l) 39 | 40 | for l in convblock(128, 2, bits=2): 41 | mdl.add(l) 42 | 43 | for l in convblock(256, 3, bits=3): 44 | mdl.add(l) 45 | 46 | for l in convblock(512, 4, bits=3): 47 | mdl.add(l) 48 | 49 | for l in convblock(512, 5, bits=3): 50 | mdl.add(l) 51 | 52 | mdl.add( Convolution2D(4096, 7, 7, border_mode='same', activation='relu', name='fc6') ) # WARNING border 53 | if withDO: 54 | mdl.add( Dropout(0.5) ) 55 | mdl.add( Convolution2D(4096, 1, 1, border_mode='same', activation='relu', name='fc7') ) # WARNING border 56 | if withDO: 57 | mdl.add( Dropout(0.5) ) 58 | 59 | # WARNING : model decapitation i.e. remove the classifier step of VGG16 (usually named fc8) 60 | 61 | mdl.add( Convolution2D(21, 1, 1, 62 | border_mode='same', # WARNING : zero or same ? does not matter for 1x1 63 | activation='relu', name='score_fr') ) 64 | 65 | convsize = mdl.layers[-1].output_shape[2] 66 | deconv_output_size = (convsize-1)*2+4 # INFO: =34 when images are 512x512 67 | mdl.add( Deconvolution2D(21, 4, 4, 68 | output_shape=(None, 21, deconv_output_size, deconv_output_size), 69 | subsample=(2, 2), 70 | border_mode='valid', # WARNING : valid, same or full ? 71 | activation=None, 72 | name = 'score2') ) 73 | 74 | extra_margin = deconv_output_size - convsize*2 # INFO: =2 when images are 512x512 75 | assert (extra_margin > 0) 76 | assert (extra_margin % 2 == 0) 77 | mdl.add( Cropping2D(cropping=((extra_margin/2, extra_margin/2), 78 | (extra_margin/2, extra_margin/2))) ) # INFO : cropping as deconv gained pixels 79 | 80 | return mdl 81 | 82 | else: 83 | # See following link for a version based on Keras functional API : 84 | # gist.github.com/EncodeTS/6bbe8cb8bebad7a672f0d872561782d9 85 | raise ValueError('not implemented') 86 | 87 | 88 | 89 | # WARNING : explanation about Deconvolution2D layer 90 | # http://stackoverflow.com/questions/39018767/deconvolution2d-layer-in-keras 91 | # the code example in the help (??Deconvolution2D) is very usefull too 92 | # ?? Deconvolution2D 93 | 94 | def fcn_32s_to_16s(fcn32model=None): 95 | 96 | if (fcn32model is None): 97 | fcn32model = fcn32_blank() 98 | 99 | fcn32shape = fcn32model.layers[-1].output_shape 100 | assert (len(fcn32shape) == 4) 101 | assert (fcn32shape[0] is None) # batch axis 102 | assert (fcn32shape[1] == 21) # number of filters 103 | assert (fcn32shape[2] == fcn32shape[3]) # must be square 104 | 105 | fcn32size = fcn32shape[2] # INFO: =32 when images are 512x512 106 | 107 | if (fcn32size != 32): 108 | print('WARNING : handling of image size different from 512x512 has not been tested') 109 | 110 | sp4 = Convolution2D(21, 1, 1, 111 | border_mode='same', # WARNING : zero or same ? does not matter for 1x1 112 | activation=None, # WARNING : to check 113 | name='score_pool4') 114 | 115 | # INFO : to replicate MatConvNet.DAGN.Sum layer see documentation at : 116 | # https://keras.io/getting-started/sequential-model-guide/ 117 | summed = merge([sp4(fcn32model.layers[14].output), fcn32model.layers[-1].output], mode='sum') 118 | 119 | # INFO : 120 | # final 16x16 upsampling of "summed" using deconv layer upsample_new (32, 32, 21, 21) 121 | # deconv setting is valid if (528-32)/16 + 1 = deconv_input_dim (= fcn32size) 122 | deconv_output_size = (fcn32size-1)*16+32 # INFO: =528 when images are 512x512 123 | upnew = Deconvolution2D(21, 32, 32, 124 | output_shape=(None, 21, deconv_output_size, deconv_output_size), 125 | border_mode='valid', # WARNING : valid, same or full ? 126 | subsample=(16, 16), 127 | activation=None, 128 | name = 'upsample_new') 129 | 130 | extra_margin = deconv_output_size - fcn32size*16 # INFO: =16 when images are 512x512 131 | assert (extra_margin > 0) 132 | assert (extra_margin % 2 == 0) 133 | crop_margin = Cropping2D(cropping=((extra_margin/2, extra_margin/2), 134 | (extra_margin/2, extra_margin/2))) # INFO : cropping as deconv gained pixels 135 | 136 | return Model(fcn32model.input, crop_margin(upnew(summed))) 137 | 138 | 139 | def fcn_32s_to_8s(fcn32model=None): 140 | 141 | if (fcn32model is None): 142 | fcn32model = fcn32_blank() 143 | 144 | fcn32shape = fcn32model.layers[-1].output_shape 145 | assert (len(fcn32shape) == 4) 146 | assert (fcn32shape[0] is None) # batch axis 147 | assert (fcn32shape[1] == 21) # number of filters 148 | assert (fcn32shape[2] == fcn32shape[3]) # must be square 149 | 150 | fcn32size = fcn32shape[2] # INFO: =32 when images are 512x512 151 | 152 | if (fcn32size != 32): 153 | print('WARNING : handling of image size different from 512x512 has not been tested') 154 | 155 | sp4 = Convolution2D(21, 1, 1, 156 | border_mode='same', # WARNING : zero or same ? does not matter for 1x1 157 | activation=None, # WARNING : to check 158 | name='score_pool4') 159 | 160 | # INFO : to replicate MatConvNet.DAGN.Sum layer see documentation at : 161 | # https://keras.io/getting-started/sequential-model-guide/ 162 | score_fused = merge([sp4(fcn32model.layers[14].output), fcn32model.layers[-1].output], mode='sum') 163 | 164 | deconv4_output_size = (fcn32size-1)*2+4 # INFO: =66 when images are 512x512 165 | s4deconv = Deconvolution2D(21, 4, 4, 166 | output_shape=(None, 21, deconv4_output_size, deconv4_output_size), 167 | border_mode='valid', # WARNING : valid, same or full ? 168 | subsample=(2, 2), 169 | activation=None, 170 | name = 'score4') 171 | 172 | extra_margin4 = deconv4_output_size - fcn32size*2 # INFO: =2 when images are 512x512 173 | assert (extra_margin4 > 0) 174 | assert (extra_margin4 % 2 == 0) 175 | crop_margin4 = Cropping2D(cropping=((extra_margin4/2, extra_margin4/2), 176 | (extra_margin4/2, extra_margin4/2))) # INFO : cropping as deconv gained pixels 177 | 178 | score4 = crop_margin4(s4deconv(score_fused)) 179 | 180 | # WARNING : check dimensions 181 | sp3 = Convolution2D(21, 1, 1, 182 | border_mode='same', # WARNING : zero or same ? does not matter for 1x1 183 | activation=None, # WARNING : to check 184 | name='score_pool3') 185 | 186 | score_final = merge([sp3(fcn32model.layers[10].output), score4], mode='sum') # WARNING : is that correct ? 187 | 188 | assert (fcn32size*2 == fcn32model.layers[10].output_shape[2]) 189 | deconvUP_output_size = (fcn32size*2-1)*8+16 # INFO: =520 when images are 512x512 190 | upsample = Deconvolution2D(21, 16, 16, 191 | output_shape=(None, 21, deconvUP_output_size, deconvUP_output_size), 192 | border_mode='valid', # WARNING : valid, same or full ? 193 | subsample=(8, 8), 194 | activation=None, 195 | name = 'upsample') 196 | 197 | bigscore = upsample(score_final) 198 | 199 | extra_marginUP = deconvUP_output_size - (fcn32size*2)*8 # INFO: =8 when images are 512x512 200 | assert (extra_marginUP > 0) 201 | assert (extra_marginUP % 2 == 0) 202 | crop_marginUP = Cropping2D(cropping=((extra_marginUP/2, extra_marginUP/2), 203 | (extra_marginUP/2, extra_marginUP/2))) # INFO : cropping as deconv gained pixels 204 | 205 | coarse = crop_marginUP(bigscore) 206 | 207 | return Model(fcn32model.input, coarse) 208 | 209 | 210 | def prediction(kmodel, crpimg, transform=False): 211 | 212 | # INFO : crpimg should be a cropped image of the right dimension 213 | 214 | # transform=True seems more robust but I think the RGB channels are not in right order 215 | 216 | imarr = np.array(crpimg).astype(np.float32) 217 | 218 | if transform: 219 | imarr[:,:,0] -= 129.1863 220 | imarr[:,:,1] -= 104.7624 221 | imarr[:,:,2] -= 93.5940 222 | # 223 | # WARNING : in this script (https://github.com/rcmalli/keras-vggface) colours are switched 224 | aux = copy.copy(imarr) 225 | imarr[:, :, 0] = aux[:, :, 2] 226 | imarr[:, :, 2] = aux[:, :, 0] 227 | 228 | #imarr[:,:,0] -= 129.1863 229 | #imarr[:,:,1] -= 104.7624 230 | #imarr[:,:,2] -= 93.5940 231 | 232 | imarr = imarr.transpose((2,0,1)) 233 | imarr = np.expand_dims(imarr, axis=0) 234 | 235 | return kmodel.predict(imarr) -------------------------------------------------------------------------------- /dogsandcats_keras/dogsandcats.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "name": "stderr", 12 | "output_type": "stream", 13 | "text": [ 14 | "Using gpu device 0: Tesla K80 (CNMeM is disabled)\n", 15 | "Using Theano backend.\n" 16 | ] 17 | } 18 | ], 19 | "source": [ 20 | "# Rather than importing everything manually, we'll make things easy\n", 21 | "# and load them all in utils.py, and just import them from there.\n", 22 | "%matplotlib inline\n", 23 | "import utils; reload(utils)\n", 24 | "from utils import *" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 2, 30 | "metadata": { 31 | "collapsed": true 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "%matplotlib inline\n", 36 | "from __future__ import division,print_function\n", 37 | "import os, json\n", 38 | "from glob import glob\n", 39 | "import numpy as np\n", 40 | "import scipy\n", 41 | "from sklearn.preprocessing import OneHotEncoder\n", 42 | "from sklearn.metrics import confusion_matrix\n", 43 | "np.set_printoptions(precision=4, linewidth=100)\n", 44 | "from matplotlib import pyplot as plt\n", 45 | "import utils; reload(utils)\n", 46 | "from utils import plots, get_batches, plot_confusion_matrix, get_data\n" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 3, 52 | "metadata": { 53 | "collapsed": true 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "from numpy.random import random, permutation\n", 58 | "from scipy import misc, ndimage\n", 59 | "from scipy.ndimage.interpolation import zoom\n", 60 | "\n", 61 | "import keras\n", 62 | "from keras import backend as K\n", 63 | "from keras.utils.data_utils import get_file\n", 64 | "from keras.models import Sequential\n", 65 | "from keras.layers import Input\n", 66 | "from keras.layers.core import Flatten, Dense, Dropout, Lambda\n", 67 | "from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D\n", 68 | "from keras.optimizers import SGD, RMSprop\n", 69 | "from keras.preprocessing import image" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 49, 75 | "metadata": { 76 | "collapsed": false 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "#path = \"data/dogscats/sample/\"\n", 81 | "path = \"\"\n", 82 | "model_path = path + \"models/\"\n", 83 | "if not os.path.exists(model_path):\n", 84 | " os.mkdir(model_path)\n", 85 | " print('Done')" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 125, 91 | "metadata": { 92 | "collapsed": false 93 | }, 94 | "outputs": [ 95 | { 96 | "name": "stderr", 97 | "output_type": "stream", 98 | "text": [ 99 | "/home/ubuntu/anaconda2/lib/python2.7/site-packages/keras/layers/core.py:622: UserWarning: `output_shape` argument not specified for layer lambda_8 and cannot be automatically inferred with the Theano backend. Defaulting to output shape `(None, 3, 224, 224)` (same as input shape). If the expected output shape is different, specify it via the `output_shape` argument.\n", 100 | " .format(self.name, input_shape))\n" 101 | ] 102 | } 103 | ], 104 | "source": [ 105 | "from vgg16 import Vgg16\n", 106 | "vgg = Vgg16()\n", 107 | "#model = vgg.model" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 102, 113 | "metadata": { 114 | "collapsed": true 115 | }, 116 | "outputs": [], 117 | "source": [ 118 | "batch_size = 100\n", 119 | "batch_size = 50" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 103, 125 | "metadata": { 126 | "collapsed": true 127 | }, 128 | "outputs": [], 129 | "source": [ 130 | "def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, \n", 131 | " batch_size=batch_size, class_mode='categorical'):\n", 132 | " return gen.flow_from_directory(path+dirname, target_size=(224,224), \n", 133 | " class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 124, 139 | "metadata": { 140 | "collapsed": false 141 | }, 142 | "outputs": [ 143 | { 144 | "name": "stdout", 145 | "output_type": "stream", 146 | "text": [ 147 | "Found 4000 images belonging to 2 classes.\n", 148 | "Found 21000 images belonging to 2 classes.\n" 149 | ] 150 | } 151 | ], 152 | "source": [ 153 | "# Use batch size of 1 since we're just doing preprocessing on the CPU\n", 154 | "val_batches = get_batches('valid', shuffle=True, batch_size=batch_size)\n", 155 | "trn_batches = get_batches('train', shuffle=True, batch_size=batch_size)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 105, 161 | "metadata": { 162 | "collapsed": false 163 | }, 164 | "outputs": [], 165 | "source": [ 166 | "#val_data = get_data(val_batches)\n", 167 | "#trn_data = get_data(trn_batches)" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 106, 173 | "metadata": { 174 | "collapsed": false 175 | }, 176 | "outputs": [], 177 | "source": [ 178 | "#model.predict(val_batches, batch_size=batch_size)" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 126, 184 | "metadata": { 185 | "collapsed": false 186 | }, 187 | "outputs": [], 188 | "source": [ 189 | "#imgs, labels = next(trn_batches)" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 127, 195 | "metadata": { 196 | "collapsed": false 197 | }, 198 | "outputs": [], 199 | "source": [ 200 | "#vgg.predict(imgs, True)[1]" 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": 128, 206 | "metadata": { 207 | "collapsed": true 208 | }, 209 | "outputs": [], 210 | "source": [ 211 | "vgg.model.pop()\n", 212 | "for layer in vgg.model.layers:\n", 213 | " layer.trainable=False\n", 214 | "vgg.model.add(Dense(2, activation='softmax'))" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 264, 220 | "metadata": { 221 | "collapsed": true 222 | }, 223 | "outputs": [], 224 | "source": [ 225 | "opt = RMSprop(lr=0.0005)\n", 226 | "vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 145, 232 | "metadata": { 233 | "collapsed": false 234 | }, 235 | "outputs": [], 236 | "source": [ 237 | "#vgg.model.summary()" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": 146, 243 | "metadata": { 244 | "collapsed": false 245 | }, 246 | "outputs": [], 247 | "source": [ 248 | "#model.fit_generator(trn_batches, samples_per_epoch=trn_batches.N, nb_epoch=1, validation_data=val_batches, nb_val_samples=val_batches.N)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": 268, 254 | "metadata": { 255 | "collapsed": false 256 | }, 257 | "outputs": [ 258 | { 259 | "name": "stdout", 260 | "output_type": "stream", 261 | "text": [ 262 | "[array(0.0020906650461256504, dtype=float32), array(1.0, dtype=float32)]\n", 263 | "[array(0.007886412553489208, dtype=float32), array(1.0, dtype=float32)]\n", 264 | "[array(1.2159348727891484e-07, dtype=float32), array(1.0, dtype=float32)]\n", 265 | "[array(0.5502254366874695, dtype=float32), array(0.9599999785423279, dtype=float32)]\n", 266 | "[array(1.358986168042975e-07, dtype=float32), array(1.0, dtype=float32)]\n", 267 | "[array(0.16351263225078583, dtype=float32), array(0.9800000190734863, dtype=float32)]\n", 268 | "[array(0.00029040570370852947, dtype=float32), array(1.0, dtype=float32)]\n", 269 | "[array(0.0001684165035840124, dtype=float32), array(1.0, dtype=float32)]\n", 270 | "[array(0.5389044284820557, dtype=float32), array(0.9399999976158142, dtype=float32)]\n", 271 | "[array(0.3258085548877716, dtype=float32), array(0.9800000190734863, dtype=float32)]\n", 272 | "------\n", 273 | "[array(0.025226211175322533, dtype=float32), array(0.9800000190734863, dtype=float32)]\n" 274 | ] 275 | } 276 | ], 277 | "source": [ 278 | "for i in range(10):\n", 279 | " imgs, labels = next(trn_batches)\n", 280 | " o = vgg.model.train_on_batch(imgs, labels)\n", 281 | " print(o)\n", 282 | "\n", 283 | "print('------')\n", 284 | "imgs, labels = next(val_batches)\n", 285 | "ov = vgg.model.test_on_batch(imgs, labels)\n", 286 | "print(ov)" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": 273, 292 | "metadata": { 293 | "collapsed": false 294 | }, 295 | "outputs": [ 296 | { 297 | "name": "stdout", 298 | "output_type": "stream", 299 | "text": [ 300 | "Epoch 1/1\n", 301 | "21000/21000 [==============================] - 661s - loss: 0.2105 - acc: 0.9800 - val_loss: 0.1585 - val_acc: 0.9843\n" 302 | ] 303 | } 304 | ], 305 | "source": [ 306 | "vgg.fit(trn_batches, val_batches, nb_epoch=1)" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": 274, 312 | "metadata": { 313 | "collapsed": true 314 | }, 315 | "outputs": [], 316 | "source": [ 317 | "vgg.model.save_weights(model_path+'finetune1.h5')\n", 318 | "#vgg.model.load_weights(model_path+'finetune1.h5')" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 294, 324 | "metadata": { 325 | "collapsed": false 326 | }, 327 | "outputs": [ 328 | { 329 | "name": "stdout", 330 | "output_type": "stream", 331 | "text": [ 332 | "Found 12500 images belonging to 1 classes.\n" 333 | ] 334 | }, 335 | { 336 | "data": { 337 | "text/plain": [ 338 | "['test/10592.jpg',\n", 339 | " 'test/7217.jpg',\n", 340 | " 'test/3653.jpg',\n", 341 | " 'test/4382.jpg',\n", 342 | " 'test/2924.jpg',\n", 343 | " 'test/10.jpg',\n", 344 | " 'test/10916.jpg',\n", 345 | " 'test/12374.jpg',\n", 346 | " 'test/1871.jpg',\n", 347 | " 'test/11645.jpg']" 348 | ] 349 | }, 350 | "execution_count": 294, 351 | "metadata": {}, 352 | "output_type": "execute_result" 353 | } 354 | ], 355 | "source": [ 356 | "test_batches = get_batches('test', shuffle=False, batch_size=100, class_mode=None)\n", 357 | "#gen = image.ImageDataGenerator()\n", 358 | "#test_batches = gen.flow_from_directory(\"test\", target_size=(224,224), class_mode=None, shuffle=False, batch_size=50)\n", 359 | "test_preds = []\n", 360 | "testfiles = test_batches.filenames\n", 361 | "testfiles[0:10]" 362 | ] 363 | }, 364 | { 365 | "cell_type": "code", 366 | "execution_count": 303, 367 | "metadata": { 368 | "collapsed": false 369 | }, 370 | "outputs": [], 371 | "source": [ 372 | "for i in range(20):\n", 373 | " imgs = next(test_batches)\n", 374 | " bps = vgg.model.predict_on_batch(imgs).tolist()\n", 375 | " test_preds.extend(bps)" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": 304, 381 | "metadata": { 382 | "collapsed": false 383 | }, 384 | "outputs": [ 385 | { 386 | "data": { 387 | "text/plain": [ 388 | "12500" 389 | ] 390 | }, 391 | "execution_count": 304, 392 | "metadata": {}, 393 | "output_type": "execute_result" 394 | } 395 | ], 396 | "source": [ 397 | "len(test_preds)" 398 | ] 399 | }, 400 | { 401 | "cell_type": "code", 402 | "execution_count": 305, 403 | "metadata": { 404 | "collapsed": false 405 | }, 406 | "outputs": [ 407 | { 408 | "data": { 409 | "text/plain": [ 410 | "[[1.0, 1.471363387541058e-43],\n", 411 | " [1.0, 0.0],\n", 412 | " [9.274744774610854e-37, 1.0],\n", 413 | " [1.0, 1.8135492388597416e-18],\n", 414 | " [6.925395368284626e-09, 1.0],\n", 415 | " [1.0, 0.0],\n", 416 | " [5.914161881561224e-18, 1.0],\n", 417 | " [1.0, 0.0],\n", 418 | " [9.36502665811189e-34, 1.0],\n", 419 | " [0.0, 1.0]]" 420 | ] 421 | }, 422 | "execution_count": 305, 423 | "metadata": {}, 424 | "output_type": "execute_result" 425 | } 426 | ], 427 | "source": [ 428 | "test_preds[0:10]" 429 | ] 430 | }, 431 | { 432 | "cell_type": "code", 433 | "execution_count": 306, 434 | "metadata": { 435 | "collapsed": false 436 | }, 437 | "outputs": [ 438 | { 439 | "data": { 440 | "text/plain": [ 441 | "[{'id': 1, 'label': 0.9999},\n", 442 | " {'id': 2, 'label': 0.9999},\n", 443 | " {'id': 3, 'label': 0.9999},\n", 444 | " {'id': 4, 'label': 0.9999},\n", 445 | " {'id': 5, 'label': 0.0001},\n", 446 | " {'id': 6, 'label': 0.0001},\n", 447 | " {'id': 7, 'label': 0.0001},\n", 448 | " {'id': 8, 'label': 0.0001},\n", 449 | " {'id': 9, 'label': 0.0001},\n", 450 | " {'id': 10, 'label': 0.0001},\n", 451 | " {'id': 11, 'label': 0.0001},\n", 452 | " {'id': 12, 'label': 0.9999},\n", 453 | " {'id': 13, 'label': 0.0001},\n", 454 | " {'id': 14, 'label': 0.0001},\n", 455 | " {'id': 15, 'label': 0.0001},\n", 456 | " {'id': 16, 'label': 0.0001},\n", 457 | " {'id': 17, 'label': 0.9999},\n", 458 | " {'id': 18, 'label': 0.9999}]" 459 | ] 460 | }, 461 | "execution_count": 306, 462 | "metadata": {}, 463 | "output_type": "execute_result" 464 | } 465 | ], 466 | "source": [ 467 | "Z0 = [{'id':int(f.split('/')[-1].split('.')[0]), 'label':min(max(round(p[0],5),0.0001),0.9999)} for f, p in zip(testfiles, test_preds)]\n", 468 | "Z1 = [{'id':int(f.split('/')[-1].split('.')[0]), 'label':min(max(round(p[1],5),0.0001),0.9999)} for f, p in zip(testfiles, test_preds)]\n", 469 | "def comp(x,y):\n", 470 | " return int(x['id']) - int(y['id'])\n", 471 | "Z0 = sorted(Z0, comp)\n", 472 | "Z1 = sorted(Z1, comp)\n", 473 | "Z1[0:18]" 474 | ] 475 | }, 476 | { 477 | "cell_type": "code", 478 | "execution_count": 307, 479 | "metadata": { 480 | "collapsed": true 481 | }, 482 | "outputs": [], 483 | "source": [ 484 | "import csv\n", 485 | "\n", 486 | "with open('predictions_0.csv', 'w') as csvfile:\n", 487 | " fieldnames = ['id', 'label']\n", 488 | " writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n", 489 | " writer.writeheader()\n", 490 | " for z in Z0:\n", 491 | " writer.writerow(z)\n", 492 | " \n", 493 | "with open('predictions_1.csv', 'w') as csvfile:\n", 494 | " fieldnames = ['id', 'label']\n", 495 | " writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n", 496 | " writer.writeheader()\n", 497 | " for z in Z1:\n", 498 | " writer.writerow(z)" 499 | ] 500 | }, 501 | { 502 | "cell_type": "code", 503 | "execution_count": null, 504 | "metadata": { 505 | "collapsed": true 506 | }, 507 | "outputs": [], 508 | "source": [] 509 | } 510 | ], 511 | "metadata": { 512 | "kernelspec": { 513 | "display_name": "Python 2", 514 | "language": "python", 515 | "name": "python2" 516 | }, 517 | "language_info": { 518 | "codemirror_mode": { 519 | "name": "ipython", 520 | "version": 2 521 | }, 522 | "file_extension": ".py", 523 | "mimetype": "text/x-python", 524 | "name": "python", 525 | "nbconvert_exporter": "python", 526 | "pygments_lexer": "ipython2", 527 | "version": "2.7.11" 528 | } 529 | }, 530 | "nbformat": 4, 531 | "nbformat_minor": 0 532 | } 533 | -------------------------------------------------------------------------------- /dogsandcats_keras/dogsandcats_v2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "name": "stderr", 12 | "output_type": "stream", 13 | "text": [ 14 | "Using gpu device 0: Tesla K80 (CNMeM is disabled)\n", 15 | "Using Theano backend.\n" 16 | ] 17 | } 18 | ], 19 | "source": [ 20 | "# Rather than importing everything manually, we'll make things easy\n", 21 | "# and load them all in utils.py, and just import them from there.\n", 22 | "%matplotlib inline\n", 23 | "import utils; reload(utils)\n", 24 | "from utils import *" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 2, 30 | "metadata": { 31 | "collapsed": true 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "%matplotlib inline\n", 36 | "from __future__ import division,print_function\n", 37 | "import os, json\n", 38 | "from glob import glob\n", 39 | "import numpy as np\n", 40 | "import scipy\n", 41 | "from sklearn.preprocessing import OneHotEncoder\n", 42 | "from sklearn.metrics import confusion_matrix\n", 43 | "np.set_printoptions(precision=4, linewidth=100)\n", 44 | "from matplotlib import pyplot as plt\n", 45 | "import utils; reload(utils)\n", 46 | "from utils import plots, get_batches, plot_confusion_matrix, get_data" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 3, 52 | "metadata": { 53 | "collapsed": true 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "from numpy.random import random, permutation\n", 58 | "from scipy import misc, ndimage\n", 59 | "from scipy.ndimage.interpolation import zoom\n", 60 | "\n", 61 | "import keras\n", 62 | "from keras import backend as K\n", 63 | "from keras.utils.data_utils import get_file\n", 64 | "from keras.models import Sequential\n", 65 | "from keras.layers import Input\n", 66 | "from keras.layers.core import Flatten, Dense, Dropout, Lambda\n", 67 | "from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D\n", 68 | "from keras.optimizers import SGD, RMSprop\n", 69 | "from keras.preprocessing import image" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 58, 75 | "metadata": { 76 | "collapsed": false 77 | }, 78 | "outputs": [ 79 | { 80 | "name": "stdout", 81 | "output_type": "stream", 82 | "text": [ 83 | "Done\n" 84 | ] 85 | } 86 | ], 87 | "source": [ 88 | "#path = \"../data/dogsandcats_small/\" # we copied a fraction of the full set for tests\n", 89 | "path = \"../data/dogsandcats/\"\n", 90 | "model_path = path + \"models/\"\n", 91 | "if not os.path.exists(model_path):\n", 92 | " os.mkdir(model_path)\n", 93 | " print('Done')" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 59, 99 | "metadata": { 100 | "collapsed": false 101 | }, 102 | "outputs": [], 103 | "source": [ 104 | "from vgg16 import Vgg16" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 60, 110 | "metadata": { 111 | "collapsed": true 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "batch_size = 100" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 61, 121 | "metadata": { 122 | "collapsed": false 123 | }, 124 | "outputs": [], 125 | "source": [ 126 | "def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, \n", 127 | " batch_size=batch_size, class_mode='categorical'):\n", 128 | " return gen.flow_from_directory(path+dirname, target_size=(224,224), \n", 129 | " class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": 62, 135 | "metadata": { 136 | "collapsed": false 137 | }, 138 | "outputs": [ 139 | { 140 | "name": "stdout", 141 | "output_type": "stream", 142 | "text": [ 143 | "Found 3000 images belonging to 2 classes.\n", 144 | "Found 3000 images belonging to 2 classes.\n" 145 | ] 146 | } 147 | ], 148 | "source": [ 149 | "# Use batch size of 1 since we're just doing preprocessing on the CPU\n", 150 | "val_batches = get_batches('valid', shuffle=False, batch_size=batch_size) # no shuffle as we store conv output\n", 151 | "trn_batches = get_batches('train', shuffle=False, batch_size=batch_size) # no shuffle as we store conv output" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 63, 157 | "metadata": { 158 | "collapsed": false 159 | }, 160 | "outputs": [ 161 | { 162 | "data": { 163 | "text/plain": [ 164 | "['cat/cat.3044.jpg',\n", 165 | " 'cat/cat.8847.jpg',\n", 166 | " 'cat/cat.308.jpg',\n", 167 | " 'cat/cat.10802.jpg',\n", 168 | " 'cat/cat.5060.jpg',\n", 169 | " 'cat/cat.10406.jpg',\n", 170 | " 'cat/cat.11054.jpg',\n", 171 | " 'cat/cat.380.jpg',\n", 172 | " 'cat/cat.6588.jpg',\n", 173 | " 'cat/cat.9693.jpg']" 174 | ] 175 | }, 176 | "execution_count": 63, 177 | "metadata": {}, 178 | "output_type": "execute_result" 179 | } 180 | ], 181 | "source": [ 182 | "val_batches.filenames[0:10]" 183 | ] 184 | }, 185 | { 186 | "cell_type": "code", 187 | "execution_count": 64, 188 | "metadata": { 189 | "collapsed": false 190 | }, 191 | "outputs": [], 192 | "source": [ 193 | "val_labels = onehot(val_batches.classes)\n", 194 | "trn_labels = onehot(trn_batches.classes)" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": 65, 200 | "metadata": { 201 | "collapsed": false 202 | }, 203 | "outputs": [], 204 | "source": [ 205 | "# DONT USE IT FOR NOW\n", 206 | "if False:\n", 207 | " realvgg = Vgg16()\n", 208 | " conv_layers, fc_layers = split_at(realvgg.model, Convolution2D)\n", 209 | " conv_model = Sequential(conv_layers)" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 66, 215 | "metadata": { 216 | "collapsed": false 217 | }, 218 | "outputs": [ 219 | { 220 | "name": "stderr", 221 | "output_type": "stream", 222 | "text": [ 223 | "/home/ubuntu/anaconda2/lib/python2.7/site-packages/keras/layers/core.py:622: UserWarning: `output_shape` argument not specified for layer lambda_2 and cannot be automatically inferred with the Theano backend. Defaulting to output shape `(None, 3, 224, 224)` (same as input shape). If the expected output shape is different, specify it via the `output_shape` argument.\n", 224 | " .format(self.name, input_shape))\n" 225 | ] 226 | } 227 | ], 228 | "source": [ 229 | "vggbase = Vgg16()\n", 230 | "vggbase.model.pop()\n", 231 | "vggbase.model.pop()" 232 | ] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "metadata": {}, 237 | "source": [ 238 | "### Will take 1 or 2 minutes to complete the 1st time" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": 67, 244 | "metadata": { 245 | "collapsed": false 246 | }, 247 | "outputs": [], 248 | "source": [ 249 | "# DONT USE IT FOR NOW\n", 250 | "if False:\n", 251 | " try:\n", 252 | " val_features = load_array(model_path+'valid_convlayer_features.bc')\n", 253 | " if False: # force update\n", 254 | " raise\n", 255 | " except:\n", 256 | " print('Missing file')\n", 257 | " val_features = conv_model.predict_generator(val_batches, val_batches.nb_sample)\n", 258 | " save_array(model_path + 'valid_convlayer_features.bc', val_features)" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": 68, 264 | "metadata": { 265 | "collapsed": false 266 | }, 267 | "outputs": [ 268 | { 269 | "name": "stdout", 270 | "output_type": "stream", 271 | "text": [ 272 | "Missing file\n" 273 | ] 274 | } 275 | ], 276 | "source": [ 277 | "try:\n", 278 | " val_vggfeatures = load_array(model_path+'valid_vggbase_features.bc')\n", 279 | " if False: # force update\n", 280 | " raise\n", 281 | "except:\n", 282 | " print('Missing file')\n", 283 | " val_vggfeatures = vggbase.model.predict_generator(val_batches, val_batches.nb_sample)\n", 284 | " save_array(model_path + 'valid_vggbase_features.bc', val_vggfeatures)" 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "metadata": {}, 290 | "source": [ 291 | "### Will take a few minutes (maybe 10) to complete the 1st time" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": 69, 297 | "metadata": { 298 | "collapsed": false 299 | }, 300 | "outputs": [], 301 | "source": [ 302 | "# DONT USE IT FOR NOW\n", 303 | "if False:\n", 304 | " try:\n", 305 | " trn_features = load_array(model_path+'train_convlayer_features.bc')\n", 306 | " if False: # force update\n", 307 | " raise\n", 308 | " except:\n", 309 | " print('Missing file')\n", 310 | " trn_features = conv_model.predict_generator(trn_batches, trn_batches.nb_sample)\n", 311 | " save_array(model_path + 'train_convlayer_features.bc', trn_features)" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": 70, 317 | "metadata": { 318 | "collapsed": false 319 | }, 320 | "outputs": [ 321 | { 322 | "name": "stdout", 323 | "output_type": "stream", 324 | "text": [ 325 | "Missing file\n" 326 | ] 327 | } 328 | ], 329 | "source": [ 330 | "try:\n", 331 | " trn_vggfeatures = load_array(model_path+'train_vggbase_features.bc')\n", 332 | " if False: # force update\n", 333 | " raise\n", 334 | "except:\n", 335 | " print('Missing file')\n", 336 | " trn_vggfeatures = vggbase.model.predict_generator(trn_batches, trn_batches.nb_sample)\n", 337 | " save_array(model_path + 'train_vggbase_features.bc', trn_vggfeatures)" 338 | ] 339 | }, 340 | { 341 | "cell_type": "markdown", 342 | "metadata": {}, 343 | "source": [ 344 | "### Ready to train the model" 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": 71, 350 | "metadata": { 351 | "collapsed": true 352 | }, 353 | "outputs": [], 354 | "source": [ 355 | "ll_layers = [BatchNormalization(input_shape=(4096,)),\n", 356 | " Dropout(0.25),\n", 357 | " Dense(2, activation='softmax')]\n", 358 | "ll_model = Sequential(ll_layers)\n", 359 | "ll_model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['accuracy'])" 360 | ] 361 | }, 362 | { 363 | "cell_type": "code", 364 | "execution_count": 110, 365 | "metadata": { 366 | "collapsed": false 367 | }, 368 | "outputs": [ 369 | { 370 | "name": "stdout", 371 | "output_type": "stream", 372 | "text": [ 373 | "Train on 3000 samples, validate on 3000 samples\n", 374 | "Epoch 1/5\n", 375 | "3000/3000 [==============================] - 0s - loss: 0.2221 - acc: 0.9093 - val_loss: 0.1989 - val_acc: 0.9270\n", 376 | "Epoch 2/5\n", 377 | "3000/3000 [==============================] - 0s - loss: 0.2220 - acc: 0.9137 - val_loss: 0.1984 - val_acc: 0.9280\n", 378 | "Epoch 3/5\n", 379 | "3000/3000 [==============================] - 0s - loss: 0.2248 - acc: 0.9103 - val_loss: 0.1984 - val_acc: 0.9277\n", 380 | "Epoch 4/5\n", 381 | "3000/3000 [==============================] - 0s - loss: 0.2263 - acc: 0.9137 - val_loss: 0.1978 - val_acc: 0.9267\n", 382 | "Epoch 5/5\n", 383 | "3000/3000 [==============================] - 0s - loss: 0.2266 - acc: 0.9087 - val_loss: 0.1979 - val_acc: 0.9263\n" 384 | ] 385 | }, 386 | { 387 | "data": { 388 | "text/plain": [ 389 | "" 390 | ] 391 | }, 392 | "execution_count": 110, 393 | "metadata": {}, 394 | "output_type": "execute_result" 395 | } 396 | ], 397 | "source": [ 398 | "ll_model.optimizer.lr = 10*1e-5\n", 399 | "ll_model.fit(trn_vggfeatures, trn_labels, validation_data=(val_vggfeatures, val_labels), nb_epoch=5)" 400 | ] 401 | }, 402 | { 403 | "cell_type": "code", 404 | "execution_count": 73, 405 | "metadata": { 406 | "collapsed": true 407 | }, 408 | "outputs": [], 409 | "source": [ 410 | "#ll_model.save_weights(model_path+'llmodel_finetune1.h5')\n", 411 | "#ll_model.load_weights(model_path+'llmodel_finetune1.h5')" 412 | ] 413 | }, 414 | { 415 | "cell_type": "code", 416 | "execution_count": 74, 417 | "metadata": { 418 | "collapsed": false 419 | }, 420 | "outputs": [ 421 | { 422 | "name": "stdout", 423 | "output_type": "stream", 424 | "text": [ 425 | "Found 1500 images belonging to 1 classes.\n" 426 | ] 427 | }, 428 | { 429 | "data": { 430 | "text/plain": [ 431 | "['test/10916.jpg',\n", 432 | " 'test/12374.jpg',\n", 433 | " 'test/1871.jpg',\n", 434 | " 'test/6164.jpg',\n", 435 | " 'test/3491.jpg',\n", 436 | " 'test/8027.jpg',\n", 437 | " 'test/1556.jpg',\n", 438 | " 'test/6560.jpg',\n", 439 | " 'test/8180.jpg',\n", 440 | " 'test/7451.jpg']" 441 | ] 442 | }, 443 | "execution_count": 74, 444 | "metadata": {}, 445 | "output_type": "execute_result" 446 | } 447 | ], 448 | "source": [ 449 | "test_batches = get_batches('test', shuffle=False, batch_size=batch_size, class_mode=None)\n", 450 | "testfiles = test_batches.filenames\n", 451 | "testfiles[0:10]" 452 | ] 453 | }, 454 | { 455 | "cell_type": "markdown", 456 | "metadata": {}, 457 | "source": [ 458 | "### Will take a few minutes (maybe 5) to complete the 1st time" 459 | ] 460 | }, 461 | { 462 | "cell_type": "code", 463 | "execution_count": 75, 464 | "metadata": { 465 | "collapsed": false 466 | }, 467 | "outputs": [ 468 | { 469 | "name": "stdout", 470 | "output_type": "stream", 471 | "text": [ 472 | "Missing file\n" 473 | ] 474 | } 475 | ], 476 | "source": [ 477 | "try:\n", 478 | " test_vggfeatures = load_array(model_path+'test_vggbase_features.bc')\n", 479 | " if False: # force update\n", 480 | " raise\n", 481 | "except:\n", 482 | " print('Missing file')\n", 483 | " test_vggfeatures = vggbase.model.predict_generator(test_batches, test_batches.nb_sample)\n", 484 | " save_array(model_path + 'test_vggbase_features.bc', test_vggfeatures)" 485 | ] 486 | }, 487 | { 488 | "cell_type": "code", 489 | "execution_count": 76, 490 | "metadata": { 491 | "collapsed": false 492 | }, 493 | "outputs": [], 494 | "source": [ 495 | "test_preds = ll_model.predict_on_batch(test_vggfeatures)" 496 | ] 497 | }, 498 | { 499 | "cell_type": "code", 500 | "execution_count": 79, 501 | "metadata": { 502 | "collapsed": false 503 | }, 504 | "outputs": [ 505 | { 506 | "ename": "AssertionError", 507 | "evalue": "", 508 | "output_type": "error", 509 | "traceback": [ 510 | "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", 511 | "\u001b[1;31mAssertionError\u001b[0m Traceback (most recent call last)", 512 | "\u001b[1;32m\u001b[0m in \u001b[0;36m\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32massert\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtest_preds\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m12500\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m", 513 | "\u001b[1;31mAssertionError\u001b[0m: " 514 | ] 515 | } 516 | ], 517 | "source": [ 518 | "assert(len(test_preds) == 12500)" 519 | ] 520 | }, 521 | { 522 | "cell_type": "code", 523 | "execution_count": 80, 524 | "metadata": { 525 | "collapsed": false 526 | }, 527 | "outputs": [ 528 | { 529 | "data": { 530 | "text/plain": [ 531 | "array([[ 0.453 , 0.547 ],\n", 532 | " [ 0.5199, 0.4801],\n", 533 | " [ 0.7576, 0.2424],\n", 534 | " [ 0.0066, 0.9934],\n", 535 | " [ 0.7282, 0.2718],\n", 536 | " [ 0.9167, 0.0833],\n", 537 | " [ 0.434 , 0.566 ],\n", 538 | " [ 0.9562, 0.0438],\n", 539 | " [ 0.8702, 0.1298],\n", 540 | " [ 0.3122, 0.6878]], dtype=float32)" 541 | ] 542 | }, 543 | "execution_count": 80, 544 | "metadata": {}, 545 | "output_type": "execute_result" 546 | } 547 | ], 548 | "source": [ 549 | "test_preds[0:10]" 550 | ] 551 | }, 552 | { 553 | "cell_type": "code", 554 | "execution_count": 44, 555 | "metadata": { 556 | "collapsed": false 557 | }, 558 | "outputs": [ 559 | { 560 | "data": { 561 | "text/plain": [ 562 | "[{'id': 1, 'label': 0.99986},\n", 563 | " {'id': 2, 'label': 0.9999},\n", 564 | " {'id': 3, 'label': 0.9999},\n", 565 | " {'id': 4, 'label': 0.99985},\n", 566 | " {'id': 5, 'label': 0.0001},\n", 567 | " {'id': 6, 'label': 0.00019},\n", 568 | " {'id': 7, 'label': 0.0001},\n", 569 | " {'id': 8, 'label': 0.0001},\n", 570 | " {'id': 9, 'label': 0.00055},\n", 571 | " {'id': 10, 'label': 0.0001},\n", 572 | " {'id': 11, 'label': 0.0001},\n", 573 | " {'id': 12, 'label': 0.99988},\n", 574 | " {'id': 13, 'label': 0.00523},\n", 575 | " {'id': 14, 'label': 0.00335},\n", 576 | " {'id': 15, 'label': 0.0001},\n", 577 | " {'id': 16, 'label': 0.00025},\n", 578 | " {'id': 17, 'label': 0.9506},\n", 579 | " {'id': 18, 'label': 0.9999}]" 580 | ] 581 | }, 582 | "execution_count": 44, 583 | "metadata": {}, 584 | "output_type": "execute_result" 585 | } 586 | ], 587 | "source": [ 588 | "dog_idx = 1\n", 589 | "Z1 = [{'id':int(f.split('/')[-1].split('.')[0]), 'label':min(max(round(p[dog_idx],5),0.0001),0.9999)} \n", 590 | " for f, p in zip(testfiles, test_preds)]\n", 591 | "def comp(x,y):\n", 592 | " return int(x['id']) - int(y['id'])\n", 593 | "Z1 = sorted(Z1, comp)\n", 594 | "Z1[0:18]" 595 | ] 596 | }, 597 | { 598 | "cell_type": "code", 599 | "execution_count": 45, 600 | "metadata": { 601 | "collapsed": true 602 | }, 603 | "outputs": [], 604 | "source": [ 605 | "import csv\n", 606 | " \n", 607 | "with open('predictions.csv', 'w') as csvfile:\n", 608 | " fieldnames = ['id', 'label']\n", 609 | " writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n", 610 | " writer.writeheader()\n", 611 | " for z in Z1:\n", 612 | " writer.writerow(z)" 613 | ] 614 | }, 615 | { 616 | "cell_type": "code", 617 | "execution_count": null, 618 | "metadata": { 619 | "collapsed": true 620 | }, 621 | "outputs": [], 622 | "source": [] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "execution_count": null, 627 | "metadata": { 628 | "collapsed": true 629 | }, 630 | "outputs": [], 631 | "source": [] 632 | } 633 | ], 634 | "metadata": { 635 | "kernelspec": { 636 | "display_name": "Python 2", 637 | "language": "python", 638 | "name": "python2" 639 | }, 640 | "language_info": { 641 | "codemirror_mode": { 642 | "name": "ipython", 643 | "version": 2 644 | }, 645 | "file_extension": ".py", 646 | "mimetype": "text/x-python", 647 | "name": "python", 648 | "nbconvert_exporter": "python", 649 | "pygments_lexer": "ipython2", 650 | "version": "2.7.11" 651 | } 652 | }, 653 | "nbformat": 4, 654 | "nbformat_minor": 0 655 | } 656 | -------------------------------------------------------------------------------- /autoencoder_keras/finance_autoencoder.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 86, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline\n", 12 | "import matplotlib.pyplot as plt" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 87, 18 | "metadata": { 19 | "collapsed": false 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "from keras.layers import Input, Dense, Convolution2D, MaxPooling2D, UpSampling2D\n", 24 | "from keras.models import Model\n", 25 | "from keras import regularizers" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 88, 31 | "metadata": { 32 | "collapsed": false 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "import numpy as np" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "### Reference :\n", 44 | "* Blog : building autoencoders in keras : https://blog.keras.io/building-autoencoders-in-keras.html" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "### Load market data from Quandl" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 89, 57 | "metadata": { 58 | "collapsed": false 59 | }, 60 | "outputs": [], 61 | "source": [ 62 | "import quandl # pip install quandl\n", 63 | "import pandas as pd" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 90, 69 | "metadata": { 70 | "collapsed": true 71 | }, 72 | "outputs": [], 73 | "source": [ 74 | "def qData(tick='XLU'):\n", 75 | " # GOOG/NYSE_XLU.4\n", 76 | " # WIKI/MSFT.4\n", 77 | " qtck = \"GOOG/NYSE_\"+tick+\".4\"\n", 78 | " return quandl.get(qtck,\n", 79 | " start_date=\"2003-01-01\",\n", 80 | " end_date=\"2016-12-31\",\n", 81 | " collapse=\"daily\")" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 91, 87 | "metadata": { 88 | "collapsed": true 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "'''TICKERS = ['MSFT','JPM','INTC','DOW','KO',\n", 93 | " 'MCD','CAT','WMT','MMM','AXP',\n", 94 | " 'BA','GE','XOM','PG','JNJ']'''\n", 95 | "TICKERS = ['XLU','XLF','XLK','XLY','XLV','XLB','XLE','XLP','XLI']" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 92, 101 | "metadata": { 102 | "collapsed": false 103 | }, 104 | "outputs": [], 105 | "source": [ 106 | "try:\n", 107 | " D.keys()\n", 108 | "except:\n", 109 | " print('create empty Quandl cache')\n", 110 | " D = {}\n", 111 | "\n", 112 | "for tckr in TICKERS:\n", 113 | " if not(tckr in D.keys()):\n", 114 | " print(tckr)\n", 115 | " qdt = qData(tckr)\n", 116 | " qdt.rename(columns={'Close': tckr}, inplace = True)\n", 117 | " D[tckr] = qdt\n", 118 | " \n", 119 | "for tck in D.keys():\n", 120 | " assert(D[tck].keys() == [tck])" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": 93, 126 | "metadata": { 127 | "collapsed": false 128 | }, 129 | "outputs": [ 130 | { 131 | "name": "stdout", 132 | "output_type": "stream", 133 | "text": [ 134 | "(3538, 1)\n", 135 | "(3538, 1)\n", 136 | "(3538, 1)\n", 137 | "(3538, 1)\n", 138 | "(3538, 1)\n", 139 | "(3538, 1)\n", 140 | "(3538, 1)\n", 141 | "(3538, 1)\n", 142 | "(3538, 1)\n" 143 | ] 144 | } 145 | ], 146 | "source": [ 147 | "for tck in D.keys():\n", 148 | " print(D[tck].shape)" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": 94, 154 | "metadata": { 155 | "collapsed": true 156 | }, 157 | "outputs": [], 158 | "source": [ 159 | "J = D[TICKERS[0]].join(D[TICKERS[1]])\n", 160 | "for tck in TICKERS[2:]:\n", 161 | " J = J.join(D[tck])" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 95, 167 | "metadata": { 168 | "collapsed": false 169 | }, 170 | "outputs": [ 171 | { 172 | "data": { 173 | "text/html": [ 174 | "
\n", 175 | "\n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | " \n", 183 | " \n", 184 | " \n", 185 | " \n", 186 | " \n", 187 | " \n", 188 | " \n", 189 | " \n", 190 | " \n", 191 | " \n", 192 | " \n", 193 | " \n", 194 | " \n", 195 | " \n", 196 | " \n", 197 | " \n", 198 | " \n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | " \n", 203 | " \n", 204 | " \n", 205 | " \n", 206 | " \n", 207 | " \n", 208 | " \n", 209 | " \n", 210 | " \n", 211 | " \n", 212 | " \n", 213 | " \n", 214 | " \n", 215 | " \n", 216 | " \n", 217 | " \n", 218 | " \n", 219 | " \n", 220 | " \n", 221 | " \n", 222 | " \n", 223 | " \n", 224 | " \n", 225 | " \n", 226 | " \n", 227 | " \n", 228 | " \n", 229 | " \n", 230 | " \n", 231 | " \n", 232 | " \n", 233 | " \n", 234 | " \n", 235 | " \n", 236 | " \n", 237 | " \n", 238 | " \n", 239 | " \n", 240 | " \n", 241 | " \n", 242 | " \n", 243 | " \n", 244 | " \n", 245 | " \n", 246 | " \n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | "
XLUXLFXLKXLYXLVXLBXLEXLPXLI
Date
2003-01-0219.6022.8015.6023.9827.2720.3622.8020.3221.12
2003-01-0319.8022.7815.6623.4327.5520.2522.7620.3021.19
2003-01-0620.6923.5516.3523.8627.8520.6422.9520.4521.38
2003-01-0720.2423.2916.5223.7327.4520.5722.2020.2721.26
2003-01-0820.3823.0515.9823.5127.2420.0421.9920.1521.00
\n", 265 | "
" 266 | ], 267 | "text/plain": [ 268 | " XLU XLF XLK XLY XLV XLB XLE XLP XLI\n", 269 | "Date \n", 270 | "2003-01-02 19.60 22.80 15.60 23.98 27.27 20.36 22.80 20.32 21.12\n", 271 | "2003-01-03 19.80 22.78 15.66 23.43 27.55 20.25 22.76 20.30 21.19\n", 272 | "2003-01-06 20.69 23.55 16.35 23.86 27.85 20.64 22.95 20.45 21.38\n", 273 | "2003-01-07 20.24 23.29 16.52 23.73 27.45 20.57 22.20 20.27 21.26\n", 274 | "2003-01-08 20.38 23.05 15.98 23.51 27.24 20.04 21.99 20.15 21.00" 275 | ] 276 | }, 277 | "execution_count": 95, 278 | "metadata": {}, 279 | "output_type": "execute_result" 280 | } 281 | ], 282 | "source": [ 283 | "J.head(5)" 284 | ] 285 | }, 286 | { 287 | "cell_type": "code", 288 | "execution_count": 96, 289 | "metadata": { 290 | "collapsed": false 291 | }, 292 | "outputs": [ 293 | { 294 | "data": { 295 | "text/plain": [ 296 | "XLU 0\n", 297 | "XLF 0\n", 298 | "XLK 0\n", 299 | "XLY 0\n", 300 | "XLV 0\n", 301 | "XLB 0\n", 302 | "XLE 0\n", 303 | "XLP 0\n", 304 | "XLI 0\n", 305 | "dtype: int64" 306 | ] 307 | }, 308 | "execution_count": 96, 309 | "metadata": {}, 310 | "output_type": "execute_result" 311 | } 312 | ], 313 | "source": [ 314 | "J.isnull().sum()" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 97, 320 | "metadata": { 321 | "collapsed": false 322 | }, 323 | "outputs": [ 324 | { 325 | "name": "stdout", 326 | "output_type": "stream", 327 | "text": [ 328 | "(3537, 9)\n", 329 | "(3537, 9)\n" 330 | ] 331 | } 332 | ], 333 | "source": [ 334 | "J2 = J.fillna(method='ffill')\n", 335 | "#J2[J['WMT'].isnull()]\n", 336 | "\n", 337 | "LogDiffJ = J2.apply(np.log).diff(periods=1, axis=0)\n", 338 | "LogDiffJ.drop(LogDiffJ.index[0:1], inplace=True)\n", 339 | "print LogDiffJ.shape\n", 340 | "\n", 341 | "MktData = LogDiffJ.as_matrix(columns=None) # as numpy.array\n", 342 | "print MktData.shape" 343 | ] 344 | }, 345 | { 346 | "cell_type": "code", 347 | "execution_count": 98, 348 | "metadata": { 349 | "collapsed": false 350 | }, 351 | "outputs": [], 352 | "source": [ 353 | "np.random.shuffle(MktData)\n", 354 | "split_index = 3000\n", 355 | "x_train = MktData[0:split_index,:]*100\n", 356 | "x_test = MktData[split_index:,:]*100" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 99, 362 | "metadata": { 363 | "collapsed": false 364 | }, 365 | "outputs": [ 366 | { 367 | "data": { 368 | "text/plain": [ 369 | "array([ 1.0982247 , 2.03701577, 1.29596141, 1.32517727, 1.04548284,\n", 370 | " 1.53840115, 1.77989192, 0.83855434, 1.3078465 ])" 371 | ] 372 | }, 373 | "execution_count": 99, 374 | "metadata": {}, 375 | "output_type": "execute_result" 376 | } 377 | ], 378 | "source": [ 379 | "np.std(x_train, axis=0)" 380 | ] 381 | }, 382 | { 383 | "cell_type": "markdown", 384 | "metadata": {}, 385 | "source": [ 386 | "## Linear auto-encoder : like PCA\n", 387 | "### We get a linear model by removing activation functions" 388 | ] 389 | }, 390 | { 391 | "cell_type": "code", 392 | "execution_count": 100, 393 | "metadata": { 394 | "collapsed": false 395 | }, 396 | "outputs": [], 397 | "source": [ 398 | "original_dim = 9\n", 399 | "\n", 400 | "# this is the size of our encoded representations\n", 401 | "encoding_dim = 3\n", 402 | "\n", 403 | "# this is our input placeholder\n", 404 | "input_data = Input(shape=(original_dim,))\n", 405 | "\n", 406 | "if True: # no sparsity constraint\n", 407 | " encoded = Dense(encoding_dim, activation=None)(input_data)\n", 408 | "else:\n", 409 | " encoded = Dense(encoding_dim, activation=None,\n", 410 | " activity_regularizer=regularizers.activity_l1(10e-5))(input_data)\n", 411 | "\n", 412 | "# \"decoded\" is the lossy reconstruction of the input\n", 413 | "decoded = Dense(original_dim, activation=None)(encoded)\n", 414 | "\n", 415 | "# this model maps an input to its reconstruction\n", 416 | "autoencoder = Model(inputs=input_data, outputs=decoded)\n", 417 | "\n", 418 | "# this model maps an input to its encoded representation\n", 419 | "encoder = Model(inputs=input_data, outputs=encoded)\n", 420 | "\n", 421 | "# create a placeholder for an encoded (32-dimensional) input\n", 422 | "encoded_input = Input(shape=(encoding_dim,))\n", 423 | "# retrieve the last layer of the autoencoder model\n", 424 | "decoder_layer = autoencoder.layers[-1]\n", 425 | "# create the decoder model\n", 426 | "decoder = Model(inputs=encoded_input, outputs=decoder_layer(encoded_input))" 427 | ] 428 | }, 429 | { 430 | "cell_type": "code", 431 | "execution_count": 101, 432 | "metadata": { 433 | "collapsed": false 434 | }, 435 | "outputs": [], 436 | "source": [ 437 | "# train autoencoder to reconstruct Stock returns\n", 438 | "# use L2 loss\n", 439 | "autoencoder.compile(optimizer='adadelta', loss='mean_squared_error')" 440 | ] 441 | }, 442 | { 443 | "cell_type": "code", 444 | "execution_count": 123, 445 | "metadata": { 446 | "collapsed": false 447 | }, 448 | "outputs": [ 449 | { 450 | "name": "stdout", 451 | "output_type": "stream", 452 | "text": [ 453 | "Train on 3000 samples, validate on 537 samples\n", 454 | "Epoch 1/50\n", 455 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 456 | "Epoch 2/50\n", 457 | "3000/3000 [==============================] - 0s - loss: 0.2340 - val_loss: 0.2279\n", 458 | "Epoch 3/50\n", 459 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 460 | "Epoch 4/50\n", 461 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 462 | "Epoch 5/50\n", 463 | "3000/3000 [==============================] - 0s - loss: 0.2340 - val_loss: 0.2279\n", 464 | "Epoch 6/50\n", 465 | "3000/3000 [==============================] - 0s - loss: 0.2340 - val_loss: 0.2279\n", 466 | "Epoch 7/50\n", 467 | "3000/3000 [==============================] - 0s - loss: 0.2340 - val_loss: 0.2280\n", 468 | "Epoch 8/50\n", 469 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 470 | "Epoch 9/50\n", 471 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 472 | "Epoch 10/50\n", 473 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 474 | "Epoch 11/50\n", 475 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 476 | "Epoch 12/50\n", 477 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 478 | "Epoch 13/50\n", 479 | "3000/3000 [==============================] - 0s - loss: 0.2340 - val_loss: 0.2279\n", 480 | "Epoch 14/50\n", 481 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 482 | "Epoch 15/50\n", 483 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 484 | "Epoch 16/50\n", 485 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 486 | "Epoch 17/50\n", 487 | "3000/3000 [==============================] - 0s - loss: 0.2340 - val_loss: 0.2279\n", 488 | "Epoch 18/50\n", 489 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 490 | "Epoch 19/50\n", 491 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 492 | "Epoch 20/50\n", 493 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 494 | "Epoch 21/50\n", 495 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 496 | "Epoch 22/50\n", 497 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2280\n", 498 | "Epoch 23/50\n", 499 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 500 | "Epoch 24/50\n", 501 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 502 | "Epoch 25/50\n", 503 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 504 | "Epoch 26/50\n", 505 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 506 | "Epoch 27/50\n", 507 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 508 | "Epoch 28/50\n", 509 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 510 | "Epoch 29/50\n", 511 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 512 | "Epoch 30/50\n", 513 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 514 | "Epoch 31/50\n", 515 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 516 | "Epoch 32/50\n", 517 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 518 | "Epoch 33/50\n", 519 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 520 | "Epoch 34/50\n", 521 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2277\n", 522 | "Epoch 35/50\n", 523 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2277\n", 524 | "Epoch 36/50\n", 525 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 526 | "Epoch 37/50\n", 527 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2279\n", 528 | "Epoch 38/50\n", 529 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 530 | "Epoch 39/50\n", 531 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2277\n", 532 | "Epoch 40/50\n", 533 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 534 | "Epoch 41/50\n", 535 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2277\n", 536 | "Epoch 42/50\n", 537 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2277\n", 538 | "Epoch 43/50\n", 539 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2277\n", 540 | "Epoch 44/50\n", 541 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 542 | "Epoch 45/50\n", 543 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 544 | "Epoch 46/50\n", 545 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 546 | "Epoch 47/50\n", 547 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 548 | "Epoch 48/50\n", 549 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n", 550 | "Epoch 49/50\n", 551 | "3000/3000 [==============================] - 0s - loss: 0.2338 - val_loss: 0.2277\n", 552 | "Epoch 50/50\n", 553 | "3000/3000 [==============================] - 0s - loss: 0.2339 - val_loss: 0.2278\n" 554 | ] 555 | }, 556 | { 557 | "data": { 558 | "text/plain": [ 559 | "" 560 | ] 561 | }, 562 | "execution_count": 123, 563 | "metadata": {}, 564 | "output_type": "execute_result" 565 | } 566 | ], 567 | "source": [ 568 | "autoencoder.fit(x_train, x_train,\n", 569 | " epochs=50,\n", 570 | " batch_size=128,\n", 571 | " shuffle=True,\n", 572 | " validation_data=(x_test, x_test))" 573 | ] 574 | }, 575 | { 576 | "cell_type": "code", 577 | "execution_count": 124, 578 | "metadata": { 579 | "collapsed": false 580 | }, 581 | "outputs": [], 582 | "source": [ 583 | "# encode and decode some digits\n", 584 | "# note that we take them from the *test* set\n", 585 | "encoded_data = encoder.predict(x_test)\n", 586 | "decoded_data = decoder.predict(encoded_data)" 587 | ] 588 | }, 589 | { 590 | "cell_type": "code", 591 | "execution_count": 125, 592 | "metadata": { 593 | "collapsed": false 594 | }, 595 | "outputs": [ 596 | { 597 | "name": "stdout", 598 | "output_type": "stream", 599 | "text": [ 600 | "0 0.769975033646\n", 601 | "1 0.99518747652\n", 602 | "2 0.901590510518\n", 603 | "3 0.947480846807\n", 604 | "4 0.82774934141\n", 605 | "5 0.900718119215\n", 606 | "6 0.989944764408\n", 607 | "7 0.854005961685\n", 608 | "8 0.930788316892\n" 609 | ] 610 | } 611 | ], 612 | "source": [ 613 | "for i in range(original_dim):\n", 614 | " print i, np.corrcoef(x_test[:,i].T, decoded_data[:,i].T)[0,1]" 615 | ] 616 | }, 617 | { 618 | "cell_type": "code", 619 | "execution_count": 126, 620 | "metadata": { 621 | "collapsed": false 622 | }, 623 | "outputs": [ 624 | { 625 | "name": "stdout", 626 | "output_type": "stream", 627 | "text": [ 628 | "0 0.11938556286\n", 629 | "1 -0.115034789311\n", 630 | "2 -0.0466444153257\n", 631 | "3 0.241456431721\n", 632 | "4 -0.140337077375\n", 633 | "5 -0.100318098446\n", 634 | "6 0.106878897453\n", 635 | "7 0.0370390714887\n", 636 | "8 -0.0743364310779\n" 637 | ] 638 | } 639 | ], 640 | "source": [ 641 | "decoding_error = x_test - decoded_data\n", 642 | "for i in range(original_dim):\n", 643 | " print i, np.corrcoef(decoded_data[:,i].T, decoding_error[:,i].T)[0,1]" 644 | ] 645 | }, 646 | { 647 | "cell_type": "code", 648 | "execution_count": null, 649 | "metadata": { 650 | "collapsed": true 651 | }, 652 | "outputs": [], 653 | "source": [] 654 | } 655 | ], 656 | "metadata": { 657 | "kernelspec": { 658 | "display_name": "Python 2", 659 | "language": "python", 660 | "name": "python2" 661 | }, 662 | "language_info": { 663 | "codemirror_mode": { 664 | "name": "ipython", 665 | "version": 2 666 | }, 667 | "file_extension": ".py", 668 | "mimetype": "text/x-python", 669 | "name": "python", 670 | "nbconvert_exporter": "python", 671 | "pygments_lexer": "ipython2", 672 | "version": "2.7.13" 673 | } 674 | }, 675 | "nbformat": 4, 676 | "nbformat_minor": 0 677 | } 678 | -------------------------------------------------------------------------------- /dogsandcats_keras/dogsandcats_v3.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "name": "stderr", 12 | "output_type": "stream", 13 | "text": [ 14 | "Using gpu device 0: Tesla K80 (CNMeM is disabled)\n", 15 | "Using Theano backend.\n" 16 | ] 17 | } 18 | ], 19 | "source": [ 20 | "# Rather than importing everything manually, we'll make things easy\n", 21 | "# and load them all in utils.py, and just import them from there.\n", 22 | "%matplotlib inline\n", 23 | "import utils; reload(utils)\n", 24 | "from utils import *" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 2, 30 | "metadata": { 31 | "collapsed": true 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "%matplotlib inline\n", 36 | "from __future__ import division,print_function\n", 37 | "import os, json\n", 38 | "from glob import glob\n", 39 | "import numpy as np\n", 40 | "import scipy\n", 41 | "from sklearn.preprocessing import OneHotEncoder\n", 42 | "from sklearn.metrics import confusion_matrix\n", 43 | "np.set_printoptions(precision=4, linewidth=100)\n", 44 | "from matplotlib import pyplot as plt\n", 45 | "import utils; reload(utils)\n", 46 | "from utils import plots, get_batches, plot_confusion_matrix, get_data" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 3, 52 | "metadata": { 53 | "collapsed": true 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "from numpy.random import random, permutation\n", 58 | "from scipy import misc, ndimage\n", 59 | "from scipy.ndimage.interpolation import zoom\n", 60 | "\n", 61 | "import keras\n", 62 | "from keras import backend as K\n", 63 | "from keras.utils.data_utils import get_file\n", 64 | "from keras.models import Sequential\n", 65 | "from keras.layers import Input\n", 66 | "from keras.layers.core import Flatten, Dense, Dropout, Lambda\n", 67 | "from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D\n", 68 | "from keras.optimizers import SGD, RMSprop\n", 69 | "from keras.preprocessing import image" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": { 76 | "collapsed": false 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "#path = \"../data/dogsandcats_small/\" # we copied a fraction of the full set for tests\n", 81 | "path = \"../data/dogsandcats/\"\n", 82 | "model_path = path + \"models/\"\n", 83 | "if not os.path.exists(model_path):\n", 84 | " os.mkdir(model_path)\n", 85 | " print('Done')" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 5, 91 | "metadata": { 92 | "collapsed": false 93 | }, 94 | "outputs": [], 95 | "source": [ 96 | "from vgg16 import Vgg16\n", 97 | "from resnet50 import Resnet50" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": 6, 103 | "metadata": { 104 | "collapsed": true 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "batch_size = 100" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 7, 114 | "metadata": { 115 | "collapsed": false 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, \n", 120 | " batch_size=batch_size, class_mode='categorical'):\n", 121 | " return gen.flow_from_directory(path+dirname, target_size=(224,224), \n", 122 | " class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 8, 128 | "metadata": { 129 | "collapsed": false 130 | }, 131 | "outputs": [ 132 | { 133 | "name": "stdout", 134 | "output_type": "stream", 135 | "text": [ 136 | "Found 4000 images belonging to 2 classes.\n", 137 | "Found 21000 images belonging to 2 classes.\n" 138 | ] 139 | } 140 | ], 141 | "source": [ 142 | "# Use batch size of 1 since we're just doing preprocessing on the CPU\n", 143 | "val_batches = get_batches('valid', shuffle=False, batch_size=batch_size) # no shuffle as we store conv output\n", 144 | "trn_batches = get_batches('train', shuffle=False, batch_size=batch_size) # no shuffle as we store conv output" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 9, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [ 154 | { 155 | "data": { 156 | "text/plain": [ 157 | "['cat/cat.1262.jpg',\n", 158 | " 'cat/cat.9495.jpg',\n", 159 | " 'cat/cat.3044.jpg',\n", 160 | " 'cat/cat.1424.jpg',\n", 161 | " 'cat/cat.8210.jpg',\n", 162 | " 'cat/cat.8847.jpg',\n", 163 | " 'cat/cat.308.jpg',\n", 164 | " 'cat/cat.10802.jpg',\n", 165 | " 'cat/cat.5060.jpg',\n", 166 | " 'cat/cat.10406.jpg']" 167 | ] 168 | }, 169 | "execution_count": 9, 170 | "metadata": {}, 171 | "output_type": "execute_result" 172 | } 173 | ], 174 | "source": [ 175 | "val_batches.filenames[0:10]" 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 10, 181 | "metadata": { 182 | "collapsed": false 183 | }, 184 | "outputs": [], 185 | "source": [ 186 | "val_labels = onehot(val_batches.classes)\n", 187 | "trn_labels = onehot(trn_batches.classes)" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": 11, 193 | "metadata": { 194 | "collapsed": false 195 | }, 196 | "outputs": [ 197 | { 198 | "data": { 199 | "text/plain": [ 200 | "'import hashlib\\ndef modelhash(mdl):\\n chaine = str(mdl.to_json())\\n return hashlib.md5(chaine).hexdigest()'" 201 | ] 202 | }, 203 | "execution_count": 11, 204 | "metadata": {}, 205 | "output_type": "execute_result" 206 | } 207 | ], 208 | "source": [ 209 | "'''import hashlib\n", 210 | "def modelhash(mdl):\n", 211 | " chaine = str(mdl.to_json())\n", 212 | " return hashlib.md5(chaine).hexdigest()'''\n", 213 | "# THE ABOVE FUNCTION DOES NOT WORK DUE TO LAYER DEFAULT NAMES" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": 12, 219 | "metadata": { 220 | "collapsed": false 221 | }, 222 | "outputs": [ 223 | { 224 | "name": "stderr", 225 | "output_type": "stream", 226 | "text": [ 227 | "/home/ubuntu/anaconda2/lib/python2.7/site-packages/keras/layers/core.py:622: UserWarning: `output_shape` argument not specified for layer lambda_1 and cannot be automatically inferred with the Theano backend. Defaulting to output shape `(None, 3, 224, 224)` (same as input shape). If the expected output shape is different, specify it via the `output_shape` argument.\n", 228 | " .format(self.name, input_shape))\n" 229 | ] 230 | } 231 | ], 232 | "source": [ 233 | "if True:\n", 234 | " realvgg = Vgg16()\n", 235 | " conv_layers, fc_layers = split_at(realvgg.model, Flatten)\n", 236 | " #conv_layers, fc_layers = split_at(realvgg.model, Convolution2D)\n", 237 | " conv_model = Sequential(conv_layers)\n", 238 | " conv_model_hash = 'conv_v3'" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": 13, 244 | "metadata": { 245 | "collapsed": false 246 | }, 247 | "outputs": [], 248 | "source": [ 249 | "if False:\n", 250 | " vggbase = Vgg16()\n", 251 | " vggbase.model.pop()\n", 252 | " vggbase.model.pop()\n", 253 | " vggbase_hash = 'vgg_v1'\n", 254 | " # optional extra pop\n", 255 | " if False:\n", 256 | " vggbase.model.pop()\n", 257 | " vggbase.model.pop()\n", 258 | " #vggbase_hash = modelhash(vggbase.model)\n", 259 | " vggbase_hash = 'vgg_v2'\n", 260 | " print(vggbase_hash)" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": 14, 266 | "metadata": { 267 | "collapsed": false 268 | }, 269 | "outputs": [], 270 | "source": [ 271 | "if False:\n", 272 | " resnetbase = Resnet50()\n", 273 | " resnetbase.model.layers.pop()\n", 274 | " for layer in resnetbase.model.layers:\n", 275 | " layer.trainable=False\n", 276 | " resnetbase.model = Model(resnetbase.model.input, resnetbase.model.layers[-1].output)\n", 277 | " resnetbase.model.compile(optimizer=RMSprop(lr=0.1), loss='categorical_crossentropy', metrics=['accuracy'])\n", 278 | " resnetbase_hash = 'resnet_v1'" 279 | ] 280 | }, 281 | { 282 | "cell_type": "markdown", 283 | "metadata": {}, 284 | "source": [ 285 | "### Will take a few minutes to complete the 1st time" 286 | ] 287 | }, 288 | { 289 | "cell_type": "code", 290 | "execution_count": 15, 291 | "metadata": { 292 | "collapsed": false 293 | }, 294 | "outputs": [], 295 | "source": [ 296 | "if True:\n", 297 | " try:\n", 298 | " val_convfeatures = load_array(model_path+'valid_'+conv_model_hash+'_features.bc')\n", 299 | " if False: # force update\n", 300 | " raise\n", 301 | " except:\n", 302 | " print('Missing file')\n", 303 | " val_convfeatures = conv_model.predict_generator(val_batches, val_batches.nb_sample)\n", 304 | " save_array(model_path+'valid_'+conv_model_hash+'_features.bc', val_convfeatures)" 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": 16, 310 | "metadata": { 311 | "collapsed": false 312 | }, 313 | "outputs": [], 314 | "source": [ 315 | "if False:\n", 316 | " try:\n", 317 | " val_vggfeatures = load_array(model_path+'valid_'+vggbase_hash+'_features.bc')\n", 318 | " if False: # force update\n", 319 | " raise\n", 320 | " except:\n", 321 | " print('Missing file')\n", 322 | " val_vggfeatures = vggbase.model.predict_generator(val_batches, val_batches.nb_sample)\n", 323 | " save_array(model_path+'valid_'+vggbase_hash+'_features.bc', val_vggfeatures)" 324 | ] 325 | }, 326 | { 327 | "cell_type": "code", 328 | "execution_count": 17, 329 | "metadata": { 330 | "collapsed": false 331 | }, 332 | "outputs": [], 333 | "source": [ 334 | "if False:\n", 335 | " try:\n", 336 | " val_resnetfeatures = load_array(model_path+'valid_'+resnetbase_hash+'_features.bc')\n", 337 | " if False: # force update\n", 338 | " raise\n", 339 | " except:\n", 340 | " print('Missing file')\n", 341 | " val_resnetfeatures = resnetbase.model.predict_generator(val_batches, val_batches.nb_sample)\n", 342 | " save_array(model_path+'valid_'+resnetbase_hash+'_features.bc', val_resnetfeatures)" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "### Will take a few minutes (maybe 10) to complete the 1st time" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": 18, 355 | "metadata": { 356 | "collapsed": false 357 | }, 358 | "outputs": [], 359 | "source": [ 360 | "if True:\n", 361 | " try:\n", 362 | " trn_convfeatures = load_array(model_path+'train_'+conv_model_hash+'_features.bc')\n", 363 | " if False: # force update\n", 364 | " raise\n", 365 | " except:\n", 366 | " print('Missing file')\n", 367 | " trn_convfeatures = conv_model.predict_generator(trn_batches, trn_batches.nb_sample)\n", 368 | " save_array(model_path+'train_'+conv_model_hash+'_features.bc', trn_convfeatures)" 369 | ] 370 | }, 371 | { 372 | "cell_type": "code", 373 | "execution_count": 19, 374 | "metadata": { 375 | "collapsed": false 376 | }, 377 | "outputs": [], 378 | "source": [ 379 | "if False:\n", 380 | " try:\n", 381 | " trn_vggfeatures = load_array(model_path+'train_'+vggbase_hash+'_features.bc')\n", 382 | " if False: # force update\n", 383 | " raise\n", 384 | " except:\n", 385 | " print('Missing file')\n", 386 | " trn_vggfeatures = vggbase.model.predict_generator(trn_batches, trn_batches.nb_sample)\n", 387 | " save_array(model_path+'train_'+vggbase_hash+'_features.bc', trn_vggfeatures)" 388 | ] 389 | }, 390 | { 391 | "cell_type": "code", 392 | "execution_count": 20, 393 | "metadata": { 394 | "collapsed": false 395 | }, 396 | "outputs": [], 397 | "source": [ 398 | "if False:\n", 399 | " try:\n", 400 | " trn_resnetfeatures = load_array(model_path+'train_'+resnetbase_hash+'_features.bc')\n", 401 | " if False: # force update\n", 402 | " raise\n", 403 | " except:\n", 404 | " print('Missing file')\n", 405 | " trn_resnetfeatures = resnetbase.model.predict_generator(trn_batches, trn_batches.nb_sample)\n", 406 | " save_array(model_path+'train_'+resnetbase_hash+'_features.bc', trn_resnetfeatures)" 407 | ] 408 | }, 409 | { 410 | "cell_type": "markdown", 411 | "metadata": {}, 412 | "source": [ 413 | "### Ready to train the model" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 30, 419 | "metadata": { 420 | "collapsed": false 421 | }, 422 | "outputs": [], 423 | "source": [ 424 | "# see : https://github.com/fastai/courses/blob/master/deeplearning1/nbs/lesson3.ipynb\n", 425 | "\n", 426 | "def proc_wgts(layer, ndo):\n", 427 | " # copy the weights from the pre-trained model\n", 428 | " # original weights are for a 50% drop out\n", 429 | " # we infer the corresponding weight for a new drop out (ndo) level\n", 430 | " return [w*0.5/(1.-ndo) for w in layer.get_weights()]\n", 431 | "\n", 432 | "def get_fc_model(ndo):\n", 433 | " model = Sequential([\n", 434 | " Dense(4096, activation='relu', input_shape=conv_layers[-1].output_shape[1:]),\n", 435 | " Dropout(ndo),\n", 436 | " Dense(4096, activation='relu'),\n", 437 | " #BatchNormalization(),\n", 438 | " Dropout(ndo),\n", 439 | " #BatchNormalization(),\n", 440 | " Dense(2, activation='softmax')\n", 441 | " ])\n", 442 | "\n", 443 | " for l_new, l_orig in zip(model.layers[0:3], fc_layers[0:3]):\n", 444 | " assert (type(l_new) == type(l_orig))\n", 445 | " l_new.set_weights(proc_wgts(l_orig, ndo))\n", 446 | " \n", 447 | " for layer in model.layers[:-1]:\n", 448 | " layer.trainable = False\n", 449 | " \n", 450 | " model.layers[-1].trainable = True\n", 451 | " \n", 452 | " #opt = RMSprop(lr=0.00001, rho=0.7)\n", 453 | " opt = Adam()\n", 454 | " model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n", 455 | " return model" 456 | ] 457 | }, 458 | { 459 | "cell_type": "markdown", 460 | "metadata": {}, 461 | "source": [ 462 | "### Train one or several models (ensembling)" 463 | ] 464 | }, 465 | { 466 | "cell_type": "code", 467 | "execution_count": 62, 468 | "metadata": { 469 | "collapsed": false 470 | }, 471 | "outputs": [ 472 | { 473 | "name": "stdout", 474 | "output_type": "stream", 475 | "text": [ 476 | "Train on 21000 samples, validate on 4000 samples\n", 477 | "Epoch 1/10\n", 478 | "21000/21000 [==============================] - 9s - loss: 0.0532 - acc: 0.9798 - val_loss: 0.0442 - val_acc: 0.9830\n", 479 | "Epoch 2/10\n", 480 | "21000/21000 [==============================] - 9s - loss: 0.0398 - acc: 0.9850 - val_loss: 0.0420 - val_acc: 0.9845\n", 481 | "Epoch 3/10\n", 482 | "21000/21000 [==============================] - 9s - loss: 0.0311 - acc: 0.9883 - val_loss: 0.0791 - val_acc: 0.9705\n", 483 | "Epoch 4/10\n", 484 | "21000/21000 [==============================] - 9s - loss: 0.0290 - acc: 0.9894 - val_loss: 0.0522 - val_acc: 0.9830\n", 485 | "Epoch 5/10\n", 486 | "21000/21000 [==============================] - 9s - loss: 0.0251 - acc: 0.9914 - val_loss: 0.0483 - val_acc: 0.9852\n", 487 | "Epoch 6/10\n", 488 | "21000/21000 [==============================] - 9s - loss: 0.0210 - acc: 0.9920 - val_loss: 0.0489 - val_acc: 0.9852\n", 489 | "Epoch 7/10\n", 490 | "21000/21000 [==============================] - 9s - loss: 0.0189 - acc: 0.9932 - val_loss: 0.0458 - val_acc: 0.9838\n", 491 | "Epoch 8/10\n", 492 | "21000/21000 [==============================] - 9s - loss: 0.0161 - acc: 0.9936 - val_loss: 0.0501 - val_acc: 0.9850\n", 493 | "Epoch 9/10\n", 494 | "21000/21000 [==============================] - 9s - loss: 0.0159 - acc: 0.9939 - val_loss: 0.0549 - val_acc: 0.9850\n", 495 | "Epoch 10/10\n", 496 | "21000/21000 [==============================] - 9s - loss: 0.0147 - acc: 0.9947 - val_loss: 0.0569 - val_acc: 0.9855\n" 497 | ] 498 | } 499 | ], 500 | "source": [ 501 | "ll_models = []\n", 502 | "for i in range(1): # INFO : change here the size of the ensemble\n", 503 | " ll_models.append( get_fc_model(0.1) )\n", 504 | " #ll_models[-1].optimizer.lr = 1*1e-5\n", 505 | " ll_models[-1].fit(trn_convfeatures, trn_labels, validation_data=(val_convfeatures, val_labels), nb_epoch=10)" 506 | ] 507 | }, 508 | { 509 | "cell_type": "code", 510 | "execution_count": 63, 511 | "metadata": { 512 | "collapsed": false 513 | }, 514 | "outputs": [ 515 | { 516 | "data": { 517 | "text/plain": [ 518 | "array(0.9854999780654907, dtype=float32)" 519 | ] 520 | }, 521 | "execution_count": 63, 522 | "metadata": {}, 523 | "output_type": "execute_result" 524 | } 525 | ], 526 | "source": [ 527 | "all_val_preds = []\n", 528 | "for mdl in ll_models:\n", 529 | " these_val_preds = mdl.predict_on_batch(val_convfeatures)\n", 530 | " assert(len(these_val_preds) == 4000)\n", 531 | " all_val_preds.append( these_val_preds )\n", 532 | "mean_val_preds = np.stack(all_val_preds).mean(axis=0)\n", 533 | "categorical_accuracy(val_labels, mean_val_preds).eval()" 534 | ] 535 | }, 536 | { 537 | "cell_type": "code", 538 | "execution_count": 46, 539 | "metadata": { 540 | "collapsed": true 541 | }, 542 | "outputs": [], 543 | "source": [ 544 | "# WARNING : should save each model of the ensemble\n", 545 | "#ll_model.save_weights(model_path+'llmodel_finetune1.h5')\n", 546 | "#ll_model.load_weights(model_path+'llmodel_finetune1.h5')" 547 | ] 548 | }, 549 | { 550 | "cell_type": "code", 551 | "execution_count": 33, 552 | "metadata": { 553 | "collapsed": false 554 | }, 555 | "outputs": [ 556 | { 557 | "name": "stdout", 558 | "output_type": "stream", 559 | "text": [ 560 | "Found 12500 images belonging to 1 classes.\n" 561 | ] 562 | }, 563 | { 564 | "data": { 565 | "text/plain": [ 566 | "['test/10592.jpg',\n", 567 | " 'test/7217.jpg',\n", 568 | " 'test/3653.jpg',\n", 569 | " 'test/4382.jpg',\n", 570 | " 'test/2924.jpg',\n", 571 | " 'test/10.jpg',\n", 572 | " 'test/10916.jpg',\n", 573 | " 'test/12374.jpg',\n", 574 | " 'test/1871.jpg',\n", 575 | " 'test/11645.jpg']" 576 | ] 577 | }, 578 | "execution_count": 33, 579 | "metadata": {}, 580 | "output_type": "execute_result" 581 | } 582 | ], 583 | "source": [ 584 | "test_batches = get_batches('test', shuffle=False, batch_size=batch_size, class_mode=None)\n", 585 | "testfiles = test_batches.filenames\n", 586 | "testfiles[0:10]" 587 | ] 588 | }, 589 | { 590 | "cell_type": "markdown", 591 | "metadata": {}, 592 | "source": [ 593 | "### Will take a few minutes (maybe 5) to complete the 1st time" 594 | ] 595 | }, 596 | { 597 | "cell_type": "code", 598 | "execution_count": 34, 599 | "metadata": { 600 | "collapsed": false 601 | }, 602 | "outputs": [ 603 | { 604 | "name": "stdout", 605 | "output_type": "stream", 606 | "text": [ 607 | "Missing file\n" 608 | ] 609 | } 610 | ], 611 | "source": [ 612 | "try:\n", 613 | " test_convfeatures = load_array(model_path+'test_'+conv_model_hash+'_features.bc')\n", 614 | " #test_vggfeatures = load_array(model_path+'test_vggbase_features.bc')\n", 615 | " if False: # force update\n", 616 | " raise\n", 617 | "except:\n", 618 | " print('Missing file')\n", 619 | " test_convfeatures = conv_model.predict_generator(test_batches, test_batches.nb_sample)\n", 620 | " save_array(model_path+'test_'+conv_model_hash+'_features.bc', test_convfeatures)\n", 621 | " #test_vggfeatures = vggbase.model.predict_generator(test_batches, test_batches.nb_sample)\n", 622 | " #save_array(model_path + 'test_vggbase_features.bc', test_vggfeatures)" 623 | ] 624 | }, 625 | { 626 | "cell_type": "code", 627 | "execution_count": 64, 628 | "metadata": { 629 | "collapsed": false 630 | }, 631 | "outputs": [], 632 | "source": [ 633 | "all_test_preds = []\n", 634 | "for mdl in ll_models:\n", 635 | " these_test_preds = mdl.predict_on_batch(test_convfeatures)\n", 636 | " assert(len(these_test_preds) == 12500)\n", 637 | " all_test_preds.append( these_test_preds )\n", 638 | "mean_test_preds = np.stack(all_test_preds).mean(axis=0)" 639 | ] 640 | }, 641 | { 642 | "cell_type": "code", 643 | "execution_count": 67, 644 | "metadata": { 645 | "collapsed": false 646 | }, 647 | "outputs": [ 648 | { 649 | "data": { 650 | "text/plain": [ 651 | "array([[ 1.0000e+00, 2.9676e-10],\n", 652 | " [ 1.0000e+00, 3.1055e-09],\n", 653 | " [ 8.6116e-09, 1.0000e+00],\n", 654 | " [ 1.0000e+00, 1.5027e-06],\n", 655 | " [ 1.1490e-03, 9.9885e-01],\n", 656 | " [ 1.0000e+00, 1.1879e-09],\n", 657 | " [ 2.2946e-04, 9.9977e-01],\n", 658 | " [ 1.0000e+00, 1.2871e-14],\n", 659 | " [ 1.6734e-06, 1.0000e+00],\n", 660 | " [ 1.2401e-08, 1.0000e+00]], dtype=float32)" 661 | ] 662 | }, 663 | "execution_count": 67, 664 | "metadata": {}, 665 | "output_type": "execute_result" 666 | } 667 | ], 668 | "source": [ 669 | "mean_test_preds[0:10]" 670 | ] 671 | }, 672 | { 673 | "cell_type": "code", 674 | "execution_count": 57, 675 | "metadata": { 676 | "collapsed": false 677 | }, 678 | "outputs": [ 679 | { 680 | "data": { 681 | "text/plain": [ 682 | "[{'id': 1, 'label': 0.9999},\n", 683 | " {'id': 2, 'label': 0.9999},\n", 684 | " {'id': 3, 'label': 0.9999},\n", 685 | " {'id': 4, 'label': 0.9999},\n", 686 | " {'id': 5, 'label': 0.0001},\n", 687 | " {'id': 6, 'label': 0.0001},\n", 688 | " {'id': 7, 'label': 0.0001},\n", 689 | " {'id': 8, 'label': 0.0001},\n", 690 | " {'id': 9, 'label': 0.0001},\n", 691 | " {'id': 10, 'label': 0.0001},\n", 692 | " {'id': 11, 'label': 0.0001},\n", 693 | " {'id': 12, 'label': 0.9999},\n", 694 | " {'id': 13, 'label': 0.0001},\n", 695 | " {'id': 14, 'label': 0.013},\n", 696 | " {'id': 15, 'label': 0.0001},\n", 697 | " {'id': 16, 'label': 0.0001},\n", 698 | " {'id': 17, 'label': 0.9999},\n", 699 | " {'id': 18, 'label': 0.9999}]" 700 | ] 701 | }, 702 | "execution_count": 57, 703 | "metadata": {}, 704 | "output_type": "execute_result" 705 | } 706 | ], 707 | "source": [ 708 | "dog_idx = 1\n", 709 | "Z1 = [{'id':int(f.split('/')[-1].split('.')[0]), 'label':min(max(round(p[dog_idx],4),0.0001),0.9999)} \n", 710 | " for f, p in zip(testfiles, mean_test_preds)]\n", 711 | "def comp(x,y):\n", 712 | " return int(x['id']) - int(y['id'])\n", 713 | "Z1 = sorted(Z1, comp)\n", 714 | "Z1[0:18]" 715 | ] 716 | }, 717 | { 718 | "cell_type": "code", 719 | "execution_count": 58, 720 | "metadata": { 721 | "collapsed": true 722 | }, 723 | "outputs": [], 724 | "source": [ 725 | "import csv\n", 726 | "\n", 727 | "with open('predictions.csv', 'w') as csvfile:\n", 728 | " fieldnames = ['id', 'label']\n", 729 | " writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n", 730 | " writer.writeheader()\n", 731 | " for z in Z1:\n", 732 | " writer.writerow(z)" 733 | ] 734 | }, 735 | { 736 | "cell_type": "code", 737 | "execution_count": null, 738 | "metadata": { 739 | "collapsed": true 740 | }, 741 | "outputs": [], 742 | "source": [] 743 | } 744 | ], 745 | "metadata": { 746 | "kernelspec": { 747 | "display_name": "Python 2", 748 | "language": "python", 749 | "name": "python2" 750 | }, 751 | "language_info": { 752 | "codemirror_mode": { 753 | "name": "ipython", 754 | "version": 2 755 | }, 756 | "file_extension": ".py", 757 | "mimetype": "text/x-python", 758 | "name": "python", 759 | "nbconvert_exporter": "python", 760 | "pygments_lexer": "ipython2", 761 | "version": "2.7.11" 762 | } 763 | }, 764 | "nbformat": 4, 765 | "nbformat_minor": 0 766 | } 767 | -------------------------------------------------------------------------------- /dogsandcats_keras/dogsandcats_v4.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "name": "stderr", 12 | "output_type": "stream", 13 | "text": [ 14 | "Using gpu device 0: Tesla K80 (CNMeM is disabled)\n", 15 | "Using Theano backend.\n" 16 | ] 17 | } 18 | ], 19 | "source": [ 20 | "# Rather than importing everything manually, we'll make things easy\n", 21 | "# and load them all in utils.py, and just import them from there.\n", 22 | "%matplotlib inline\n", 23 | "import utils; reload(utils)\n", 24 | "from utils import *" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 2, 30 | "metadata": { 31 | "collapsed": true 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "%matplotlib inline\n", 36 | "from __future__ import division,print_function\n", 37 | "import os, json\n", 38 | "from glob import glob\n", 39 | "import numpy as np\n", 40 | "import scipy\n", 41 | "from sklearn.preprocessing import OneHotEncoder\n", 42 | "from sklearn.metrics import confusion_matrix\n", 43 | "np.set_printoptions(precision=4, linewidth=100)\n", 44 | "from matplotlib import pyplot as plt\n", 45 | "import utils; reload(utils)\n", 46 | "from utils import plots, get_batches, plot_confusion_matrix, get_data" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 3, 52 | "metadata": { 53 | "collapsed": true 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "from numpy.random import random, permutation\n", 58 | "from scipy import misc, ndimage\n", 59 | "from scipy.ndimage.interpolation import zoom\n", 60 | "\n", 61 | "import keras\n", 62 | "from keras import backend as K\n", 63 | "from keras.utils.data_utils import get_file\n", 64 | "from keras.models import Sequential\n", 65 | "from keras.layers import Input\n", 66 | "from keras.layers.core import Flatten, Dense, Dropout, Lambda\n", 67 | "from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D\n", 68 | "from keras.optimizers import SGD, RMSprop\n", 69 | "from keras.preprocessing import image" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": { 76 | "collapsed": false 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "#path = \"../data/dogsandcats_small/\" # we copied a fraction of the full set for tests\n", 81 | "path = \"../data/dogsandcats/\"\n", 82 | "model_path = path + \"models/\"\n", 83 | "if not os.path.exists(model_path):\n", 84 | " os.mkdir(model_path)\n", 85 | " print('Done')" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 6, 91 | "metadata": { 92 | "collapsed": false 93 | }, 94 | "outputs": [], 95 | "source": [ 96 | "from vgg16 import Vgg16" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 7, 102 | "metadata": { 103 | "collapsed": true 104 | }, 105 | "outputs": [], 106 | "source": [ 107 | "batch_size = 100" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 8, 113 | "metadata": { 114 | "collapsed": false 115 | }, 116 | "outputs": [], 117 | "source": [ 118 | "def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, \n", 119 | " batch_size=batch_size, class_mode='categorical'):\n", 120 | " return gen.flow_from_directory(path+dirname, target_size=(224,224), \n", 121 | " class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 9, 127 | "metadata": { 128 | "collapsed": false 129 | }, 130 | "outputs": [ 131 | { 132 | "name": "stdout", 133 | "output_type": "stream", 134 | "text": [ 135 | "Found 4000 images belonging to 2 classes.\n", 136 | "Found 21000 images belonging to 2 classes.\n" 137 | ] 138 | } 139 | ], 140 | "source": [ 141 | "# Use batch size of 1 since we're just doing preprocessing on the CPU\n", 142 | "val_batches = get_batches('valid', shuffle=False, batch_size=batch_size) # no shuffle as we store conv output\n", 143 | "trn_batches = get_batches('train', shuffle=False, batch_size=batch_size) # no shuffle as we store conv output" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": 10, 149 | "metadata": { 150 | "collapsed": false 151 | }, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "['cat/cat.1262.jpg',\n", 157 | " 'cat/cat.9495.jpg',\n", 158 | " 'cat/cat.3044.jpg',\n", 159 | " 'cat/cat.1424.jpg',\n", 160 | " 'cat/cat.8210.jpg',\n", 161 | " 'cat/cat.8847.jpg',\n", 162 | " 'cat/cat.308.jpg',\n", 163 | " 'cat/cat.10802.jpg',\n", 164 | " 'cat/cat.5060.jpg',\n", 165 | " 'cat/cat.10406.jpg']" 166 | ] 167 | }, 168 | "execution_count": 10, 169 | "metadata": {}, 170 | "output_type": "execute_result" 171 | } 172 | ], 173 | "source": [ 174 | "val_batches.filenames[0:10]" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 11, 180 | "metadata": { 181 | "collapsed": false 182 | }, 183 | "outputs": [], 184 | "source": [ 185 | "val_labels = onehot(val_batches.classes)\n", 186 | "trn_labels = onehot(trn_batches.classes)" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": 12, 192 | "metadata": { 193 | "collapsed": false 194 | }, 195 | "outputs": [], 196 | "source": [ 197 | "'''try:\n", 198 | " trn = load_array(model_path+'train_data.bc')\n", 199 | "except:\n", 200 | " trn = get_data(path+'train')\n", 201 | " save_array(model_path+'train_data.bc', trn)'''" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": 13, 207 | "metadata": { 208 | "collapsed": false 209 | }, 210 | "outputs": [], 211 | "source": [ 212 | "'''try:\n", 213 | " val = load_array(model_path+'valid_data.bc')\n", 214 | "except:\n", 215 | " val = get_data(path+'valid')\n", 216 | " save_array(model_path+'valid_data.bc', val)'''" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": 14, 222 | "metadata": { 223 | "collapsed": false 224 | }, 225 | "outputs": [], 226 | "source": [ 227 | "'''gen = image.ImageDataGenerator(rotation_range=10, width_shift_range=0.05, \n", 228 | " zoom_range=0.05,\n", 229 | " #channel_shift_range=10,\n", 230 | " height_shift_range=0.05, shear_range=0.05, horizontal_flip=False)\n", 231 | "trn_batchesRND = gen.flow(trn, trn_labels, batch_size=batch_size)\n", 232 | "val_batchesRND = gen.flow(val, val_labels, batch_size=batch_size)'''" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": 13, 238 | "metadata": { 239 | "collapsed": false 240 | }, 241 | "outputs": [ 242 | { 243 | "name": "stderr", 244 | "output_type": "stream", 245 | "text": [ 246 | "/home/ubuntu/anaconda2/lib/python2.7/site-packages/keras/layers/core.py:622: UserWarning: `output_shape` argument not specified for layer lambda_2 and cannot be automatically inferred with the Theano backend. Defaulting to output shape `(None, 3, 224, 224)` (same as input shape). If the expected output shape is different, specify it via the `output_shape` argument.\n", 247 | " .format(self.name, input_shape))\n" 248 | ] 249 | } 250 | ], 251 | "source": [ 252 | "if True:\n", 253 | " realvgg = Vgg16()\n", 254 | " conv_layers, fc_layers = split_at(realvgg.model, Flatten)\n", 255 | " #conv_layers, fc_layers = split_at(realvgg.model, Convolution2D)\n", 256 | " conv_model = Sequential(conv_layers)\n", 257 | " conv_model_hash = 'conv_v3'" 258 | ] 259 | }, 260 | { 261 | "cell_type": "markdown", 262 | "metadata": {}, 263 | "source": [ 264 | "### Will take a few minutes to complete the 1st time" 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": 14, 270 | "metadata": { 271 | "collapsed": false 272 | }, 273 | "outputs": [], 274 | "source": [ 275 | "if True:\n", 276 | " try:\n", 277 | " val_convfeatures = load_array(model_path+'valid_'+conv_model_hash+'_features.bc')\n", 278 | " if False: # force update\n", 279 | " raise\n", 280 | " except:\n", 281 | " print('Missing file')\n", 282 | " val_convfeatures = conv_model.predict_generator(val_batches, val_batches.nb_sample)\n", 283 | " save_array(model_path+'valid_'+conv_model_hash+'_features.bc', val_convfeatures)" 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "metadata": {}, 289 | "source": [ 290 | "### Will take a few minutes (maybe 10) to complete the 1st time" 291 | ] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "execution_count": 15, 296 | "metadata": { 297 | "collapsed": false 298 | }, 299 | "outputs": [], 300 | "source": [ 301 | "if True:\n", 302 | " try:\n", 303 | " trn_convfeatures = load_array(model_path+'train_'+conv_model_hash+'_features.bc')\n", 304 | " if False: # force update\n", 305 | " raise\n", 306 | " except:\n", 307 | " print('Missing file')\n", 308 | " trn_convfeatures = conv_model.predict_generator(trn_batches, trn_batches.nb_sample)\n", 309 | " save_array(model_path+'train_'+conv_model_hash+'_features.bc', trn_convfeatures)" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "### Ready to train the model\n", 317 | "#### We use VGG top layers but we insert BatchNorm layers\n", 318 | "#### BatchNorm layers needs to be initialized properly so we first estimate\n", 319 | "#### the mean/var of the layers feeding into them" 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "execution_count": 18, 325 | "metadata": { 326 | "collapsed": false 327 | }, 328 | "outputs": [], 329 | "source": [ 330 | "# see : https://github.com/fastai/courses/blob/master/deeplearning1/nbs/lesson3.ipynb\n", 331 | "\n", 332 | "def proc_wgts(layer, ndo):\n", 333 | " # copy the weights from the pre-trained model\n", 334 | " # original weights are for a 50% drop out\n", 335 | " # we infer the corresponding weight for a new drop out (ndo) level\n", 336 | " return [w*0.5/(1.-ndo) for w in layer.get_weights()]\n", 337 | "\n", 338 | "def get_fc_model(ndo):\n", 339 | " model = Sequential([\n", 340 | " Dense(4096, activation='relu', input_shape=conv_layers[-1].output_shape[1:]),\n", 341 | " Dropout(ndo),\n", 342 | " Dense(4096, activation='relu'),\n", 343 | " Dropout(ndo),\n", 344 | " Dense(2, activation='softmax')\n", 345 | " ])\n", 346 | "\n", 347 | " for l_new, l_orig in zip(model.layers[0:3], fc_layers[0:3]):\n", 348 | " assert (type(l_new) == type(l_orig))\n", 349 | " l_new.set_weights(proc_wgts(l_orig, ndo))\n", 350 | " \n", 351 | " for layer in model.layers[:-1]:\n", 352 | " layer.trainable = False\n", 353 | " \n", 354 | " model.layers[-1].trainable = True\n", 355 | " \n", 356 | " #opt = RMSprop(lr=0.00001, rho=0.7)\n", 357 | " opt = Adam()\n", 358 | " model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n", 359 | " return model" 360 | ] 361 | }, 362 | { 363 | "cell_type": "code", 364 | "execution_count": 62, 365 | "metadata": { 366 | "collapsed": true 367 | }, 368 | "outputs": [], 369 | "source": [ 370 | "def get_bn_model(p):\n", 371 | " dense_model = get_fc_model(p)\n", 372 | "\n", 373 | " k_layer_out0 = K.function([dense_model.layers[0].input, K.learning_phase()],\n", 374 | " [dense_model.layers[0].output])\n", 375 | " d0_out = k_layer_out0([trn_convfeatures, 0])[0]\n", 376 | " mu0, var0 = d0_out.mean(axis=0), d0_out.var(axis=0)\n", 377 | "\n", 378 | "\n", 379 | " k_layer_out2 = K.function([dense_model.layers[0].input, K.learning_phase()],\n", 380 | " [dense_model.layers[2].output])\n", 381 | " d2_out = k_layer_out2([trn_convfeatures, 0])[0]\n", 382 | " mu2, var2 = d2_out.mean(axis=0), d2_out.var(axis=0)\n", 383 | "\n", 384 | " bn_model = insert_layer(dense_model, BatchNormalization(), 1)\n", 385 | " bn_model = insert_layer(bn_model, BatchNormalization(), 4) # shifted due to insertion\n", 386 | "\n", 387 | " bnl1 = bn_model.layers[1]\n", 388 | " bnl4 = bn_model.layers[4]\n", 389 | "\n", 390 | " #After inserting the layers, we can set their weights to the variance and mean we just calculated.\n", 391 | " bnl1.set_weights([var0, mu0, mu0, var0])\n", 392 | " bnl4.set_weights([var2, mu2, mu2, var2])\n", 393 | "\n", 394 | " bn_model.compile(Adam(1e-3), 'categorical_crossentropy', ['accuracy'])\n", 395 | " \n", 396 | " for layer in bn_model.layers:\n", 397 | " layer.trainable = False\n", 398 | " bn_model.layers[-1].trainable = True\n", 399 | " \n", 400 | " return bn_model" 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": 64, 406 | "metadata": { 407 | "collapsed": false 408 | }, 409 | "outputs": [], 410 | "source": [ 411 | "def train_fresh_bn(mdl, top=2, full=5):\n", 412 | " # top\n", 413 | " for layer in mdl.layers:\n", 414 | " layer.trainable = False\n", 415 | " mdl.layers[-1].trainable = True\n", 416 | " mdl.optimizer.lr = 1e-3\n", 417 | " mdl.fit(trn_convfeatures, trn_labels, validation_data=(val_convfeatures, val_labels), nb_epoch=top)\n", 418 | " # full\n", 419 | " for layer in mdl.layers:\n", 420 | " layer.trainable = True\n", 421 | " mdl.optimizer.lr = 0.01*1e-3\n", 422 | " mdl.fit(trn_convfeatures, trn_labels, validation_data=(val_convfeatures, val_labels), nb_epoch=full)" 423 | ] 424 | }, 425 | { 426 | "cell_type": "code", 427 | "execution_count": null, 428 | "metadata": { 429 | "collapsed": true 430 | }, 431 | "outputs": [], 432 | "source": [ 433 | "#bn_model = get_bn_model(0.30)" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": 72, 439 | "metadata": { 440 | "collapsed": false 441 | }, 442 | "outputs": [], 443 | "source": [ 444 | "#train_fresh_bn(bn_model, 2, 5)" 445 | ] 446 | }, 447 | { 448 | "cell_type": "markdown", 449 | "metadata": {}, 450 | "source": [ 451 | "### Train one or several models (ensembling)" 452 | ] 453 | }, 454 | { 455 | "cell_type": "code", 456 | "execution_count": 66, 457 | "metadata": { 458 | "collapsed": false 459 | }, 460 | "outputs": [ 461 | { 462 | "name": "stdout", 463 | "output_type": "stream", 464 | "text": [ 465 | "Train on 21000 samples, validate on 4000 samples\n", 466 | "Epoch 1/2\n", 467 | "21000/21000 [==============================] - 11s - loss: 0.1018 - acc: 0.9642 - val_loss: 0.0605 - val_acc: 0.9805\n", 468 | "Epoch 2/2\n", 469 | "21000/21000 [==============================] - 11s - loss: 0.0902 - acc: 0.9708 - val_loss: 0.0663 - val_acc: 0.9798\n", 470 | "Train on 21000 samples, validate on 4000 samples\n", 471 | "Epoch 1/5\n", 472 | "21000/21000 [==============================] - 11s - loss: 0.0998 - acc: 0.9685 - val_loss: 0.0456 - val_acc: 0.9828\n", 473 | "Epoch 2/5\n", 474 | "21000/21000 [==============================] - 11s - loss: 0.0870 - acc: 0.9715 - val_loss: 0.0576 - val_acc: 0.9818\n", 475 | "Epoch 3/5\n", 476 | "21000/21000 [==============================] - 11s - loss: 0.0747 - acc: 0.9751 - val_loss: 0.0516 - val_acc: 0.9815\n", 477 | "Epoch 4/5\n", 478 | "21000/21000 [==============================] - 11s - loss: 0.0810 - acc: 0.9735 - val_loss: 0.0456 - val_acc: 0.9845\n", 479 | "Epoch 5/5\n", 480 | "21000/21000 [==============================] - 11s - loss: 0.0852 - acc: 0.9730 - val_loss: 0.0515 - val_acc: 0.9810\n", 481 | "Train on 21000 samples, validate on 4000 samples\n", 482 | "Epoch 1/2\n", 483 | "21000/21000 [==============================] - 11s - loss: 0.1048 - acc: 0.9648 - val_loss: 0.0449 - val_acc: 0.9840\n", 484 | "Epoch 2/2\n", 485 | "21000/21000 [==============================] - 11s - loss: 0.0898 - acc: 0.9708 - val_loss: 0.0597 - val_acc: 0.9762\n", 486 | "Train on 21000 samples, validate on 4000 samples\n", 487 | "Epoch 1/5\n", 488 | "21000/21000 [==============================] - 11s - loss: 0.0903 - acc: 0.9701 - val_loss: 0.0465 - val_acc: 0.9840\n", 489 | "Epoch 2/5\n", 490 | "21000/21000 [==============================] - 11s - loss: 0.0882 - acc: 0.9712 - val_loss: 0.0453 - val_acc: 0.9838\n", 491 | "Epoch 3/5\n", 492 | "21000/21000 [==============================] - 11s - loss: 0.0826 - acc: 0.9728 - val_loss: 0.0520 - val_acc: 0.9822\n", 493 | "Epoch 4/5\n", 494 | "21000/21000 [==============================] - 11s - loss: 0.0868 - acc: 0.9721 - val_loss: 0.0610 - val_acc: 0.9792\n", 495 | "Epoch 5/5\n", 496 | "21000/21000 [==============================] - 11s - loss: 0.0947 - acc: 0.9703 - val_loss: 0.0677 - val_acc: 0.9792\n", 497 | "Train on 21000 samples, validate on 4000 samples\n", 498 | "Epoch 1/2\n", 499 | "21000/21000 [==============================] - 11s - loss: 0.1010 - acc: 0.9630 - val_loss: 0.0511 - val_acc: 0.9815\n", 500 | "Epoch 2/2\n", 501 | "21000/21000 [==============================] - 11s - loss: 0.0929 - acc: 0.9690 - val_loss: 0.0463 - val_acc: 0.9825\n", 502 | "Train on 21000 samples, validate on 4000 samples\n", 503 | "Epoch 1/5\n", 504 | "21000/21000 [==============================] - 11s - loss: 0.0833 - acc: 0.9725 - val_loss: 0.0454 - val_acc: 0.9830\n", 505 | "Epoch 2/5\n", 506 | "21000/21000 [==============================] - 11s - loss: 0.0859 - acc: 0.9718 - val_loss: 0.0524 - val_acc: 0.9818\n", 507 | "Epoch 3/5\n", 508 | "21000/21000 [==============================] - 11s - loss: 0.0917 - acc: 0.9703 - val_loss: 0.0456 - val_acc: 0.9835\n", 509 | "Epoch 4/5\n", 510 | "21000/21000 [==============================] - 11s - loss: 0.0903 - acc: 0.9725 - val_loss: 0.0509 - val_acc: 0.9832\n", 511 | "Epoch 5/5\n", 512 | "21000/21000 [==============================] - 11s - loss: 0.0848 - acc: 0.9729 - val_loss: 0.0447 - val_acc: 0.9835\n", 513 | "Train on 21000 samples, validate on 4000 samples\n", 514 | "Epoch 1/2\n", 515 | "21000/21000 [==============================] - 11s - loss: 0.1083 - acc: 0.9622 - val_loss: 0.0476 - val_acc: 0.9822\n", 516 | "Epoch 2/2\n", 517 | "21000/21000 [==============================] - 11s - loss: 0.0991 - acc: 0.9678 - val_loss: 0.0425 - val_acc: 0.9845\n", 518 | "Train on 21000 samples, validate on 4000 samples\n", 519 | "Epoch 1/5\n", 520 | "21000/21000 [==============================] - 11s - loss: 0.0919 - acc: 0.9704 - val_loss: 0.0507 - val_acc: 0.9832\n", 521 | "Epoch 2/5\n", 522 | "21000/21000 [==============================] - 11s - loss: 0.0827 - acc: 0.9729 - val_loss: 0.0472 - val_acc: 0.9842\n", 523 | "Epoch 3/5\n", 524 | "21000/21000 [==============================] - 11s - loss: 0.0885 - acc: 0.9713 - val_loss: 0.0502 - val_acc: 0.9810\n", 525 | "Epoch 4/5\n", 526 | "21000/21000 [==============================] - 11s - loss: 0.0754 - acc: 0.9746 - val_loss: 0.0520 - val_acc: 0.9805\n", 527 | "Epoch 5/5\n", 528 | "21000/21000 [==============================] - 11s - loss: 0.0755 - acc: 0.9754 - val_loss: 0.0470 - val_acc: 0.9820\n", 529 | "Train on 21000 samples, validate on 4000 samples\n", 530 | "Epoch 1/2\n", 531 | "21000/21000 [==============================] - 11s - loss: 0.1034 - acc: 0.9629 - val_loss: 0.0592 - val_acc: 0.9765\n", 532 | "Epoch 2/2\n", 533 | "21000/21000 [==============================] - 11s - loss: 0.0855 - acc: 0.9705 - val_loss: 0.0450 - val_acc: 0.9835\n", 534 | "Train on 21000 samples, validate on 4000 samples\n", 535 | "Epoch 1/5\n", 536 | "21000/21000 [==============================] - 11s - loss: 0.0877 - acc: 0.9712 - val_loss: 0.0464 - val_acc: 0.9810\n", 537 | "Epoch 2/5\n", 538 | "21000/21000 [==============================] - 11s - loss: 0.0914 - acc: 0.9714 - val_loss: 0.0459 - val_acc: 0.9840\n", 539 | "Epoch 3/5\n", 540 | "21000/21000 [==============================] - 11s - loss: 0.0879 - acc: 0.9712 - val_loss: 0.0557 - val_acc: 0.9805\n", 541 | "Epoch 4/5\n", 542 | "21000/21000 [==============================] - 11s - loss: 0.0840 - acc: 0.9715 - val_loss: 0.0513 - val_acc: 0.9772\n", 543 | "Epoch 5/5\n", 544 | "21000/21000 [==============================] - 11s - loss: 0.0821 - acc: 0.9732 - val_loss: 0.0515 - val_acc: 0.9830\n", 545 | "Train on 21000 samples, validate on 4000 samples\n", 546 | "Epoch 1/2\n", 547 | "21000/21000 [==============================] - 11s - loss: 0.0994 - acc: 0.9640 - val_loss: 0.0688 - val_acc: 0.9768\n", 548 | "Epoch 2/2\n", 549 | "21000/21000 [==============================] - 11s - loss: 0.0912 - acc: 0.9691 - val_loss: 0.0539 - val_acc: 0.9778\n", 550 | "Train on 21000 samples, validate on 4000 samples\n", 551 | "Epoch 1/5\n", 552 | "21000/21000 [==============================] - 11s - loss: 0.0899 - acc: 0.9708 - val_loss: 0.0485 - val_acc: 0.9828\n", 553 | "Epoch 2/5\n", 554 | "21000/21000 [==============================] - 11s - loss: 0.0850 - acc: 0.9732 - val_loss: 0.0494 - val_acc: 0.9818\n", 555 | "Epoch 3/5\n", 556 | "21000/21000 [==============================] - 11s - loss: 0.0774 - acc: 0.9755 - val_loss: 0.0713 - val_acc: 0.9775\n", 557 | "Epoch 4/5\n", 558 | "21000/21000 [==============================] - 11s - loss: 0.0826 - acc: 0.9715 - val_loss: 0.0530 - val_acc: 0.9790\n", 559 | "Epoch 5/5\n", 560 | "21000/21000 [==============================] - 11s - loss: 0.0801 - acc: 0.9736 - val_loss: 0.0465 - val_acc: 0.9820\n", 561 | "Train on 21000 samples, validate on 4000 samples\n", 562 | "Epoch 1/2\n", 563 | "21000/21000 [==============================] - 11s - loss: 0.1125 - acc: 0.9622 - val_loss: 0.0463 - val_acc: 0.9820\n", 564 | "Epoch 2/2\n", 565 | "21000/21000 [==============================] - 11s - loss: 0.0960 - acc: 0.9678 - val_loss: 0.0528 - val_acc: 0.9840\n", 566 | "Train on 21000 samples, validate on 4000 samples\n", 567 | "Epoch 1/5\n", 568 | "21000/21000 [==============================] - 11s - loss: 0.0886 - acc: 0.9693 - val_loss: 0.0424 - val_acc: 0.9845\n", 569 | "Epoch 2/5\n", 570 | "21000/21000 [==============================] - 11s - loss: 0.0836 - acc: 0.9723 - val_loss: 0.0436 - val_acc: 0.9852\n", 571 | "Epoch 3/5\n", 572 | "21000/21000 [==============================] - 11s - loss: 0.0801 - acc: 0.9733 - val_loss: 0.0497 - val_acc: 0.9795\n", 573 | "Epoch 4/5\n", 574 | "21000/21000 [==============================] - 11s - loss: 0.0883 - acc: 0.9719 - val_loss: 0.0595 - val_acc: 0.9810\n", 575 | "Epoch 5/5\n", 576 | "21000/21000 [==============================] - 11s - loss: 0.0890 - acc: 0.9725 - val_loss: 0.0481 - val_acc: 0.9802\n", 577 | "Train on 21000 samples, validate on 4000 samples\n", 578 | "Epoch 1/2\n", 579 | "21000/21000 [==============================] - 11s - loss: 0.0994 - acc: 0.9650 - val_loss: 0.0934 - val_acc: 0.9663\n", 580 | "Epoch 2/2\n", 581 | "21000/21000 [==============================] - 11s - loss: 0.0967 - acc: 0.9683 - val_loss: 0.0469 - val_acc: 0.9832\n", 582 | "Train on 21000 samples, validate on 4000 samples\n", 583 | "Epoch 1/5\n", 584 | "21000/21000 [==============================] - 11s - loss: 0.0821 - acc: 0.9737 - val_loss: 0.0485 - val_acc: 0.9842\n", 585 | "Epoch 2/5\n", 586 | "21000/21000 [==============================] - 11s - loss: 0.0818 - acc: 0.9730 - val_loss: 0.0527 - val_acc: 0.9812\n", 587 | "Epoch 3/5\n", 588 | "21000/21000 [==============================] - 11s - loss: 0.0936 - acc: 0.9710 - val_loss: 0.0558 - val_acc: 0.9828\n", 589 | "Epoch 4/5\n", 590 | "21000/21000 [==============================] - 11s - loss: 0.0776 - acc: 0.9746 - val_loss: 0.0553 - val_acc: 0.9825\n", 591 | "Epoch 5/5\n", 592 | "21000/21000 [==============================] - 11s - loss: 0.0866 - acc: 0.9731 - val_loss: 0.0699 - val_acc: 0.9780\n", 593 | "Train on 21000 samples, validate on 4000 samples\n", 594 | "Epoch 1/2\n", 595 | "21000/21000 [==============================] - 11s - loss: 0.1162 - acc: 0.9590 - val_loss: 0.0487 - val_acc: 0.9812\n", 596 | "Epoch 2/2\n", 597 | "21000/21000 [==============================] - 11s - loss: 0.0872 - acc: 0.9712 - val_loss: 0.0473 - val_acc: 0.9828\n", 598 | "Train on 21000 samples, validate on 4000 samples\n", 599 | "Epoch 1/5\n", 600 | "21000/21000 [==============================] - 11s - loss: 0.0852 - acc: 0.9720 - val_loss: 0.0453 - val_acc: 0.9832\n", 601 | "Epoch 2/5\n", 602 | "21000/21000 [==============================] - 11s - loss: 0.0811 - acc: 0.9730 - val_loss: 0.0457 - val_acc: 0.9845\n", 603 | "Epoch 3/5\n", 604 | "21000/21000 [==============================] - 11s - loss: 0.0754 - acc: 0.9742 - val_loss: 0.0513 - val_acc: 0.9828\n", 605 | "Epoch 4/5\n", 606 | "21000/21000 [==============================] - 11s - loss: 0.0861 - acc: 0.9723 - val_loss: 0.0514 - val_acc: 0.9795\n", 607 | "Epoch 5/5\n", 608 | "21000/21000 [==============================] - 11s - loss: 0.0758 - acc: 0.9751 - val_loss: 0.0521 - val_acc: 0.9838\n", 609 | "Train on 21000 samples, validate on 4000 samples\n", 610 | "Epoch 1/2\n", 611 | "21000/21000 [==============================] - 11s - loss: 0.0997 - acc: 0.9637 - val_loss: 0.0922 - val_acc: 0.9698\n", 612 | "Epoch 2/2\n", 613 | "21000/21000 [==============================] - 11s - loss: 0.0913 - acc: 0.9688 - val_loss: 0.0534 - val_acc: 0.9848\n", 614 | "Train on 21000 samples, validate on 4000 samples\n", 615 | "Epoch 1/5\n", 616 | "21000/21000 [==============================] - 11s - loss: 0.0881 - acc: 0.9718 - val_loss: 0.0479 - val_acc: 0.9820\n", 617 | "Epoch 2/5\n", 618 | "21000/21000 [==============================] - 11s - loss: 0.0807 - acc: 0.9725 - val_loss: 0.0710 - val_acc: 0.9782\n", 619 | "Epoch 3/5\n", 620 | "21000/21000 [==============================] - 11s - loss: 0.0895 - acc: 0.9710 - val_loss: 0.0477 - val_acc: 0.9842\n", 621 | "Epoch 4/5\n", 622 | "21000/21000 [==============================] - 11s - loss: 0.0786 - acc: 0.9743 - val_loss: 0.0455 - val_acc: 0.9830\n", 623 | "Epoch 5/5\n", 624 | "21000/21000 [==============================] - 11s - loss: 0.0809 - acc: 0.9738 - val_loss: 0.0537 - val_acc: 0.9808\n" 625 | ] 626 | } 627 | ], 628 | "source": [ 629 | "bn_models = []\n", 630 | "for i in range(10): # INFO : change here the size of the ensemble\n", 631 | " bn_models.append( get_bn_model(0.30) )\n", 632 | " train_fresh_bn(bn_models[-1], 2, 8)" 633 | ] 634 | }, 635 | { 636 | "cell_type": "code", 637 | "execution_count": 64, 638 | "metadata": { 639 | "collapsed": false 640 | }, 641 | "outputs": [], 642 | "source": [ 643 | "'''i = 0\n", 644 | "\n", 645 | "x_conv_model = Sequential(conv_layers)\n", 646 | "for layer in x_conv_model.layers:\n", 647 | " layer.trainable = False\n", 648 | "\n", 649 | "for layer in ll_models[i].layers:\n", 650 | " x_conv_model.add(layer)\n", 651 | " \n", 652 | "#for l1,l2 in zip(conv_model.layers[last_conv_idx+1:], fc_model.layers): \n", 653 | "# l1.set_weights(l2.get_weights())\n", 654 | "x_conv_model.compile(optimizer=Adam(1e-5), loss='categorical_crossentropy', metrics=['accuracy'])\n", 655 | "#x_conv_model.save_weights(model_path+'no_dropout_bn' + i + '.h5')'''" 656 | ] 657 | }, 658 | { 659 | "cell_type": "code", 660 | "execution_count": 65, 661 | "metadata": { 662 | "collapsed": false 663 | }, 664 | "outputs": [], 665 | "source": [ 666 | "'''for layer in x_conv_model.layers[-5:]:\n", 667 | " layer.trainable = True\n", 668 | "x_conv_model.optimizer.lr = 1e-6'''" 669 | ] 670 | }, 671 | { 672 | "cell_type": "code", 673 | "execution_count": 68, 674 | "metadata": { 675 | "collapsed": false 676 | }, 677 | "outputs": [ 678 | { 679 | "name": "stdout", 680 | "output_type": "stream", 681 | "text": [ 682 | "Epoch 1/1\n", 683 | "4000/4000 [==============================] - 167s - loss: 0.0266 - acc: 0.9888 - val_loss: 0.0518 - val_acc: 0.9790\n" 684 | ] 685 | }, 686 | { 687 | "data": { 688 | "text/plain": [ 689 | "" 690 | ] 691 | }, 692 | "execution_count": 68, 693 | "metadata": {}, 694 | "output_type": "execute_result" 695 | } 696 | ], 697 | "source": [ 698 | "'''x_conv_model.fit_generator(trn_batchesRND,\n", 699 | " samples_per_epoch = min(40*batch_size,trn_batchesRND.n),\n", 700 | " nb_epoch = 1,\n", 701 | " validation_data = val_batchesRND,\n", 702 | " nb_val_samples = min(20*batch_size,val_batchesRND.n))'''" 703 | ] 704 | }, 705 | { 706 | "cell_type": "code", 707 | "execution_count": 69, 708 | "metadata": { 709 | "collapsed": false 710 | }, 711 | "outputs": [ 712 | { 713 | "name": "stdout", 714 | "output_type": "stream", 715 | "text": [ 716 | "-5\n", 717 | "-4\n", 718 | "-3\n", 719 | "-2\n", 720 | "-1\n" 721 | ] 722 | } 723 | ], 724 | "source": [ 725 | "'''for mdl in ll_models:\n", 726 | " for k in range(-len(mdl.layers),0):\n", 727 | " print(k)\n", 728 | " #x_conv_model.layers[k].get_weights()\n", 729 | " #mdl.layers[k].set_weights\n", 730 | " mdl.layers[k].set_weights( x_conv_model.layers[k].get_weights() )'''" 731 | ] 732 | }, 733 | { 734 | "cell_type": "code", 735 | "execution_count": 73, 736 | "metadata": { 737 | "collapsed": false 738 | }, 739 | "outputs": [ 740 | { 741 | "data": { 742 | "text/plain": [ 743 | "array(0.984499990940094, dtype=float32)" 744 | ] 745 | }, 746 | "execution_count": 73, 747 | "metadata": {}, 748 | "output_type": "execute_result" 749 | } 750 | ], 751 | "source": [ 752 | "if False:\n", 753 | " models = [bn_model] # without ensembling\n", 754 | "else:\n", 755 | " models = bn_models # with ensembling\n", 756 | "\n", 757 | "all_val_preds = []\n", 758 | "for mdl in models:\n", 759 | " these_val_preds = mdl.predict_on_batch(val_convfeatures)\n", 760 | " assert(len(these_val_preds) == 4000)\n", 761 | " all_val_preds.append( these_val_preds )\n", 762 | "mean_val_preds = np.stack(all_val_preds).mean(axis=0)\n", 763 | "categorical_accuracy(val_labels, mean_val_preds).eval()" 764 | ] 765 | }, 766 | { 767 | "cell_type": "code", 768 | "execution_count": null, 769 | "metadata": { 770 | "collapsed": true 771 | }, 772 | "outputs": [], 773 | "source": [ 774 | "# WARNING : should save each model of the ensemble\n", 775 | "#ll_model.save_weights(model_path+'llmodel_finetune1.h5')\n", 776 | "#ll_model.load_weights(model_path+'llmodel_finetune1.h5')" 777 | ] 778 | }, 779 | { 780 | "cell_type": "code", 781 | "execution_count": 39, 782 | "metadata": { 783 | "collapsed": false 784 | }, 785 | "outputs": [ 786 | { 787 | "name": "stdout", 788 | "output_type": "stream", 789 | "text": [ 790 | "Found 12500 images belonging to 1 classes.\n" 791 | ] 792 | }, 793 | { 794 | "data": { 795 | "text/plain": [ 796 | "['test/10592.jpg',\n", 797 | " 'test/7217.jpg',\n", 798 | " 'test/3653.jpg',\n", 799 | " 'test/4382.jpg',\n", 800 | " 'test/2924.jpg',\n", 801 | " 'test/10.jpg',\n", 802 | " 'test/10916.jpg',\n", 803 | " 'test/12374.jpg',\n", 804 | " 'test/1871.jpg',\n", 805 | " 'test/11645.jpg']" 806 | ] 807 | }, 808 | "execution_count": 39, 809 | "metadata": {}, 810 | "output_type": "execute_result" 811 | } 812 | ], 813 | "source": [ 814 | "test_batches = get_batches('test', shuffle=False, batch_size=batch_size, class_mode=None)\n", 815 | "testfiles = test_batches.filenames\n", 816 | "testfiles[0:10]" 817 | ] 818 | }, 819 | { 820 | "cell_type": "markdown", 821 | "metadata": {}, 822 | "source": [ 823 | "### Will take a few minutes (maybe 5) to complete the 1st time" 824 | ] 825 | }, 826 | { 827 | "cell_type": "code", 828 | "execution_count": 40, 829 | "metadata": { 830 | "collapsed": false 831 | }, 832 | "outputs": [], 833 | "source": [ 834 | "try:\n", 835 | " test_convfeatures = load_array(model_path+'test_'+conv_model_hash+'_features.bc')\n", 836 | " if False: # force update\n", 837 | " raise\n", 838 | "except:\n", 839 | " print('Missing file')\n", 840 | " test_convfeatures = conv_model.predict_generator(test_batches, test_batches.nb_sample)\n", 841 | " save_array(model_path+'test_'+conv_model_hash+'_features.bc', test_convfeatures)" 842 | ] 843 | }, 844 | { 845 | "cell_type": "code", 846 | "execution_count": 74, 847 | "metadata": { 848 | "collapsed": false 849 | }, 850 | "outputs": [], 851 | "source": [ 852 | "if False:\n", 853 | " models = [bn_model] # without ensembling\n", 854 | "else:\n", 855 | " models = bn_models # with ensembling\n", 856 | "\n", 857 | "all_test_preds = []\n", 858 | "for mdl in models:\n", 859 | " these_test_preds = mdl.predict_on_batch(test_convfeatures)\n", 860 | " assert(len(these_test_preds) == 12500)\n", 861 | " all_test_preds.append( these_test_preds )\n", 862 | "mean_test_preds = np.stack(all_test_preds).mean(axis=0)" 863 | ] 864 | }, 865 | { 866 | "cell_type": "code", 867 | "execution_count": 75, 868 | "metadata": { 869 | "collapsed": false 870 | }, 871 | "outputs": [ 872 | { 873 | "data": { 874 | "text/plain": [ 875 | "array([[ 9.9996e-01, 3.8756e-05],\n", 876 | " [ 9.9993e-01, 6.8629e-05],\n", 877 | " [ 2.0637e-04, 9.9979e-01],\n", 878 | " [ 9.9551e-01, 4.4935e-03],\n", 879 | " [ 1.4125e-02, 9.8587e-01],\n", 880 | " [ 1.0000e+00, 3.3480e-06],\n", 881 | " [ 2.2238e-03, 9.9778e-01],\n", 882 | " [ 1.0000e+00, 1.2753e-07],\n", 883 | " [ 1.8051e-04, 9.9982e-01],\n", 884 | " [ 6.9930e-05, 9.9993e-01]], dtype=float32)" 885 | ] 886 | }, 887 | "execution_count": 75, 888 | "metadata": {}, 889 | "output_type": "execute_result" 890 | } 891 | ], 892 | "source": [ 893 | "mean_test_preds[0:10]" 894 | ] 895 | }, 896 | { 897 | "cell_type": "code", 898 | "execution_count": 76, 899 | "metadata": { 900 | "collapsed": false 901 | }, 902 | "outputs": [ 903 | { 904 | "data": { 905 | "text/plain": [ 906 | "(-0.018694336826138514, -0.018949097448258439)" 907 | ] 908 | }, 909 | "execution_count": 76, 910 | "metadata": {}, 911 | "output_type": "execute_result" 912 | } 913 | ], 914 | "source": [ 915 | "dog_idx = 1\n", 916 | "eps = 1e-3 # WARNING : this has significant impact\n", 917 | "digits = 3 # WARNING : this has significant impact\n", 918 | "\n", 919 | "cut = lambda x : round(min(max(x,eps),1-eps),digits)\n", 920 | "\n", 921 | "a = sum([p[dog_idx]*math.log(p[dog_idx]) for p in mean_test_preds])/len(mean_test_preds)\n", 922 | "b = sum([p[dog_idx]*math.log(cut(p[dog_idx])) for p in mean_test_preds])/len(mean_test_preds)\n", 923 | "a, b" 924 | ] 925 | }, 926 | { 927 | "cell_type": "code", 928 | "execution_count": 77, 929 | "metadata": { 930 | "collapsed": false 931 | }, 932 | "outputs": [ 933 | { 934 | "data": { 935 | "text/plain": [ 936 | "[{'id': 1, 'label': 0.999},\n", 937 | " {'id': 2, 'label': 0.999},\n", 938 | " {'id': 3, 'label': 0.999},\n", 939 | " {'id': 4, 'label': 0.999},\n", 940 | " {'id': 5, 'label': 0.001},\n", 941 | " {'id': 6, 'label': 0.001},\n", 942 | " {'id': 7, 'label': 0.001},\n", 943 | " {'id': 8, 'label': 0.001},\n", 944 | " {'id': 9, 'label': 0.001},\n", 945 | " {'id': 10, 'label': 0.001},\n", 946 | " {'id': 11, 'label': 0.001},\n", 947 | " {'id': 12, 'label': 0.999},\n", 948 | " {'id': 13, 'label': 0.001},\n", 949 | " {'id': 14, 'label': 0.003},\n", 950 | " {'id': 15, 'label': 0.001},\n", 951 | " {'id': 16, 'label': 0.001},\n", 952 | " {'id': 17, 'label': 0.998},\n", 953 | " {'id': 18, 'label': 0.999}]" 954 | ] 955 | }, 956 | "execution_count": 77, 957 | "metadata": {}, 958 | "output_type": "execute_result" 959 | } 960 | ], 961 | "source": [ 962 | "Z1 = [{'id':int(f.split('/')[-1].split('.')[0]), 'label':cut(p[dog_idx])} for f, p in zip(testfiles, mean_test_preds)]\n", 963 | "def comp(x,y):\n", 964 | " return int(x['id']) - int(y['id'])\n", 965 | "Z1 = sorted(Z1, comp)\n", 966 | "Z1[0:18]" 967 | ] 968 | }, 969 | { 970 | "cell_type": "code", 971 | "execution_count": 79, 972 | "metadata": { 973 | "collapsed": false 974 | }, 975 | "outputs": [], 976 | "source": [ 977 | "import csv\n", 978 | "\n", 979 | "with open('predictions_v4_9.csv', 'w') as csvfile:\n", 980 | " fieldnames = ['id', 'label']\n", 981 | " writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n", 982 | " writer.writeheader()\n", 983 | " for z in Z1:\n", 984 | " writer.writerow(z)" 985 | ] 986 | }, 987 | { 988 | "cell_type": "code", 989 | "execution_count": null, 990 | "metadata": { 991 | "collapsed": true 992 | }, 993 | "outputs": [], 994 | "source": [] 995 | } 996 | ], 997 | "metadata": { 998 | "kernelspec": { 999 | "display_name": "Python 2", 1000 | "language": "python", 1001 | "name": "python2" 1002 | }, 1003 | "language_info": { 1004 | "codemirror_mode": { 1005 | "name": "ipython", 1006 | "version": 2 1007 | }, 1008 | "file_extension": ".py", 1009 | "mimetype": "text/x-python", 1010 | "name": "python", 1011 | "nbconvert_exporter": "python", 1012 | "pygments_lexer": "ipython2", 1013 | "version": "2.7.11" 1014 | } 1015 | }, 1016 | "nbformat": 4, 1017 | "nbformat_minor": 0 1018 | } 1019 | --------------------------------------------------------------------------------