├── LICENSE ├── README.md ├── datasets ├── SEM_image.npy └── SEM_mask.npy ├── docs ├── activelearning_fig0.png └── neuropoly.png ├── notebooks ├── Dataset_preparation_v2.0.ipynb ├── deep_active_learning_simulation_framework.ipynb └── exploratory_work │ └── active_learning_30_11.ipynb ├── requirements.txt └── results ├── index_uncert_code_refactoring.npy ├── score_average_code_refactoring.npy ├── score_global_code_refactoring.npy ├── uncertainty_iter_code_refactoring.npy └── uncertainty_map_x_unlabelled_code_refactoring.npy /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 NeuroPoly, École Polytechnique, Université de Montréal 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deep Active Learning for Myelin Segmentation on Histology Data 2 | 3 | Open-source Active Learning simulation framework for segmenting myelin from histology data based on uncertainty sampling. Written in Python. Using the Keras framework. Based on a convolutional neural network architecture. Pixels are classified either as myelin or background. 4 | 5 | ![alt tag](https://github.com/neuropoly/deep_active_learning/blob/master/docs/activelearning_fig0.png) 6 | 7 | ## Installation 8 | 9 | The following lines will help you install all you need to ensure that the Notebooks are working. Test data, instructions and results examples are provided to help you use this framework. 10 | 11 | #### Python 12 | First, you should make sure that Python 2.7 is installed on your computer. 13 | Run the following command in the terminal: 14 | ``` 15 | python -V 16 | ``` 17 | If you have the Anaconda distribution installed on your system, you can specify the version of Python that you want installed in your virtual environment set up below, even if it differs from the version displayed by the “python -V” command. To see the list of Python versions available to be installed in your conda virtual environment, run: 18 | ``` 19 | conda search python 20 | ``` 21 | #### Virtual Environment 22 | 23 | We recommand you to set up a virtual environment. A virtual environment is a tool that lets you install specific versions of the python modules you want. It will allow to run this code with respect to its module requirements, without affecting the rest of your python installation. 24 | 25 | If you have the Anaconda Distribution installed on your system, you can use the conda virtual environment manager, which allows you to specify a different Python version to be installed in your virtual environment than what’s available by default on your system. 26 | 27 | To create a virtual environment called “dal_venv” with the Anaconda Distribution, run: 28 | ``` 29 | conda create -n dal_venv python=2.7 30 | ``` 31 | To activate it, run the following command: 32 | 33 | ``` 34 | source activate dal_venv 35 | ``` 36 | #### Git Clone 37 | 38 | To use this framework, you first need to clone the deep_active_learning repository using the following command: 39 | ``` 40 | git clone https://github.com/neuropoly/deep_active_learning.git 41 | ``` 42 | Then, go to the newly created git repository and install the requirements using the following commands: 43 | 44 | ``` 45 | pip install -r /path/to/requirements.txt 46 | ``` 47 | ## Getting Started 48 | 49 | #### Toy Datasets 50 | 51 | A toy dataset is made available to run the notebooks. It is composed of 2 SEM-acquired images of spinal-cord histology and their corresponding ground-truths (masks). The 2 images are already pre-processed and stored as .npy files under the *./dataset* folder. However, if you want to test this code on your own images, an example of raw images pre-processing is given in the **Dataset_preparation_v2.0.ipynb** notebook. 52 | 53 | #### Notebooks 54 | 55 | * **Dataset_preparation_v2.0.ipynb**: Example of raw images pre-processing to obtain normalized patches store in numpy-arrays. 56 | * **deep_active_learning_simulation_framework.ipynb**: Deep Active Learning simulation framework to simulate active learning procedure for image segmentation. 57 | 58 | #### GPUs 59 | It is advised to run this code on GPUs since the models are heavy and trained multiple times (active learning iteration). Therefore, tensorflow-gpu must be used instead of tensorflow. If you are having issues installing tensorflow-gpu, you can refer to this page: https://www.tensorflow.org/install/pip 60 | 61 | ## Help 62 | 63 | If you experience issues during installation and/or use of this code, you can post a new issue on the deep_active_learning GitHub issues webpage. We will reply to you as soon as possible. 64 | 65 | ## Authors 66 | 67 | * **Mélanie Lubrano** - [MelanieLu](https://github.com/MelanieLu) 68 | * **Christian S. Perone** - [perone](https://github.com/perone) 69 | * **Mathieu Boudreau** - [mathieuboudreau](https://github.com/mathieuboudreau) 70 | * **Julien Cohen-Adad** - [jcohenadad](https://github.com/jcohenadad) 71 | 72 | See also the list of [contributors](https://github.com/neuropoly/deep_active_learning/graphs/contributors) who participated in this project. 73 | 74 | ## License 75 | 76 | This project is licensed under the MIT License 77 | 78 | Copyright (c) 2018 NeuroPoly, École Polytechnique, Université de Montréal 79 | 80 | Permission is hereby granted, free of charge, to any person obtaining a copy 81 | of this software and associated documentation files (the "Software"), to deal 82 | in the Software without restriction, including without limitation the rights 83 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 84 | copies of the Software, and to permit persons to whom the Software is 85 | furnished to do so, subject to the following conditions: 86 | 87 | The above copyright notice and this permission notice shall be included in all 88 | copies or substantial portions of the Software. 89 | 90 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 91 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 92 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 93 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 94 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 95 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 96 | SOFTWARE. 97 | 98 | ## Acknowledgments 99 | 100 | We thank Shawn Mikula for sharing the histology data. Funded by the Canada Research Chair in Quantitative Magnetic Resonance Imaging [950-230815], the Canadian Institute of Health Research [CIHR FDN-143263], the Canada Foundation for Innovation [32454, 34824], the Fonds de Recherche du Québec - Santé [28826], the Fonds de Recherche du Québec - Nature et Technologies [2015-PR-182754], the Natural Sciences and Engineering Research Council of Canada [435897-2013], the Canada First Research Excellence Fund (IVADO and TransMedTech) and the Quebec BioImaging Network [5886]. 101 | 102 | 103 | -------------------------------------------------------------------------------- /datasets/SEM_image.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/datasets/SEM_image.npy -------------------------------------------------------------------------------- /datasets/SEM_mask.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/datasets/SEM_mask.npy -------------------------------------------------------------------------------- /docs/activelearning_fig0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/docs/activelearning_fig0.png -------------------------------------------------------------------------------- /docs/neuropoly.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/docs/neuropoly.png -------------------------------------------------------------------------------- /notebooks/deep_active_learning_simulation_framework.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Deep Active Learning simulation framework for Image Segmentation\n", 8 | "\n", 9 | "---\n", 10 | "**This notebook provides useful functions to implement active learning features for image segmentation.**
\n", 11 | "**It is applied in this case for histology segmentation of myelin sheath along the spinal cord.**\n", 12 | "\n", 13 | "Active Learning helps to efficiently train deep neural networks by selecting wisely the sample to annotate. \n", 14 | "\n", 15 | "---\n", 16 | "\n", 17 | "### Preliminary remarks\n", 18 | "##### Datasets\n", 19 | "* This code takes as input patches of images and their corresponding masks, preprocessed to specific shapes.
\n", 20 | "* It performs myelin segmentation (binary segmentation) on such images
\n", 21 | "* Raw data preprocessing is detailled in the **Dataset_preparation_v2.0.ipynb** notebook.
\n", 22 | "* A toy dataset composed of 2 SEM histology images preprocessed into 18 patches is available at **./deep_active_learning/datasets/** to run the notebook. \n", 23 | "\n", 24 | "\n", 25 | "##### Objectives\n", 26 | "* The objective of the notebook is to **SIMULATE** active learning on histology data to efficiently train models to do segmentation. \n", 27 | "* This active learning simulation framework is based on convolutional neural network to perform binary segmentation and MC-dropout to measure the uncertainty on the samples. \n", 28 | "\n", 29 | "\n", 30 | "##### GPUs\n", 31 | "* It is advised to run this notebook on GPUs since the models are trained multiple times (active learning iteration).\n", 32 | "* **Training time**: the results where obtain by realizing 10 experiments, of 15 active-learning iteration each (ie. adding up to 15 patches to the initial dataset). On 2 NVIDIA GPUs, 1 active learning iteration was taking about 20 minutes to run.\n", 33 | "\n", 34 | "---\n", 35 | "\n", 36 | "This notebook is organized in several parts as listed here: \n", 37 | "\n", 38 | "**1. Data Loading**
\n", 39 | "**2. Image segmentation**
\n", 40 | " - Utils\n", 41 | " - U-Net\n", 42 | " - Score measures\n", 43 | "**3. Uncertainty measure**
\n", 44 | "**4. Active Learning Loop**
\n", 45 | "**5. Configuration parameters**
\n", 46 | "**6. Main - Training**
\n", 47 | "**7. Results analysis**
\n" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 2, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "from __future__ import division\n", 57 | "\n", 58 | "import json\n", 59 | "import os\n", 60 | "import matplotlib.pyplot as plt\n", 61 | "import numpy as np\n", 62 | "import cv2\n", 63 | "import gzip\n", 64 | "import time\n", 65 | "import random\n", 66 | "\n", 67 | "from keras import backend as K\n", 68 | "# from keras.layers import Input, merge, Convolution2D, MaxPooling2D, UpSampling2D, Dropout\n", 69 | "from keras.layers import *\n", 70 | "from keras.layers.merge import concatenate\n", 71 | "from keras.models import *\n", 72 | "from keras.optimizers import *\n", 73 | "from keras.callbacks import ModelCheckpoint, LearningRateScheduler, TensorBoard, Callback\n", 74 | "from keras import losses\n", 75 | "from keras.preprocessing.image import ImageDataGenerator\n", 76 | "\n", 77 | "import scipy.ndimage\n", 78 | "from scipy.misc import imsave\n", 79 | "from scipy.ndimage import morphology\n", 80 | "from numpy import linalg\n", 81 | "\n", 82 | "from itertools import islice\n", 83 | "from scipy.ndimage.morphology import distance_transform_edt as edt" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 2, 89 | "metadata": { 90 | "collapsed": true 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "# Run this line if you are using a GPU\n", 95 | "#os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"3, 4\"" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 3, 101 | "metadata": {}, 102 | "outputs": [ 103 | { 104 | "name": "stdout", 105 | "output_type": "stream", 106 | "text": [ 107 | "/Users/mel/Documents/neuropoly/deep_active_learning\n" 108 | ] 109 | } 110 | ], 111 | "source": [ 112 | "cd .." 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 4, 118 | "metadata": {}, 119 | "outputs": [ 120 | { 121 | "data": { 122 | "text/plain": [ 123 | "u'/Users/mel/Documents/neuropoly/deep_active_learning'" 124 | ] 125 | }, 126 | "execution_count": 4, 127 | "metadata": {}, 128 | "output_type": "execute_result" 129 | } 130 | ], 131 | "source": [ 132 | "pwd" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "# 1. Data Loading" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 5, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "def preprocessor(input_img, img_rows, img_cols):\n", 149 | " \"\"\"\n", 150 | " Resize input images to constants sizes\n", 151 | " :param input_img: numpy array of images\n", 152 | " :return: numpy array of preprocessed images\n", 153 | " \"\"\"\n", 154 | " output_img = np.ndarray((input_img.shape[0], input_img.shape[1], img_rows, img_cols), dtype=np.uint8)\n", 155 | " \n", 156 | " for i in range(input_img.shape[0]):\n", 157 | " output_img[i, 0] = cv2.resize(input_img[i, 0], (img_cols, img_rows), interpolation=cv2.INTER_CUBIC)\n", 158 | " \n", 159 | " return output_img\n", 160 | "\n", 161 | "def load_data(path_img, path_mask, img_rows, img_cols):\n", 162 | " \"\"\"\n", 163 | " Load data from project path\n", 164 | " :return: [X, y] numpy arrays containing the [training, validation, test] data and their respective masks.\n", 165 | " \"\"\"\n", 166 | " print(\"\\nLoading data...\\n\")\n", 167 | " X = np.load(path_img)\n", 168 | " y = np.load(path_mask)\n", 169 | "\n", 170 | " X = preprocessor(X, img_rows, img_cols)\n", 171 | " y = preprocessor(y, img_rows, img_cols)\n", 172 | "\n", 173 | " X = X.astype('float32')\n", 174 | "\n", 175 | " mean = np.mean(X) # mean for data centering\n", 176 | " std = np.std(X) # std for data normalization\n", 177 | "\n", 178 | " X -= mean\n", 179 | " X /= std\n", 180 | "\n", 181 | " y = y.astype('float32')\n", 182 | " y /= 255. # scale masks to [0, 1]\n", 183 | " return X, y\n" 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": {}, 189 | "source": [ 190 | "# 2. Image Segmentation" 191 | ] 192 | }, 193 | { 194 | "cell_type": "markdown", 195 | "metadata": {}, 196 | "source": [ 197 | "### 2.1. Utils" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": 23, 203 | "metadata": {}, 204 | "outputs": [], 205 | "source": [ 206 | "#### loss and metrics #####\n", 207 | "\n", 208 | "def dice_coef(y_true, y_pred, smooth=1):\n", 209 | " \"\"\"Compute Dice Coefficient between prediction and Ground Truth\n", 210 | " :param y_true: ground truth\n", 211 | " :param y_pred: prediction\n", 212 | " :param smooth: avoid division by 0\n", 213 | " :return: dice coefficient\n", 214 | " \"\"\"\n", 215 | " y_true_f = K.flatten(y_true)\n", 216 | " y_pred_f = K.flatten(y_pred)\n", 217 | " intersection = K.sum(y_true_f * y_pred_f)\n", 218 | " return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n", 219 | "\n", 220 | "\n", 221 | "def dice_coef_loss(y_true, y_pred):\n", 222 | " \"\"\"Dice loss to train a network\n", 223 | " :param y_true: ground truth\n", 224 | " :param y_pred: prediction\n", 225 | " :return: dice loss function\n", 226 | " \"\"\"\n", 227 | " return 1 - dice_coef(y_true, y_pred)\n", 228 | "\n", 229 | "\n", 230 | "def weighted_binary_crossentropy(y_true, y_pred):\n", 231 | " \"\"\"Compute the weighted binary cross-entropy\n", 232 | " :param y_true: ground truth\n", 233 | " :param y_pred: prediction\n", 234 | " :return: weighted binary cross-entropy\n", 235 | " \"\"\"\n", 236 | " b_ce = K.binary_crossentropy(y_true, y_pred) # Calculate the binary crossentropy\n", 237 | " weight_vector = y_true * 0.30 + (1. - y_true) * 0.70 # Apply the weights\n", 238 | " weighted_b_ce = weight_vector * b_ce \n", 239 | " return K.mean(weighted_b_ce) # Return the mean error\n", 240 | "\n", 241 | "\n", 242 | "##### data augmentation ######\n", 243 | "def data_generator():\n", 244 | " \"\"\"\n", 245 | " :return: Keras data generator. Data augmentation parameters.\n", 246 | " \"\"\"\n", 247 | " return ImageDataGenerator(\n", 248 | " fill_mode = fill_mode,\n", 249 | " rotation_range = rotation_range,\n", 250 | " horizontal_flip = horizontal_flip,\n", 251 | " vertical_flip = vertical_flip,\n", 252 | " rescale = rescale,\n", 253 | " zoom_range = zoom_range,\n", 254 | " channel_shift_range = channel_shift_range,\n", 255 | " width_shift_range = width_shift_range,\n", 256 | " height_shift_range = height_shift_range)\n", 257 | "\n", 258 | "\n", 259 | "##### Monte-Carlo Dropout : keep dropout active at test time #####\n", 260 | "def call(self, inputs, training=None):\n", 261 | " \"\"\"Override Dropout. Make it able at test time\n", 262 | " \"\"\"\n", 263 | " if 0. < self.rate < 1.:\n", 264 | " noise_shape = self._get_noise_shape(inputs)\n", 265 | " def dropped_inputs():\n", 266 | " return K.dropout(inputs, self.rate, noise_shape,\n", 267 | " seed=self.seed)\n", 268 | " if (training):\n", 269 | " return K.in_train_phase(dropped_inputs, inputs, training=training)\n", 270 | " else:\n", 271 | " return K.in_test_phase(dropped_inputs, inputs, training=None)\n", 272 | " return inputs\n", 273 | "\n", 274 | "\n", 275 | "def reset_weights(model):\n", 276 | " \"\"\"Initialize weights of Neural Networks\n", 277 | " \"\"\"\n", 278 | " session = K.get_session()\n", 279 | " for layer in model.layers: \n", 280 | " if hasattr(layer, 'kernel_initializer'):\n", 281 | " layer.kernel.initializer.run(session=session)\n", 282 | " \n", 283 | " \n", 284 | "def predict(data, model):\n", 285 | " \"\"\"\n", 286 | " Data prediction for a given model\n", 287 | " :param data: input data to predict.\n", 288 | " :param model: unet model.\n", 289 | " :return: predictions.\n", 290 | " \"\"\"\n", 291 | " return model.predict(data, verbose=0)\n", 292 | "\n", 293 | "\n", 294 | "def save_parameters(timestamp, params):\n", 295 | " \"\"\"Save current parameters to text file\n", 296 | " :param timestamp:timestamp of the current run session\n", 297 | " :param params: dictionary of current parameters\n", 298 | " \"\"\"\n", 299 | " \n", 300 | " if not os.path.exists(param_path):\n", 301 | " os.makedirs(param_path)\n", 302 | " print(\"Directory \" , param_path , \" Created \")\n", 303 | " \n", 304 | " param_file = str(param_path + 'params_' + timestamp + '.txt')\n", 305 | " txt_file = open(param_file,'w')\n", 306 | " txt_file.write(str(params))\n", 307 | " txt_file.close()\n", 308 | " \n", 309 | " \n", 310 | "def get_params():\n", 311 | " \"\"\"Convert list of current parameters to dictionary \n", 312 | " \"\"\"\n", 313 | " param_dict = {\n", 314 | " 'img_rows':img_rows,\n", 315 | " 'img_cols' : img_cols,\n", 316 | " 'batch_size' : batch_size,\n", 317 | " 'fill_mode' : fill_mode,\n", 318 | " 'rotation_range': rotation_range,\n", 319 | " 'horizontal_flip': horizontal_flip,\n", 320 | " 'vertical_flip' : vertical_flip,\n", 321 | " 'rescale' : rescale,\n", 322 | " 'zoom_range': zoom_range,\n", 323 | " 'channel_shift_range' : channel_shift_range,\n", 324 | " 'width_shift_range' : width_shift_range,\n", 325 | " 'height_shift_range' : height_shift_range,\n", 326 | " 'nb_initial_epochs' : nb_initial_epochs,\n", 327 | " 'apply_augmentation' : apply_augmentation,\n", 328 | " 'nb_step_predictions': nb_step_predictions,\n", 329 | " 'steps_per_epoch' : steps_per_epoch,\n", 330 | " 'learning_rate' : learning_rate,\n", 331 | " 'decay_rate' : decay_rate,\n", 332 | " 'apply_edt' : apply_edt\n", 333 | " }\n", 334 | " return param_dict\n", 335 | "\n", 336 | "\n", 337 | "def save_history(history, timestamp, iteration, history_path):\n", 338 | " \"\"\" Save training history in text file\n", 339 | " :param history: training history (numpy array)\n", 340 | " :param timestamp: timestamp of current run session\n", 341 | " :param iteration: active learning iteration number\n", 342 | " :param history_path: path to history text file\n", 343 | " \"\"\"\n", 344 | " history_file = str(history_path + timestamp + '_history_iter_' + str(iteration) +'_' + '.txt')\n", 345 | " txt_file = open(history_file,'w')\n", 346 | " txt_file.write(str(history))\n", 347 | " txt_file.close()" 348 | ] 349 | }, 350 | { 351 | "cell_type": "markdown", 352 | "metadata": {}, 353 | "source": [ 354 | "### 2.2. U-Net" 355 | ] 356 | }, 357 | { 358 | "cell_type": "code", 359 | "execution_count": 24, 360 | "metadata": {}, 361 | "outputs": [], 362 | "source": [ 363 | "def unet_full_bn(dropout):\n", 364 | " inputs = Input((1, img_rows, img_cols))\n", 365 | " conv1 = Conv2D(32, (3, 3), padding=\"same\", activation=\"relu\")(inputs)\n", 366 | " conv1 = Conv2D(32, (3, 3), padding=\"same\", activation=\"relu\")(conv1)\n", 367 | " batch1 = BatchNormalization(axis=1)(conv1)\n", 368 | " pool1 = MaxPooling2D(pool_size=(2, 2))(batch1)\n", 369 | " pool1 = Dropout(dropout_proba)(pool1)\n", 370 | "\n", 371 | "\n", 372 | " conv2 = Conv2D(64, (3, 3), padding=\"same\", activation=\"relu\")(pool1)\n", 373 | " conv2 = Conv2D(64, (3, 3), padding=\"same\", activation=\"relu\")(conv2)\n", 374 | " batch2 = BatchNormalization(axis=1)(conv2)\n", 375 | " pool2 = MaxPooling2D(pool_size=(2, 2))(batch2)\n", 376 | " pool2 = Dropout(dropout_proba)(pool2)\n", 377 | "\n", 378 | " \n", 379 | " conv3 = Conv2D(128, (3, 3), padding=\"same\", activation=\"relu\")(pool2)\n", 380 | " conv3 = Conv2D(128, (3, 3), padding=\"same\", activation=\"relu\")(conv3)\n", 381 | " batch3 = BatchNormalization(axis=1)(conv3)\n", 382 | " pool3 = MaxPooling2D(pool_size=(2, 2))(batch3)\n", 383 | " pool3 = Dropout(dropout_proba)(pool3)\n", 384 | "\n", 385 | " conv4 = Conv2D(256, (3, 3), padding=\"same\", activation=\"relu\")(pool3)\n", 386 | " conv4 = Conv2D(256, (3, 3), padding=\"same\", activation=\"relu\")(conv4)\n", 387 | " batch4 = BatchNormalization(axis=1)(conv4)\n", 388 | " pool4 = MaxPooling2D(pool_size=(2, 2))(batch4)\n", 389 | " \n", 390 | " if dropout:\n", 391 | " pool4 = Dropout(dropout_proba)(pool4)\n", 392 | " \n", 393 | " conv5 = Conv2D(512, (3, 3), padding=\"same\", activation=\"relu\")(pool4)\n", 394 | " conv5 = Conv2D(512, (3, 3), padding=\"same\", activation=\"relu\")(conv5)\n", 395 | "\n", 396 | " if dropout:\n", 397 | " conv5 = Dropout(dropout_proba)(conv5) \n", 398 | "\n", 399 | " up6_interm = UpSampling2D(size=(2, 2))(conv5)\n", 400 | " \n", 401 | " up6 = concatenate([up6_interm, conv4], axis=1)\n", 402 | "\n", 403 | " conv6 = Conv2D(256, (3, 3), padding=\"same\", activation=\"relu\")(up6)\n", 404 | " conv6 = Conv2D(256, (3, 3), padding=\"same\", activation=\"relu\")(conv6)\n", 405 | " batch6 = BatchNormalization(axis=1)(conv6)\n", 406 | "\n", 407 | " up7 = concatenate([UpSampling2D(size=(2, 2))(conv6), conv3], axis=1)\n", 408 | " \n", 409 | " if dropout:\n", 410 | " up7 = Dropout(dropout_proba)(up7) \n", 411 | " \n", 412 | " conv7 = Conv2D(128, (3, 3), padding=\"same\", activation=\"relu\")(up7)\n", 413 | " conv7 = Conv2D(128, (3, 3), padding=\"same\", activation=\"relu\")(conv7)\n", 414 | " batch7 = BatchNormalization(axis=1)(conv7)\n", 415 | "\n", 416 | " up8 = concatenate([UpSampling2D(size=(2, 2))(batch7), conv2], axis=1)\n", 417 | " up8 = Dropout(dropout_proba)(up8)\n", 418 | "\n", 419 | " conv8 = Conv2D(64, (3, 3), padding=\"same\", activation=\"relu\")(up8)\n", 420 | " conv8 = Conv2D(64, (3, 3), padding=\"same\", activation=\"relu\")(conv8)\n", 421 | " batch8 = BatchNormalization(axis=1)(conv8)\n", 422 | "\n", 423 | " up9 = concatenate([UpSampling2D(size=(2, 2))(batch8), conv1], axis=1)\n", 424 | " up9 = Dropout(dropout_proba)(up9)\n", 425 | "\n", 426 | " conv9 = Conv2D(32, (3, 3), padding=\"same\", activation=\"relu\")(up9)\n", 427 | " conv9 = Conv2D(32, (3, 3), padding=\"same\", activation=\"relu\")(conv9)\n", 428 | " batch9 = BatchNormalization(axis=1)(conv9)\n", 429 | "\n", 430 | " conv10 = Conv2D(1, (1, 1), activation=\"sigmoid\")(batch9)\n", 431 | "\n", 432 | " model = Model(outputs=conv10, inputs=inputs)\n", 433 | "\n", 434 | " model.compile(optimizer=Adam(lr = learning_rate, decay=decay_rate), loss= dice_coef_loss, \n", 435 | " metrics = [dice_coef])\n", 436 | "\n", 437 | " return model" 438 | ] 439 | }, 440 | { 441 | "cell_type": "markdown", 442 | "metadata": {}, 443 | "source": [ 444 | "### 2.3. Score Measure" 445 | ] 446 | }, 447 | { 448 | "cell_type": "code", 449 | "execution_count": 25, 450 | "metadata": {}, 451 | "outputs": [], 452 | "source": [ 453 | "## Score measure\n", 454 | "\n", 455 | "def numeric_score(y_pred, y_true):\n", 456 | " \"\"\"Compute True Positive, True Negative, False Positive, False Negative classifications\n", 457 | " between a prediction and its ground truth\n", 458 | " :param y_pred: prediction\n", 459 | " :param y_true: ground truth\n", 460 | " :return: True Positive, True Negative, False Positive, False Negative\n", 461 | " \"\"\"\n", 462 | " y_pred = y_pred.astype(int)\n", 463 | " y_true = y_true.astype(int)\n", 464 | " FP = float(np.sum((y_pred == 1) & (y_true == 0)))\n", 465 | " FN = float(np.sum((y_pred == 0) & (y_true == 1)))\n", 466 | " TP = float(np.sum((y_pred == 1) & (y_true == 1)))\n", 467 | " TN = float(np.sum((y_pred == 0) & (y_true == 0)))\n", 468 | " return FP, FN, TP, TN\n", 469 | "\n", 470 | "\n", 471 | "def jaccard_score(y_pred, y_true):\n", 472 | " \"\"\"Compute Jaccard Score (= Intersection / Union) between a prediction and its ground truth\n", 473 | " :param y_pred: prediction\n", 474 | " :param y_true: ground truth\n", 475 | " :return: Jaccard score value\n", 476 | " \"\"\"\n", 477 | " intersection = (y_pred * y_true).sum()\n", 478 | " union = y_pred.sum() + y_true.sum() - intersection\n", 479 | " if union == 0:\n", 480 | " return 1.\n", 481 | " else:\n", 482 | " return float(intersection)/union\n", 483 | " \n", 484 | " \n", 485 | "def pixel_wise_accuracy(y_true, y_pred):\n", 486 | " \"\"\"Compute Pixel-wise accuracy (= number of well classified pixel / total number of pixel) \n", 487 | " between a prediction and its ground truth\n", 488 | " :param y_pred: prediction\n", 489 | " :param y_true: ground truth\n", 490 | " :return: Pixel-wise accuracy value\n", 491 | " \"\"\"\n", 492 | " y_true_f = y_true.reshape([1, img_rows * img_cols])\n", 493 | " y_pred_f = y_pred.reshape([1, img_rows * img_cols])\n", 494 | " return 1 - np.count_nonzero(y_pred_f - y_true_f) / y_true_f.shape[1]\n", 495 | " \n", 496 | "\n", 497 | "def precision_score(y_pred, y_true):\n", 498 | " \"\"\"Compute precision (= TP / (TP+FP)) between a prediction and its ground truth\n", 499 | " :param y_pred: prediction\n", 500 | " :param y_true: ground truth\n", 501 | " :return: Precision score value\n", 502 | " \"\"\"\n", 503 | " FP, FN, TP, TN = numeric_score(y_pred, y_true)\n", 504 | " if (TP + FP) <= 0:\n", 505 | " return 0.\n", 506 | " else:\n", 507 | " return np.divide(TP, TP + FP) \n", 508 | " \n", 509 | "\n", 510 | "def sensitivity_score(y_pred, y_true):\n", 511 | " \"\"\"Compute sensitivity (= TP / (TP+FN)) between a prediction and its ground truth\n", 512 | " :param y_pred: prediction\n", 513 | " :param y_true: ground truth\n", 514 | " :return: Sensitivity score value\n", 515 | " \"\"\"\n", 516 | " FP, FN, TP, TN = numeric_score(y_pred, y_true)\n", 517 | " if (TP + FN) <= 0:\n", 518 | " return 0.\n", 519 | " else:\n", 520 | " return np.divide(TP, TP + FN) \n" 521 | ] 522 | }, 523 | { 524 | "cell_type": "markdown", 525 | "metadata": {}, 526 | "source": [ 527 | "# 3. Uncertainty Measure" 528 | ] 529 | }, 530 | { 531 | "cell_type": "code", 532 | "execution_count": 26, 533 | "metadata": {}, 534 | "outputs": [], 535 | "source": [ 536 | "def range_transform(sample):\n", 537 | " \"\"\"\n", 538 | " Range normalization for 255 range of values\n", 539 | " :param sample: numpy array for normalize\n", 540 | " :return: normalize numpy array\n", 541 | " \"\"\"\n", 542 | " if (np.max(sample) == 1):\n", 543 | " sample = sample * 255\n", 544 | "\n", 545 | " m = 255 / (np.max(sample) - np.min(sample))\n", 546 | " n = 255 - m * np.max(sample)\n", 547 | " return (m * sample + n) / 255\n", 548 | "\n", 549 | "\n", 550 | "def compute_dice_coef(y_true, y_pred):\n", 551 | " \"\"\"\n", 552 | " Computes the Dice-Coefficient of a prediction given its ground truth.\n", 553 | " :param y_true: Ground truth.\n", 554 | " :param y_pred: Prediction.\n", 555 | " :return: Dice-Coefficient value.\n", 556 | " \"\"\"\n", 557 | " smooth = 1. # smoothing value to deal zero denominators.\n", 558 | " y_true_f = y_true.reshape([1, img_rows * img_cols])\n", 559 | " y_pred_f = y_pred.reshape([1, img_rows * img_cols])\n", 560 | " intersection = np.sum(y_true_f * y_pred_f)\n", 561 | " return (2. * intersection + smooth) / (np.sum(y_true_f) + np.sum(y_pred_f) + smooth)\n", 562 | "\n", 563 | "def compute_uncertain(sample, model):\n", 564 | " \"\"\"\n", 565 | " Computes uncertainty map for a given sample and its prediction for a given model, based on the\n", 566 | " number of step predictions defined in constants file.\n", 567 | " :param sample: input sample.\n", 568 | " :param model: unet model with Dropout layers.\n", 569 | " :return: averaged-thresholded predictions after nb_steps_prediction samples\n", 570 | " :return: overall uncertainty (not map)\n", 571 | " :return: uncertainty_map.\n", 572 | " \"\"\"\n", 573 | " X = np.zeros([1, img_rows, img_cols])\n", 574 | "\n", 575 | " for t in range(nb_step_predictions):\n", 576 | " prediction = model.predict(sample, verbose=0).reshape([1, img_rows, img_cols])\n", 577 | " X = np.concatenate((X, prediction))\n", 578 | "\n", 579 | " X = np.delete(X, [0], 0)\n", 580 | " # averaged-thresholded predictions after np_step_prediction predicted samples\n", 581 | " X_prediction = cv2.threshold(np.mean(X, axis=0), threshold, 1, cv2.THRESH_BINARY)[1].astype('uint8')\n", 582 | "\n", 583 | " if (apply_edt):\n", 584 | " # apply distance transform normalization.\n", 585 | " var = np.var(X, axis=0)\n", 586 | " transform = range_transform(edt(1 - prediction))\n", 587 | " return X_prediction, np.sum(np.multiply(var, transform)), np.multiply(var, transform)\n", 588 | "\n", 589 | " else:\n", 590 | " return X_prediction, np.sum(np.sqrt(np.var(X, axis=0))), np.var(X, axis=0)\n", 591 | "\n", 592 | "\n", 593 | "def compute_uncertain_map(sample, prediction, model):\n", 594 | " \"\"\"\n", 595 | " Computes uncertainty map for a given sample and its prediction for a given model, based on the\n", 596 | " number of step predictions defined in constants file.\n", 597 | " :param sample: input sample.\n", 598 | " :param prediction: input sample prediction.\n", 599 | " :param model: unet model with Dropout layers.\n", 600 | " :return: uncertainty map.\n", 601 | " \"\"\"\n", 602 | " X = np.zeros([1, img_rows, img_cols])\n", 603 | "\n", 604 | " for t in range(nb_step_predictions):\n", 605 | " prediction = model.predict(sample, verbose=0).reshape([1, img_rows, img_cols])\n", 606 | " X = np.concatenate((X, prediction))\n", 607 | "\n", 608 | " X = np.delete(X, [0], 0)\n", 609 | "\n", 610 | " if (apply_edt):\n", 611 | " # apply distance transform normalization.\n", 612 | " var = np.var(X, axis=0)\n", 613 | " transform = range_transform(edt(prediction))\n", 614 | " return var * transform\n", 615 | "\n", 616 | " else:\n", 617 | " return np.var(X, axis=0) " 618 | ] 619 | }, 620 | { 621 | "cell_type": "markdown", 622 | "metadata": {}, 623 | "source": [ 624 | "# 4. Active Learning Loop" 625 | ] 626 | }, 627 | { 628 | "cell_type": "code", 629 | "execution_count": 27, 630 | "metadata": {}, 631 | "outputs": [], 632 | "source": [ 633 | "def add_annotated_sample(original, added, added_index):\n", 634 | " \"\"\"append the annotated samples to the training set\n", 635 | " \"\"\"\n", 636 | " return np.vstack((original, added[added_index] ))\n", 637 | "\n", 638 | "\n", 639 | "def data_splitting(X_set, training_size, validation_size, test_size):\n", 640 | " \"\"\"generate indexes to split initial dataset in train, validation and test sets\n", 641 | " :param X_set: initial dataset (all the samples)\n", 642 | " :param training_size: size of the training set\n", 643 | " :param validation_size: size of the validation set\n", 644 | " :param test_size: size of the test set\n", 645 | " :return: indexes for labelled, unlabelled, test and validation sets\n", 646 | " \"\"\"\n", 647 | " unlabelled_size = len(X_set) - training_size - validation_size - test_size\n", 648 | " index_labelled = np.arange(0, training_size, 1)\n", 649 | " index_unlabelled = np.arange(training_size, training_size + unlabelled_size, 1)\n", 650 | " index_test = np.arange(training_size + unlabelled_size, training_size + unlabelled_size + test_size, 1)\n", 651 | " index_validation = np.arange(training_size + unlabelled_size + test_size, len(X_set), 1)\n", 652 | " return index_labelled, index_unlabelled, index_test, index_validation\n", 653 | "\n", 654 | "def split_for_simulation(X, y, index_labelled, index_unlabelled, index_test, index_validation):\n", 655 | " \"\"\"split initial dataset in train, validation and test sets based on indexes \n", 656 | " \"\"\"\n", 657 | " X_labelled_0 = X[index_labelled]\n", 658 | " y_labelled_0 = y[index_labelled]\n", 659 | "\n", 660 | " X_unlabelled_0 = X[index_unlabelled]\n", 661 | " y_unlabelled_0 = y[index_unlabelled]\n", 662 | "\n", 663 | " X_test = X[index_test]\n", 664 | " y_test = y[index_test]\n", 665 | " \n", 666 | " X_validation = X[index_validation]\n", 667 | " y_validation = y[index_validation]\n", 668 | " \n", 669 | " return X_labelled_0, y_labelled_0, X_unlabelled_0, y_unlabelled_0, X_test, y_test, X_validation, y_validation\n" 670 | ] 671 | }, 672 | { 673 | "cell_type": "code", 674 | "execution_count": 28, 675 | "metadata": {}, 676 | "outputs": [], 677 | "source": [ 678 | "def scores(model, X_test, y_test, score_global):\n", 679 | " \"\"\"Compute predictions scores on test set for the current active learning iteration\n", 680 | " :param model: trained Unet model\n", 681 | " :param X_test: test set\n", 682 | " :param y_test: test ground truth\n", 683 | " :param score global: numpy array with previous scores\n", 684 | " :return: numpy array with scores\n", 685 | " \"\"\"\n", 686 | " uncertain_map = []\n", 687 | " uncertain = np.zeros(len(X_test))\n", 688 | " dice = np.zeros(len(X_test))\n", 689 | " jaccard = np.zeros(len(X_test))\n", 690 | " accuracy = np.zeros(len(X_test))\n", 691 | " sensitivity = np.zeros(len(X_test))\n", 692 | " precision = np.zeros(len(X_test))\n", 693 | " \n", 694 | " print(\"Computing predictions on test data ...\\n\")\n", 695 | " predictions = predict(X_test, model)\n", 696 | "\n", 697 | " for i in range(len(X_test)):\n", 698 | " sample = X_test[i].reshape([1, 1, img_rows, img_cols])\n", 699 | " sample_prediction = cv2.threshold(predictions[i], threshold, 1, cv2.THRESH_BINARY)[1].astype('uint8')\n", 700 | " dice[i] = compute_dice_coef(y_test[i][0], sample_prediction)\n", 701 | " jaccard[i] = jaccard_score(sample_prediction, y_test[i][0, :, :])\n", 702 | " accuracy[i] = pixel_wise_accuracy(sample_prediction, y_test[i][0, :, :])\n", 703 | " sensitivity[i] = sensitivity_score(sample_prediction, y_test[i][0, :, :])\n", 704 | " precision[i] = precision_score(sample_prediction, y_test[i][0, :, :])\n", 705 | " _ , uncertain[i], uncertain_map_i = compute_uncertain(sample, model)\n", 706 | " #uncertain_map_i = compute_uncertain_map(sample, sample_prediction, model)\n", 707 | " uncertain_map.append(uncertain_map_i)\n", 708 | "\n", 709 | " uncertain_map_array = np.asarray(uncertain_map)\n", 710 | " print(\"Done computing predictions on test data\")\n", 711 | " metrics_array = np.array([np.mean(dice), np.mean(jaccard), np.mean(accuracy), np.mean(precision),\n", 712 | " np.mean(sensitivity)])\n", 713 | " score_global = np.vstack((score_global, metrics_array))\n", 714 | " return score_global\n", 715 | "\n", 716 | "\n", 717 | "def balance_uncertainty(sample_pred, uncertainty, apply_edt):\n", 718 | " \"\"\"Multiply uncertainty by a ratio to overcome unbalanced class issues\n", 719 | " :param sample_pred: prediction for a sample\n", 720 | " :param uncertainty: associated uncertainty measure\n", 721 | " :return: scaled uncertainty measure\n", 722 | " \"\"\"\n", 723 | " if apply_edt == True:\n", 724 | " ratio = 1\n", 725 | " else:\n", 726 | " ratio = (1.0 / ((np.count_nonzero(sample_pred) + 1.0) / (512*512* 1.0)))\n", 727 | " return uncertainty * ratio\n", 728 | "\n", 729 | "\n", 730 | "def uncertainty_for_ranking(model, X_set):\n", 731 | " \"\"\"Compute uncertainty and uncertainty map for a set of images\n", 732 | " :param model: trained Unet model\n", 733 | " :param X_set: set of data\n", 734 | " :return: numpy array with uncertainty value and numpy array of uncertainty maps for each image of X_set,\n", 735 | " and final prediction for X_set, which is an AVERAGE OF THE nb_step_prediction FORWARD PASS PREDICTIONS\n", 736 | " \"\"\"\n", 737 | " # uncertainty computation for unlabelled\n", 738 | " print(\"Computing predictions for unlabelled data ...\\n\")\n", 739 | " uncertain_map = []\n", 740 | " uncertainty = np.zeros(len(X_set))\n", 741 | " predictions = predict(X_set, model)\n", 742 | " X = []\n", 743 | " for i in range(len(X_set)):\n", 744 | " sample = X_set[i].reshape([1, 1, img_rows, img_cols])\n", 745 | " sample_prediction = cv2.threshold(predictions[i], threshold, 1, cv2.THRESH_BINARY)[1].astype('uint8')\n", 746 | " X_i, uncertainty_i, uncertain_map_i = compute_uncertain(sample, model)\n", 747 | " uncertainty[i] = balance_uncertainty(X_i, uncertainty_i, apply_edt) # scale uncertainty\n", 748 | " uncertain_map.append(uncertain_map_i)\n", 749 | " X.append(X_i)\n", 750 | " uncertain_map_array = np.asarray(uncertain_map)\n", 751 | " return uncertainty, uncertain_map, X\n", 752 | "\n", 753 | "\n", 754 | "\n", 755 | "\n", 756 | "def to_be_added_index(active_learning_iter_size, index, uncertain_unlabelled, dont_random):\n", 757 | " \"\"\" Select the most uncertain samples, retrieve its indice in order to add it to the training set\n", 758 | " :param active_learning_iter_size: number of samples selected to be added to the training set after each AL iteration\n", 759 | " :param index: list of already selected unlabelled samples\n", 760 | " :param uncertain_unlabelled: Uncertainty measures for all unlabelled samples\n", 761 | " :param dont_random: number of randomly selected samples to be added to the training set (< active_learning_iter_size)\n", 762 | " :return: list of selected samples indices after each AL iteration\n", 763 | " \"\"\"\n", 764 | " uncert_unlab = np.copy(uncertain_unlabelled)\n", 765 | " if dont_random == 0:\n", 766 | " for m in range(active_learning_iter_size):\n", 767 | " to_be_added = np.argmax(uncert_unlab)\n", 768 | " while to_be_added in index:\n", 769 | " uncert_unlab[to_be_added] = 0\n", 770 | " to_be_added = np.argmax(uncert_unlab)\n", 771 | " index.append(to_be_added)\n", 772 | " return index\n", 773 | " else:\n", 774 | " for m in range(active_learning_iter_size - dont_random):\n", 775 | " to_be_added = np.argmax(uncertain_unlabelled)\n", 776 | " while to_be_added in index:\n", 777 | " uncertain_unlabelled[to_be_added] = 0\n", 778 | " to_be_added = np.argmax(uncertain_unlabelled)\n", 779 | " index = np.append(np.asarray(index), np.asarray(to_be_added))\n", 780 | " left_to_pick = np.setdiff1d(np.arange(len(uncertain_unlabelled)), np.asarray(index))\n", 781 | " random_index = np.random.choice(left_to_pick, dont_random, replace=False)\n", 782 | " return np.hstack((np.asarray(index, dtype = int), np.asarray(random_index, dtype = int)))\n", 783 | "\n", 784 | "\n", 785 | "def to_be_added_random(active_learning_iter_size, index, uncertain_unlabelled):\n", 786 | " \"\"\" Select randomly samples to add to the training set (baseline)\n", 787 | " :param active_learning_iter_size: number of samples selected to be added to the training set after each AL iteration\n", 788 | " :param index: list of already selected unlabelled samples\n", 789 | " :param uncertain_unlabelled: Uncertainty measures for all unlabelled samples\n", 790 | " :return: list of selected samples indices after each AL iteration\n", 791 | " \"\"\"\n", 792 | " lim = 0\n", 793 | " for m in range(active_learning_iter_size):\n", 794 | " to_be_added = random.randint(0, len(uncertain_unlabelled) -1)\n", 795 | " while to_be_added in index:\n", 796 | " if lim > 100: \n", 797 | " to_be_added = np.delete[np.arange(len(uncertain_unlabelled), index)[0]]\n", 798 | " to_be_added = random.randint(0, len(uncertain_unlabelled)-1)\n", 799 | " lim = lim + 1\n", 800 | " index.append(to_be_added)\n", 801 | " return index\n", 802 | "\n", 803 | "def save_results(result, name, session_name, random_bool): \n", 804 | " if random_bool == True:\n", 805 | " np.save(str(results_path + name + session_name + \"_random\" ), result)\n", 806 | " else:\n", 807 | " np.save(str(results_path + name + session_name), result)\n" 808 | ] 809 | }, 810 | { 811 | "cell_type": "markdown", 812 | "metadata": {}, 813 | "source": [ 814 | "# 5. Configuration\n", 815 | "**Default toy values for each parameters**
\n" 816 | ] 817 | }, 818 | { 819 | "cell_type": "code", 820 | "execution_count": 29, 821 | "metadata": {}, 822 | "outputs": [], 823 | "source": [ 824 | "# PATH definition\n", 825 | "param_path = '../params/'\n", 826 | "results_path = './results/'\n", 827 | "history_path = '../history/'\n", 828 | "\n", 829 | "# paths to datasets\n", 830 | "path_img = \"./datasets/SEM_image.npy\"\n", 831 | "path_mask = \"./datasets/SEM_mask.npy\"\n", 832 | "\n", 833 | "\n", 834 | "# image dimension (array)\n", 835 | "img_rows = 512\n", 836 | "img_cols = 512\n", 837 | "\n", 838 | "# data augmentation\n", 839 | "fill_mode = 'reflect'\n", 840 | "rotation_range= 10\n", 841 | "horizontal_flip= True\n", 842 | "vertical_flip = True\n", 843 | "rescale = 0\n", 844 | "zoom_range= 0.2\n", 845 | "channel_shift_range = 0.1\n", 846 | "width_shift_range = 0.1\n", 847 | "height_shift_range = 0.1\n", 848 | "\n", 849 | "# Dropout at test time\n", 850 | "K.set_image_dim_ordering('th') # Theano dimension ordering in this code\n", 851 | "smooth = 1.\n", 852 | "Dropout.call = call\n", 853 | "\n", 854 | "# training hyperparameters\n", 855 | "batch_size = 5 \n", 856 | "nb_initial_epochs = 3 # number of epochs\n", 857 | "steps_per_epoch = 2 # number of steps per epochs\n", 858 | "apply_augmentation = True # True or False: Apply Data Augmentation\n", 859 | "nb_step_predictions = 5 # Number of Monte-Carlo samples to compute the uncertainty\n", 860 | "\n", 861 | "dropout_proba = 0.2 # dropout probability\n", 862 | "\n", 863 | "learning_rate = 1e-2\n", 864 | "decay_rate = learning_rate / nb_initial_epochs\n", 865 | "\n", 866 | "apply_edt = True # apply euclidean distance transform when computing the uncertainty\n", 867 | "threshold = 0.5 # prediction threshold\n", 868 | "\n", 869 | "nb_active_learning_iter = 3 # number of active learning iteration\n", 870 | "active_learning_iter_size = 1 # number of patch added during each active learning iteration\n", 871 | "\n", 872 | "\n", 873 | "# Datasets splitting\n", 874 | "init_labelled_size = 3 \n", 875 | "test_size = 3\n", 876 | "validation_size = 4\n", 877 | "\n", 878 | "random_bool = False # True: Baseline (random selection), False: Uncertainty based selection\n", 879 | "dont_random = 0 # Number of randomly picked samples among the selected sample (e.g. adding 2 patches: \n", 880 | " #1 selected with uncertainty, 1 picked randomly)\n", 881 | "\n", 882 | "# name to save the results\n", 883 | "session_name = \"toy_trial\"\n", 884 | "\n", 885 | "# number of experiments: multiple experiments should be run to obtain a proper mean and \n", 886 | "# standard deviation of the results and overcome randomness of the network.\n", 887 | "nb_experiement = 2" 888 | ] 889 | }, 890 | { 891 | "cell_type": "markdown", 892 | "metadata": {}, 893 | "source": [ 894 | "# 6. Main\n" 895 | ] 896 | }, 897 | { 898 | "cell_type": "code", 899 | "execution_count": 30, 900 | "metadata": { 901 | "scrolled": false 902 | }, 903 | "outputs": [ 904 | { 905 | "name": "stdout", 906 | "output_type": "stream", 907 | "text": [ 908 | "\n", 909 | "Loading data...\n", 910 | "\n", 911 | "\n", 912 | "\n", 913 | "*****************EXPERIMENT 1 IS STARTING********************\n", 914 | "('X_labelled shape = ', (3, 1, 512, 512), 'X_unlabelled shape = ', (8, 1, 512, 512))\n", 915 | "('X_test shape = ', (3, 1, 512, 512), 'X_validation shape = ', (4, 1, 512, 512))\n", 916 | "------------DATA SPLITTING DONE---------\n", 917 | "\n", 918 | "TIMESTAMP: 0118_1705_23\n", 919 | "\n", 920 | "---------Starting AL Iteration number 0----------\n", 921 | "------------TRAINING -----------\n", 922 | "Epoch 1/3\n" 923 | ] 924 | }, 925 | { 926 | "ename": "KeyboardInterrupt", 927 | "evalue": "", 928 | "output_type": "error", 929 | "traceback": [ 930 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 931 | "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", 932 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 96\u001b[0m \u001b[0mshuffle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 97\u001b[0m \u001b[0msteps_per_epoch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 98\u001b[0;31m callbacks = [tensorboard])\n\u001b[0m\u001b[1;32m 99\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 100\u001b[0m \u001b[0msave_history\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhistory\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimestamp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhistory_path\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# save training history\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 933 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/keras/legacy/interfaces.pyc\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 89\u001b[0m warnings.warn('Update your `' + object_name +\n\u001b[1;32m 90\u001b[0m '` call to the Keras 2 API: ' + signature, stacklevel=2)\n\u001b[0;32m---> 91\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 92\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_original_function\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 93\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 934 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/keras/engine/training.pyc\u001b[0m in \u001b[0;36mfit_generator\u001b[0;34m(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\u001b[0m\n\u001b[1;32m 1413\u001b[0m \u001b[0muse_multiprocessing\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_multiprocessing\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1414\u001b[0m \u001b[0mshuffle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mshuffle\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1415\u001b[0;31m initial_epoch=initial_epoch)\n\u001b[0m\u001b[1;32m 1416\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1417\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0minterfaces\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlegacy_generator_methods_support\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 935 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/keras/engine/training_generator.pyc\u001b[0m in \u001b[0;36mfit_generator\u001b[0;34m(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\u001b[0m\n\u001b[1;32m 236\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 237\u001b[0m \u001b[0msample_weight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mval_sample_weights\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 238\u001b[0;31m verbose=0)\n\u001b[0m\u001b[1;32m 239\u001b[0m \u001b[0mval_outs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mto_list\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval_outs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 240\u001b[0m \u001b[0;31m# Same labels assumed.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 936 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/keras/engine/training.pyc\u001b[0m in \u001b[0;36mevaluate\u001b[0;34m(self, x, y, batch_size, verbose, sample_weight, steps)\u001b[0m\n\u001b[1;32m 1109\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1110\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1111\u001b[0;31m steps=steps)\n\u001b[0m\u001b[1;32m 1112\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1113\u001b[0m def predict(self, x,\n", 937 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/keras/engine/training_arrays.pyc\u001b[0m in \u001b[0;36mtest_loop\u001b[0;34m(model, f, ins, batch_size, verbose, steps)\u001b[0m\n\u001b[1;32m 390\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 391\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 392\u001b[0;31m \u001b[0mbatch_outs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 393\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_outs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 394\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mbatch_index\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 938 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m 2664\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_legacy_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2665\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2666\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2667\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2668\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mpy_any\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mis_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 939 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m 2634\u001b[0m \u001b[0msymbol_vals\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2635\u001b[0m session)\n\u001b[0;32m-> 2636\u001b[0;31m \u001b[0mfetched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2637\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfetched\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2638\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 940 | "\u001b[0;32m/Users/mel/anaconda3/envs/dal_venv/lib/python2.7/site-packages/tensorflow/python/client/session.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1449\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_created_with_new_api\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1450\u001b[0m return tf_session.TF_SessionRunCallable(\n\u001b[0;32m-> 1451\u001b[0;31m self._session._session, self._handle, args, status, None)\n\u001b[0m\u001b[1;32m 1452\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1453\u001b[0m return tf_session.TF_DeprecatedSessionRunCallable(\n", 941 | "\u001b[0;31mKeyboardInterrupt\u001b[0m: " 942 | ] 943 | } 944 | ], 945 | "source": [ 946 | "# load data\n", 947 | "X_full, y_full = load_data(path_img, path_mask, img_rows, img_cols )\n", 948 | "\n", 949 | "index_labelled, index_unlabelled, index_test, index_validation = data_splitting(X_full, init_labelled_size,\n", 950 | " validation_size, test_size)\n", 951 | "\n", 952 | "# split data into train (labelled, 30%), unlabelled (active learning simulation, 50%), test (20%)\n", 953 | "X_labelled_0, y_labelled_0, X_unlabelled_0, y_unlabelled_0, X_test, y_test , X_validation, y_validation = split_for_simulation(\n", 954 | " X_full, \n", 955 | " y_full,\n", 956 | " index_labelled,\n", 957 | " index_unlabelled,\n", 958 | " index_test, \n", 959 | " index_validation)\n", 960 | "\n", 961 | "score_average = []\n", 962 | "indexes_experiments = []\n", 963 | "\n", 964 | "for r in range(nb_experiement):\n", 965 | " print(\"\\n\\n*****************EXPERIMENT \" + str(r+1) + \" IS STARTING********************\")\n", 966 | " \n", 967 | " rank_iter = np.zeros((nb_active_learning_iter, len(index_unlabelled)))\n", 968 | " uncertain_iter = np.zeros((nb_active_learning_iter, len(index_unlabelled)))\n", 969 | "\n", 970 | " # full experience\n", 971 | " uncertainty_unlabelled = np.zeros((nb_active_learning_iter, len(index_unlabelled)))\n", 972 | " to_be_annotated_index = []\n", 973 | "\n", 974 | " print(\"X_labelled shape = \", X_labelled_0.shape, \"X_unlabelled shape = \", X_unlabelled_0.shape)\n", 975 | " print(\"X_test shape = \", X_test.shape,\"X_validation shape = \", X_validation.shape )\n", 976 | " print(\"------------DATA SPLITTING DONE---------\\n\")\n", 977 | "\n", 978 | " timestamp = format(time.strftime('%m%d_%H%M_%S'))\n", 979 | " print \"TIMESTAMP:\", timestamp\n", 980 | "\n", 981 | "\n", 982 | " # START ACTIVE LEARNING ITERATION\n", 983 | " for i in range(nb_active_learning_iter):\n", 984 | " \n", 985 | " if random_bool == True:\n", 986 | " model_path = \"../models/AL_model_random_\" + str(i) + \".hdf5\"\n", 987 | " else:\n", 988 | " model_path = \"../models/AL_model_test_\" + str(i) + \".hdf5\" # save trained model after each training\n", 989 | " \n", 990 | " print(\"\\n---------Starting AL Iteration number \" + str(i) + '----------')\n", 991 | " \n", 992 | " if i == 0: #initialization \n", 993 | " X_labelled = X_labelled_0\n", 994 | " y_labelled = y_labelled_0\n", 995 | " X_unlabelled = X_unlabelled_0\n", 996 | " y_unlabelled = y_unlabelled_0\n", 997 | " score_global = np.zeros(5)\n", 998 | " else:\n", 999 | " # select the samples that are going to be annotated by expert \n", 1000 | " uncertain_iter[i] = uncertain_unlabelled\n", 1001 | " if random_bool == True:\n", 1002 | " to_be_annotated_index = to_be_added_random(active_learning_iter_size, to_be_annotated_index, \n", 1003 | " uncertain_unlabelled)\n", 1004 | " else:\n", 1005 | " to_be_annotated_index = to_be_added_index(active_learning_iter_size, to_be_annotated_index, \n", 1006 | " uncertain_unlabelled, dont_random)\n", 1007 | " \n", 1008 | " \n", 1009 | " print \"List of samples to be annotated by oracle=\", to_be_annotated_index\n", 1010 | " # add this samples + masks to the train set \n", 1011 | " X_labelled = add_annotated_sample(X_labelled_0, X_unlabelled_0, to_be_annotated_index)\n", 1012 | " y_labelled = add_annotated_sample(y_labelled_0, y_unlabelled_0, to_be_annotated_index)\n", 1013 | "\n", 1014 | "\n", 1015 | " print(\"new X_labelled shape = \", X_labelled.shape, \"new y_labelled shape =\", y_labelled.shape)\n", 1016 | " print(\"new X_unlabelled shape=\", X_unlabelled.shape, \"new y_unlabelled shape =\", y_unlabelled.shape)\n", 1017 | "\n", 1018 | "\n", 1019 | " ## Retrain with new dataset\n", 1020 | "\n", 1021 | " # data augmentation\n", 1022 | " seed = 1\n", 1023 | " train_img_generator = data_generator().flow(X_labelled, seed = seed, batch_size = batch_size, shuffle=False)\n", 1024 | " train_mask_generator = data_generator().flow(y_labelled, seed = seed, batch_size = batch_size, shuffle=False)\n", 1025 | " train_generator = zip(train_img_generator, train_mask_generator)\n", 1026 | "\n", 1027 | "\n", 1028 | " # load model\n", 1029 | " model = unet_full_bn(dropout = True)\n", 1030 | "\n", 1031 | " # retrain from scratch after each iteration\n", 1032 | " reset_weights(model)\n", 1033 | "\n", 1034 | " print(\"------------TRAINING -----------\")\n", 1035 | " save_parameters(timestamp, get_params()) # save parameters\n", 1036 | " tensorboard = TensorBoard(log_dir = str(\"../logs2/\" + str(timestamp)) + \"_iter_\" + str(i), write_images=True) # tensorboard\n", 1037 | " history = model.fit_generator(train_generator, \n", 1038 | " validation_data = (X_validation, y_validation), \n", 1039 | " epochs = nb_initial_epochs, \n", 1040 | " verbose = 2,\n", 1041 | " shuffle = True,\n", 1042 | " steps_per_epoch = steps_per_epoch, \n", 1043 | " callbacks = [tensorboard])\n", 1044 | "\n", 1045 | " save_history(history.history, timestamp, i, history_path) # save training history\n", 1046 | " model.save(model_path) # save fully trained model\n", 1047 | " \n", 1048 | " save_results(uncertain_iter, \"uncertainty_iter_\", session_name, random_bool)\n", 1049 | "\n", 1050 | " print(\"----------- Training done --------------\")\n", 1051 | "\n", 1052 | " print(\"\\n----------- RANKING -----------\")\n", 1053 | " # compute predictions / uncertainty for ranking on unlabelled dataset\n", 1054 | " uncertain_unlabelled, uncertain_map, unlabelled_prediction = uncertainty_for_ranking(model, X_unlabelled)\n", 1055 | " save_results(uncertain_map, \"uncertainty_map_x_unlabelled_\", session_name, random_bool)\n", 1056 | "\n", 1057 | "\n", 1058 | " print(\"------------Computing scores on test set -----------\")\n", 1059 | " # score computation on X_test\n", 1060 | " score_global = scores(model, X_test, y_test, score_global)\n", 1061 | " save_results(score_global, \"score_global_\", session_name, random_bool)\n", 1062 | " \n", 1063 | " # save score global on test set for each experiment\n", 1064 | " score_average.append(score_global)\n", 1065 | " save_results(score_average, \"score_average_\", session_name, random_bool)\n", 1066 | "\n", 1067 | " \n", 1068 | " # track selected samples over experiments\n", 1069 | " indexes_experiments.append(to_be_annotated_index)\n", 1070 | " print \"Selected samples over experiments=\", indexes_experiments\n", 1071 | " save_results(indexes_experiments, \"index_uncert_\", session_name, random_bool)\n" 1072 | ] 1073 | }, 1074 | { 1075 | "cell_type": "markdown", 1076 | "metadata": {}, 1077 | "source": [ 1078 | "# 7. Results" 1079 | ] 1080 | }, 1081 | { 1082 | "cell_type": "markdown", 1083 | "metadata": {}, 1084 | "source": [ 1085 | "### 7.1. Dice on Test Set, compared with Random Baseline" 1086 | ] 1087 | }, 1088 | { 1089 | "cell_type": "code", 1090 | "execution_count": null, 1091 | "metadata": { 1092 | "collapsed": true 1093 | }, 1094 | "outputs": [], 1095 | "source": [ 1096 | "score_average_random = np.load(results_path + \"score_average_\"+ session_name + \"_random.npy\")\n", 1097 | "score_average_uncert = np.load(results_path + \"score_average_\" + session_name + \".npy\")\n", 1098 | "\n", 1099 | "mean_uncert = np.mean(score_average_uncert, axis = 0)\n", 1100 | "mean_rand = np.mean(score_average_random, axis = 0)\n", 1101 | "\n", 1102 | "std_uncert= np.std(score_average_uncert, axis = 0)\n", 1103 | "std_rand = np.std(score_average_random, axis = 0)\n", 1104 | "\n", 1105 | "print\"Random Baseline array (Mean and Std) shapes: \", mean_rand.shape, std_rand.shape\n", 1106 | "print\"Uncertainty method array (Mean and Std) shapes: \", score_average_uncert.shape, score_average_random.shape" 1107 | ] 1108 | }, 1109 | { 1110 | "cell_type": "code", 1111 | "execution_count": null, 1112 | "metadata": { 1113 | "collapsed": true 1114 | }, 1115 | "outputs": [], 1116 | "source": [ 1117 | "mu_rand = mean_rand[1:, 0]\n", 1118 | "sigma_rand = std_rand[1:, 0]\n", 1119 | "\n", 1120 | "mu_uncert = mean_uncert[1:, 0]\n", 1121 | "sigma_uncert = std_uncert[1:, 0]\n", 1122 | "\n", 1123 | "fig, ax = plt.subplots(figsize = (10, 5))\n", 1124 | "ax.plot(range(1, len(mu_rand)+1), mu_rand, lw=2, label='Random', color='blue')\n", 1125 | "ax.plot(range(1, len(mu_uncert)+1), mu_uncert, lw=2, label='Uncert', color='green')\n", 1126 | "ax.fill_between(range(1, len(mu_rand)+1), mu_rand+sigma_rand, mu_rand-sigma_rand, facecolor='blue', alpha=0.1)\n", 1127 | "ax.fill_between(range(1, len(mu_uncert)+1),mu_uncert+sigma_uncert, mu_uncert-sigma_uncert, facecolor='green', alpha=0.1)\n", 1128 | "ax.set_title('Segmentation results on Test Set - Mean and std over multiple experiments', fontsize = 20)\n", 1129 | "ax.legend(loc='lower right', fontsize = 20)\n", 1130 | "ax.set_xlabel('Iteration', fontsize = 20)\n", 1131 | "ax.set_ylabel('Dice', fontsize = 20)\n", 1132 | "ax.set_xticks(range(1, len(mu_uncert)))\n", 1133 | "ax.grid()" 1134 | ] 1135 | }, 1136 | { 1137 | "cell_type": "markdown", 1138 | "metadata": {}, 1139 | "source": [ 1140 | "### 7.2. Segmentation results on Test set for Top 3 MOST UNCERTAIN" 1141 | ] 1142 | }, 1143 | { 1144 | "cell_type": "code", 1145 | "execution_count": null, 1146 | "metadata": { 1147 | "collapsed": true 1148 | }, 1149 | "outputs": [], 1150 | "source": [ 1151 | "uncertain_test, uncertain_map_test, predictions_test = uncertainty_for_ranking(model, X_test)\n", 1152 | "predictions_test = np.asarray(predictions_test)\n", 1153 | "print \"Top 3 most uncertain samples in test set: \" , np.argsort(uncertain_test)[-3:]" 1154 | ] 1155 | }, 1156 | { 1157 | "cell_type": "code", 1158 | "execution_count": null, 1159 | "metadata": { 1160 | "collapsed": true 1161 | }, 1162 | "outputs": [], 1163 | "source": [ 1164 | "for indice in np.argsort(uncertain_test)[-3:]:\n", 1165 | " plt.figure(figsize=(15,15))\n", 1166 | " plt.subplot(141)\n", 1167 | " plt.axis('off')\n", 1168 | " plt.imshow(predictions_test[indice][:,:])\n", 1169 | " plt.title(\"Prediction sample \" + str(indice))\n", 1170 | " plt.subplot(142)\n", 1171 | " plt.axis('off')\n", 1172 | " plt.imshow(np.asarray(uncertain_map_test)[indice, 0])\n", 1173 | " plt.title(\"Uncertainty map sample \"+ str(indice))\n", 1174 | " plt.subplot(143)\n", 1175 | " plt.axis('off')\n", 1176 | " plt.imshow(X_test[indice][0,:,:])\n", 1177 | " plt.title(\"Sample \"+ str(indice))\n", 1178 | " plt.subplot(144)\n", 1179 | " plt.axis('off')\n", 1180 | " plt.imshow(y_test[indice][0,:,:])\n", 1181 | " plt.title(\"Ground Truth sample \"+ str(indice))\n", 1182 | " plt.tight_layout()\n", 1183 | " plt.show()" 1184 | ] 1185 | }, 1186 | { 1187 | "cell_type": "markdown", 1188 | "metadata": {}, 1189 | "source": [ 1190 | "### 7.3. Segmentation results on Test set at a specific iteration" 1191 | ] 1192 | }, 1193 | { 1194 | "cell_type": "code", 1195 | "execution_count": null, 1196 | "metadata": { 1197 | "collapsed": true 1198 | }, 1199 | "outputs": [], 1200 | "source": [ 1201 | "model_iter = unet_full_bn(dropout = True)\n", 1202 | "# load weights of your model at a certain iteration\n", 1203 | "model_iter.load_weights('../models/AL_model_test_10.hdf5')\n", 1204 | "uncertain_test, uncertain_map_test, predictions_test = uncertainty_for_ranking(model_iter, X_test)\n", 1205 | "predictions_test = np.asarray(predictions_test)\n", 1206 | "plt.figure(figsize=(10,10))\n", 1207 | "plt.subplot(221)\n", 1208 | "plt.imshow(predictions_test[2][:,:])\n", 1209 | "plt.title(\"Prediction sample \" + str(2))\n", 1210 | "plt.axis('off')\n", 1211 | "plt.subplot(222)\n", 1212 | "plt.imshow(np.asarray(uncertain_map_test)[2, 0])\n", 1213 | "plt.title(\"Uncertainty map sample \"+ str(2))\n", 1214 | "plt.axis('off')\n", 1215 | "plt.subplot(223)\n", 1216 | "plt.imshow(X_test[2][0,:,:])\n", 1217 | "plt.title(\"Sample \"+ str(2))\n", 1218 | "plt.axis('off')\n", 1219 | "plt.subplot(224)\n", 1220 | "plt.imshow(y_test[2][0,:,:])\n", 1221 | "plt.title(\"Ground Truth sample \"+ str(2))\n", 1222 | "plt.axis('off')\n", 1223 | "plt.tight_layout()\n", 1224 | "plt.show()" 1225 | ] 1226 | } 1227 | ], 1228 | "metadata": { 1229 | "kernelspec": { 1230 | "display_name": "Python (dal_venv)", 1231 | "language": "python", 1232 | "name": "dal_venv" 1233 | }, 1234 | "language_info": { 1235 | "codemirror_mode": { 1236 | "name": "ipython", 1237 | "version": 2 1238 | }, 1239 | "file_extension": ".py", 1240 | "mimetype": "text/x-python", 1241 | "name": "python", 1242 | "nbconvert_exporter": "python", 1243 | "pygments_lexer": "ipython2", 1244 | "version": "2.7.15" 1245 | } 1246 | }, 1247 | "nbformat": 4, 1248 | "nbformat_minor": 2 1249 | } 1250 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | jupyter==1.0.0 2 | numpy==1.14.2 3 | scipy==1.1.0 4 | matplotlib==2.2.2 5 | opencv-python==3.4.2.17 6 | Keras==2.2.2 7 | tensorflow-gpu==1.1.0 8 | AxonDeepSeg==1.1 9 | Pillow==5.2.0 10 | tensorflow-tensorboard==0.1.8 11 | -------------------------------------------------------------------------------- /results/index_uncert_code_refactoring.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/results/index_uncert_code_refactoring.npy -------------------------------------------------------------------------------- /results/score_average_code_refactoring.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/results/score_average_code_refactoring.npy -------------------------------------------------------------------------------- /results/score_global_code_refactoring.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/results/score_global_code_refactoring.npy -------------------------------------------------------------------------------- /results/uncertainty_iter_code_refactoring.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/results/uncertainty_iter_code_refactoring.npy -------------------------------------------------------------------------------- /results/uncertainty_map_x_unlabelled_code_refactoring.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/neuropoly/deep-active-learning/2aff400ca57dc60e1b50dbd2c4002e1ece1dcaec/results/uncertainty_map_x_unlabelled_code_refactoring.npy --------------------------------------------------------------------------------