├── .DS_Store ├── Convolutional Neural Networks ├── .DS_Store ├── Certificate.pdf ├── week1 │ ├── Convolution+model+-+Application+-+v1.ipynb │ └── Convolution+model+-+Step+by+Step+-+v2.ipynb ├── week2 │ ├── Keras+-+Tutorial+-+Happy+House+v2.ipynb │ └── Residual+Networks+-+v2.ipynb ├── week3 │ └── Autonomous+driving+application+-+Car+detection+-+v3.ipynb └── week4 │ ├── Art+Generation+with+Neural+Style+Transfer+-+v4.ipynb │ └── Face+Recognition+for+the+Happy+House+-+v4.ipynb ├── Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization ├── .DS_Store ├── Certificate.pdf ├── Week 1 Programming Assignments │ ├── .DS_Store │ ├── Gradient+Checking.ipynb │ ├── Initialization.ipynb │ └── Regularization.ipynb ├── Week 2 Programming Assignments │ ├── .DS_Store │ └── Optimization+methods.ipynb └── Week 3 Programming Assignments │ ├── .DS_Store │ └── Tensorflow+Tutorial.ipynb ├── Neural Networks and Deep Learning ├── .DS_Store ├── Certificate.pdf ├── Deep-learning-notation.pdf ├── README.md ├── Week 2 Programming Assignments │ ├── Logistic+Regression+with+a+Neural+Network+mindset+week 2 Programming Assignment.ipynb │ └── Python+Basics+With+Numpy_Practice.ipynb ├── Week 3 Programming Assignment │ ├── Planar data classification with one hidden layer_Programming Assignments Week3.ipynb │ └── test └── Week 4 Programming Assignments │ ├── Building+your+Deep+Neural+Network+-+Step+by+Step+week4_1.ipynb │ └── Deep+Neural+Network+-+Application+week4_2.ipynb ├── README.md └── Structuring Machine Learning Projects_Certificate.pdf /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/.DS_Store -------------------------------------------------------------------------------- /Convolutional Neural Networks/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Convolutional Neural Networks/.DS_Store -------------------------------------------------------------------------------- /Convolutional Neural Networks/Certificate.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Convolutional Neural Networks/Certificate.pdf -------------------------------------------------------------------------------- /Convolutional Neural Networks/week2/Keras+-+Tutorial+-+Happy+House+v2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Keras tutorial - the Happy House\n", 8 | "\n", 9 | "Welcome to the first assignment of week 2. In this assignment, you will:\n", 10 | "1. Learn to use Keras, a high-level neural networks API (programming framework), written in Python and capable of running on top of several lower-level frameworks including TensorFlow and CNTK. \n", 11 | "2. See how you can in a couple of hours build a deep learning algorithm.\n", 12 | "\n", 13 | "Why are we using Keras? Keras was developed to enable deep learning engineers to build and experiment with different models very quickly. Just as TensorFlow is a higher-level framework than Python, Keras is an even higher-level framework and provides additional abstractions. Being able to go from idea to result with the least possible delay is key to finding good models. However, Keras is more restrictive than the lower-level frameworks, so there are some very complex models that you can implement in TensorFlow but not (without more difficulty) in Keras. That being said, Keras will work fine for many common models. \n", 14 | "\n", 15 | "In this exercise, you'll work on the \"Happy House\" problem, which we'll explain below. Let's load the required packages and solve the problem of the Happy House!" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "metadata": {}, 22 | "outputs": [ 23 | { 24 | "name": "stderr", 25 | "output_type": "stream", 26 | "text": [ 27 | "Using TensorFlow backend.\n" 28 | ] 29 | } 30 | ], 31 | "source": [ 32 | "import numpy as np\n", 33 | "from keras import layers\n", 34 | "from keras.layers import Input, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D\n", 35 | "from keras.layers import AveragePooling2D, MaxPooling2D, Dropout, GlobalMaxPooling2D, GlobalAveragePooling2D\n", 36 | "from keras.models import Model\n", 37 | "from keras.preprocessing import image\n", 38 | "from keras.utils import layer_utils\n", 39 | "from keras.utils.data_utils import get_file\n", 40 | "from keras.applications.imagenet_utils import preprocess_input\n", 41 | "import pydot\n", 42 | "from IPython.display import SVG\n", 43 | "from keras.utils.vis_utils import model_to_dot\n", 44 | "from keras.utils import plot_model\n", 45 | "from kt_utils import *\n", 46 | "\n", 47 | "import keras.backend as K\n", 48 | "K.set_image_data_format('channels_last')\n", 49 | "import matplotlib.pyplot as plt\n", 50 | "from matplotlib.pyplot import imshow\n", 51 | "\n", 52 | "%matplotlib inline" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "**Note**: As you can see, we've imported a lot of functions from Keras. You can use them easily just by calling them directly in the notebook. Ex: `X = Input(...)` or `X = ZeroPadding2D(...)`." 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "## 1 - The Happy House \n", 67 | "\n", 68 | "For your next vacation, you decided to spend a week with five of your friends from school. It is a very convenient house with many things to do nearby. But the most important benefit is that everybody has commited to be happy when they are in the house. So anyone wanting to enter the house must prove their current state of happiness.\n", 69 | "\n", 70 | "\n", 71 | "
**Figure 1** : **the Happy House**
\n", 72 | "\n", 73 | "\n", 74 | "As a deep learning expert, to make sure the \"Happy\" rule is strictly applied, you are going to build an algorithm which that uses pictures from the front door camera to check if the person is happy or not. The door should open only if the person is happy. \n", 75 | "\n", 76 | "You have gathered pictures of your friends and yourself, taken by the front-door camera. The dataset is labbeled. \n", 77 | "\n", 78 | "\n", 79 | "\n", 80 | "Run the following code to normalize the dataset and learn about its shapes." 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 2, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "name": "stdout", 90 | "output_type": "stream", 91 | "text": [ 92 | "number of training examples = 600\n", 93 | "number of test examples = 150\n", 94 | "X_train shape: (600, 64, 64, 3)\n", 95 | "Y_train shape: (600, 1)\n", 96 | "X_test shape: (150, 64, 64, 3)\n", 97 | "Y_test shape: (150, 1)\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, classes = load_dataset()\n", 103 | "\n", 104 | "# Normalize image vectors\n", 105 | "X_train = X_train_orig/255.\n", 106 | "X_test = X_test_orig/255.\n", 107 | "\n", 108 | "# Reshape\n", 109 | "Y_train = Y_train_orig.T\n", 110 | "Y_test = Y_test_orig.T\n", 111 | "\n", 112 | "print (\"number of training examples = \" + str(X_train.shape[0]))\n", 113 | "print (\"number of test examples = \" + str(X_test.shape[0]))\n", 114 | "print (\"X_train shape: \" + str(X_train.shape))\n", 115 | "print (\"Y_train shape: \" + str(Y_train.shape))\n", 116 | "print (\"X_test shape: \" + str(X_test.shape))\n", 117 | "print (\"Y_test shape: \" + str(Y_test.shape))" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": {}, 123 | "source": [ 124 | "**Details of the \"Happy\" dataset**:\n", 125 | "- Images are of shape (64,64,3)\n", 126 | "- Training: 600 pictures\n", 127 | "- Test: 150 pictures\n", 128 | "\n", 129 | "It is now time to solve the \"Happy\" Challenge." 130 | ] 131 | }, 132 | { 133 | "cell_type": "markdown", 134 | "metadata": {}, 135 | "source": [ 136 | "## 2 - Building a model in Keras\n", 137 | "\n", 138 | "Keras is very good for rapid prototyping. In just a short time you will be able to build a model that achieves outstanding results.\n", 139 | "\n", 140 | "Here is an example of a model in Keras:\n", 141 | "\n", 142 | "```python\n", 143 | "def model(input_shape):\n", 144 | " # Define the input placeholder as a tensor with shape input_shape. Think of this as your input image!\n", 145 | " X_input = Input(input_shape)\n", 146 | "\n", 147 | " # Zero-Padding: pads the border of X_input with zeroes\n", 148 | " X = ZeroPadding2D((3, 3))(X_input)\n", 149 | "\n", 150 | " # CONV -> BN -> RELU Block applied to X\n", 151 | " X = Conv2D(32, (7, 7), strides = (1, 1), name = 'conv0')(X)\n", 152 | " X = BatchNormalization(axis = 3, name = 'bn0')(X)\n", 153 | " X = Activation('relu')(X)\n", 154 | "\n", 155 | " # MAXPOOL\n", 156 | " X = MaxPooling2D((2, 2), name='max_pool')(X)\n", 157 | "\n", 158 | " # FLATTEN X (means convert it to a vector) + FULLYCONNECTED\n", 159 | " X = Flatten()(X)\n", 160 | " X = Dense(1, activation='sigmoid', name='fc')(X)\n", 161 | "\n", 162 | " # Create model. This creates your Keras model instance, you'll use this instance to train/test the model.\n", 163 | " model = Model(inputs = X_input, outputs = X, name='HappyModel')\n", 164 | " \n", 165 | " return model\n", 166 | "```\n", 167 | "\n", 168 | "Note that Keras uses a different convention with variable names than we've previously used with numpy and TensorFlow. In particular, rather than creating and assigning a new variable on each step of forward propagation such as `X`, `Z1`, `A1`, `Z2`, `A2`, etc. for the computations for the different layers, in Keras code each line above just reassigns `X` to a new value using `X = ...`. In other words, during each step of forward propagation, we are just writing the latest value in the commputation into the same variable `X`. The only exception was `X_input`, which we kept separate and did not overwrite, since we needed it at the end to create the Keras model instance (`model = Model(inputs = X_input, ...)` above). \n", 169 | "\n", 170 | "**Exercise**: Implement a `HappyModel()`. This assignment is more open-ended than most. We suggest that you start by implementing a model using the architecture we suggest, and run through the rest of this assignment using that as your initial model. But after that, come back and take initiative to try out other model architectures. For example, you might take inspiration from the model above, but then vary the network architecture and hyperparameters however you wish. You can also use other functions such as `AveragePooling2D()`, `GlobalMaxPooling2D()`, `Dropout()`. \n", 171 | "\n", 172 | "**Note**: You have to be careful with your data's shapes. Use what you've learned in the videos to make sure your convolutional, pooling and fully-connected layers are adapted to the volumes you're applying it to." 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 3, 178 | "metadata": { 179 | "collapsed": true 180 | }, 181 | "outputs": [], 182 | "source": [ 183 | "# GRADED FUNCTION: HappyModel\n", 184 | "\n", 185 | "def HappyModel(input_shape):\n", 186 | " \"\"\"\n", 187 | " Implementation of the HappyModel.\n", 188 | " \n", 189 | " Arguments:\n", 190 | " input_shape -- shape of the images of the dataset\n", 191 | "\n", 192 | " Returns:\n", 193 | " model -- a Model() instance in Keras\n", 194 | " \"\"\"\n", 195 | " \n", 196 | " ### START CODE HERE ###\n", 197 | " # Feel free to use the suggested outline in the text above to get started, and run through the whole\n", 198 | " # exercise (including the later portions of this notebook) once. The come back also try out other\n", 199 | " # network architectures as well. \n", 200 | "\n", 201 | " # Define the input placeholder as a tensor with shape input_shape. Think of this as your input image!\n", 202 | " X_input = Input(input_shape)\n", 203 | "\n", 204 | " # Zero-Padding: pads the border of X_input with zeroes\n", 205 | " X = ZeroPadding2D((3, 3))(X_input)\n", 206 | "\n", 207 | " # CONV -> BN -> RELU Block applied to X\n", 208 | " X = Conv2D(32, (7, 7), strides = (1, 1), name = 'conv0')(X)\n", 209 | " X = BatchNormalization(axis = 3, name = 'bn0')(X)\n", 210 | " X = Activation('relu')(X)\n", 211 | "\n", 212 | " # MAXPOOL\n", 213 | " X = MaxPooling2D((2, 2), name='max_pool')(X)\n", 214 | "\n", 215 | " # FLATTEN X (means convert it to a vector) + FULLYCONNECTED\n", 216 | " X = Flatten()(X)\n", 217 | " X = Dense(1, activation='sigmoid', name='fc')(X)\n", 218 | "\n", 219 | " # Create model. This creates your Keras model instance, you'll use this instance to train/test the model.\n", 220 | " model = Model(inputs = X_input, outputs = X, name='HappyModel')\n", 221 | "\n", 222 | " ### END CODE HERE ###\n", 223 | " \n", 224 | " return model" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "You have now built a function to describe your model. To train and test this model, there are four steps in Keras:\n", 232 | "1. Create the model by calling the function above\n", 233 | "2. Compile the model by calling `model.compile(optimizer = \"...\", loss = \"...\", metrics = [\"accuracy\"])`\n", 234 | "3. Train the model on train data by calling `model.fit(x = ..., y = ..., epochs = ..., batch_size = ...)`\n", 235 | "4. Test the model on test data by calling `model.evaluate(x = ..., y = ...)`\n", 236 | "\n", 237 | "If you want to know more about `model.compile()`, `model.fit()`, `model.evaluate()` and their arguments, refer to the official [Keras documentation](https://keras.io/models/model/).\n", 238 | "\n", 239 | "**Exercise**: Implement step 1, i.e. create the model." 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": 4, 245 | "metadata": { 246 | "collapsed": true 247 | }, 248 | "outputs": [], 249 | "source": [ 250 | "### START CODE HERE ### (1 line)\n", 251 | "happyModel = HappyModel(X_train.shape[1:])\n", 252 | "### END CODE HERE ###" 253 | ] 254 | }, 255 | { 256 | "cell_type": "markdown", 257 | "metadata": {}, 258 | "source": [ 259 | "**Exercise**: Implement step 2, i.e. compile the model to configure the learning process. Choose the 3 arguments of `compile()` wisely. Hint: the Happy Challenge is a binary classification problem." 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": 5, 265 | "metadata": { 266 | "collapsed": true 267 | }, 268 | "outputs": [], 269 | "source": [ 270 | "### START CODE HERE ### (1 line)\n", 271 | "happyModel.compile(optimizer='adam',\n", 272 | " loss='binary_crossentropy',\n", 273 | " metrics=['accuracy'])\n", 274 | "### END CODE HERE ###" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": {}, 280 | "source": [ 281 | "**Exercise**: Implement step 3, i.e. train the model. Choose the number of epochs and the batch size." 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 6, 287 | "metadata": {}, 288 | "outputs": [ 289 | { 290 | "name": "stdout", 291 | "output_type": "stream", 292 | "text": [ 293 | "Epoch 1/40\n", 294 | "600/600 [==============================] - 11s - loss: 2.2106 - acc: 0.5650 \n", 295 | "Epoch 2/40\n", 296 | "600/600 [==============================] - 11s - loss: 0.6308 - acc: 0.7683 \n", 297 | "Epoch 3/40\n", 298 | "600/600 [==============================] - 11s - loss: 0.3249 - acc: 0.8617 \n", 299 | "Epoch 4/40\n", 300 | "600/600 [==============================] - 11s - loss: 0.1754 - acc: 0.9417 \n", 301 | "Epoch 5/40\n", 302 | "600/600 [==============================] - 11s - loss: 0.1406 - acc: 0.9483 \n", 303 | "Epoch 6/40\n", 304 | "600/600 [==============================] - 11s - loss: 0.1115 - acc: 0.9650 \n", 305 | "Epoch 7/40\n", 306 | "600/600 [==============================] - 11s - loss: 0.0921 - acc: 0.9750 \n", 307 | "Epoch 8/40\n", 308 | "600/600 [==============================] - 11s - loss: 0.0823 - acc: 0.9817 \n", 309 | "Epoch 9/40\n", 310 | "600/600 [==============================] - 11s - loss: 0.0688 - acc: 0.9850 \n", 311 | "Epoch 10/40\n", 312 | "600/600 [==============================] - 11s - loss: 0.0656 - acc: 0.9867 \n", 313 | "Epoch 11/40\n", 314 | "600/600 [==============================] - 11s - loss: 0.0717 - acc: 0.9817 \n", 315 | "Epoch 12/40\n", 316 | "600/600 [==============================] - 11s - loss: 0.0665 - acc: 0.9767 \n", 317 | "Epoch 13/40\n", 318 | "600/600 [==============================] - 12s - loss: 0.0488 - acc: 0.9900 \n", 319 | "Epoch 14/40\n", 320 | "600/600 [==============================] - 12s - loss: 0.0506 - acc: 0.9917 \n", 321 | "Epoch 15/40\n", 322 | "600/600 [==============================] - 12s - loss: 0.0509 - acc: 0.9917 \n", 323 | "Epoch 16/40\n", 324 | "600/600 [==============================] - 12s - loss: 0.0401 - acc: 0.9900 \n", 325 | "Epoch 17/40\n", 326 | "600/600 [==============================] - 11s - loss: 0.0398 - acc: 0.9900 \n", 327 | "Epoch 18/40\n", 328 | "600/600 [==============================] - 11s - loss: 0.0352 - acc: 0.9883 \n", 329 | "Epoch 19/40\n", 330 | "600/600 [==============================] - 11s - loss: 0.0325 - acc: 0.9933 \n", 331 | "Epoch 20/40\n", 332 | "600/600 [==============================] - 11s - loss: 0.0295 - acc: 0.9917 \n", 333 | "Epoch 21/40\n", 334 | "600/600 [==============================] - 11s - loss: 0.0293 - acc: 0.9933 \n", 335 | "Epoch 22/40\n", 336 | "600/600 [==============================] - 12s - loss: 0.0381 - acc: 0.9900 \n", 337 | "Epoch 23/40\n", 338 | "600/600 [==============================] - 12s - loss: 0.0664 - acc: 0.9733 \n", 339 | "Epoch 24/40\n", 340 | "600/600 [==============================] - 12s - loss: 0.0499 - acc: 0.9867 \n", 341 | "Epoch 25/40\n", 342 | "600/600 [==============================] - 11s - loss: 0.0281 - acc: 0.9933 \n", 343 | "Epoch 26/40\n", 344 | "600/600 [==============================] - 11s - loss: 0.0302 - acc: 0.9933 \n", 345 | "Epoch 27/40\n", 346 | "600/600 [==============================] - 11s - loss: 0.0326 - acc: 0.9950 \n", 347 | "Epoch 28/40\n", 348 | "600/600 [==============================] - 11s - loss: 0.0261 - acc: 0.9917 \n", 349 | "Epoch 29/40\n", 350 | "600/600 [==============================] - 12s - loss: 0.0197 - acc: 0.9983 \n", 351 | "Epoch 30/40\n", 352 | "600/600 [==============================] - 13s - loss: 0.0181 - acc: 0.9950 \n", 353 | "Epoch 31/40\n", 354 | "600/600 [==============================] - 13s - loss: 0.0211 - acc: 0.9933 \n", 355 | "Epoch 32/40\n", 356 | "600/600 [==============================] - 13s - loss: 0.0139 - acc: 0.9983 \n", 357 | "Epoch 33/40\n", 358 | "600/600 [==============================] - 12s - loss: 0.0158 - acc: 0.9983 \n", 359 | "Epoch 34/40\n", 360 | "600/600 [==============================] - 12s - loss: 0.0171 - acc: 0.9950 \n", 361 | "Epoch 35/40\n", 362 | "600/600 [==============================] - 12s - loss: 0.0155 - acc: 0.9950 \n", 363 | "Epoch 36/40\n", 364 | "600/600 [==============================] - 12s - loss: 0.0091 - acc: 0.9983 \n", 365 | "Epoch 37/40\n", 366 | "600/600 [==============================] - 12s - loss: 0.0167 - acc: 0.9933 \n", 367 | "Epoch 38/40\n", 368 | "600/600 [==============================] - 12s - loss: 0.0123 - acc: 0.9983 \n", 369 | "Epoch 39/40\n", 370 | "600/600 [==============================] - 12s - loss: 0.0150 - acc: 0.9950 \n", 371 | "Epoch 40/40\n", 372 | "600/600 [==============================] - 13s - loss: 0.0093 - acc: 0.9983 \n" 373 | ] 374 | }, 375 | { 376 | "data": { 377 | "text/plain": [ 378 | "" 379 | ] 380 | }, 381 | "execution_count": 6, 382 | "metadata": {}, 383 | "output_type": "execute_result" 384 | } 385 | ], 386 | "source": [ 387 | "### START CODE HERE ### (1 line)\n", 388 | "happyModel.fit(x = X_train, y = Y_train, epochs = 40, batch_size = 50)\n", 389 | "### END CODE HERE ###" 390 | ] 391 | }, 392 | { 393 | "cell_type": "markdown", 394 | "metadata": {}, 395 | "source": [ 396 | "Note that if you run `fit()` again, the `model` will continue to train with the parameters it has already learnt instead of reinitializing them.\n", 397 | "\n", 398 | "**Exercise**: Implement step 4, i.e. test/evaluate the model." 399 | ] 400 | }, 401 | { 402 | "cell_type": "code", 403 | "execution_count": 7, 404 | "metadata": { 405 | "scrolled": true 406 | }, 407 | "outputs": [ 408 | { 409 | "name": "stdout", 410 | "output_type": "stream", 411 | "text": [ 412 | "150/150 [==============================] - 1s \n", 413 | "\n", 414 | "Loss = 0.12383275648\n", 415 | "Test Accuracy = 0.953333337307\n" 416 | ] 417 | } 418 | ], 419 | "source": [ 420 | "### START CODE HERE ### (1 line)\n", 421 | "preds = happyModel.evaluate(x = X_test, y = Y_test)\n", 422 | "### END CODE HERE ###\n", 423 | "print()\n", 424 | "print (\"Loss = \" + str(preds[0]))\n", 425 | "print (\"Test Accuracy = \" + str(preds[1]))" 426 | ] 427 | }, 428 | { 429 | "cell_type": "markdown", 430 | "metadata": {}, 431 | "source": [ 432 | "If your `happyModel()` function worked, you should have observed much better than random-guessing (50%) accuracy on the train and test sets.\n", 433 | "\n", 434 | "To give you a point of comparison, our model gets around **95% test accuracy in 40 epochs** (and 99% train accuracy) with a mini batch size of 16 and \"adam\" optimizer. But our model gets decent accuracy after just 2-5 epochs, so if you're comparing different models you can also train a variety of models on just a few epochs and see how they compare. \n", 435 | "\n", 436 | "If you have not yet achieved a very good accuracy (let's say more than 80%), here're some things you can play around with to try to achieve it:\n", 437 | "\n", 438 | "- Try using blocks of CONV->BATCHNORM->RELU such as:\n", 439 | "```python\n", 440 | "X = Conv2D(32, (3, 3), strides = (1, 1), name = 'conv0')(X)\n", 441 | "X = BatchNormalization(axis = 3, name = 'bn0')(X)\n", 442 | "X = Activation('relu')(X)\n", 443 | "```\n", 444 | "until your height and width dimensions are quite low and your number of channels quite large (≈32 for example). You are encoding useful information in a volume with a lot of channels. You can then flatten the volume and use a fully-connected layer.\n", 445 | "- You can use MAXPOOL after such blocks. It will help you lower the dimension in height and width.\n", 446 | "- Change your optimizer. We find Adam works well. \n", 447 | "- If the model is struggling to run and you get memory issues, lower your batch_size (12 is usually a good compromise)\n", 448 | "- Run on more epochs, until you see the train accuracy plateauing. \n", 449 | "\n", 450 | "Even if you have achieved a good accuracy, please feel free to keep playing with your model to try to get even better results. \n", 451 | "\n", 452 | "**Note**: If you perform hyperparameter tuning on your model, the test set actually becomes a dev set, and your model might end up overfitting to the test (dev) set. But just for the purpose of this assignment, we won't worry about that here.\n" 453 | ] 454 | }, 455 | { 456 | "cell_type": "markdown", 457 | "metadata": {}, 458 | "source": [ 459 | "## 3 - Conclusion\n", 460 | "\n", 461 | "Congratulations, you have solved the Happy House challenge! \n", 462 | "\n", 463 | "Now, you just need to link this model to the front-door camera of your house. We unfortunately won't go into the details of how to do that here. " 464 | ] 465 | }, 466 | { 467 | "cell_type": "markdown", 468 | "metadata": {}, 469 | "source": [ 470 | "\n", 471 | "**What we would like you to remember from this assignment:**\n", 472 | "- Keras is a tool we recommend for rapid prototyping. It allows you to quickly try out different model architectures. Are there any applications of deep learning to your daily life that you'd like to implement using Keras? \n", 473 | "- Remember how to code a model in Keras and the four steps leading to the evaluation of your model on the test set. Create->Compile->Fit/Train->Evaluate/Test." 474 | ] 475 | }, 476 | { 477 | "cell_type": "markdown", 478 | "metadata": {}, 479 | "source": [ 480 | "## 4 - Test with your own image (Optional)\n", 481 | "\n", 482 | "Congratulations on finishing this assignment. You can now take a picture of your face and see if you could enter the Happy House. To do that:\n", 483 | " 1. Click on \"File\" in the upper bar of this notebook, then click \"Open\" to go on your Coursera Hub.\n", 484 | " 2. Add your image to this Jupyter Notebook's directory, in the \"images\" folder\n", 485 | " 3. Write your image's name in the following code\n", 486 | " 4. Run the code and check if the algorithm is right (0 is unhappy, 1 is happy)!\n", 487 | " \n", 488 | "The training/test sets were quite similar; for example, all the pictures were taken against the same background (since a front door camera is always mounted in the same position). This makes the problem easier, but a model trained on this data may or may not work on your own data. But feel free to give it a try! " 489 | ] 490 | }, 491 | { 492 | "cell_type": "code", 493 | "execution_count": 8, 494 | "metadata": {}, 495 | "outputs": [ 496 | { 497 | "name": "stdout", 498 | "output_type": "stream", 499 | "text": [ 500 | "[[ 0.]]\n" 501 | ] 502 | }, 503 | { 504 | "data": { 505 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfWeYXNWRdp3O3dOTc5A0ygFljbIQQiQJARJgMGaxAcOy\nXliv7XXArD/bC044LKyxcWCBBdtgTBBGsCRJSEigOMpppBlJI82MJscO0/l8P7q5VXWkGQ1BPdL2\neZ9Hj6qnTt97+vY9fatOVb0lpJSgoaGRejAN9gQ0NDQGB3rxa2ikKPTi19BIUejFr6GRotCLX0Mj\nRaEXv4ZGikIvfg2NFMWnWvxCiCVCiMNCiBohxHc/q0lpaGice4hPmuQjhDADwBEAuAIA6gFgOwB8\nQUp58LObnoaGxrmC5VO8dxYA1EgpjwEACCFeAIDlANDn4s/KcsuikhwAAIj5Q0wXs9txUlb+vq4O\nnyFn57oMOc2ewcbVnzxlyJEYP0ZaBh7fBHgCYQ6wcQEP/himZZmZzhR1GrLX5zFku9PJxgliT2W4\n3Uzn9eBniYkw01lMOEd3Gr7P09vFxrldqDMJfvxAsAWPZ0adxeICDvycwQD/LqKxqCEL8mHCkSAb\nZ3PiRQ75+bUKRfCYTkfEkL2hCBvncjpwHvzwQJ9LEvB4GelFbJzVhNfRDHama/Pi9QgF/Hhel42N\nC5MbJjs9i+kCYXxfb6+P6fIzS3H+EfyerBb+vQjA+ywY5feckHjt7OTm7w3y+8Nhx2vVG+plOpct\nvq5qa+uhra1DwADwaRZ/KQDUkdf1ADC7vzcUleTAU899CwAAfLsamC5QPtyQs0q4NfL6C9sN+YYv\nTjXkuSOXsHHfvvf/GXJ7kB9j9qWjDdkBhYZsyz7Mxh15D2/OmSvSmc7VNcmQP9i53pCHj5vKxtlc\neCNdvnAB0215f4sh91gbma7INQrnWzHXkN/fv4qNmz/9YkN2W+Yz3aHa3xpynht1+XnT2LgYWUzH\nqk8wXU+P15AtVrzhmtqOsnFlU/AmrtvFb/ZTzfj9jhvXbshb65rZuGlTxhty9VH+nYUD5IfHVGvI\nVy28n40rSGs15GzTMKZ7aiNej7rDuw15+rShbFxDOy7wzy1eznTVDbsMefe+bUx37zU/NuTjHa8Z\ncmEW/94tAp+JRz1VTGcN5xjyiGy8N/cda2Hjxo8aY8gH6w8w3dSyWwAAoKJiGQwU53zDTwhxjxCi\nUghR2dXpPfsbNDQ0koJP4/PPBYD/kFJelXj9AACAlPJnfb1n5Pgi+fOnvwQAADMmjGe6dX8/ZMjX\nfv5apnvs2ScNOT2AJvbSf7yFjSsgZmj1+3uZbuXqNw35S/feZsjPPvx3Nm74PPwVnjdlJtOtenaD\nIV9y6dWG3OXn5tnxdnLuYJTpJo6dQuSLmM7fiU/F17euMeSpU/PYuHXr8En04/v/zHQxqDdkE4xj\nGoqTx4+gJsr9LFMuHkP2lBtyJreGodm31pCDPfzJX1NFjpmO9nzj8XY2zhvFp1vUzd2PG1bgE/7g\nnk2GXN30IRv37RUPGvIPn/sO0/m6enCOPnz45A0tZOPmTsDXe6qrme5zK2YZcollCNNVefA6Tkv/\noiGHBb//GsJbDTnkG8F0MTNe79FufHKbJTfMQwKvVYv3HaYrTY9bwTMrvgyVlVUDMvs/zZN/OwCM\nFkIMF0LYAOAWAFh1lvdoaGicJ/jEPr+UMiKE+BcAeAcAzADwtJTywFnepqGhcZ7g02z4gZTyTQB4\n86wDNTQ0zjt8Yp//k2DMuKHyd0/Hd/tdGXxXdszEXEP+YNXLTNduwV3O0lL0C4/u4LvU02d0GrI7\nk+9uv/lUDb6Ioo9VMqOcjYt50LEVaTysU0jmvOatzYY8clEBG7d40QpDPrSlkulqDmCA5KJZY5nu\nvXW4p7BgDu72jy7nIc0Dx08aclEa90FtLtwTmTIddS0tPFzYGcPf7MlFjzGdlOgyhgX62rWNb7Nx\nkInRiTEuvk8TA/zc697BsGi44BgbV3UAvxdh415oetocQ84fgtd+39HNbJzXc9yQc+08hGdLw/2G\ngowyQ26M8dDk0Ow0Qz52qIbp5l2Fn9Nm4vs7Q20YNXA7MDLS6OFRJH8MQ62TskYzXUBi2M4scH+n\nLfIBG+ftwTmW55QxnVnGjzlr5i1QWXngnPv8GhoaFzD04tfQSFEk1ewfPWGofOy5ePim6RBPGKkL\noBnWeYLvG9bVoMl61x1fQkVRBxs3cQSaRWVpX2E6DzxtyK8/i8k1Dd089DRj2kRDrm7ayHTXL8RQ\nTnM9JoWsWs0TPzKzMVGoYmEx0zUcTiOv/Ew3cQqakMeqSVgnzENP48svNeSIlZvzJ4/tMeTZ89E0\nfOs9nni5bH6FIYssHvYym/GeMIXQlaqu56bygpFolncHeWir3oMu2Yg8dAkONB1i4yamTzBkZ1o2\n0/3Hbx4yZH8YXbDeFp4KOO5SDDN+9arfMd19/3kTjivCENv0y+ewcTW1eM+VZPIMwnfeecuQ586+\nlenyR6ILNsyGSWGhMHfpmr14j2Rmq9mFKBeklxtytonf31XNOw15RuldTNcQiYeGr579GOzZUa/N\nfg0Njb6hF7+GRopCL34NjRRFUn3+caOHySf/K+7zv/HhOqb7f9//qiG/9SoPcWQWoY/0+koMIeUX\n83zTb3/vy4Z8uHEH000t/idDjpJ6pJ/8iIe5rOVYkJLdwcNo0Uz06aZPQ1915uRZbNyTf3jGkHOy\n+fXddRArDyvm8zTPqy7F8OS69Zg6O3ESGwZdUGLIpcW8+GjjB+hA9vTuN+SLZ/FU6LpTuJ9R08j3\nFLLz8RjUd0038UoyqwNfjx+zlOkyYlj0YxU4X2Hm7uhTT6M/XTCFX6v8XNyLoFV3gSCvEelow/0F\nc2A607U78HoXWzGc3N11io0LdWFqtVtynzyaR1KXHd1Mt/ziG3Ee3bhP09SyiY3bvgvnXDScf2c5\nAlPAr5z1BUMOS74OnAJDhNXN+5iuqDAeSryk4lHYVVmnfX4NDY2+oRe/hkaKIqlm/9Rpo+W77/8a\nAAB6DvJwjT8LM6f27+LmjsOBtf49gO87vGE/G5c7CrOoGmt4GO3O72EWXmkW1t+nWbiZ+PivMVTU\n2sMr4WZdcrkh+7reM+SAhw2DDDMSPNiHceUlc+YZ8mP/+QrTff9bGJ7cuv2Phjxj5iVsXGM9fmer\ntr7PdP944xOG/J/P3W3IsTQeprt6BrotxVncr9hbjyZrhx/nf/OM29i43c1vGPLQwnFMZ5Hlhhz2\nowvTFORhtGOn0Hw9uZ2TohwNoDsyejS6ACqRRU46ZsUdPMxr4P2kjDwnD0OJVdu4q7N4BV7jrHx+\nb35++j2G/OL255hu0jB8fja24P2Smcbn2EVukivH/ZTp/vbmfxhywQQkI7FZ+DxmlWF4sjWay3R1\nJ+L34x0rnoJD+xq12a+hodE39OLX0EhRJNXsHzFmiPzR41+Lv+jku629ATTPwmFuMmUVIvHH8UrM\nqBo6jZs+UyeiWff3F44w3ZKr/9WQq6sfNeSbbvohG7f54LOGHJU5TFdT02TI/hhaVv46XggyceRC\nQ84v5XOcPQ1dh67IU0y38k+YrWfLxILLUVPy2biDm9C0NRfy65idiUVArUfQbLz4cu7ejClBKrCI\n3M50PQEsZPnTG0iDZbZyN+iaK79uyFHBOQIP7f6DIc+ac6Uhr9vAz+U/grvn3Vl8J92eg9mRx45j\noUxmfjkbV16K5w6382KsmibMkrtiMX7m3fu5y3h4HRZ75Q7hxCTdhHexaAS/J750y1WGfGAjRqJm\nzueFZTkmzBw97ueZoz1dGH2ykMKe8cWcTObZdzBL9a6rfs90Xoi7Vgsrvgo7K49os19DQ6Nv6MWv\noZGi0ItfQyNFkVSfv6g4R37prrj/V1TOK7jCEfRVpZ/TO1ujGKa7+V+uM+TVz/+Njbvzy/9iyPWt\nPFzodqJv1tiKFMv7T/GKvHkXoU8edXB/vf4w+pYHDv7FkO+59QY27v0PkFwC7JyUcvsmJOwoGVbC\ndCId/VNrD5J5nFBomgtLMg05N4uHx3JG4uvycqwgdAc4CWiHxH2EIQ5ODOENIMW1tKGf39pznI2T\nMSTYGJnLsyE7oli1uXkrUlUvnTOZjWsk3Bj/u5JTWk+aidf/3a1Y0fa1f7iDjdt9AMkxm738etef\nxIzNxZdjFqI7O5ONe/TnvzbkbBsnTxkxCl+n56UxXcyB2Xn5EdwraBd8HlYbhp5nKimbxw/gdT1c\nhePuvulf2bioxHObgH+f0Wg8BDl79g9hx47j2ufX0NDoG3rxa2ikKD4VgefHhRQxCCbaY/V6O5ku\nGsMMqIZ9PFwzdRqalz/+CmZHTbuauw5CYBHKkAJu1v328UdQNwFDdjELLxJZuQ657eZN5eGaWj+G\nZFZce40h/+H1X7Fx49Kw2MOqcK+3NSHPW7ibW2fLrrvDkN/ahe6B08W5/4G0fjJHeJHI4T342aIR\ndOk+3PAuGzd1GmZNOi7iZB69rXi+xhjyJ4pupQdaBmYCOqKctMRVgJ/z0jmY1bdKIRUZW4DXZ3Qh\nD6NlOvB8bY34PaVFTrJxNsK/t/1Dzu8HHnSD3iK9GzJtvK3XnNkYissqG8l07Q1IQCJD3CUwu7Co\n6GgLmuyBKL+Hl87E7LwchQewkvQy+NxNGCb+5TM8DH3PHei6/v6xNUx3+z/HM0cjSn+G/qCf/Boa\nKQq9+DU0UhR68WtopCiS6vPHwgC+xrhP0tvbxnTdMfTBPH7u46YTQsmhU9GXHJXLfVUA9KVUz2fK\nfPQn1+7G0Mq8cbxnoGs48sqPGs4r0DbuwpDSH/b+xpD/9SvcN/vLUz/HeUjuk89dhL7x3u1NTFdV\njT60kJjqGjbzysBuK/q/1UpfuYVLsRfg3s14jZcv4bz6L72BPf5k+EqmyynB0Oq0UrxFTrh5mKu1\nG1+vOcjDkcscSHDiBeyiu2zxRDbO2ol7M3mjuS/8659i/4Y8N/ru3/wZD/FmpeF3uOi6FUz3yp/x\nGNkRvF8+XLOHjVt662JD9nfydO2xpRgWvWgUT5Nu6MZ9pud2PW/IhW5eRTmi+GZDfmfLL5luxbVY\nwRkFTN2+Yxkne0kD3Cdbdi3v83jgYDydvTeg9DnvB2d98gshnhZCtAgh9pO/5QghVgshqhP/Z/d3\nDA0NjfMPAzH7nwGAJcrfvgsAa6WUowFgbeK1hobGBYQBZfgJIcoB4A0p5cTE68MAsEhK2SiEKAaA\n9VLKsf0cAgAA8vOy5fXL45zz/izOl29uw4yluRP5oWzDMWzS1IRmUQFw92DWIiRkiKb1MF1jN2bT\n5afjMVa+xEMm08egWVc4jYd82uqQF7DVh2EdTwc3yyO9+JuaZuNmf30d9gwYOpKHIzPcSDLy3ioM\nS100m5vKva0YIjSV8Kq+hmNYGZiZRardFHNwEclkzMpUqhdPYobfmFGzDXnve7vZuKKhmBlYfhkP\n3U7OR1dCWjDLzizns3G/fw5DsNNmOZhuw3v4WTwBdINGXDaXjdu3CXkAo928ArKjG12faUPxuy3K\n4vf9W2ux7fmkK3mINxzC72zyVB7SzLShaxiJ4D29aMw/sXGH6vH7LCjgFZA/+CVW602dPMOQm2s4\nb7+NtI+bs+QKpivKizu6n1v4I9i/s/acZvgVSik/uiJNAKA63xoaGuc5PvVuv4ybDn2aD0KIe4QQ\nlUKIysDH2IzQ0NA4t/iku/3NQohiYva39DVQSvkEADwBAJBflCNjeVYAALjx0sVs3K4NaOKUl/Ku\ntysrkS9v1Ag0ow/UccKO3U9gJtZDv/oe0/3ukb8b8r13YLZVVy9vdxUV+Hs4adgyplvbjmZ/ZiNm\nejV18I9vCyH186YDvI3V2Ark91v3biPTTV2A5qCTkHKMM3Ozvzkbd8+bu/kO+YkaLI65/jo0DY9u\nb2Xjek7h53zxvzkvnTsL3YA//w53pvNz+L5uwWHcFc/bzTPf3hYYTVj2Nfw+3133v2zcuLFIHDJp\n+Bimm3wLnu+Bp5805JrdvPNxRwtauUVKp9+yXGyBBlZ0K97dsIWNi4XwGP9w3eeZ7qEHf2LIU0Zz\nbshjVRg1yC9AHsNfb/sxGzd7BEZbVv75Sab7aTaeO3s5mv1v7OP8jNU7MYbla+XRspVr41Gqzg4+\nv/7wSZ/8qwDg9oR8OwC81s9YDQ2N8xADCfX9FQA2A8BYIUS9EOIuAHgYAK4QQlQDwOWJ1xoaGhcQ\nzmr2Sym/0Ifqss94LhoaGklEUsk8iocWyLu+Fc90atrLfZb8dCRCcGbxkE/Yiv7qjo1IEmGO8Syq\nheMx9FTXxX/XPGHMinOMQaIFGeWkkd/4Lrbv+umD32a6669AQsVX/4r7EKKAtw3LbECD6qKFPBNr\n3xb0yf3KPmmrB6v1hpZiaDJi5cQQx3fhXse1t/IUjJYjGG5qasD9jKqqY2xcWwBbcjmtVqYDifsI\nQ4ahz3zyRD0bZrXg5yws5CE2kx99z+tXYAVkqGgDG5ddWm7IFdM5IYgjhqHQndtqDXnlBzwDb8Yo\nfF/tEb7/Mn/W5wy5tQ5JO32dPAfUYsUM0wwLvx5eD+5HBbv4/ZI9DOdv6kXi2Qmj+Wc52obh4Ovy\n+F7Pob1Y6fhMPe5zLLyFh7wtPbiv8v7bnAi1fFI84PbLh5+DkyeaNZmHhoZG39CLX0MjRZFcs78w\nT97xhXjIIxKMMF2YTKMrxM3ch+6705CffxHDUjOm8BBYj8Tij/UbPmS65pNIunDRLOSRqz/FC2Ms\nFjSt0hw8A+9wI5rbF+WhWX6onmdiebsxu7B0CueAD3Th2K5O3g120bVIsHH4Q3QBojGeH5Ep0C0q\nKlR4BhuJiRpD12fdFt612OVAMzfYG2C6mRWYCdfcjHyK3T08azIYRPdAvY9cLgxbhmJoYhdn8M98\n93cw07DhIA9H1h/B8106s8KQjzfzcFYB4eOzS35feYL42SaUYAiz5vgJNm7WWCSMefMAJwtJT8Pr\nPT6LZ+cVl2J+m4e4BG/u48e3ubAIqr2hjum6AjjH3BzMGOzq4WFomwmt+fJinlfnzolnyH7/l7+B\nYyfrtdmvoaHRN/Ti19BIUejFr6GRokgqmUcoHIG6pniIb9zIYUzntmDYLg14yuqrK18yZBdJ0ayt\n5n6VL4x7BVdO5BV5PZMxbFJbh37s3MkL2bhD1Uj00auEdZbehoQMm/+EVVoiwl0sKfCyBju5P912\nBH3S4aNKmS6jFee87GLcU3C4eOpsTQ2GujZv4ymg6Vn4vq07sVLNZuZfdRYJrRaU8XnICFZLWqIo\n25W7xWrBfYMhQznxxL4DGFYTEq9PJJdXEL72NPLxTyJhRQCAZXdilea+NzBUGVFIUev9GGKzm/l3\n0UH2XxZOxXtgTzX36wW5r76kVA2u2YWhOLuDh6EzcnDP5XgD7lksmcJblj+/DtOJr57BCWTS7HjM\nP6/HcYuVY2w7jHM2m3k4MjOxL2E2Dfx5rp/8GhopCr34NTRSFEk1+9NcDpgzJW7yNDZy/jqTHU3I\naeO5Gbp2F4bj/u0OzNiqOsaz1tZvwHBWznAekpFRDDcFCBHHXj8nFQEHmuUBJQus5zhmZo1fMs+Q\nT9Tyll9zLUgGkZXDL7FzHoa6Whp5iHDte2jCZ2QhuYndxT9Leydmi7VzzwQOHcUqMz/JsnM6+TGC\nQQwf1rfyEFv3UXR98rIwjOb1chfGH0L3rLOLVy/abfi5nYQj36SQK3YT4pP0LM4RuOGveEyLCd1C\nh2J6l+fi+yomcpP6/R14PcJBdAvHlfJQ2Zpd6Epdv5iHT6+/ESvyZCfPIDx2FF1PTxBdjkCQZ7Au\nGIcZf8OH8PZoG/bXGnJGGrpF++t4qK+kEF06M/Ds1qbG+PnCYR7q7A/6ya+hkaLQi19DI0WR3HZd\nEiASi+8ez5vJedIaatC07/by3X4HKbT47VMvGvLCCl40k1mC5tq2bm52ZfjQHMrLQxPYnsH599LL\nsYNq2weHmK7XRDIPfWgqDy/iRRyt9VgA03GUfxYRQhPYncHNXGHBDMWTjXj8jg7uHpSWYaTEYuHm\nn92Kr63k+FYr7+YbIVl3HR3cvKQZbW0d6FfElEIkmnHWq5ibsRiODYXxXFYHb49mFngMm51fD5MJ\nj+Fwors0bQyP5FD3b1u1kj3nx3NH/Xjuzk7OOXjZVMyuzCpXoh/U1Ldz98kiMRrS3objXDa+G+8R\n+NnW7TnKdOSrgB4SucjK4BmmeXlIimKJ8qxPbyJF1qR3+zU0NM4Gvfg1NFIUevFraKQokurzRyIR\naG2J+5eTCzkBRm4+ZpxVH+dkByPLsNLJbUEf8ZSfh56qIxjaKmjnlV8He9A3awtieC8vjWecnazF\nlllTL7+U6fZtfRvfNwL3BgpLOJFF00Ek+hg+nPunDScJFz13f6FkGPryXkKO6XTxdtIQRr9wyuRZ\nTBULoa6xFvdRhFJ15wngXkcswPcNpMCxkQjuWZhMfJyfZAJmOPgchcCxkuxDxCTPwOvxYQZerIC3\nwoLG9YY4mvjh5nSe8SgB9xtcJn78L1+7wJCdaeh3lxGyFACAcBT3c0xKKFEMQZLRaAP31zPTcA+g\nm4RCc8aWs3ExEnbt7uDtuzsDeB1LCvBecrj4POoacV3kZvD1E4zFr0HsY1Tp6ie/hkaKQi9+DY0U\nRVLNfpPJDHZXnHe/8jAvyikcgZlZFsmzqNo60DQMjsXij6CFp4sN8WM468O2w0xny0HzviOAYUC5\njxOHlM5H7viNq99jOrMTzbCufViQcmobP0a+QJOs/hh3YQ4cxszG1i7umsQATdZuL4bYZpXzUOK4\nicglOG7KbKbbvRk58gIka7KrnWdUCkIg4RbcVA5F0IzOt+LzIaKE+qIWvH0iER7qk8RDEETlJ6Es\nAIAQIW5paeTcfDkODO8V5CG3XSjCr3dZNoZ4L5s7lemceaRdWj622rLU8zZnkpj9QrkeB1ZhzwdL\nhF+DuhYMGQZNeH8EeCc5SCdkHq093Ox3ZeJ1NMXwjcLEw4XXfP4GQz5V38B03t64G2dVXJb+oJ/8\nGhopCr34NTRSFHrxa2ikKJLq88dkDIKhuK9piXCnqL4KyR9CIe4/moMYlqpbj33acip4ddTBLiTp\nWD6Mp/6+1YqVatle9B+bI7yqL6sWz9Vcy/319HKSstqI+w1DR/KwUelFGJba/EoV09W34Pmkcvlb\n21E3oRj3KObM5G2tnRlY8bd73ZtM19qMhA+trbh3EgjxdNAoIelI4xE8sBOXN8pCZ/xZESOvA1Hu\nC5sI80eQ+MmBEP/e7Tbcpzm4ZyfTLZ2D/ru3C+Oi7UHu888eV27I1hLeTzB0Cvc6TCT8KASfh3kc\n7jnFBPe1a07idVx20w1M5yDVqfVRrCB0ZvJQ3KSpEwzZVctTkLfvxDTySeNxHu4CTnjTEsFrZR82\nmeliEA8Rmhy/gIFiIO26hggh1gkhDgohDgghvpb4e44QYrUQojrxf/bZjqWhoXH+YCBmfwQAviml\nnAAAcwDgPiHEBAD4LgCslVKOBoC1idcaGhoXCAbSq68RABoTskcIcQgASgFgOQAsSgx7FgDWA8D9\n/R3LarVBUXHcRO48rvDlWzGsE1VCStRUDBNzNXCYV6PNuBTN7aZtvBIucwiGfMw2DHNlpXOTPbsQ\nK6nmTeTmZeWHGJ4cOgSzzI5t49V/gWbMiuv28WNQfvt2hZc9zYFfx4RhGN7z+nloqL0Zw1TN9fzc\nfpL1GCLhPHrd4vMgYVIlK85KMvko/x4NRQIAmKL42qlUF9pIVdv0adh2evfB/WzcyHFoDtfXcXP4\nb+8iucmtVyCf3+hR3Bwuv3oFvpBK1V2x4tMMAOoTcd7Sqw3ZH+FLppOQkRTk4L0zLJ2Hoeu3rzXk\n/PIZTHfzF28x5JJS/GzdSrbefV//KR5//CKmC4i4W9fl4fdbf/hYG35CiHIAmAYAWwGgMPHDAADQ\nBACFfbxNQ0PjPMSAF78Qwg0ArwDA16WUrHWLjD/OzphULIS4RwhRKYSo9Hp9ZxqioaExCBjQ4hdC\nWCG+8J+TUq5M/LlZCFGc0BcDQMuZ3iulfEJKWSGlrHC70840RENDYxBwVp9fxHMdnwKAQ1LKR4hq\nFQDcDgAPJ/5/7WzHkrEYhBMhJ7OZ+2LhMPqk/l4elqIVYiHS1K+ljbPwBOvwfTYbZ65p24phQIsJ\n9xc6u3g4r2gF8vjv2cJTYn/2AJJ2PvlrDEtFlN6C1bXok1vsPOTTTchDrVbety7dRsgs0/GHMi+N\nj6s6gb+zdqWddNiC18dsxlRaoaSlAklhlQqpZgTwOgpyi5zW15HsFVgVvnwr2SuwmfFzza6Yyca9\n+8EHhpymVAYWEI7/jCL0KqcvW8HGCUinLwYEv/cD9trlXtDHSID8kVP71E0txYq/qXMX9jmuP1yy\n4m5DdrvxWvV081Roux3vg1MnnmS6j74Zfze/n/vDQOL88wHgiwCwTwixO/G3f4f4on9RCHEXAJwA\ngJv7eL+GhsZ5iIHs9n8Aff+eXvbZTkdDQyNZSG67rlAQao/FyRBK0rn/7yWtlFXCBz8hqBASzSKz\n0o65/j10A9qUdtLjlmEIpSeEFXPNmzgxxLYtBwzZ6efhsZ/922ZDtmfi+0bM4G2mTEEMH55o4iFH\naqZ7FPcmQqq4TBHUnVIIJGJh3DhVq+koG6SD8OX3KucKW/A6mmLcnKcvSeEemGTfW0RC8Qioi9Bc\nT9qjT+PErQX5WJHX2cXZTTwkxLl+625DvvYL/9jnPAaK/sz8ZCMWQ7fRbMbrIUK88jBIQt4uJ3cn\nba7sxPs1gaeGhsZZoBe/hkaKIqlmv9lkhlx33FzuCXAzNEjMxIgSCQCJ5nCERAWCyja1l3DHS8X8\n2f0mZhSGCdNC1MovwcgiLFGwubk5v/8IFge5Q+hWjBjKyxqqjh8xZH9YyXyjvPohrrOYSZYjyciz\nKNl5FmKLu3hQg0VRvIQ3TijX1Eral4Vj/DpaaQQhhtfUrJr9hFcfFAIMRwa6RRYHTrLmECdZ8bQj\nGUZObgGCza1DAAAgAElEQVTT9XjQ7Dfb/u+EiV99+y32evgIJJDZuwOzGq+/7go2bs+efYbc0sx3\n9Xt64u5lJMzXVX/QT34NjRSFXvwaGikKvfg1NFIUySXziMWgJ5HhZrbwU+cPQf9a7R3nIP3QvD0Y\npgv4eK1AjGTIScl93BzCz9/ahscIRXgvvb3HMDR382LOI3/8JGYJOmPon3v8PDNt3izk0t+17wDT\ntXfhucN2hY+fEEzQluJqRV4vObdFIWwMhNHPp/sqoTD/nHYr+vUWJUzH9htM+D0JJZ5HyTzCEb5v\n0N2NnzOzZIQhDxnCyUhrCRe9r5cTmtrJfswxso9yoeP6JUvZ6/95FbMNHYS09LXXXmfjll19nSHP\nUjIl16yL7xXoXn0aGhpnhV78GhopiqSa/a5sG8y4sRwAAI6t4xl4EZrFp2SchUnhjN2JYSNfLy98\nSHMgkUNY4U3PHIO8d8WlyP23a8dBPg8/hrZeeJWHZBbPn2vIJxuwDfe2Ss7TFx2HroNZaaWck4Ft\nydo6mpnOSbIe6RdjdaWzce29eD1qqnjorLUTXaYYCX1mu3hxUJZAM92mhExtxCWjX4VKE9ETIm2+\nFU7GcC+e+8T2bYZs3rmDjZtQjBltbW3cjSscia6gy0afU0pWY3Jv40+NTfv4d5Zvw8zU1hjeLzd9\n/m427uCB7YbsUNy9nkRGa1RxEfuDfvJraKQo9OLX0EhR6MWvoZGiSKqz5O8Kwu7X42m2BbmcOLOx\nGQkqTErYKIf0yKtrw3TQrg5O5mEj7btz7TzU17AFK6SsJMxlVsJcBVmYqmu28t/GnkbsjzaiCOfo\nGssrxHIzMWS1/U+7mW5kKR5/YcVEppPd+NnMNvwsa3bxfYnuDhxnF3yOQ9JxT0FGcE/EZeV5wL4g\nJTHlIUe/D8OCwRghUpHcn+wMou9tMfNbqYjsMZgECRcqBCY+K547FONVfX4vzv+BH9+H7wkcZ+M6\nO7HfgSnKP0sx2S8ROSOIZvCee/MmjeWvf/OYIdOm88dO8fv70EHsWbHpw41M91HKt1SZWfqBfvJr\naKQo9OLX0EhRJLeqLxaFdF/clFk6m4drthHzcvy4uUx3vAbJLErzMZzX4+VVYFPGIGFH+cgRTFdb\ng+GVTi8GrabO4Fl8haTdU0Mz5yTNysVz1x5Ffr+hJIwIALBlE7aa/jCD8+pvrsL3FWfyVmETC5Gg\nIUAITabkc8IRDwl7NXd2M50vRDgCiUsQCvIwWr4b3YMcNzfFg71obgriF1li/HYpJKa+ycR1QdK+\ny2FDOTuNm+VZ2ch139jGP0uXD43g1loMrUZHlLJxTlLx5+3hpjLkjDTEsETXzyz4PJob8fgSeHi2\nqBh7C5iAfxefNewkjDmhhId4//T4f+ELpdfCO4fjnH5fvfEnAz6XfvJraKQo9OLX0EhRJNXsT093\nw+JFcZM+7OQm++U3YqaXqYfTUY8eiWQHq/dgEcTkq3gnXjPJlBJufvyKMUWGHHLhrrXZx00rLyGQ\nSMvmTYiy89Dks7kxSzA9ne+kT5iIJnVGzqt8js24o93h4cU2w2bjHL09OC4rg5uop0in3zZlc7ed\npOG5ooQbzsLNRC8xqe1WrgsBPSg+H1oVHkALcSu6Yjz/LwLoxmXFMFpTUsbNZredRF5M/Hv3EPdj\n4UKkxTZZA2xcljPfkLOz+PcuAF0aqygiGu52FhZj11tTP/zfUTjBXgsyVpBisk6FqCXbXkTe0zd8\ncIy84gVuJhO6pIdb1zBdjqsNAADMJjX7sW/oJ7+GRopCL34NjRSFXvwaGimKpPr8VpcdiqfFQ3A2\nF/en21qR1CG/jPvhAcJhP/cSJDEoHckJNo8dxXBeZgH/XXNkYUgpQ6IP6rVwAgnIwLBRpIv7uC3d\n+DrNTUgp67kf2Esy6zJinNwzLR33FCzAHXYHIfcIW0lVXBMPCbZ6cR5SqabLIWHASIh6lwqRKOl5\nEFOcUMq5bybEnE6F7NQTwD2LLBuvMhOEqJSSgJjs/BitTRhOddr4d+ZORwIWP6lkLMooYuNigPMw\nm/OZTgKtHnUTmZ+L+vkS+DUVgNWXZuD7LxLwc8d72cZhlbwV+dFWDP/6u/j+SD3J2Lxq5jU4JzGG\njdvW8u+GvO8AP8YXF98LAAAu24swUJz1yS+EcAghtgkh9gghDgghHkz8PUcIsVoIUZ34P/tsx9LQ\n0Dh/MBCzPwgAi6WUUwBgKgAsEULMAYDvAsBaKeVoAFibeK2hoXGBYCC9+iQAfBR3sib+SQBYDgCL\nEn9/FgDWA8D9/R3L7rTDsClxU8bTyYs40kvKDTkjbTbTRQNIlmE2YZZd8wnexqoof5whO6w8bCQC\naGJH7WikRCOcEKS5rc2Qe308pJSRi2ZdWVExjgtzYhLwkY7DCo96XhaGunw+fm5akxEiUcC8nFw2\nzk3M7+5O7ra0edGt6CSmrEWpYBqSiZ8lpLgOZlqIQx4PboUfLka6xjqUVgvpNjwG7S5rUj5zKIrH\nHF1exnTZOWjeF+YRV1DyLD4LdWkEN4cFEIIXQLfQDOV8woCZlyc79zKNIxvHHmvkuunFlxuypxeL\nuPKcvLNvuoMsNe6ZwGQib2l9yZDn5N/Exk0oXG7INvM+pnt5/f8AAECnh7uI/WFAG35CCHOiQ28L\nAKyWUm4FgEIp5UeOehMAFPZ5AA0NjfMOA1r8UsqolHIqAJQBwCwhxERFLwFbhDMIIe4RQlQKISpb\nW7vONERDQ2MQ8LFCfVLKLgBYBwBLAKBZCFEMAJD4v6WP9zwhpayQUlbk52edaYiGhsYg4Kw+vxAi\nHwDCUsouIYQTAK4AgJ8DwCoAuB0AHk78/9rZT+cCc8LDycriVWYSCIEn8LbWFgdWzXW2ov8fM3FH\ns6n1pCHn5fFKu05CAuKw1RlyezP368fPmmfIddV1TOdpIPMain54yRB+GYuz8LMEI7xSLTsX/dh2\n0qcOAABoiM2BexYxP08DDpLehS1e7vNHCd+/m6Tt5ist0TNISnJ3D/eTpcT5SxIHdNn4PoqZbBWo\nSaV+SiRJZK+H+/w5hEzVrvQxaDmEfi1tXQ1SvW7kXIJXKALgHk6bF+8Xh7WejdpRg+3XF41fxnQW\n4tEWFnMCFgq7krL+STCC3B8n/KuZrsyFYW6Hnfv8X1j0fQAAeDT9zQGfayBx/mIAeFYIYYa4pfCi\nlPINIcRmAHhRCHEXAJwAgJsHfFYNDY1Bx0B2+/cCwLQz/L0dAC47F5PS0NA490gu4bnshWgwbq6Y\nrZzHTJqo+c2zxbzd2F7bnb/YkLPzucl+Yhe2N87M4W2hcvIxNAchdDmKcvg+pdmM5vCIsaOZLkqY\n6yUJo23buI6Nm33x7YZc2MUN4pgLzXSbhW+5OGlPAh++LxxTuNhJG64hJcOYyutBk9hE2mvblGxC\nVzruv3h8PDxEO2/HBMnUU9qeUyNd7bVgIi3BaZRR5ZWPBNANKBvBMzbd5PocrNlkyCETd6W6O/Gz\njRjbxHSlaXMMudjNMwMpLr9ofJ+6T4Ko9Cp/we/dLPp2D44c2WnIY8ZMZroQ4P2Rkzaf6U764u3M\nQjG+JvqDzu3X0EhR6MWvoZGiSK7ZL1xgtsc582SYt7gymcad6R0AAODOxGwpwcxXHl0cOm02Gady\nreH7KBWdOVv5/SMFL6Dw0pnJMWgX4JkXX8+PEcWWVDfcvISp/vbK24Z8UTk3QynFtUWQHX7F7M8i\nLb96FM46WwRdkyCJEjSGuDmYW4w8eBGlg68k/IHUTA9GlUIkkllns/FrFSNjLRaMEsgYP0Y0gPOK\nKakivU2YFzJh1JVwIaFX8CjMzkPPGPK8sf/MdK9V/Qh15Z835FVv/ZaN8/uRyMbm59/7mNFxNyDo\n5xml/UE/+TU0UhR68WtopCj04tfQSFEMQm/j+O+NsE7oZwz3CwVY+xg3VBnX9zHoRzX195N3WoYY\nBb6RcFycvr9gnmGIdd1/ZKoh5cgjX+fjLbp7g+gnxsh8LRYlPEZCeA4Lz3IMEbc5SFp0V0zmxBA+\n0gqLfRgAECS8R9s/5RNCFACADlKZmRZVLiottCOHt9r49aV7ClWHeJXmS6//L1xI6AghgUeVZwvT\n5TkrDPnxd7/CdMNzLsb3daNfP2I0zyYsKcC0mgO7PmS6qr0bAAAg0KuGGPuGfvJraKQo9OLX0EhR\nDILZf2ZQrrXTw3R9vEfy0JBg5ms/v2vEI4iZKpnKBBVnHqiem/C8idOY2PHcd/74l0zzyo8eMmTf\nNp5Zd/QUZudluzHLMSb4PMLEFBeKyU6vSYDMcdchzjM4pAAz/NIFz0L0k/O5iFvR1sJLsiXh6QtF\n+TEcJjTvzSTbL6SELe3k+CW5F3bVp48URM3N+jLTiVy8ppX73me6oweOGPLiAgz1bW/7NRvX1oHu\nb1V9DdNNnTAJAACsTp4d2x/0k19DI0WhF7+GRopCL34NjRTFeePzUz9fQljRkVAfcfO7GngfvOyy\nG/o5A03bpb95M5Rx/fj50R4yCn1ts5mTUMRasC23KbOY6T73Hw8a8sFlK5iul6TBhkM4j3CQ+9M0\nHddiVUg1aWouIc7sCPJrWn0Mq9+y7PwY0wuR4PSDOtyXCCt7GxYSpstUKhRtZsKDHyHVhTYXGyfI\nZ37kqWfgQsPO4xiObGjFfZWymTxdvaoJSTaco44z3bwFSALa1oIkHbEGXpl62U13GfJbK3m6c7gq\nTqfp71bIZPuBfvJraKQo9OLX0EhRnDdmP4Wa0UfdACFQl112LR8Xq8VxpvI+j0/DdCaljRXESOab\nibfeFmYMRZlYS2puDptyS/CF0j6Ktl3+wavPMs1NVyMvextJ1SvMz2HjbE78zZYBbs43BrGqq4OE\n33oCSgsqwtjRE+K6d49htWSQZhNaeXZeL+H7jyo9v8ISq/WyHXgdTUrYsidIW231lcl57nGw4T1D\nHlc6k+k2HsR22FkZ3I0Lk5brY4rQ1G/taGTj3t2C4b2YnbtxLcfWG3J5ydWG7PVxnst333jOkC2d\nPBy+4v54u65XVvG+Av1BP/k1NFIUevFraKQoBsHs/8jsU393+t5lF4SqGpjZyE1NSmTgcPPjSXJ8\nwT62cl4TUlx3dm5nqsxszP4zmfq5dOb+mhcRE16pIXppDWYb3n/bUkPu7uDEELWtrYac7+aRBj+J\nZDhJFCLDopBrSxzXrpiXYdqZl/AKOpSMSpr9F1G+i+4wugRpdkJJrrgHkYhK+v3poEwR2lv34LnJ\n9W7v5PTfnm4kxwhkc3rx+aORzt1k4VmIb+zF3flNnncMOeLlRVDfWP6oIf/mxe8wndOExV61WzFS\n1KUQdgTNyBV55XWcwy+nYAoAAJgt3FXtD/rJr6GRotCLX0MjRaEXv4ZGimIQQ32qT06r5JTwG3MT\n6fv4b5fDPbKPcQBCdbD7OAZFVjbnV+fVe2ol32eLQBCvh93J515ciK3IRIR/zpx0dHpzTYRw1MPD\naE2EOLMsk4eNMqP42boCGNJskrytV6kFK8iCSoZfByHwpG64WokZjPTNM3+keb0h28mtGhb8s4ws\nwJ4ynm6ePVd7El9XVFxnyDZTKxuXnYat2Lyd3OcXJvzcuYXpTDdtPPreY8zIs994nB//1An05a9Y\nMovp3l69zZDHT8CsvhHuS9m4BZNwv+jNSt67oMWzFgAAIjG+T9AfBvzkT7Tp3iWEeCPxOkcIsVoI\nUZ34P/tsx9DQ0Dh/8HHM/q8BwCHy+rsAsFZKORoA1iZea2hoXCAYkNkvhCgDgGUA8BMA+LfEn5cD\nwKKE/CwArAeA+/s/kgQ0xy2KBjPfBPRH6kB/r5QwHSG56I+LTwIxyaRKfoBmnRDD4JOBZt19sqw1\nL+Hfc1u4i9FLTHaLmbtIuaSYp7EJw1l1XZzbjbbeykvjHXy9pPVWmE4/wp8Vu7vwO3MKPo/8HHQl\nKE+fp5eb+c+8/Iwhv/XW/zDdlPEXGXKGGzMlwxHF5SJdep32XKaaOnESTt/XgPPw8QIYbzfOy2pS\nsiFJ6NPn4B2kO9rwmBMmoTlvj3JXym7FEFzzEe4+ndqCZB5lNyAvZVPtJjbOYv6cIe/ZzYk+ls19\nHgAAnFY3DBQDffL/FwB8B/hqK5RSfpTD2AQA/QW3NTQ0zjOcdfELIa4BgBYp5Y6+xsj4Lo48k04I\ncY8QolIIUdna2namIRoaGoOAgTz55wPAdUKIWgB4AQAWCyH+AgDNQohiAIDE/y1nerOU8gkpZYWU\nsiI/P+9MQzQ0NAYBZ/X5pZQPAMADAABCiEUA8C0p5W1CiF8CwO0A8HDi/9fOdqzOUDW8dCKetrqi\n5EWms1rRRwp3rOG6nLmGHCTEFnY7D7tw8JDH2jeeMOTLrvkGKsS5SHX49NVpl1+M1V3rN77JdLSC\nzmbj5wqQ65PuQj/8ksJSNs7rwX2J9m6+H9BJKgNDhDjEbuXnGu5CYg6zEo70kfeFBRqFUcn99eGj\nMDyW4VSqFx24H/P4H39lyF+89SY2Llvi+1o6OPd/5ykMiQ0tx7bwLhs/V+FwnP/6dZwTP52kUIeV\nvoZ5uRiaa21F4pPhw4ezcdsO/t2QRxZdxHTf+8FPDPkUae0dq+fXavN+7AHRdoKHI9/fHV8zHv85\nCPWdAQ8DwBVCiGoAuDzxWkND4wLBx0rykVKuh/iuPkgp2wHgsv7Ga2honL9IaoZflm0orBj2OAAA\nWIFXPXV3njTkjJxFTEd3Em02GsrgPPI+L1anpaWXMd1l13zz40846UCTPS8Tzcs9JzgxRD4JzfmD\nPNOLVjbmu0mFl8LTR+nzLTF+G2SSl1LYyTg+2yjh6euJcHPYG8FvLUYIQcDGqxDv/AqGrEIBviH8\n/NPfN+Qrly42ZHc6v3eamzDc5rBzXclwzNLs9aOpvL9mLRs3ZiS6HxMmT2G6VlJFmZnNc9nsaeh6\nbt6Lx9x4+D02LsOJn+1QLa8WtVrxy7DkYYuuzGFK2G4fzl/28tBqzfZ4u65gksx+DQ2NCxh68Wto\npCiSavYLsIFVfmSOcxMvI5sUqyjTikE70dFdWm6CpaVfCOUFaDvPnbOYabKI2fvg/djJdekEnmlY\n1YI7whGFm8/uxh34CCHb6A1w0gxC4QcRhWCj1o/kIWSjnr0HACDbhSZ8THmOpDlw97zTh3OcMoRz\n4FVXI4HJyPHTmG7O4lsM2UwKs4aOGsvG/eLBGw25MI8fo8uHGelRQQg8XJxC/J2NGKy6+7YfMp3T\ngZ+tuZkX7NiDpww524mf01XOKeG7D2KazC1Xf47pooDX5Pm/Y5Zj9UnOx5ebhdmK02dx9+bVtesB\nAMDvDcJAoZ/8GhopCr34NTRSFHrxa2ikKJLq83siPljfuRUAAC7J4QSEEXjZkIM+nrnnTiP8/OeW\nQ+Mzx8yLeUuuaRWXGHLMyyvLfvH9rxqyh2TZWZSsuCOncG9AAPfls0jb7BYSmrMprbzNMVR2+TlB\nKCmSgxh5UezmoacOUnnoj/JQHwDO3+3G7M0MK5/HgWYMY9Ye4Tq/F0NbhYTAZHclD6N95yHSTyGe\nhmKg5iASZfz299g2OzeriI3zZuM+yh+f+inTTZqGewyVVRuY7ppL/sGQj4QwXA01+9i4zXtwb6Pb\nxjMq62qxqs9MKlMjgoft7FHsB9Fg5/fOpCWzAQDg3ff3w0Chn/waGikKvfg1NFIUSTX7g4FTUF31\nEAAA5E24hemOH0Kz8Zq5vHDjfMd9DzzNXjfUI2/chCkLmc7Tg2HLIVMWMF1tLZqNghJ4RHlq3dJx\nGBra0coLPCJ+NAd9XjTLm7w8G7IoB03lqMKrFyMxvV4/nrsmys3VEMnqsyqtvGgRULmNzN/KyVPM\ngFmZbU21TNfZgZ2Eu9rx2oybdDEbV7nhBTyelYfwVtyGHPmvrMJrc2AHN98f+xWGVjOdvDjIRMKp\n6S7uLhwxYxHa4qn34rn++n027uoltxlywMN7BgwrxmO+/SIWcRUP5eeqrj+Ax4jy7yy7KB4Cj4ZU\n96tv6Ce/hkaKQi9+DY0UhV78GhopiqT6/DFwQECOBgCAw90lTNcqMK0xCv/EdBbq3iQ11Md97efe\nOmHIL/wFiRUiIe53e7rRv+5pO8V0VhvpW6e0q+70ok6GMVQmlDBdiHDpl1u4j/cB4Zzv7UU5HOPj\nOnrQ15ZhniJsIjm9Vhs+H3qVvYcI2StwK0QfI/Ix1VqS6+NycLLQro4aHCf5s8hMyEkzM5GYc/+e\n9Wxcegap3FOIOV/47wcMuXQ4psdOmraIjfvL6/g99XRxgs2778A5T8icyHTdp3Bf5dW9uL9gc/Oe\nedEYEoTmZPMU5w07txjy7d/EfYO9u7eyceVFGHKk+yEAAK+/sxEAOLnr2aCf/BoaKQq9+DU0UhRJ\nNfujIT94Tu4GAIDr53+b6f7mQQKFlw5dw3Q3jnnFkG3mgfOSfxI8/HvMHtu9ixMW19fVGrJFoHnl\nU1pcW80kVKaY1A0NyHNaVsrNv237MHNvdBlmxUUkN7dtpA13moWTOswZTcKAR+twToKTaDR1Y1bf\nsAJeDdnYgtz0Jtbymz8rwiY8tz/Mq8nSQnh8akSv2syvaS95n4zxz+Jwobnd04NziijhrJ5ONIEd\nTp4dGiOuyanjBw25rYG39bI78VxjJvPs05dfx5BsWfE2pnPl4nW9aAJWaZYrPH07D2OYrq35INP1\nNuDn2bR7lyF7u3gF4f7Kw4Y8c/50pktL3IImhXClP+gnv4ZGikIvfg2NFEVSzX6zxQQZufEMrNWN\njzJdQOJO5rXD/o3p/rLzZkP+8kxOY/1J8MiT7xty1dGTTNfTgaZWRibfmXa0oAnpI9TXZsmLa2Kk\n3ZXFwi9xejqapZEIf9+2bjSQhxehTqnrARMJedjM/PjZZFd/WgkWw1S18+y84gJ0n0408Z1j2knX\nRkz7YIgXAOW6kFBiXA7PrKNZiWHyOXvDfCddknEmpS0Z0PcF0LUSFh5ZiJCiokAvL4axkihEMITH\n8PTwtltm0mU4HOLZkGZS3FSTzhtT/fuD5YZcEEG67jUbP2DjQkG8dh4/vwY33PZFQ/7F735hyM4w\nvx72IF7j9sOK6xOLRzmikruZ/UE/+TU0UhR68WtopCj04tfQSFEk1efvDURhf3Xc1/J7OSGDNQfD\nGDsyJjHdyFysGJMkcCSgvzbcHJcsuc+Q5y683JBtZu4j+f0kO6+bZ+6lObEiLRhEXVTwbC4XIYfs\naq9nuhB5H0SUcxN/9XAXht9KonxvQJLMQLOZ+4WkGzYUunG+WQpvf1sP+qBZZbx6zONH3zifEHFG\notyvz7SQ669UBvaSkNO25r655Ol+hhrCMxF/3WxGWa1CjBKfXyr9A8J98FmaldbmtI14dxevuqs9\ngkQcOcWjmO7rdyMRx29+9TVDHjliLhv30p9+bMjBXiWT0Y/nLs/Da7rihjvZuD8+/Iwh93r4Hk5Z\nXjxz1mbhe1j9YUCLP9Gk0wMAUQCISCkrhBA5APA3ACgHgFoAuFlK2dnXMTQ0NM4vfByz/1Ip5VQp\nZUXi9XcBYK2UcjQArE281tDQuEDwacz+5QCwKCE/C/Eefvf39wa3ww7zR40BAIB3Nh1mOocLM+Yu\ncr7NdAtmY0snIbm5RvG75zYb8po165luynQ0w7ZveceQc3N5dltLI7Z+spuU30ZytQTp7iuBm5q0\ni64J+HyFIK+VjLmODjSc3t2ObbhKHDxta2E5Zv+ZYvwYFjpnEkazCj6PsiwM9dnaeMuvXhuODZCw\nolXpaCwJD6AnxMNXW5rJMcm5o8DHmfp13ch1Jdl/Ugmtmsg81LAoDVuaTcR1iHKXS5rwjdEQ9xWa\nGrGgq11xCRwuvH9O1CNvn9nFXcawC12fzhgvPjpwEsO/XT24Dja+xwlHJl2NLsemDdVcVzoOAAAs\nloF3iB7ok18CwBohxA4hxD2JvxVKKT9iX2wCgMIzv1VDQ+N8xECf/AuklA1CiAIAWC2EqKJKKaUU\nQqh7bAAAkPixuAcAILfAeaYhGhoag4ABPfmllA2J/1sA4FUAmAUAzUKIYgCAxP8tfbz3CSllhZSy\nIj3TfqYhGhoagwAh5Rkf2DhAiDQAMEkpPQl5NQA8BACXAUC7lPJhIcR3ASBHSvmd/o5VNtIt7/vF\nVAAAiAZ56uzEirsMOXT8XaabMhO57sfk3GDI997/HBsXiaDv51e46Hu9mLbbcgoJJDq6m/kkSWqu\nUPzkzAxMlw2SdE31Glqt+CMXjXGdJAQhajvlMAnpZadhWM3Tw39XfYQgZEpeFtNNKsWU2yipBrQq\nacBhsh9AOfwBAE40ol9rc+Bn8SuhuDZyjXd28tCTJM63iYTV6HekwqTssVASEyrHlPlSP18lPmHH\no/svCpEKTcOOKMd32fG7iCpBZJcTr3dB8VBDfujRu9i4w1UrDXlkOq/Iq2rG6sv8qbg31XyEc/AH\nyX1QaJ/AdM2n4mMff3wl1De0DojyZiBmfyEAvJq4qBYAeF5K+bYQYjsAvCiEuAsATgDAzf0cQ0ND\n4zzDWRe/lPIYAEw5w9/bIf7019DQuACR1Aw/i9UJRfnxkMSOjTuZLgTPG/LyW/+F6UZGkPv+zq/+\nxpCVBDlmavZ0cu687nY0rWhGmMvO9yG8hH8vHOEhn6APw1d5uUiaEYzxsE4rqZLLzuahxFAE5xgJ\ncb61KMlOSytCF6PHw8NhVtKKfFcrdwlqe3COM/KxnblNCStmpqMp2+nj888m7bXqSbbftvpGNi5i\nJhl/kt9KEvDLUasXOei8FHOeWNjUXRKCh7NiJGxnVqooaUjPRLIhpeKO9dL2aApRZEDi96RmBnYH\n8PqbSGhyVAE37UULbna/deSXTFdePNuQ9777lCGPHT+LjVu/EYk+2obz7zOWIIkJh/u71hw6t19D\nI0WhF7+GRopCL34NjRTFWUN9nyVGlGfJh34Y77NWn3M50+U6/mTIPR2ZTNedWWrIh19GtpQuL/dV\ne8aBB5UAAA7qSURBVD3or7scPCx1ovaYIdtIKicN2QEARPoqAwMA6pMyths7D1tS9hhWxQcAUdJj\nzWLmvrzPj/66xdR3s4JeShiqMLeYyBytJA02qvD207BaROn7FiEpuCZAXzUS4+ciWcynhTuBcPDH\nSHjMrvT0CxJfW2XyoSE9kwl9efVc/YX36LlpGpow9/3cU49n6qdZBD2OhWyh2dP5PfHI7+8w5KFT\nljLds4/80JDbm3BvKmTme0KFhbgOJg1ZxHRDZ8T3iP71Sz+D6kMnBhTq009+DY0UhV78GhopiqSG\n+tx5w+CSO54EAIAj1W8x3fCRPzDk1gbeIvnRH2NVVTCMFVFhhaxRhvF1XRPnPDcJQgYZQvNVRlTy\nTWraqyY1gpqGXqXlMg0H2RRefb8P3QynEmYUMTSBQyQTLqxUzDkcSNIRU6r6eokbE2UZc7w1NsQI\nQahaKUncERqmU/sHmEiVn2oqU2JOmiiphk/Z+1RznrgOvAqPz4Oa3qdVUVJTn05EyeKj8zApZSqS\n6pQsROoSOEhWptPOr7dJ4D28+m8/Yrq8Inyf7MQQ8pTps9m4TDKPF1a+yHQLRDwzMKBktvYH/eTX\n0EhR6MWvoZGiSKrZ39Z6Ap78w90AADBv2Y1Md6wJzaRMHzeHWxraDVmYCNdaBze3o2RnXSqkEbEQ\nmnkBD7bFiirmqt1GduoVUgeaGWi34BzVX1BKqEE75QIAmMkufkDpqBohO/KUKEOh6YMQyQw8rciF\ntAezpBHOQeVckhGOKGQkJAsxRFwOq1Xly0eXgHLsAQBIE+oEMY1jp+3U4+uYknUXI1mC1Hc4bfed\nfBYpuKtGTf0YcXVO29En0YSowpkoBI068PeFqTsiSf8AH5/HvV9aZciP/PBWptt5AFuYFY3Jh75Q\nPHmYIX9j7FeZ7t13451+o6GBR+/0k19DI0WhF7+GRopCL34NjRRFUn1+W1o2lM+Ml/1XHfsN0wU7\n0Lds9vAwXVEx7g9UH8D2xt09CpmiDT9OWMncAxI6MxNii6gS6guSKraokvlGwzw09KT63dS3VFQg\nKJmH0r7bYsXz0fCeVCrJqM8bVnjqTSTs5fMSgg3Jf+fNFnKuCPdjzeQYZpJ1Ryv1ToMSFpXkuSLJ\ndZRKmM5MworhmELMCWQfgYXf+n5mqd8F/c7MlHRV3TaIkeutKGlGYTiq7BWQ08Uo2amNz9FKQpAB\nM89gHVExzZD370QC2bwZfB9lw2qshJ0+qYDpnLnx+8Bk0T6/hobGWaAXv4ZGiiKpZj/4eyCyJ86Z\nP25mBVPVeA4Z8vRpDzLdj371kCE7nVho4nbxIpHODgwJqpT7rICEWEZmq3oJ0NQK+7jr4HAg1z01\nBdU23JQH0GzhZmhvL5mHEjaiRS8W0hpbDQmykh8lZMWIM2jljVI0I1lm4MB49VSue8arp3gE9LPF\nSPjNBCqXIA0Xqj0ZcF4RwtyiDuOFPmofA9IzgGQoqp+Ftj2TKr8fOYZJ6V0QJa5QYT4SsLS1tbFx\nfvK9RzJ49t+S4UhW0xPD1t7tbYwkGzKcGOpbt/YI042bWJCY62fP26+hofF/DHrxa2ikKPTi19BI\nUSTV5w/FABp88d+bQCNPv+0+hn7ty//9PNPJCAm/hYkvrBBlhAkRh1mtzCIvYyQsFfJzf9pK0nZP\n86eJzxjqxfedztuPfpfTwfcl6Ng0F/f9fJRIkzrRyvGliab+Kr/fJP05RkJWKmEl3fgQJuX4xI9l\n3PlKxVyEpDvbrEofP3KN6fU4bd+AzCsW5WFLQfYbTGQepxPQ0OpCrgtHSUv3/qrziF8vT9sPIH60\nSd07wbG+Xpp2zWdIp/Xoj55luptfw16UVnKPyQA/14QJ6PN7O3mYe3vNXgAA8Ksh7n6gn/waGikK\nvfg1NFIUySXzSLfBvEVlAADgO8FbDI++9iJD3rGVm25HCEGBxYamcm+vj42jJmRY4ayjYSSnC8kT\nurq62DiaIWaz88sTY9l0OC493c3Geb043yD3btg8fD4+/76yBtVQoqRVbWqVHDHNY4BhNKtJ+Sy0\nzbWS/UfdIuZ9KBl+DgdpYxXhH5RlxZHwo1RJNIj5LU97FhGXg6hUk51n9SnmvKDZhX1DMLeCf07q\nTtJWbPGT41H9Xvw+1WrOcBjvnfZWHgZ8ac17hjy9CDkqT/h5r5wHvo8h704/v46TxpfEzxPoO2yr\nYkBPfiFElhDiZSFElRDikBBirhAiRwixWghRnfg/++xH0tDQOF8wULP/1wDwtpRyHMRbdx0CgO8C\nwFop5WgAWJt4raGhcYHgrGa/ECITABYCwB0AAFLKEACEhBDLAWBRYtizALAeAO7v71gebyes/+Al\nAABYftM/Mp0MYsZfZ9fjTGemmWSkHZHTzrOZ7Blofjc38+67dLeVmonqjj41Xy1mbuLFSDaazYa7\n+D6l3RXdcVYsSHZulQBD9LGjre6Qc53i3hCT2Goh2XmquU1NfSWjLUb49ywkA1LdwWamfpQrWW2M\npJEFxWSPEJ1ZNczPzEAdU64H9BMJoHTj9HrH1CIi4hapPIBO2qm4lxO80M+pmvoc+LnDSkFXcwdy\nVv7q508bsi/Av1tnOrZRu/tWTv99IBo/t3k7z/zrDwN58g8HgFYA+B8hxC4hxJOJVt2FUsqPmrc1\nQbybr4aGxgWCgSx+CwBMB4DfSymnAYAPFBNfxn9uz7ifIoS4RwhRKYSo9Hv7KQnV0NBIKgay+OsB\noF5KuTXx+mWI/xg0CyGKAQAS/7ec6c1SyieklBVSygqXWy3c0NDQGCyc1eeXUjYJIeqEEGOllIcB\n4DIAOJj4dzsAPJz4/7WzHas0fyL89N5KAAD41qormG7muL8asgnKmI62T6bISM9ir72EiENJ9IIo\nCc3RsIuaIRdllWTcL6RureiH399m7ZsE1Ea4+iNh7tPxngHoTNKqMgDOfS+UMB3dN6BtuKj/H9eR\nLEel17nbieHU0lwkjTh6qoGNG5ZPqhyVW2nssBJDXrcLKzZVsg1KpW9S226xll99W41Fbgw5NvV4\nmW5SGc7/QBNWfZqVvQcaTlX3WGioUm3zRdpBsGvfXwWkmiVYuQP3jyZOHGfIN16/gI076MW+FDVK\nuLAwLT4vy2n7Jn1joHH+rwLAc0IIGwAcA4A7IW41vCiEuAsATgDAzQM+q4aGxqBjQItfSrkbACrO\noLrss52OhoZGspDUDL/urlZ4/bUnAADAfYgXIFQevdSQaWsjAID09HRD7iEZeWnpnAvNRLKt1Cyw\nQJSY3704jpKDAAB4/WhaBRSz3+1GMzcYOHPhCgAP+ZiV7LwQJRVR5khNRRpKjClmOeXcX375PKZb\n8/5mQ77uEuSGO9jAMxn31Rw35H++greFOn4Kw6TvVR0nGm5SdnSgmzVrwnCmW7MduRYXjEHd5mO1\nbBzlUDwt0Cco9z9ex2CA3zuNJLynhvoONqKpT7MLY0oxE/1e1JZiNiu6FeEgd9VstjO7C+o9Qb9q\nNWS6Y9tGQ374G8sN+dje42xc+cShhjx+Cr/eu44mOjyLge+r6dx+DY0UhV78GhopCr34NTRSFEn1\n+SNBD3QdXQsAAGUVtzDdyseQuNCsNKcrKUFfx9OJvmtYIfNgfrLCvR7sQj8uakany+VOY+O8Pejz\nW5T22jQV1e5AnUKdz1po25U23LQSceGs6Uy3euMmQ6YVYirsTvycb67bxJUkTbUjhD6zydfOhtGq\nvlfXbWe6sSOLDJlWCYoor9y7bAHuFby6YTPTXUKIJzZV1xrykimj2Lg3dh42ZLeZ77FcNBz9WjMh\nHN1cy0OONJ06qrQRp736rCSNWSpElwHiy5vUCkige0n8+JlpZB+IpH8HlfA0TzNWQrck5Dtq4nxD\n7g7y76XcjUm03m6+h9PdGt8ni4aVMtJ+oJ/8GhopCr34NTRSFOJ0PrRzeDIhWiGeEJQHAG1nGZ4M\n6Hlw6HlwnA/z+LhzGCal7LvPN0FSF79xUiEqpZRnShrS89Dz0PNI0hy02a+hkaLQi19DI0UxWIv/\niUE6rwo9Dw49D47zYR7nbA6D4vNraGgMPrTZr6GRokjq4hdCLBFCHBZC1Aghksb2K4R4WgjRIoTY\nT/6WdOpxIcQQIcQ6IcRBIcQBIcTXBmMuQgiHEGKbEGJPYh4PDsY8yHzMCX7INwZrHkKIWiHEPiHE\nbiFE5SDOI2k0+Ulb/CKeZ/k4ACwFgAkA8AUhxIQknf4ZAFii/G0wqMcjAPBNKeUEAJgDAPclrkGy\n5xIEgMVSyikAMBUAlggh5gzCPD7C1yBOB/8RBmsel0opp5LQ2mDMI3k0+VLKpPwDgLkA8A55/QAA\nPJDE85cDwH7y+jAAFCfkYgA4nKy5kDm8BgBXDOZcAMAFADsBYPZgzAMAyhI39GIAeGOwvhsAqAWA\nPOVvSZ0HAGQCwHFI7MWd63kk0+wvBYA68ro+8bfBwqBSjwshygFgGgBsHYy5JEzt3RAnXl0t4wSt\ng3FN/gsAvgO0/9ngzEMCwBohxA4hxD2DNI+k0uTrDT/on3r8XEAI4QaAVwDg61LKnsGYi5QyKqWc\nCvEn7ywhxMRkz0MIcQ0AtEgpd/Qzz2R9NwsS12MpxN2xhYMwj09Fk/9xkczF3wAAQ8jrssTfBgsD\noh7/rCGEsEJ84T8npVw5mHMBAJBSdgHAOojviSR7HvMB4DohRC0AvAAAi4UQfxmEeYCUsiHxfwsA\nvAoAswZhHp+KJv/jIpmLfzsAjBZCDE+wAN8CAKuSeH4VqyBOOQ4wQOrxTwsR529+CgAOSSkfGay5\nCCHyhRBZCdkJ8X2HqmTPQ0r5gJSyTEpZDvH74T0p5W3JnocQIk0Ikf6RDABXAsD+ZM9DStkEAHVC\niLGJP31Ek39u5nGuN1KUjYurAeAIABwFgO8l8bx/BYBGAAhD/Nf1LgDIhfhGUzUArAGAnCTMYwHE\nTba9ALA78e/qZM8FACYDwK7EPPYDwA8Sf0/6NSFzWgS44Zfs6zECAPYk/h346N4cpHtkKgBUJr6b\nvwNA9rmah87w09BIUegNPw2NFIVe/BoaKQq9+DU0UhR68WtopCj04tfQSFHoxa+hkaLQi19DI0Wh\nF7+GRori/wOMhrX8XuEH+AAAAABJRU5ErkJggg==\n", 506 | "text/plain": [ 507 | "" 508 | ] 509 | }, 510 | "metadata": {}, 511 | "output_type": "display_data" 512 | } 513 | ], 514 | "source": [ 515 | "### START CODE HERE ###\n", 516 | "img_path = 'images/my_image.jpg'\n", 517 | "### END CODE HERE ###\n", 518 | "img = image.load_img(img_path, target_size=(64, 64))\n", 519 | "imshow(img)\n", 520 | "\n", 521 | "x = image.img_to_array(img)\n", 522 | "x = np.expand_dims(x, axis=0)\n", 523 | "x = preprocess_input(x)\n", 524 | "\n", 525 | "print(happyModel.predict(x))" 526 | ] 527 | }, 528 | { 529 | "cell_type": "markdown", 530 | "metadata": {}, 531 | "source": [ 532 | "## 5 - Other useful functions in Keras (Optional)\n", 533 | "\n", 534 | "Two other basic features of Keras that you'll find useful are:\n", 535 | "- `model.summary()`: prints the details of your layers in a table with the sizes of its inputs/outputs\n", 536 | "- `plot_model()`: plots your graph in a nice layout. You can even save it as \".png\" using SVG() if you'd like to share it on social media ;). It is saved in \"File\" then \"Open...\" in the upper bar of the notebook.\n", 537 | "\n", 538 | "Run the following code." 539 | ] 540 | }, 541 | { 542 | "cell_type": "code", 543 | "execution_count": 9, 544 | "metadata": { 545 | "scrolled": false 546 | }, 547 | "outputs": [ 548 | { 549 | "name": "stdout", 550 | "output_type": "stream", 551 | "text": [ 552 | "_________________________________________________________________\n", 553 | "Layer (type) Output Shape Param # \n", 554 | "=================================================================\n", 555 | "input_1 (InputLayer) (None, 64, 64, 3) 0 \n", 556 | "_________________________________________________________________\n", 557 | "zero_padding2d_1 (ZeroPaddin (None, 70, 70, 3) 0 \n", 558 | "_________________________________________________________________\n", 559 | "conv0 (Conv2D) (None, 64, 64, 32) 4736 \n", 560 | "_________________________________________________________________\n", 561 | "bn0 (BatchNormalization) (None, 64, 64, 32) 128 \n", 562 | "_________________________________________________________________\n", 563 | "activation_1 (Activation) (None, 64, 64, 32) 0 \n", 564 | "_________________________________________________________________\n", 565 | "max_pool (MaxPooling2D) (None, 32, 32, 32) 0 \n", 566 | "_________________________________________________________________\n", 567 | "flatten_1 (Flatten) (None, 32768) 0 \n", 568 | "_________________________________________________________________\n", 569 | "fc (Dense) (None, 1) 32769 \n", 570 | "=================================================================\n", 571 | "Total params: 37,633\n", 572 | "Trainable params: 37,569\n", 573 | "Non-trainable params: 64\n", 574 | "_________________________________________________________________\n" 575 | ] 576 | } 577 | ], 578 | "source": [ 579 | "happyModel.summary()" 580 | ] 581 | }, 582 | { 583 | "cell_type": "code", 584 | "execution_count": 10, 585 | "metadata": {}, 586 | "outputs": [ 587 | { 588 | "data": { 589 | "image/svg+xml": [ 590 | "\n", 591 | "\n", 592 | "G\n", 593 | "\n", 594 | "\n", 595 | "139944265371040\n", 596 | "\n", 597 | "input_1: InputLayer\n", 598 | "\n", 599 | "\n", 600 | "139944265371208\n", 601 | "\n", 602 | "zero_padding2d_1: ZeroPadding2D\n", 603 | "\n", 604 | "\n", 605 | "139944265371040->139944265371208\n", 606 | "\n", 607 | "\n", 608 | "\n", 609 | "\n", 610 | "139944265371376\n", 611 | "\n", 612 | "conv0: Conv2D\n", 613 | "\n", 614 | "\n", 615 | "139944265371208->139944265371376\n", 616 | "\n", 617 | "\n", 618 | "\n", 619 | "\n", 620 | "139944265367960\n", 621 | "\n", 622 | "bn0: BatchNormalization\n", 623 | "\n", 624 | "\n", 625 | "139944265371376->139944265367960\n", 626 | "\n", 627 | "\n", 628 | "\n", 629 | "\n", 630 | "139944265368744\n", 631 | "\n", 632 | "activation_1: Activation\n", 633 | "\n", 634 | "\n", 635 | "139944265367960->139944265368744\n", 636 | "\n", 637 | "\n", 638 | "\n", 639 | "\n", 640 | "139944265407120\n", 641 | "\n", 642 | "max_pool: MaxPooling2D\n", 643 | "\n", 644 | "\n", 645 | "139944265368744->139944265407120\n", 646 | "\n", 647 | "\n", 648 | "\n", 649 | "\n", 650 | "139944264735768\n", 651 | "\n", 652 | "flatten_1: Flatten\n", 653 | "\n", 654 | "\n", 655 | "139944265407120->139944264735768\n", 656 | "\n", 657 | "\n", 658 | "\n", 659 | "\n", 660 | "139944265158784\n", 661 | "\n", 662 | "fc: Dense\n", 663 | "\n", 664 | "\n", 665 | "139944264735768->139944265158784\n", 666 | "\n", 667 | "\n", 668 | "\n", 669 | "\n", 670 | "" 671 | ], 672 | "text/plain": [ 673 | "" 674 | ] 675 | }, 676 | "execution_count": 10, 677 | "metadata": {}, 678 | "output_type": "execute_result" 679 | } 680 | ], 681 | "source": [ 682 | "plot_model(happyModel, to_file='HappyModel.png')\n", 683 | "SVG(model_to_dot(happyModel).create(prog='dot', format='svg'))" 684 | ] 685 | }, 686 | { 687 | "cell_type": "code", 688 | "execution_count": null, 689 | "metadata": { 690 | "collapsed": true 691 | }, 692 | "outputs": [], 693 | "source": [] 694 | } 695 | ], 696 | "metadata": { 697 | "kernelspec": { 698 | "display_name": "Python 3", 699 | "language": "python", 700 | "name": "python3" 701 | }, 702 | "language_info": { 703 | "codemirror_mode": { 704 | "name": "ipython", 705 | "version": 3 706 | }, 707 | "file_extension": ".py", 708 | "mimetype": "text/x-python", 709 | "name": "python", 710 | "nbconvert_exporter": "python", 711 | "pygments_lexer": "ipython3", 712 | "version": "3.6.0" 713 | } 714 | }, 715 | "nbformat": 4, 716 | "nbformat_minor": 2 717 | } 718 | -------------------------------------------------------------------------------- /Convolutional Neural Networks/week2/Residual+Networks+-+v2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Residual Networks\n", 8 | "\n", 9 | "Welcome to the second assignment of this week! You will learn how to build very deep convolutional networks, using Residual Networks (ResNets). In theory, very deep networks can represent very complex functions; but in practice, they are hard to train. Residual Networks, introduced by [He et al.](https://arxiv.org/pdf/1512.03385.pdf), allow you to train much deeper networks than were previously practically feasible.\n", 10 | "\n", 11 | "**In this assignment, you will:**\n", 12 | "- Implement the basic building blocks of ResNets. \n", 13 | "- Put together these building blocks to implement and train a state-of-the-art neural network for image classification. \n", 14 | "\n", 15 | "This assignment will be done in Keras. \n", 16 | "\n", 17 | "Before jumping into the problem, let's run the cell below to load the required packages." 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "metadata": {}, 24 | "outputs": [ 25 | { 26 | "name": "stderr", 27 | "output_type": "stream", 28 | "text": [ 29 | "Using TensorFlow backend.\n" 30 | ] 31 | } 32 | ], 33 | "source": [ 34 | "import numpy as np\n", 35 | "from keras import layers\n", 36 | "from keras.layers import Input, Add, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D\n", 37 | "from keras.models import Model, load_model\n", 38 | "from keras.preprocessing import image\n", 39 | "from keras.utils import layer_utils\n", 40 | "from keras.utils.data_utils import get_file\n", 41 | "from keras.applications.imagenet_utils import preprocess_input\n", 42 | "import pydot\n", 43 | "from IPython.display import SVG\n", 44 | "from keras.utils.vis_utils import model_to_dot\n", 45 | "from keras.utils import plot_model\n", 46 | "from resnets_utils import *\n", 47 | "from keras.initializers import glorot_uniform\n", 48 | "import scipy.misc\n", 49 | "from matplotlib.pyplot import imshow\n", 50 | "%matplotlib inline\n", 51 | "\n", 52 | "import keras.backend as K\n", 53 | "K.set_image_data_format('channels_last')\n", 54 | "K.set_learning_phase(1)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "## 1 - The problem of very deep neural networks\n", 62 | "\n", 63 | "Last week, you built your first convolutional neural network. In recent years, neural networks have become deeper, with state-of-the-art networks going from just a few layers (e.g., AlexNet) to over a hundred layers.\n", 64 | "\n", 65 | "The main benefit of a very deep network is that it can represent very complex functions. It can also learn features at many different levels of abstraction, from edges (at the lower layers) to very complex features (at the deeper layers). However, using a deeper network doesn't always help. A huge barrier to training them is vanishing gradients: very deep networks often have a gradient signal that goes to zero quickly, thus making gradient descent unbearably slow. More specifically, during gradient descent, as you backprop from the final layer back to the first layer, you are multiplying by the weight matrix on each step, and thus the gradient can decrease exponentially quickly to zero (or, in rare cases, grow exponentially quickly and \"explode\" to take very large values). \n", 66 | "\n", 67 | "During training, you might therefore see the magnitude (or norm) of the gradient for the earlier layers descrease to zero very rapidly as training proceeds: " 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "\n", 75 | "
**Figure 1** : **Vanishing gradient**
The speed of learning decreases very rapidly for the early layers as the network trains
\n", 76 | "\n", 77 | "You are now going to solve this problem by building a Residual Network!" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "## 2 - Building a Residual Network\n", 85 | "\n", 86 | "In ResNets, a \"shortcut\" or a \"skip connection\" allows the gradient to be directly backpropagated to earlier layers: \n", 87 | "\n", 88 | "\n", 89 | "
**Figure 2** : A ResNet block showing a **skip-connection**
\n", 90 | "\n", 91 | "The image on the left shows the \"main path\" through the network. The image on the right adds a shortcut to the main path. By stacking these ResNet blocks on top of each other, you can form a very deep network. \n", 92 | "\n", 93 | "We also saw in lecture that having ResNet blocks with the shortcut also makes it very easy for one of the blocks to learn an identity function. This means that you can stack on additional ResNet blocks with little risk of harming training set performance. (There is also some evidence that the ease of learning an identity function--even more than skip connections helping with vanishing gradients--accounts for ResNets' remarkable performance.)\n", 94 | "\n", 95 | "Two main types of blocks are used in a ResNet, depending mainly on whether the input/output dimensions are same or different. You are going to implement both of them. " 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "### 2.1 - The identity block\n", 103 | "\n", 104 | "The identity block is the standard block used in ResNets, and corresponds to the case where the input activation (say $a^{[l]}$) has the same dimension as the output activation (say $a^{[l+2]}$). To flesh out the different steps of what happens in a ResNet's identity block, here is an alternative diagram showing the individual steps:\n", 105 | "\n", 106 | "\n", 107 | "
**Figure 3** : **Identity block.** Skip connection \"skips over\" 2 layers.
\n", 108 | "\n", 109 | "The upper path is the \"shortcut path.\" The lower path is the \"main path.\" In this diagram, we have also made explicit the CONV2D and ReLU steps in each layer. To speed up training we have also added a BatchNorm step. Don't worry about this being complicated to implement--you'll see that BatchNorm is just one line of code in Keras! \n", 110 | "\n", 111 | "In this exercise, you'll actually implement a slightly more powerful version of this identity block, in which the skip connection \"skips over\" 3 hidden layers rather than 2 layers. It looks like this: \n", 112 | "\n", 113 | "\n", 114 | "
**Figure 4** : **Identity block.** Skip connection \"skips over\" 3 layers.
\n", 115 | "\n", 116 | "Here're the individual steps.\n", 117 | "\n", 118 | "First component of main path: \n", 119 | "- The first CONV2D has $F_1$ filters of shape (1,1) and a stride of (1,1). Its padding is \"valid\" and its name should be `conv_name_base + '2a'`. Use 0 as the seed for the random initialization. \n", 120 | "- The first BatchNorm is normalizing the channels axis. Its name should be `bn_name_base + '2a'`.\n", 121 | "- Then apply the ReLU activation function. This has no name and no hyperparameters. \n", 122 | "\n", 123 | "Second component of main path:\n", 124 | "- The second CONV2D has $F_2$ filters of shape $(f,f)$ and a stride of (1,1). Its padding is \"same\" and its name should be `conv_name_base + '2b'`. Use 0 as the seed for the random initialization. \n", 125 | "- The second BatchNorm is normalizing the channels axis. Its name should be `bn_name_base + '2b'`.\n", 126 | "- Then apply the ReLU activation function. This has no name and no hyperparameters. \n", 127 | "\n", 128 | "Third component of main path:\n", 129 | "- The third CONV2D has $F_3$ filters of shape (1,1) and a stride of (1,1). Its padding is \"valid\" and its name should be `conv_name_base + '2c'`. Use 0 as the seed for the random initialization. \n", 130 | "- The third BatchNorm is normalizing the channels axis. Its name should be `bn_name_base + '2c'`. Note that there is no ReLU activation function in this component. \n", 131 | "\n", 132 | "Final step: \n", 133 | "- The shortcut and the input are added together.\n", 134 | "- Then apply the ReLU activation function. This has no name and no hyperparameters. \n", 135 | "\n", 136 | "**Exercise**: Implement the ResNet identity block. We have implemented the first component of the main path. Please read over this carefully to make sure you understand what it is doing. You should implement the rest. \n", 137 | "- To implement the Conv2D step: [See reference](https://keras.io/layers/convolutional/#conv2d)\n", 138 | "- To implement BatchNorm: [See reference](https://faroit.github.io/keras-docs/1.2.2/layers/normalization/) (axis: Integer, the axis that should be normalized (typically the channels axis))\n", 139 | "- For the activation, use: `Activation('relu')(X)`\n", 140 | "- To add the value passed forward by the shortcut: [See reference](https://keras.io/layers/merge/#add)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 2, 146 | "metadata": { 147 | "collapsed": true 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "# GRADED FUNCTION: identity_block\n", 152 | "\n", 153 | "def identity_block(X, f, filters, stage, block):\n", 154 | " \"\"\"\n", 155 | " Implementation of the identity block as defined in Figure 3\n", 156 | " \n", 157 | " Arguments:\n", 158 | " X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)\n", 159 | " f -- integer, specifying the shape of the middle CONV's window for the main path\n", 160 | " filters -- python list of integers, defining the number of filters in the CONV layers of the main path\n", 161 | " stage -- integer, used to name the layers, depending on their position in the network\n", 162 | " block -- string/character, used to name the layers, depending on their position in the network\n", 163 | " \n", 164 | " Returns:\n", 165 | " X -- output of the identity block, tensor of shape (n_H, n_W, n_C)\n", 166 | " \"\"\"\n", 167 | " \n", 168 | " # defining name basis\n", 169 | " conv_name_base = 'res' + str(stage) + block + '_branch'\n", 170 | " bn_name_base = 'bn' + str(stage) + block + '_branch'\n", 171 | " \n", 172 | " # Retrieve Filters\n", 173 | " F1, F2, F3 = filters\n", 174 | " \n", 175 | " # Save the input value. You'll need this later to add back to the main path. \n", 176 | " X_shortcut = X\n", 177 | " \n", 178 | " # First component of main path\n", 179 | " X = Conv2D(filters = F1, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2a', kernel_initializer = glorot_uniform(seed=0))(X)\n", 180 | " X = BatchNormalization(axis = 3, name = bn_name_base + '2a')(X)\n", 181 | " X = Activation('relu')(X)\n", 182 | " \n", 183 | " ### START CODE HERE ###\n", 184 | " \n", 185 | " # Second component of main path (≈3 lines)\n", 186 | " X = Conv2D(filters = F2, kernel_size = (f, f), strides = (1,1), padding = 'same', name = conv_name_base + '2b', kernel_initializer = glorot_uniform(seed=0))(X)\n", 187 | " X = BatchNormalization(axis = 3, name = bn_name_base + '2b')(X)\n", 188 | " X = Activation('relu')(X)\n", 189 | "\n", 190 | " # Third component of main path (≈2 lines)\n", 191 | " X = Conv2D(filters = F3, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2c', kernel_initializer = glorot_uniform(seed=0))(X)\n", 192 | " X = BatchNormalization(axis = 3, name = bn_name_base + '2c')(X)\n", 193 | "\n", 194 | " # Final step: Add shortcut value to main path, and pass it through a RELU activation (≈2 lines)\n", 195 | " X = Add()([X, X_shortcut])\n", 196 | " X = Activation('relu')(X)\n", 197 | " \n", 198 | " ### END CODE HERE ###\n", 199 | " \n", 200 | " return X" 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": 3, 206 | "metadata": {}, 207 | "outputs": [ 208 | { 209 | "name": "stdout", 210 | "output_type": "stream", 211 | "text": [ 212 | "out = [ 0.94822985 0. 1.16101444 2.747859 0. 1.36677003]\n" 213 | ] 214 | } 215 | ], 216 | "source": [ 217 | "tf.reset_default_graph()\n", 218 | "\n", 219 | "with tf.Session() as test:\n", 220 | " np.random.seed(1)\n", 221 | " A_prev = tf.placeholder(\"float\", [3, 4, 4, 6])\n", 222 | " X = np.random.randn(3, 4, 4, 6)\n", 223 | " A = identity_block(A_prev, f = 2, filters = [2, 4, 6], stage = 1, block = 'a')\n", 224 | " test.run(tf.global_variables_initializer())\n", 225 | " out = test.run([A], feed_dict={A_prev: X, K.learning_phase(): 0})\n", 226 | " print(\"out = \" + str(out[0][1][1][0]))" 227 | ] 228 | }, 229 | { 230 | "cell_type": "markdown", 231 | "metadata": {}, 232 | "source": [ 233 | "**Expected Output**:\n", 234 | "\n", 235 | "\n", 236 | " \n", 237 | " \n", 240 | " \n", 243 | " \n", 244 | "\n", 245 | "
\n", 238 | " **out**\n", 239 | " \n", 241 | " [ 0.94822985 0. 1.16101444 2.747859 0. 1.36677003]\n", 242 | "
" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "## 2.2 - The convolutional block\n", 253 | "\n", 254 | "You've implemented the ResNet identity block. Next, the ResNet \"convolutional block\" is the other type of block. You can use this type of block when the input and output dimensions don't match up. The difference with the identity block is that there is a CONV2D layer in the shortcut path: \n", 255 | "\n", 256 | "\n", 257 | "
**Figure 4** : **Convolutional block**
\n", 258 | "\n", 259 | "The CONV2D layer in the shortcut path is used to resize the input $x$ to a different dimension, so that the dimensions match up in the final addition needed to add the shortcut value back to the main path. (This plays a similar role as the matrix $W_s$ discussed in lecture.) For example, to reduce the activation dimensions's height and width by a factor of 2, you can use a 1x1 convolution with a stride of 2. The CONV2D layer on the shortcut path does not use any non-linear activation function. Its main role is to just apply a (learned) linear function that reduces the dimension of the input, so that the dimensions match up for the later addition step. \n", 260 | "\n", 261 | "The details of the convolutional block are as follows. \n", 262 | "\n", 263 | "First component of main path:\n", 264 | "- The first CONV2D has $F_1$ filters of shape (1,1) and a stride of (s,s). Its padding is \"valid\" and its name should be `conv_name_base + '2a'`. \n", 265 | "- The first BatchNorm is normalizing the channels axis. Its name should be `bn_name_base + '2a'`.\n", 266 | "- Then apply the ReLU activation function. This has no name and no hyperparameters. \n", 267 | "\n", 268 | "Second component of main path:\n", 269 | "- The second CONV2D has $F_2$ filters of (f,f) and a stride of (1,1). Its padding is \"same\" and it's name should be `conv_name_base + '2b'`.\n", 270 | "- The second BatchNorm is normalizing the channels axis. Its name should be `bn_name_base + '2b'`.\n", 271 | "- Then apply the ReLU activation function. This has no name and no hyperparameters. \n", 272 | "\n", 273 | "Third component of main path:\n", 274 | "- The third CONV2D has $F_3$ filters of (1,1) and a stride of (1,1). Its padding is \"valid\" and it's name should be `conv_name_base + '2c'`.\n", 275 | "- The third BatchNorm is normalizing the channels axis. Its name should be `bn_name_base + '2c'`. Note that there is no ReLU activation function in this component. \n", 276 | "\n", 277 | "Shortcut path:\n", 278 | "- The CONV2D has $F_3$ filters of shape (1,1) and a stride of (s,s). Its padding is \"valid\" and its name should be `conv_name_base + '1'`.\n", 279 | "- The BatchNorm is normalizing the channels axis. Its name should be `bn_name_base + '1'`. \n", 280 | "\n", 281 | "Final step: \n", 282 | "- The shortcut and the main path values are added together.\n", 283 | "- Then apply the ReLU activation function. This has no name and no hyperparameters. \n", 284 | " \n", 285 | "**Exercise**: Implement the convolutional block. We have implemented the first component of the main path; you should implement the rest. As before, always use 0 as the seed for the random initialization, to ensure consistency with our grader.\n", 286 | "- [Conv Hint](https://keras.io/layers/convolutional/#conv2d)\n", 287 | "- [BatchNorm Hint](https://keras.io/layers/normalization/#batchnormalization) (axis: Integer, the axis that should be normalized (typically the features axis))\n", 288 | "- For the activation, use: `Activation('relu')(X)`\n", 289 | "- [Addition Hint](https://keras.io/layers/merge/#add)" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": 4, 295 | "metadata": { 296 | "collapsed": true 297 | }, 298 | "outputs": [], 299 | "source": [ 300 | "# GRADED FUNCTION: convolutional_block\n", 301 | "\n", 302 | "def convolutional_block(X, f, filters, stage, block, s = 2):\n", 303 | " \"\"\"\n", 304 | " Implementation of the convolutional block as defined in Figure 4\n", 305 | " \n", 306 | " Arguments:\n", 307 | " X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)\n", 308 | " f -- integer, specifying the shape of the middle CONV's window for the main path\n", 309 | " filters -- python list of integers, defining the number of filters in the CONV layers of the main path\n", 310 | " stage -- integer, used to name the layers, depending on their position in the network\n", 311 | " block -- string/character, used to name the layers, depending on their position in the network\n", 312 | " s -- Integer, specifying the stride to be used\n", 313 | " \n", 314 | " Returns:\n", 315 | " X -- output of the convolutional block, tensor of shape (n_H, n_W, n_C)\n", 316 | " \"\"\"\n", 317 | " \n", 318 | " # defining name basis\n", 319 | " conv_name_base = 'res' + str(stage) + block + '_branch'\n", 320 | " bn_name_base = 'bn' + str(stage) + block + '_branch'\n", 321 | " \n", 322 | " # Retrieve Filters\n", 323 | " F1, F2, F3 = filters\n", 324 | " \n", 325 | " # Save the input value\n", 326 | " X_shortcut = X\n", 327 | "\n", 328 | "\n", 329 | " ##### MAIN PATH #####\n", 330 | " # First component of main path \n", 331 | " X = Conv2D(F1, (1, 1), strides = (s,s), name = conv_name_base + '2a', padding = 'valid', kernel_initializer = glorot_uniform(seed=0))(X)\n", 332 | " X = BatchNormalization(axis = 3, name = bn_name_base + '2a')(X)\n", 333 | " X = Activation('relu')(X)\n", 334 | " \n", 335 | " ### START CODE HERE ###\n", 336 | "\n", 337 | " # Second component of main path (≈3 lines)\n", 338 | " X = Conv2D(F2, (f, f), strides = (1,1), name = conv_name_base + '2b', padding = 'same', kernel_initializer = glorot_uniform(seed=0))(X)\n", 339 | " X = BatchNormalization(axis = 3, name = bn_name_base + '2b')(X)\n", 340 | " X = Activation('relu')(X)\n", 341 | "\n", 342 | " # Third component of main path (≈2 lines)\n", 343 | " X = Conv2D(F3, (1, 1), strides = (1,1), name = conv_name_base + '2c', padding = 'valid', kernel_initializer = glorot_uniform(seed=0))(X)\n", 344 | " X = BatchNormalization(axis = 3, name = bn_name_base + '2c')(X)\n", 345 | "\n", 346 | " ##### SHORTCUT PATH #### (≈2 lines)\n", 347 | " X_shortcut = Conv2D(F3, (1, 1), strides = (s,s), name = conv_name_base + '1', padding = 'valid', kernel_initializer = glorot_uniform(seed=0))(X_shortcut)\n", 348 | " X_shortcut = BatchNormalization(axis = 3, name = bn_name_base + '1')(X_shortcut)\n", 349 | "\n", 350 | " # Final step: Add shortcut value to main path, and pass it through a RELU activation (≈2 lines)\n", 351 | " X = Add()([X, X_shortcut])\n", 352 | " X = Activation('relu')(X)\n", 353 | " \n", 354 | " ### END CODE HERE ###\n", 355 | " \n", 356 | " return X" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 5, 362 | "metadata": {}, 363 | "outputs": [ 364 | { 365 | "name": "stdout", 366 | "output_type": "stream", 367 | "text": [ 368 | "out = [ 0.09018463 1.23489773 0.46822017 0.0367176 0. 0.65516603]\n" 369 | ] 370 | } 371 | ], 372 | "source": [ 373 | "tf.reset_default_graph()\n", 374 | "\n", 375 | "with tf.Session() as test:\n", 376 | " np.random.seed(1)\n", 377 | " A_prev = tf.placeholder(\"float\", [3, 4, 4, 6])\n", 378 | " X = np.random.randn(3, 4, 4, 6)\n", 379 | " A = convolutional_block(A_prev, f = 2, filters = [2, 4, 6], stage = 1, block = 'a')\n", 380 | " test.run(tf.global_variables_initializer())\n", 381 | " out = test.run([A], feed_dict={A_prev: X, K.learning_phase(): 0})\n", 382 | " print(\"out = \" + str(out[0][1][1][0]))" 383 | ] 384 | }, 385 | { 386 | "cell_type": "markdown", 387 | "metadata": {}, 388 | "source": [ 389 | "**Expected Output**:\n", 390 | "\n", 391 | "\n", 392 | " \n", 393 | " \n", 396 | " \n", 399 | " \n", 400 | "\n", 401 | "
\n", 394 | " **out**\n", 395 | " \n", 397 | " [ 0.09018463 1.23489773 0.46822017 0.0367176 0. 0.65516603]\n", 398 | "
" 402 | ] 403 | }, 404 | { 405 | "cell_type": "markdown", 406 | "metadata": {}, 407 | "source": [ 408 | "## 3 - Building your first ResNet model (50 layers)\n", 409 | "\n", 410 | "You now have the necessary blocks to build a very deep ResNet. The following figure describes in detail the architecture of this neural network. \"ID BLOCK\" in the diagram stands for \"Identity block,\" and \"ID BLOCK x3\" means you should stack 3 identity blocks together.\n", 411 | "\n", 412 | "\n", 413 | "
**Figure 5** : **ResNet-50 model**
\n", 414 | "\n", 415 | "The details of this ResNet-50 model are:\n", 416 | "- Zero-padding pads the input with a pad of (3,3)\n", 417 | "- Stage 1:\n", 418 | " - The 2D Convolution has 64 filters of shape (7,7) and uses a stride of (2,2). Its name is \"conv1\".\n", 419 | " - BatchNorm is applied to the channels axis of the input.\n", 420 | " - MaxPooling uses a (3,3) window and a (2,2) stride.\n", 421 | "- Stage 2:\n", 422 | " - The convolutional block uses three set of filters of size [64,64,256], \"f\" is 3, \"s\" is 1 and the block is \"a\".\n", 423 | " - The 2 identity blocks use three set of filters of size [64,64,256], \"f\" is 3 and the blocks are \"b\" and \"c\".\n", 424 | "- Stage 3:\n", 425 | " - The convolutional block uses three set of filters of size [128,128,512], \"f\" is 3, \"s\" is 2 and the block is \"a\".\n", 426 | " - The 3 identity blocks use three set of filters of size [128,128,512], \"f\" is 3 and the blocks are \"b\", \"c\" and \"d\".\n", 427 | "- Stage 4:\n", 428 | " - The convolutional block uses three set of filters of size [256, 256, 1024], \"f\" is 3, \"s\" is 2 and the block is \"a\".\n", 429 | " - The 5 identity blocks use three set of filters of size [256, 256, 1024], \"f\" is 3 and the blocks are \"b\", \"c\", \"d\", \"e\" and \"f\".\n", 430 | "- Stage 5:\n", 431 | " - The convolutional block uses three set of filters of size [512, 512, 2048], \"f\" is 3, \"s\" is 2 and the block is \"a\".\n", 432 | " - The 2 identity blocks use three set of filters of size [512, 512, 2048], \"f\" is 3 and the blocks are \"b\" and \"c\".\n", 433 | "- The 2D Average Pooling uses a window of shape (2,2) and its name is \"avg_pool\".\n", 434 | "- The flatten doesn't have any hyperparameters or name.\n", 435 | "- The Fully Connected (Dense) layer reduces its input to the number of classes using a softmax activation. Its name should be `'fc' + str(classes)`.\n", 436 | "\n", 437 | "**Exercise**: Implement the ResNet with 50 layers described in the figure above. We have implemented Stages 1 and 2. Please implement the rest. (The syntax for implementing Stages 3-5 should be quite similar to that of Stage 2.) Make sure you follow the naming convention in the text above. \n", 438 | "\n", 439 | "You'll need to use this function: \n", 440 | "- Average pooling [see reference](https://keras.io/layers/pooling/#averagepooling2d)\n", 441 | "\n", 442 | "Here're some other functions we used in the code below:\n", 443 | "- Conv2D: [See reference](https://keras.io/layers/convolutional/#conv2d)\n", 444 | "- BatchNorm: [See reference](https://keras.io/layers/normalization/#batchnormalization) (axis: Integer, the axis that should be normalized (typically the features axis))\n", 445 | "- Zero padding: [See reference](https://keras.io/layers/convolutional/#zeropadding2d)\n", 446 | "- Max pooling: [See reference](https://keras.io/layers/pooling/#maxpooling2d)\n", 447 | "- Fully conected layer: [See reference](https://keras.io/layers/core/#dense)\n", 448 | "- Addition: [See reference](https://keras.io/layers/merge/#add)" 449 | ] 450 | }, 451 | { 452 | "cell_type": "code", 453 | "execution_count": 6, 454 | "metadata": { 455 | "collapsed": true 456 | }, 457 | "outputs": [], 458 | "source": [ 459 | "# GRADED FUNCTION: ResNet50\n", 460 | "\n", 461 | "def ResNet50(input_shape = (64, 64, 3), classes = 6):\n", 462 | " \"\"\"\n", 463 | " Implementation of the popular ResNet50 the following architecture:\n", 464 | " CONV2D -> BATCHNORM -> RELU -> MAXPOOL -> CONVBLOCK -> IDBLOCK*2 -> CONVBLOCK -> IDBLOCK*3\n", 465 | " -> CONVBLOCK -> IDBLOCK*5 -> CONVBLOCK -> IDBLOCK*2 -> AVGPOOL -> TOPLAYER\n", 466 | "\n", 467 | " Arguments:\n", 468 | " input_shape -- shape of the images of the dataset\n", 469 | " classes -- integer, number of classes\n", 470 | "\n", 471 | " Returns:\n", 472 | " model -- a Model() instance in Keras\n", 473 | " \"\"\"\n", 474 | " \n", 475 | " # Define the input as a tensor with shape input_shape\n", 476 | " X_input = Input(input_shape)\n", 477 | "\n", 478 | " \n", 479 | " # Zero-Padding\n", 480 | " X = ZeroPadding2D((3, 3))(X_input)\n", 481 | " \n", 482 | " # Stage 1\n", 483 | " X = Conv2D(64, (7, 7), strides = (2, 2), name = 'conv1', kernel_initializer = glorot_uniform(seed=0))(X)\n", 484 | " X = BatchNormalization(axis = 3, name = 'bn_conv1')(X)\n", 485 | " X = Activation('relu')(X)\n", 486 | " X = MaxPooling2D((3, 3), strides=(2, 2))(X)\n", 487 | "\n", 488 | " # Stage 2\n", 489 | " X = convolutional_block(X, f = 3, filters = [64, 64, 256], stage = 2, block='a', s = 1)\n", 490 | " X = identity_block(X, 3, [64, 64, 256], stage=2, block='b')\n", 491 | " X = identity_block(X, 3, [64, 64, 256], stage=2, block='c')\n", 492 | "\n", 493 | " ### START CODE HERE ###\n", 494 | "\n", 495 | " # Stage 3 (≈4 lines)\n", 496 | " X = convolutional_block(X, f = 3, filters = [128,128,512], stage = 3, block='a', s = 2)\n", 497 | " X = identity_block(X, 3, [128,128,512], stage=3, block='b')\n", 498 | " X = identity_block(X, 3, [128,128,512], stage=3, block='c')\n", 499 | " X = identity_block(X, 3, [128,128,512], stage=3, block='d')\n", 500 | "\n", 501 | " # Stage 4 (≈6 lines)\n", 502 | " X = convolutional_block(X, f = 3, filters = [256, 256, 1024], stage = 4, block='a', s = 2)\n", 503 | " X = identity_block(X, 3, [256, 256, 1024], stage=4, block='b')\n", 504 | " X = identity_block(X, 3, [256, 256, 1024], stage=4, block='c')\n", 505 | " X = identity_block(X, 3, [256, 256, 1024], stage=4, block='d')\n", 506 | " X = identity_block(X, 3, [256, 256, 1024], stage=4, block='e')\n", 507 | " X = identity_block(X, 3, [256, 256, 1024], stage=4, block='f')\n", 508 | "\n", 509 | " # Stage 5 (≈3 lines)\n", 510 | " X = convolutional_block(X, f = 3, filters = [512, 512, 2048], stage = 5, block='a', s = 2)\n", 511 | " X = identity_block(X, 3, [512, 512, 2048], stage=5, block='b')\n", 512 | " X = identity_block(X, 3, [512, 512, 2048], stage=5, block='c')\n", 513 | "\n", 514 | " # AVGPOOL (≈1 line). Use \"X = AveragePooling2D(...)(X)\"\n", 515 | " X = AveragePooling2D((2,2))(X)\n", 516 | " \n", 517 | " ### END CODE HERE ###\n", 518 | "\n", 519 | " # output layer\n", 520 | " X = Flatten()(X)\n", 521 | " X = Dense(classes, activation='softmax', name='fc' + str(classes), kernel_initializer = glorot_uniform(seed=0))(X)\n", 522 | " \n", 523 | " \n", 524 | " # Create model\n", 525 | " model = Model(inputs = X_input, outputs = X, name='ResNet50')\n", 526 | "\n", 527 | " return model" 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | "Run the following code to build the model's graph. If your implementation is not correct you will know it by checking your accuracy when running `model.fit(...)` below." 535 | ] 536 | }, 537 | { 538 | "cell_type": "code", 539 | "execution_count": 7, 540 | "metadata": {}, 541 | "outputs": [], 542 | "source": [ 543 | "model = ResNet50(input_shape = (64, 64, 3), classes = 6)" 544 | ] 545 | }, 546 | { 547 | "cell_type": "markdown", 548 | "metadata": {}, 549 | "source": [ 550 | "As seen in the Keras Tutorial Notebook, prior training a model, you need to configure the learning process by compiling the model." 551 | ] 552 | }, 553 | { 554 | "cell_type": "code", 555 | "execution_count": 8, 556 | "metadata": { 557 | "collapsed": true 558 | }, 559 | "outputs": [], 560 | "source": [ 561 | "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])" 562 | ] 563 | }, 564 | { 565 | "cell_type": "markdown", 566 | "metadata": {}, 567 | "source": [ 568 | "The model is now ready to be trained. The only thing you need is a dataset." 569 | ] 570 | }, 571 | { 572 | "cell_type": "markdown", 573 | "metadata": {}, 574 | "source": [ 575 | "Let's load the SIGNS Dataset.\n", 576 | "\n", 577 | "\n", 578 | "
**Figure 6** : **SIGNS dataset**
\n" 579 | ] 580 | }, 581 | { 582 | "cell_type": "code", 583 | "execution_count": 9, 584 | "metadata": { 585 | "scrolled": true 586 | }, 587 | "outputs": [ 588 | { 589 | "name": "stdout", 590 | "output_type": "stream", 591 | "text": [ 592 | "number of training examples = 1080\n", 593 | "number of test examples = 120\n", 594 | "X_train shape: (1080, 64, 64, 3)\n", 595 | "Y_train shape: (1080, 6)\n", 596 | "X_test shape: (120, 64, 64, 3)\n", 597 | "Y_test shape: (120, 6)\n" 598 | ] 599 | } 600 | ], 601 | "source": [ 602 | "X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, classes = load_dataset()\n", 603 | "\n", 604 | "# Normalize image vectors\n", 605 | "X_train = X_train_orig/255.\n", 606 | "X_test = X_test_orig/255.\n", 607 | "\n", 608 | "# Convert training and test labels to one hot matrices\n", 609 | "Y_train = convert_to_one_hot(Y_train_orig, 6).T\n", 610 | "Y_test = convert_to_one_hot(Y_test_orig, 6).T\n", 611 | "\n", 612 | "print (\"number of training examples = \" + str(X_train.shape[0]))\n", 613 | "print (\"number of test examples = \" + str(X_test.shape[0]))\n", 614 | "print (\"X_train shape: \" + str(X_train.shape))\n", 615 | "print (\"Y_train shape: \" + str(Y_train.shape))\n", 616 | "print (\"X_test shape: \" + str(X_test.shape))\n", 617 | "print (\"Y_test shape: \" + str(Y_test.shape))" 618 | ] 619 | }, 620 | { 621 | "cell_type": "markdown", 622 | "metadata": {}, 623 | "source": [ 624 | "Run the following cell to train your model on 2 epochs with a batch size of 32. On a CPU it should take you around 5min per epoch. " 625 | ] 626 | }, 627 | { 628 | "cell_type": "code", 629 | "execution_count": 10, 630 | "metadata": { 631 | "scrolled": false 632 | }, 633 | "outputs": [ 634 | { 635 | "name": "stdout", 636 | "output_type": "stream", 637 | "text": [ 638 | "Epoch 1/2\n", 639 | "1080/1080 [==============================] - 278s - loss: 2.8332 - acc: 0.2815 \n", 640 | "Epoch 2/2\n", 641 | "1080/1080 [==============================] - 271s - loss: 2.0919 - acc: 0.3593 \n" 642 | ] 643 | }, 644 | { 645 | "data": { 646 | "text/plain": [ 647 | "" 648 | ] 649 | }, 650 | "execution_count": 10, 651 | "metadata": {}, 652 | "output_type": "execute_result" 653 | } 654 | ], 655 | "source": [ 656 | "model.fit(X_train, Y_train, epochs = 2, batch_size = 32)" 657 | ] 658 | }, 659 | { 660 | "cell_type": "markdown", 661 | "metadata": {}, 662 | "source": [ 663 | "**Expected Output**:\n", 664 | "\n", 665 | "\n", 666 | " \n", 667 | " \n", 670 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 678 | " \n", 681 | " \n", 682 | "\n", 683 | "
\n", 668 | " ** Epoch 1/2**\n", 669 | " \n", 671 | " loss: between 1 and 5, acc: between 0.2 and 0.5, although your results can be different from ours.\n", 672 | "
\n", 676 | " ** Epoch 2/2**\n", 677 | " \n", 679 | " loss: between 1 and 5, acc: between 0.2 and 0.5, you should see your loss decreasing and the accuracy increasing.\n", 680 | "
" 684 | ] 685 | }, 686 | { 687 | "cell_type": "markdown", 688 | "metadata": {}, 689 | "source": [ 690 | "Let's see how this model (trained on only two epochs) performs on the test set." 691 | ] 692 | }, 693 | { 694 | "cell_type": "code", 695 | "execution_count": 11, 696 | "metadata": { 697 | "scrolled": true 698 | }, 699 | "outputs": [ 700 | { 701 | "name": "stdout", 702 | "output_type": "stream", 703 | "text": [ 704 | "120/120 [==============================] - 10s \n", 705 | "Loss = 2.4183625857\n", 706 | "Test Accuracy = 0.166666666667\n" 707 | ] 708 | } 709 | ], 710 | "source": [ 711 | "preds = model.evaluate(X_test, Y_test)\n", 712 | "print (\"Loss = \" + str(preds[0]))\n", 713 | "print (\"Test Accuracy = \" + str(preds[1]))" 714 | ] 715 | }, 716 | { 717 | "cell_type": "markdown", 718 | "metadata": {}, 719 | "source": [ 720 | "**Expected Output**:\n", 721 | "\n", 722 | "\n", 723 | " \n", 724 | " \n", 727 | " \n", 730 | " \n", 731 | "\n", 732 | "
\n", 725 | " **Test Accuracy**\n", 726 | " \n", 728 | " between 0.16 and 0.25\n", 729 | "
" 733 | ] 734 | }, 735 | { 736 | "cell_type": "markdown", 737 | "metadata": {}, 738 | "source": [ 739 | "For the purpose of this assignment, we've asked you to train the model only for two epochs. You can see that it achieves poor performances. Please go ahead and submit your assignment; to check correctness, the online grader will run your code only for a small number of epochs as well." 740 | ] 741 | }, 742 | { 743 | "cell_type": "markdown", 744 | "metadata": {}, 745 | "source": [ 746 | "After you have finished this official (graded) part of this assignment, you can also optionally train the ResNet for more iterations, if you want. We get a lot better performance when we train for ~20 epochs, but this will take more than an hour when training on a CPU. \n", 747 | "\n", 748 | "Using a GPU, we've trained our own ResNet50 model's weights on the SIGNS dataset. You can load and run our trained model on the test set in the cells below. It may take ≈1min to load the model." 749 | ] 750 | }, 751 | { 752 | "cell_type": "code", 753 | "execution_count": null, 754 | "metadata": { 755 | "collapsed": true 756 | }, 757 | "outputs": [], 758 | "source": [ 759 | "model = load_model('ResNet50.h5') " 760 | ] 761 | }, 762 | { 763 | "cell_type": "code", 764 | "execution_count": null, 765 | "metadata": { 766 | "collapsed": true, 767 | "scrolled": true 768 | }, 769 | "outputs": [], 770 | "source": [ 771 | "preds = model.evaluate(X_test, Y_test)\n", 772 | "print (\"Loss = \" + str(preds[0]))\n", 773 | "print (\"Test Accuracy = \" + str(preds[1]))" 774 | ] 775 | }, 776 | { 777 | "cell_type": "markdown", 778 | "metadata": {}, 779 | "source": [ 780 | "ResNet50 is a powerful model for image classification when it is trained for an adequate number of iterations. We hope you can use what you've learnt and apply it to your own classification problem to perform state-of-the-art accuracy.\n", 781 | "\n", 782 | "Congratulations on finishing this assignment! You've now implemented a state-of-the-art image classification system! " 783 | ] 784 | }, 785 | { 786 | "cell_type": "markdown", 787 | "metadata": {}, 788 | "source": [ 789 | "## 4 - Test on your own image (Optional/Ungraded)" 790 | ] 791 | }, 792 | { 793 | "cell_type": "markdown", 794 | "metadata": {}, 795 | "source": [ 796 | "If you wish, you can also take a picture of your own hand and see the output of the model. To do this:\n", 797 | " 1. Click on \"File\" in the upper bar of this notebook, then click \"Open\" to go on your Coursera Hub.\n", 798 | " 2. Add your image to this Jupyter Notebook's directory, in the \"images\" folder\n", 799 | " 3. Write your image's name in the following code\n", 800 | " 4. Run the code and check if the algorithm is right! " 801 | ] 802 | }, 803 | { 804 | "cell_type": "code", 805 | "execution_count": null, 806 | "metadata": { 807 | "collapsed": true 808 | }, 809 | "outputs": [], 810 | "source": [ 811 | "img_path = 'images/my_image.jpg'\n", 812 | "img = image.load_img(img_path, target_size=(64, 64))\n", 813 | "x = image.img_to_array(img)\n", 814 | "x = np.expand_dims(x, axis=0)\n", 815 | "x = preprocess_input(x)\n", 816 | "print('Input image shape:', x.shape)\n", 817 | "my_image = scipy.misc.imread(img_path)\n", 818 | "imshow(my_image)\n", 819 | "print(\"class prediction vector [p(0), p(1), p(2), p(3), p(4), p(5)] = \")\n", 820 | "print(model.predict(x))" 821 | ] 822 | }, 823 | { 824 | "cell_type": "markdown", 825 | "metadata": {}, 826 | "source": [ 827 | "You can also print a summary of your model by running the following code." 828 | ] 829 | }, 830 | { 831 | "cell_type": "code", 832 | "execution_count": null, 833 | "metadata": { 834 | "collapsed": true, 835 | "scrolled": true 836 | }, 837 | "outputs": [], 838 | "source": [ 839 | "model.summary()" 840 | ] 841 | }, 842 | { 843 | "cell_type": "markdown", 844 | "metadata": {}, 845 | "source": [ 846 | "Finally, run the code below to visualize your ResNet50. You can also download a .png picture of your model by going to \"File -> Open...-> model.png\"." 847 | ] 848 | }, 849 | { 850 | "cell_type": "code", 851 | "execution_count": null, 852 | "metadata": { 853 | "collapsed": true 854 | }, 855 | "outputs": [], 856 | "source": [ 857 | "plot_model(model, to_file='model.png')\n", 858 | "SVG(model_to_dot(model).create(prog='dot', format='svg'))" 859 | ] 860 | }, 861 | { 862 | "cell_type": "markdown", 863 | "metadata": {}, 864 | "source": [ 865 | "\n", 866 | "**What you should remember:**\n", 867 | "- Very deep \"plain\" networks don't work in practice because they are hard to train due to vanishing gradients. \n", 868 | "- The skip-connections help to address the Vanishing Gradient problem. They also make it easy for a ResNet block to learn an identity function. \n", 869 | "- There are two main type of blocks: The identity block and the convolutional block. \n", 870 | "- Very deep Residual Networks are built by stacking these blocks together." 871 | ] 872 | }, 873 | { 874 | "cell_type": "markdown", 875 | "metadata": {}, 876 | "source": [ 877 | "### References \n", 878 | "\n", 879 | "This notebook presents the ResNet algorithm due to He et al. (2015). The implementation here also took significant inspiration and follows the structure given in the github repository of Francois Chollet: \n", 880 | "\n", 881 | "- Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun - [Deep Residual Learning for Image Recognition (2015)](https://arxiv.org/abs/1512.03385)\n", 882 | "- Francois Chollet's github repository: https://github.com/fchollet/deep-learning-models/blob/master/resnet50.py\n" 883 | ] 884 | } 885 | ], 886 | "metadata": { 887 | "coursera": { 888 | "course_slug": "convolutional-neural-networks", 889 | "graded_item_id": "OEpi5", 890 | "launcher_item_id": "jK9EQ" 891 | }, 892 | "kernelspec": { 893 | "display_name": "Python 3", 894 | "language": "python", 895 | "name": "python3" 896 | }, 897 | "language_info": { 898 | "codemirror_mode": { 899 | "name": "ipython", 900 | "version": 3 901 | }, 902 | "file_extension": ".py", 903 | "mimetype": "text/x-python", 904 | "name": "python", 905 | "nbconvert_exporter": "python", 906 | "pygments_lexer": "ipython3", 907 | "version": "3.6.0" 908 | } 909 | }, 910 | "nbformat": 4, 911 | "nbformat_minor": 2 912 | } 913 | -------------------------------------------------------------------------------- /Convolutional Neural Networks/week4/Face+Recognition+for+the+Happy+House+-+v4.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Face Recognition for the Happy House\n", 8 | "\n", 9 | "Welcome to the first assignment of week 4! Here you will build a face recognition system. Many of the ideas presented here are from [FaceNet](https://arxiv.org/pdf/1503.03832.pdf). In lecture, we also talked about [DeepFace](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf). \n", 10 | "\n", 11 | "Face recognition problems commonly fall into two categories: \n", 12 | "\n", 13 | "- **Face Verification** - \"is this the claimed person?\". For example, at some airports, you can pass through customs by letting a system scan your passport and then verifying that you (the person carrying the passport) are the correct person. A mobile phone that unlocks using your face is also using face verification. This is a 1:1 matching problem. \n", 14 | "- **Face Recognition** - \"who is this person?\". For example, the video lecture showed a face recognition video (https://www.youtube.com/watch?v=wr4rx0Spihs) of Baidu employees entering the office without needing to otherwise identify themselves. This is a 1:K matching problem. \n", 15 | "\n", 16 | "FaceNet learns a neural network that encodes a face image into a vector of 128 numbers. By comparing two such vectors, you can then determine if two pictures are of the same person.\n", 17 | " \n", 18 | "**In this assignment, you will:**\n", 19 | "- Implement the triplet loss function\n", 20 | "- Use a pretrained model to map face images into 128-dimensional encodings\n", 21 | "- Use these encodings to perform face verification and face recognition\n", 22 | "\n", 23 | "In this exercise, we will be using a pre-trained model which represents ConvNet activations using a \"channels first\" convention, as opposed to the \"channels last\" convention used in lecture and previous programming assignments. In other words, a batch of images will be of shape $(m, n_C, n_H, n_W)$ instead of $(m, n_H, n_W, n_C)$. Both of these conventions have a reasonable amount of traction among open-source implementations; there isn't a uniform standard yet within the deep learning community. \n", 24 | "\n", 25 | "Let's load the required packages. \n" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 1, 31 | "metadata": {}, 32 | "outputs": [ 33 | { 34 | "name": "stderr", 35 | "output_type": "stream", 36 | "text": [ 37 | "Using TensorFlow backend.\n" 38 | ] 39 | } 40 | ], 41 | "source": [ 42 | "from keras.models import Sequential\n", 43 | "from keras.layers import Conv2D, ZeroPadding2D, Activation, Input, concatenate\n", 44 | "from keras.models import Model\n", 45 | "from keras.layers.normalization import BatchNormalization\n", 46 | "from keras.layers.pooling import MaxPooling2D, AveragePooling2D\n", 47 | "from keras.layers.merge import Concatenate\n", 48 | "from keras.layers.core import Lambda, Flatten, Dense\n", 49 | "from keras.initializers import glorot_uniform\n", 50 | "from keras.engine.topology import Layer\n", 51 | "from keras import backend as K\n", 52 | "K.set_image_data_format('channels_first')\n", 53 | "import cv2\n", 54 | "import os\n", 55 | "import numpy as np\n", 56 | "from numpy import genfromtxt\n", 57 | "import pandas as pd\n", 58 | "import tensorflow as tf\n", 59 | "from fr_utils import *\n", 60 | "from inception_blocks_v2 import *\n", 61 | "\n", 62 | "%matplotlib inline\n", 63 | "%load_ext autoreload\n", 64 | "%autoreload 2\n", 65 | "\n", 66 | "np.set_printoptions(threshold=np.nan)" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "## 0 - Naive Face Verification\n", 74 | "\n", 75 | "In Face Verification, you're given two images and you have to tell if they are of the same person. The simplest way to do this is to compare the two images pixel-by-pixel. If the distance between the raw images are less than a chosen threshold, it may be the same person! \n", 76 | "\n", 77 | "\n", 78 | "
**Figure 1**
" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": { 84 | "collapsed": true 85 | }, 86 | "source": [ 87 | "Of course, this algorithm performs really poorly, since the pixel values change dramatically due to variations in lighting, orientation of the person's face, even minor changes in head position, and so on. \n", 88 | "\n", 89 | "You'll see that rather than using the raw image, you can learn an encoding $f(img)$ so that element-wise comparisons of this encoding gives more accurate judgements as to whether two pictures are of the same person." 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": {}, 95 | "source": [ 96 | "## 1 - Encoding face images into a 128-dimensional vector \n", 97 | "\n", 98 | "### 1.1 - Using an ConvNet to compute encodings\n", 99 | "\n", 100 | "The FaceNet model takes a lot of data and a long time to train. So following common practice in applied deep learning settings, let's just load weights that someone else has already trained. The network architecture follows the Inception model from [Szegedy *et al.*](https://arxiv.org/abs/1409.4842). We have provided an inception network implementation. You can look in the file `inception_blocks.py` to see how it is implemented (do so by going to \"File->Open...\" at the top of the Jupyter notebook). \n" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "The key things you need to know are:\n", 108 | "\n", 109 | "- This network uses 96x96 dimensional RGB images as its input. Specifically, inputs a face image (or batch of $m$ face images) as a tensor of shape $(m, n_C, n_H, n_W) = (m, 3, 96, 96)$ \n", 110 | "- It outputs a matrix of shape $(m, 128)$ that encodes each input face image into a 128-dimensional vector\n", 111 | "\n", 112 | "Run the cell below to create the model for face images." 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 2, 118 | "metadata": { 119 | "collapsed": true 120 | }, 121 | "outputs": [], 122 | "source": [ 123 | "FRmodel = faceRecoModel(input_shape=(3, 96, 96))" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 3, 129 | "metadata": {}, 130 | "outputs": [ 131 | { 132 | "name": "stdout", 133 | "output_type": "stream", 134 | "text": [ 135 | "Total Params: 3743280\n" 136 | ] 137 | } 138 | ], 139 | "source": [ 140 | "print(\"Total Params:\", FRmodel.count_params())" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "** Expected Output **\n", 148 | "\n", 149 | "
\n", 150 | "Total Params: 3743280\n", 151 | "
\n", 152 | "
\n" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "By using a 128-neuron fully connected layer as its last layer, the model ensures that the output is an encoding vector of size 128. You then use the encodings the compare two face images as follows:\n", 160 | "\n", 161 | "\n", 162 | "
**Figure 2**:
By computing a distance between two encodings and thresholding, you can determine if the two pictures represent the same person
\n", 163 | "\n", 164 | "So, an encoding is a good one if: \n", 165 | "- The encodings of two images of the same person are quite similar to each other \n", 166 | "- The encodings of two images of different persons are very different\n", 167 | "\n", 168 | "The triplet loss function formalizes this, and tries to \"push\" the encodings of two images of the same person (Anchor and Positive) closer together, while \"pulling\" the encodings of two images of different persons (Anchor, Negative) further apart. \n", 169 | "\n", 170 | "\n", 171 | "
\n", 172 | "
**Figure 3**:
In the next part, we will call the pictures from left to right: Anchor (A), Positive (P), Negative (N)
" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "\n", 180 | "\n", 181 | "### 1.2 - The Triplet Loss\n", 182 | "\n", 183 | "For an image $x$, we denote its encoding $f(x)$, where $f$ is the function computed by the neural network.\n", 184 | "\n", 185 | "\n", 186 | "\n", 187 | "\n", 190 | "\n", 191 | "Training will use triplets of images $(A, P, N)$: \n", 192 | "\n", 193 | "- A is an \"Anchor\" image--a picture of a person. \n", 194 | "- P is a \"Positive\" image--a picture of the same person as the Anchor image.\n", 195 | "- N is a \"Negative\" image--a picture of a different person than the Anchor image.\n", 196 | "\n", 197 | "These triplets are picked from our training dataset. We will write $(A^{(i)}, P^{(i)}, N^{(i)})$ to denote the $i$-th training example. \n", 198 | "\n", 199 | "You'd like to make sure that an image $A^{(i)}$ of an individual is closer to the Positive $P^{(i)}$ than to the Negative image $N^{(i)}$) by at least a margin $\\alpha$:\n", 200 | "\n", 201 | "$$\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 + \\alpha < \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$$\n", 202 | "\n", 203 | "You would thus like to minimize the following \"triplet cost\":\n", 204 | "\n", 205 | "$$\\mathcal{J} = \\sum^{m}_{i=1} \\large[ \\small \\underbrace{\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2}_\\text{(1)} - \\underbrace{\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2}_\\text{(2)} + \\alpha \\large ] \\small_+ \\tag{3}$$\n", 206 | "\n", 207 | "Here, we are using the notation \"$[z]_+$\" to denote $max(z,0)$. \n", 208 | "\n", 209 | "Notes:\n", 210 | "- The term (1) is the squared distance between the anchor \"A\" and the positive \"P\" for a given triplet; you want this to be small. \n", 211 | "- The term (2) is the squared distance between the anchor \"A\" and the negative \"N\" for a given triplet, you want this to be relatively large, so it thus makes sense to have a minus sign preceding it. \n", 212 | "- $\\alpha$ is called the margin. It is a hyperparameter that you should pick manually. We will use $\\alpha = 0.2$. \n", 213 | "\n", 214 | "Most implementations also normalize the encoding vectors to have norm equal one (i.e., $\\mid \\mid f(img)\\mid \\mid_2$=1); you won't have to worry about that here.\n", 215 | "\n", 216 | "**Exercise**: Implement the triplet loss as defined by formula (3). Here are the 4 steps:\n", 217 | "1. Compute the distance between the encodings of \"anchor\" and \"positive\": $\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2$\n", 218 | "2. Compute the distance between the encodings of \"anchor\" and \"negative\": $\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$\n", 219 | "3. Compute the formula per training example: $ \\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid - \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2 + \\alpha$\n", 220 | "3. Compute the full formula by taking the max with zero and summing over the training examples:\n", 221 | "$$\\mathcal{J} = \\sum^{m}_{i=1} \\large[ \\small \\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 - \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2+ \\alpha \\large ] \\small_+ \\tag{3}$$\n", 222 | "\n", 223 | "Useful functions: `tf.reduce_sum()`, `tf.square()`, `tf.subtract()`, `tf.add()`, `tf.maximum()`.\n", 224 | "For steps 1 and 2, you will need to sum over the entries of $\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2$ and $\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$ while for step 4 you will need to sum over the training examples." 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": 4, 230 | "metadata": { 231 | "collapsed": true 232 | }, 233 | "outputs": [], 234 | "source": [ 235 | "# GRADED FUNCTION: triplet_loss\n", 236 | "\n", 237 | "def triplet_loss(y_true, y_pred, alpha = 0.2):\n", 238 | " \"\"\"\n", 239 | " Implementation of the triplet loss as defined by formula (3)\n", 240 | " \n", 241 | " Arguments:\n", 242 | " y_true -- true labels, required when you define a loss in Keras, you don't need it in this function.\n", 243 | " y_pred -- python list containing three objects:\n", 244 | " anchor -- the encodings for the anchor images, of shape (None, 128)\n", 245 | " positive -- the encodings for the positive images, of shape (None, 128)\n", 246 | " negative -- the encodings for the negative images, of shape (None, 128)\n", 247 | " \n", 248 | " Returns:\n", 249 | " loss -- real number, value of the loss\n", 250 | " \"\"\"\n", 251 | " \n", 252 | " anchor, positive, negative = y_pred[0], y_pred[1], y_pred[2]\n", 253 | " \n", 254 | " ### START CODE HERE ### (≈ 4 lines)\n", 255 | " # Step 1: Compute the (encoding) distance between the anchor and the positive, you will need to sum over axis=-1\n", 256 | " pos_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, positive)), axis = -1)\n", 257 | " # Step 2: Compute the (encoding) distance between the anchor and the negative, you will need to sum over axis=-1\n", 258 | " neg_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, negative)), axis = -1)\n", 259 | " # Step 3: subtract the two previous distances and add alpha.\n", 260 | " basic_loss = tf.add(tf.subtract(pos_dist, neg_dist), alpha)\n", 261 | " # Step 4: Take the maximum of basic_loss and 0.0. Sum over the training examples.\n", 262 | " loss = tf.reduce_sum(tf.maximum(basic_loss, 0))\n", 263 | " ### END CODE HERE ###\n", 264 | " \n", 265 | " return loss" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": 5, 271 | "metadata": {}, 272 | "outputs": [ 273 | { 274 | "name": "stdout", 275 | "output_type": "stream", 276 | "text": [ 277 | "loss = 528.143\n" 278 | ] 279 | } 280 | ], 281 | "source": [ 282 | "with tf.Session() as test:\n", 283 | " tf.set_random_seed(1)\n", 284 | " y_true = (None, None, None)\n", 285 | " y_pred = (tf.random_normal([3, 128], mean=6, stddev=0.1, seed = 1),\n", 286 | " tf.random_normal([3, 128], mean=1, stddev=1, seed = 1),\n", 287 | " tf.random_normal([3, 128], mean=3, stddev=4, seed = 1))\n", 288 | " loss = triplet_loss(y_true, y_pred)\n", 289 | " \n", 290 | " print(\"loss = \" + str(loss.eval()))" 291 | ] 292 | }, 293 | { 294 | "cell_type": "markdown", 295 | "metadata": {}, 296 | "source": [ 297 | "**Expected Output**:\n", 298 | "\n", 299 | "\n", 300 | " \n", 301 | " \n", 304 | " \n", 307 | " \n", 308 | "\n", 309 | "
\n", 302 | " **loss**\n", 303 | " \n", 305 | " 528.143\n", 306 | "
" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "## 2 - Loading the trained model\n", 317 | "\n", 318 | "FaceNet is trained by minimizing the triplet loss. But since training requires a lot of data and a lot of computation, we won't train it from scratch here. Instead, we load a previously trained model. Load a model using the following cell; this might take a couple of minutes to run. " 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 6, 324 | "metadata": { 325 | "collapsed": true 326 | }, 327 | "outputs": [], 328 | "source": [ 329 | "FRmodel.compile(optimizer = 'adam', loss = triplet_loss, metrics = ['accuracy'])\n", 330 | "load_weights_from_FaceNet(FRmodel)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "Here're some examples of distances between the encodings between three individuals:\n", 338 | "\n", 339 | "\n", 340 | "
\n", 341 | "
**Figure 4**:
Example of distance outputs between three individuals' encodings
\n", 342 | "\n", 343 | "Let's now use this model to perform face verification and face recognition! " 344 | ] 345 | }, 346 | { 347 | "cell_type": "markdown", 348 | "metadata": {}, 349 | "source": [ 350 | "## 3 - Applying the model" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "Back to the Happy House! Residents are living blissfully since you implemented happiness recognition for the house in an earlier assignment. \n", 358 | "\n", 359 | "However, several issues keep coming up: The Happy House became so happy that every happy person in the neighborhood is coming to hang out in your living room. It is getting really crowded, which is having a negative impact on the residents of the house. All these random happy people are also eating all your food. \n", 360 | "\n", 361 | "So, you decide to change the door entry policy, and not just let random happy people enter anymore, even if they are happy! Instead, you'd like to build a **Face verification** system so as to only let people from a specified list come in. To get admitted, each person has to swipe an ID card (identification card) to identify themselves at the door. The face recognition system then checks that they are who they claim to be. " 362 | ] 363 | }, 364 | { 365 | "cell_type": "markdown", 366 | "metadata": {}, 367 | "source": [ 368 | "### 3.1 - Face Verification\n", 369 | "\n", 370 | "Let's build a database containing one encoding vector for each person allowed to enter the happy house. To generate the encoding we use `img_to_encoding(image_path, model)` which basically runs the forward propagation of the model on the specified image. \n", 371 | "\n", 372 | "Run the following code to build the database (represented as a python dictionary). This database maps each person's name to a 128-dimensional encoding of their face." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": 7, 378 | "metadata": { 379 | "collapsed": true 380 | }, 381 | "outputs": [], 382 | "source": [ 383 | "database = {}\n", 384 | "database[\"danielle\"] = img_to_encoding(\"images/danielle.png\", FRmodel)\n", 385 | "database[\"younes\"] = img_to_encoding(\"images/younes.jpg\", FRmodel)\n", 386 | "database[\"tian\"] = img_to_encoding(\"images/tian.jpg\", FRmodel)\n", 387 | "database[\"andrew\"] = img_to_encoding(\"images/andrew.jpg\", FRmodel)\n", 388 | "database[\"kian\"] = img_to_encoding(\"images/kian.jpg\", FRmodel)\n", 389 | "database[\"dan\"] = img_to_encoding(\"images/dan.jpg\", FRmodel)\n", 390 | "database[\"sebastiano\"] = img_to_encoding(\"images/sebastiano.jpg\", FRmodel)\n", 391 | "database[\"bertrand\"] = img_to_encoding(\"images/bertrand.jpg\", FRmodel)\n", 392 | "database[\"kevin\"] = img_to_encoding(\"images/kevin.jpg\", FRmodel)\n", 393 | "database[\"felix\"] = img_to_encoding(\"images/felix.jpg\", FRmodel)\n", 394 | "database[\"benoit\"] = img_to_encoding(\"images/benoit.jpg\", FRmodel)\n", 395 | "database[\"arnaud\"] = img_to_encoding(\"images/arnaud.jpg\", FRmodel)" 396 | ] 397 | }, 398 | { 399 | "cell_type": "markdown", 400 | "metadata": {}, 401 | "source": [ 402 | "Now, when someone shows up at your front door and swipes their ID card (thus giving you their name), you can look up their encoding in the database, and use it to check if the person standing at the front door matches the name on the ID.\n", 403 | "\n", 404 | "**Exercise**: Implement the verify() function which checks if the front-door camera picture (`image_path`) is actually the person called \"identity\". You will have to go through the following steps:\n", 405 | "1. Compute the encoding of the image from image_path\n", 406 | "2. Compute the distance about this encoding and the encoding of the identity image stored in the database\n", 407 | "3. Open the door if the distance is less than 0.7, else do not open.\n", 408 | "\n", 409 | "As presented above, you should use the L2 distance (np.linalg.norm). (Note: In this implementation, compare the L2 distance, not the square of the L2 distance, to the threshold 0.7.) " 410 | ] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "execution_count": 8, 415 | "metadata": { 416 | "collapsed": true 417 | }, 418 | "outputs": [], 419 | "source": [ 420 | "# GRADED FUNCTION: verify\n", 421 | "\n", 422 | "def verify(image_path, identity, database, model):\n", 423 | " \"\"\"\n", 424 | " Function that verifies if the person on the \"image_path\" image is \"identity\".\n", 425 | " \n", 426 | " Arguments:\n", 427 | " image_path -- path to an image\n", 428 | " identity -- string, name of the person you'd like to verify the identity. Has to be a resident of the Happy house.\n", 429 | " database -- python dictionary mapping names of allowed people's names (strings) to their encodings (vectors).\n", 430 | " model -- your Inception model instance in Keras\n", 431 | " \n", 432 | " Returns:\n", 433 | " dist -- distance between the image_path and the image of \"identity\" in the database.\n", 434 | " door_open -- True, if the door should open. False otherwise.\n", 435 | " \"\"\"\n", 436 | " \n", 437 | " ### START CODE HERE ###\n", 438 | " \n", 439 | " # Step 1: Compute the encoding for the image. Use img_to_encoding() see example above. (≈ 1 line)\n", 440 | " encoding = img_to_encoding(image_path, model)\n", 441 | " \n", 442 | " # Step 2: Compute distance with identity's image (≈ 1 line)\n", 443 | " dist = np.linalg.norm(encoding - database[identity])\n", 444 | " \n", 445 | " # Step 3: Open the door if dist < 0.7, else don't open (≈ 3 lines)\n", 446 | " if dist < 0.7:\n", 447 | " print(\"It's \" + str(identity) + \", welcome home!\")\n", 448 | " door_open = True\n", 449 | " else:\n", 450 | " print(\"It's not \" + str(identity) + \", please go away\")\n", 451 | " door_open = False\n", 452 | " \n", 453 | " ### END CODE HERE ###\n", 454 | " \n", 455 | " return dist, door_open" 456 | ] 457 | }, 458 | { 459 | "cell_type": "markdown", 460 | "metadata": {}, 461 | "source": [ 462 | "Younes is trying to enter the Happy House and the camera takes a picture of him (\"images/camera_0.jpg\"). Let's run your verification algorithm on this picture:\n", 463 | "\n", 464 | "" 465 | ] 466 | }, 467 | { 468 | "cell_type": "code", 469 | "execution_count": 9, 470 | "metadata": {}, 471 | "outputs": [ 472 | { 473 | "name": "stdout", 474 | "output_type": "stream", 475 | "text": [ 476 | "It's younes, welcome home!\n" 477 | ] 478 | }, 479 | { 480 | "data": { 481 | "text/plain": [ 482 | "(0.65939283, True)" 483 | ] 484 | }, 485 | "execution_count": 9, 486 | "metadata": {}, 487 | "output_type": "execute_result" 488 | } 489 | ], 490 | "source": [ 491 | "verify(\"images/camera_0.jpg\", \"younes\", database, FRmodel)" 492 | ] 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "metadata": { 497 | "collapsed": true 498 | }, 499 | "source": [ 500 | "**Expected Output**:\n", 501 | "\n", 502 | "\n", 503 | " \n", 504 | " \n", 507 | " \n", 510 | " \n", 511 | "\n", 512 | "
\n", 505 | " **It's younes, welcome home!**\n", 506 | " \n", 508 | " (0.65939283, True)\n", 509 | "
" 513 | ] 514 | }, 515 | { 516 | "cell_type": "markdown", 517 | "metadata": { 518 | "collapsed": true 519 | }, 520 | "source": [ 521 | "Benoit, who broke the aquarium last weekend, has been banned from the house and removed from the database. He stole Kian's ID card and came back to the house to try to present himself as Kian. The front-door camera took a picture of Benoit (\"images/camera_2.jpg). Let's run the verification algorithm to check if benoit can enter.\n", 522 | "" 523 | ] 524 | }, 525 | { 526 | "cell_type": "code", 527 | "execution_count": 10, 528 | "metadata": {}, 529 | "outputs": [ 530 | { 531 | "name": "stdout", 532 | "output_type": "stream", 533 | "text": [ 534 | "It's not kian, please go away\n" 535 | ] 536 | }, 537 | { 538 | "data": { 539 | "text/plain": [ 540 | "(0.86224014, False)" 541 | ] 542 | }, 543 | "execution_count": 10, 544 | "metadata": {}, 545 | "output_type": "execute_result" 546 | } 547 | ], 548 | "source": [ 549 | "verify(\"images/camera_2.jpg\", \"kian\", database, FRmodel)" 550 | ] 551 | }, 552 | { 553 | "cell_type": "markdown", 554 | "metadata": {}, 555 | "source": [ 556 | "**Expected Output**:\n", 557 | "\n", 558 | "\n", 559 | " \n", 560 | " \n", 563 | " \n", 566 | " \n", 567 | "\n", 568 | "
\n", 561 | " **It's not kian, please go away**\n", 562 | " \n", 564 | " (0.86224014, False)\n", 565 | "
" 569 | ] 570 | }, 571 | { 572 | "cell_type": "markdown", 573 | "metadata": {}, 574 | "source": [ 575 | "### 3.2 - Face Recognition\n", 576 | "\n", 577 | "Your face verification system is mostly working well. But since Kian got his ID card stolen, when he came back to the house that evening he couldn't get in! \n", 578 | "\n", 579 | "To reduce such shenanigans, you'd like to change your face verification system to a face recognition system. This way, no one has to carry an ID card anymore. An authorized person can just walk up to the house, and the front door will unlock for them! \n", 580 | "\n", 581 | "You'll implement a face recognition system that takes as input an image, and figures out if it is one of the authorized persons (and if so, who). Unlike the previous face verification system, we will no longer get a person's name as another input. \n", 582 | "\n", 583 | "**Exercise**: Implement `who_is_it()`. You will have to go through the following steps:\n", 584 | "1. Compute the target encoding of the image from image_path\n", 585 | "2. Find the encoding from the database that has smallest distance with the target encoding. \n", 586 | " - Initialize the `min_dist` variable to a large enough number (100). It will help you keep track of what is the closest encoding to the input's encoding.\n", 587 | " - Loop over the database dictionary's names and encodings. To loop use `for (name, db_enc) in database.items()`.\n", 588 | " - Compute L2 distance between the target \"encoding\" and the current \"encoding\" from the database.\n", 589 | " - If this distance is less than the min_dist, then set min_dist to dist, and identity to name." 590 | ] 591 | }, 592 | { 593 | "cell_type": "code", 594 | "execution_count": 11, 595 | "metadata": { 596 | "collapsed": true 597 | }, 598 | "outputs": [], 599 | "source": [ 600 | "# GRADED FUNCTION: who_is_it\n", 601 | "\n", 602 | "def who_is_it(image_path, database, model):\n", 603 | " \"\"\"\n", 604 | " Implements face recognition for the happy house by finding who is the person on the image_path image.\n", 605 | " \n", 606 | " Arguments:\n", 607 | " image_path -- path to an image\n", 608 | " database -- database containing image encodings along with the name of the person on the image\n", 609 | " model -- your Inception model instance in Keras\n", 610 | " \n", 611 | " Returns:\n", 612 | " min_dist -- the minimum distance between image_path encoding and the encodings from the database\n", 613 | " identity -- string, the name prediction for the person on image_path\n", 614 | " \"\"\"\n", 615 | " \n", 616 | " ### START CODE HERE ### \n", 617 | " \n", 618 | " ## Step 1: Compute the target \"encoding\" for the image. Use img_to_encoding() see example above. ## (≈ 1 line)\n", 619 | " encoding = img_to_encoding(image_path, model)\n", 620 | " \n", 621 | " ## Step 2: Find the closest encoding ##\n", 622 | " \n", 623 | " # Initialize \"min_dist\" to a large value, say 100 (≈1 line)\n", 624 | " min_dist = 100\n", 625 | " \n", 626 | " # Loop over the database dictionary's names and encodings.\n", 627 | " for (name, db_enc) in database.items():\n", 628 | " \n", 629 | " # Compute L2 distance between the target \"encoding\" and the current \"emb\" from the database. (≈ 1 line)\n", 630 | " dist = np.linalg.norm(encoding - db_enc)\n", 631 | "\n", 632 | " # If this distance is less than the min_dist, then set min_dist to dist, and identity to name. (≈ 3 lines)\n", 633 | " if dist < min_dist:\n", 634 | " min_dist = dist\n", 635 | " identity = name\n", 636 | "\n", 637 | " ### END CODE HERE ###\n", 638 | " \n", 639 | " if min_dist > 0.7:\n", 640 | " print(\"Not in the database.\")\n", 641 | " else:\n", 642 | " print (\"it's \" + str(identity) + \", the distance is \" + str(min_dist))\n", 643 | " \n", 644 | " return min_dist, identity" 645 | ] 646 | }, 647 | { 648 | "cell_type": "markdown", 649 | "metadata": {}, 650 | "source": [ 651 | "Younes is at the front-door and the camera takes a picture of him (\"images/camera_0.jpg\"). Let's see if your who_it_is() algorithm identifies Younes. " 652 | ] 653 | }, 654 | { 655 | "cell_type": "code", 656 | "execution_count": 12, 657 | "metadata": { 658 | "scrolled": false 659 | }, 660 | "outputs": [ 661 | { 662 | "name": "stdout", 663 | "output_type": "stream", 664 | "text": [ 665 | "it's younes, the distance is 0.659393\n" 666 | ] 667 | }, 668 | { 669 | "data": { 670 | "text/plain": [ 671 | "(0.65939283, 'younes')" 672 | ] 673 | }, 674 | "execution_count": 12, 675 | "metadata": {}, 676 | "output_type": "execute_result" 677 | } 678 | ], 679 | "source": [ 680 | "who_is_it(\"images/camera_0.jpg\", database, FRmodel)" 681 | ] 682 | }, 683 | { 684 | "cell_type": "markdown", 685 | "metadata": {}, 686 | "source": [ 687 | "**Expected Output**:\n", 688 | "\n", 689 | "\n", 690 | " \n", 691 | " \n", 694 | " \n", 697 | " \n", 698 | "\n", 699 | "
\n", 692 | " **it's younes, the distance is 0.659393**\n", 693 | " \n", 695 | " (0.65939283, 'younes')\n", 696 | "
" 700 | ] 701 | }, 702 | { 703 | "cell_type": "markdown", 704 | "metadata": {}, 705 | "source": [ 706 | "You can change \"`camera_0.jpg`\" (picture of younes) to \"`camera_1.jpg`\" (picture of bertrand) and see the result." 707 | ] 708 | }, 709 | { 710 | "cell_type": "markdown", 711 | "metadata": {}, 712 | "source": [ 713 | "Your Happy House is running well. It only lets in authorized persons, and people don't need to carry an ID card around anymore! \n", 714 | "\n", 715 | "You've now seen how a state-of-the-art face recognition system works.\n", 716 | "\n", 717 | "Although we won't implement it here, here're some ways to further improve the algorithm:\n", 718 | "- Put more images of each person (under different lighting conditions, taken on different days, etc.) into the database. Then given a new image, compare the new face to multiple pictures of the person. This would increae accuracy.\n", 719 | "- Crop the images to just contain the face, and less of the \"border\" region around the face. This preprocessing removes some of the irrelevant pixels around the face, and also makes the algorithm more robust.\n" 720 | ] 721 | }, 722 | { 723 | "cell_type": "markdown", 724 | "metadata": {}, 725 | "source": [ 726 | "\n", 727 | "**What you should remember**:\n", 728 | "- Face verification solves an easier 1:1 matching problem; face recognition addresses a harder 1:K matching problem. \n", 729 | "- The triplet loss is an effective loss function for training a neural network to learn an encoding of a face image.\n", 730 | "- The same encoding can be used for verification and recognition. Measuring distances between two images' encodings allows you to determine whether they are pictures of the same person. " 731 | ] 732 | }, 733 | { 734 | "cell_type": "markdown", 735 | "metadata": {}, 736 | "source": [ 737 | "Congrats on finishing this assignment! \n" 738 | ] 739 | }, 740 | { 741 | "cell_type": "markdown", 742 | "metadata": {}, 743 | "source": [ 744 | "### References:\n", 745 | "\n", 746 | "- Florian Schroff, Dmitry Kalenichenko, James Philbin (2015). [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/pdf/1503.03832.pdf)\n", 747 | "- Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, Lior Wolf (2014). [DeepFace: Closing the gap to human-level performance in face verification](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf) \n", 748 | "- The pretrained model we use is inspired by Victor Sy Wang's implementation and was loaded using his code: https://github.com/iwantooxxoox/Keras-OpenFace.\n", 749 | "- Our implementation also took a lot of inspiration from the official FaceNet github repository: https://github.com/davidsandberg/facenet \n" 750 | ] 751 | }, 752 | { 753 | "cell_type": "code", 754 | "execution_count": null, 755 | "metadata": { 756 | "collapsed": true 757 | }, 758 | "outputs": [], 759 | "source": [] 760 | } 761 | ], 762 | "metadata": { 763 | "coursera": { 764 | "course_slug": "convolutional-neural-networks", 765 | "graded_item_id": "IaknP", 766 | "launcher_item_id": "5UMr4" 767 | }, 768 | "kernelspec": { 769 | "display_name": "Python 3", 770 | "language": "python", 771 | "name": "python3" 772 | }, 773 | "language_info": { 774 | "codemirror_mode": { 775 | "name": "ipython", 776 | "version": 3 777 | }, 778 | "file_extension": ".py", 779 | "mimetype": "text/x-python", 780 | "name": "python", 781 | "nbconvert_exporter": "python", 782 | "pygments_lexer": "ipython3", 783 | "version": "3.6.0" 784 | } 785 | }, 786 | "nbformat": 4, 787 | "nbformat_minor": 2 788 | } 789 | -------------------------------------------------------------------------------- /Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/.DS_Store -------------------------------------------------------------------------------- /Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Certificate.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Certificate.pdf -------------------------------------------------------------------------------- /Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Week 1 Programming Assignments/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Week 1 Programming Assignments/.DS_Store -------------------------------------------------------------------------------- /Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Week 1 Programming Assignments/Gradient+Checking.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Gradient Checking\n", 8 | "\n", 9 | "Welcome to the final assignment for this week! In this assignment you will learn to implement and use gradient checking. \n", 10 | "\n", 11 | "You are part of a team working to make mobile payments available globally, and are asked to build a deep learning model to detect fraud--whenever someone makes a payment, you want to see if the payment might be fraudulent, such as if the user's account has been taken over by a hacker. \n", 12 | "\n", 13 | "But backpropagation is quite challenging to implement, and sometimes has bugs. Because this is a mission-critical application, your company's CEO wants to be really certain that your implementation of backpropagation is correct. Your CEO says, \"Give me a proof that your backpropagation is actually working!\" To give this reassurance, you are going to use \"gradient checking\".\n", 14 | "\n", 15 | "Let's do it!" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "metadata": { 22 | "collapsed": true 23 | }, 24 | "outputs": [], 25 | "source": [ 26 | "# Packages\n", 27 | "import numpy as np\n", 28 | "from testCases import *\n", 29 | "from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "## 1) How does gradient checking work?\n", 37 | "\n", 38 | "Backpropagation computes the gradients $\\frac{\\partial J}{\\partial \\theta}$, where $\\theta$ denotes the parameters of the model. $J$ is computed using forward propagation and your loss function.\n", 39 | "\n", 40 | "Because forward propagation is relatively easy to implement, you're confident you got that right, and so you're almost 100% sure that you're computing the cost $J$ correctly. Thus, you can use your code for computing $J$ to verify the code for computing $\\frac{\\partial J}{\\partial \\theta}$. \n", 41 | "\n", 42 | "Let's look back at the definition of a derivative (or gradient):\n", 43 | "$$ \\frac{\\partial J}{\\partial \\theta} = \\lim_{\\varepsilon \\to 0} \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon} \\tag{1}$$\n", 44 | "\n", 45 | "If you're not familiar with the \"$\\displaystyle \\lim_{\\varepsilon \\to 0}$\" notation, it's just a way of saying \"when $\\varepsilon$ is really really small.\"\n", 46 | "\n", 47 | "We know the following:\n", 48 | "\n", 49 | "- $\\frac{\\partial J}{\\partial \\theta}$ is what you want to make sure you're computing correctly. \n", 50 | "- You can compute $J(\\theta + \\varepsilon)$ and $J(\\theta - \\varepsilon)$ (in the case that $\\theta$ is a real number), since you're confident your implementation for $J$ is correct. \n", 51 | "\n", 52 | "Lets use equation (1) and a small value for $\\varepsilon$ to convince your CEO that your code for computing $\\frac{\\partial J}{\\partial \\theta}$ is correct!" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "## 2) 1-dimensional gradient checking\n", 60 | "\n", 61 | "Consider a 1D linear function $J(\\theta) = \\theta x$. The model contains only a single real-valued parameter $\\theta$, and takes $x$ as input.\n", 62 | "\n", 63 | "You will implement code to compute $J(.)$ and its derivative $\\frac{\\partial J}{\\partial \\theta}$. You will then use gradient checking to make sure your derivative computation for $J$ is correct. \n", 64 | "\n", 65 | "\n", 66 | "
**Figure 1** : **1D linear model**
\n", 67 | "\n", 68 | "The diagram above shows the key computation steps: First start with $x$, then evaluate the function $J(x)$ (\"forward propagation\"). Then compute the derivative $\\frac{\\partial J}{\\partial \\theta}$ (\"backward propagation\"). \n", 69 | "\n", 70 | "**Exercise**: implement \"forward propagation\" and \"backward propagation\" for this simple function. I.e., compute both $J(.)$ (\"forward propagation\") and its derivative with respect to $\\theta$ (\"backward propagation\"), in two separate functions. " 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 3, 76 | "metadata": { 77 | "collapsed": true 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "# GRADED FUNCTION: forward_propagation\n", 82 | "\n", 83 | "def forward_propagation(x, theta):\n", 84 | " \"\"\"\n", 85 | " Implement the linear forward propagation (compute J) presented in Figure 1 (J(theta) = theta * x)\n", 86 | " \n", 87 | " Arguments:\n", 88 | " x -- a real-valued input\n", 89 | " theta -- our parameter, a real number as well\n", 90 | " \n", 91 | " Returns:\n", 92 | " J -- the value of function J, computed using the formula J(theta) = theta * x\n", 93 | " \"\"\"\n", 94 | " \n", 95 | " ### START CODE HERE ### (approx. 1 line)\n", 96 | " J = x * theta\n", 97 | " ### END CODE HERE ###\n", 98 | " \n", 99 | " return J" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 4, 105 | "metadata": {}, 106 | "outputs": [ 107 | { 108 | "name": "stdout", 109 | "output_type": "stream", 110 | "text": [ 111 | "J = 8\n" 112 | ] 113 | } 114 | ], 115 | "source": [ 116 | "x, theta = 2, 4\n", 117 | "J = forward_propagation(x, theta)\n", 118 | "print (\"J = \" + str(J))" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "**Expected Output**:\n", 126 | "\n", 127 | "\n", 128 | " \n", 129 | " \n", 130 | " \n", 131 | " \n", 132 | "
** J ** 8
" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "**Exercise**: Now, implement the backward propagation step (derivative computation) of Figure 1. That is, compute the derivative of $J(\\theta) = \\theta x$ with respect to $\\theta$. To save you from doing the calculus, you should get $dtheta = \\frac { \\partial J }{ \\partial \\theta} = x$." 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 5, 145 | "metadata": { 146 | "collapsed": true 147 | }, 148 | "outputs": [], 149 | "source": [ 150 | "# GRADED FUNCTION: backward_propagation\n", 151 | "\n", 152 | "def backward_propagation(x, theta):\n", 153 | " \"\"\"\n", 154 | " Computes the derivative of J with respect to theta (see Figure 1).\n", 155 | " \n", 156 | " Arguments:\n", 157 | " x -- a real-valued input\n", 158 | " theta -- our parameter, a real number as well\n", 159 | " \n", 160 | " Returns:\n", 161 | " dtheta -- the gradient of the cost with respect to theta\n", 162 | " \"\"\"\n", 163 | " \n", 164 | " ### START CODE HERE ### (approx. 1 line)\n", 165 | " dtheta = x\n", 166 | " ### END CODE HERE ###\n", 167 | " \n", 168 | " return dtheta" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": 6, 174 | "metadata": { 175 | "scrolled": true 176 | }, 177 | "outputs": [ 178 | { 179 | "name": "stdout", 180 | "output_type": "stream", 181 | "text": [ 182 | "dtheta = 2\n" 183 | ] 184 | } 185 | ], 186 | "source": [ 187 | "x, theta = 2, 4\n", 188 | "dtheta = backward_propagation(x, theta)\n", 189 | "print (\"dtheta = \" + str(dtheta))" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "**Expected Output**:\n", 197 | "\n", 198 | "\n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | " \n", 203 | "
** dtheta ** 2
" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "**Exercise**: To show that the `backward_propagation()` function is correctly computing the gradient $\\frac{\\partial J}{\\partial \\theta}$, let's implement gradient checking.\n", 211 | "\n", 212 | "**Instructions**:\n", 213 | "- First compute \"gradapprox\" using the formula above (1) and a small value of $\\varepsilon$. Here are the Steps to follow:\n", 214 | " 1. $\\theta^{+} = \\theta + \\varepsilon$\n", 215 | " 2. $\\theta^{-} = \\theta - \\varepsilon$\n", 216 | " 3. $J^{+} = J(\\theta^{+})$\n", 217 | " 4. $J^{-} = J(\\theta^{-})$\n", 218 | " 5. $gradapprox = \\frac{J^{+} - J^{-}}{2 \\varepsilon}$\n", 219 | "- Then compute the gradient using backward propagation, and store the result in a variable \"grad\"\n", 220 | "- Finally, compute the relative difference between \"gradapprox\" and the \"grad\" using the following formula:\n", 221 | "$$ difference = \\frac {\\mid\\mid grad - gradapprox \\mid\\mid_2}{\\mid\\mid grad \\mid\\mid_2 + \\mid\\mid gradapprox \\mid\\mid_2} \\tag{2}$$\n", 222 | "You will need 3 Steps to compute this formula:\n", 223 | " - 1'. compute the numerator using np.linalg.norm(...)\n", 224 | " - 2'. compute the denominator. You will need to call np.linalg.norm(...) twice.\n", 225 | " - 3'. divide them.\n", 226 | "- If this difference is small (say less than $10^{-7}$), you can be quite confident that you have computed your gradient correctly. Otherwise, there may be a mistake in the gradient computation. \n" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 9, 232 | "metadata": { 233 | "collapsed": true 234 | }, 235 | "outputs": [], 236 | "source": [ 237 | "# GRADED FUNCTION: gradient_check\n", 238 | "\n", 239 | "def gradient_check(x, theta, epsilon = 1e-7):\n", 240 | " \"\"\"\n", 241 | " Implement the backward propagation presented in Figure 1.\n", 242 | " \n", 243 | " Arguments:\n", 244 | " x -- a real-valued input\n", 245 | " theta -- our parameter, a real number as well\n", 246 | " epsilon -- tiny shift to the input to compute approximated gradient with formula(1)\n", 247 | " \n", 248 | " Returns:\n", 249 | " difference -- difference (2) between the approximated gradient and the backward propagation gradient\n", 250 | " \"\"\"\n", 251 | " \n", 252 | " # Compute gradapprox using left side of formula (1). epsilon is small enough, you don't need to worry about the limit.\n", 253 | " ### START CODE HERE ### (approx. 5 lines)\n", 254 | " thetaplus = theta + epsilon # Step 1\n", 255 | " thetaminus = theta - epsilon # Step 2\n", 256 | " J_plus = x * (theta + epsilon) # Step 3\n", 257 | " J_minus = x * (theta - epsilon) # Step 4\n", 258 | " gradapprox = (J_plus - J_minus) / (2 * epsilon) # Step 5\n", 259 | " ### END CODE HERE ###\n", 260 | " \n", 261 | " # Check if gradapprox is close enough to the output of backward_propagation()\n", 262 | " ### START CODE HERE ### (approx. 1 line)\n", 263 | " grad = x\n", 264 | " ### END CODE HERE ###\n", 265 | " \n", 266 | " ### START CODE HERE ### (approx. 1 line)\n", 267 | " numerator = np.linalg.norm(grad - gradapprox) # Step 1'\n", 268 | " denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'\n", 269 | " difference = numerator / denominator # Step 3'\n", 270 | " ### END CODE HERE ###\n", 271 | " \n", 272 | " if difference < 1e-7:\n", 273 | " print (\"The gradient is correct!\")\n", 274 | " else:\n", 275 | " print (\"The gradient is wrong!\")\n", 276 | " \n", 277 | " return difference" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 10, 283 | "metadata": { 284 | "scrolled": true 285 | }, 286 | "outputs": [ 287 | { 288 | "name": "stdout", 289 | "output_type": "stream", 290 | "text": [ 291 | "The gradient is correct!\n", 292 | "difference = 2.91933588329e-10\n" 293 | ] 294 | } 295 | ], 296 | "source": [ 297 | "x, theta = 2, 4\n", 298 | "difference = gradient_check(x, theta)\n", 299 | "print(\"difference = \" + str(difference))" 300 | ] 301 | }, 302 | { 303 | "cell_type": "markdown", 304 | "metadata": {}, 305 | "source": [ 306 | "**Expected Output**:\n", 307 | "The gradient is correct!\n", 308 | "\n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | "
** difference ** 2.9193358103083e-10
" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": {}, 319 | "source": [ 320 | "Congrats, the difference is smaller than the $10^{-7}$ threshold. So you can have high confidence that you've correctly computed the gradient in `backward_propagation()`. \n", 321 | "\n", 322 | "Now, in the more general case, your cost function $J$ has more than a single 1D input. When you are training a neural network, $\\theta$ actually consists of multiple matrices $W^{[l]}$ and biases $b^{[l]}$! It is important to know how to do a gradient check with higher-dimensional inputs. Let's do it!" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "## 3) N-dimensional gradient checking" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "metadata": { 335 | "collapsed": true 336 | }, 337 | "source": [ 338 | "The following figure describes the forward and backward propagation of your fraud detection model.\n", 339 | "\n", 340 | "\n", 341 | "
**Figure 2** : **deep neural network**
*LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID*
\n", 342 | "\n", 343 | "Let's look at your implementations for forward propagation and backward propagation. " 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 11, 349 | "metadata": { 350 | "collapsed": true 351 | }, 352 | "outputs": [], 353 | "source": [ 354 | "def forward_propagation_n(X, Y, parameters):\n", 355 | " \"\"\"\n", 356 | " Implements the forward propagation (and computes the cost) presented in Figure 3.\n", 357 | " \n", 358 | " Arguments:\n", 359 | " X -- training set for m examples\n", 360 | " Y -- labels for m examples \n", 361 | " parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n", 362 | " W1 -- weight matrix of shape (5, 4)\n", 363 | " b1 -- bias vector of shape (5, 1)\n", 364 | " W2 -- weight matrix of shape (3, 5)\n", 365 | " b2 -- bias vector of shape (3, 1)\n", 366 | " W3 -- weight matrix of shape (1, 3)\n", 367 | " b3 -- bias vector of shape (1, 1)\n", 368 | " \n", 369 | " Returns:\n", 370 | " cost -- the cost function (logistic cost for one example)\n", 371 | " \"\"\"\n", 372 | " \n", 373 | " # retrieve parameters\n", 374 | " m = X.shape[1]\n", 375 | " W1 = parameters[\"W1\"]\n", 376 | " b1 = parameters[\"b1\"]\n", 377 | " W2 = parameters[\"W2\"]\n", 378 | " b2 = parameters[\"b2\"]\n", 379 | " W3 = parameters[\"W3\"]\n", 380 | " b3 = parameters[\"b3\"]\n", 381 | "\n", 382 | " # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID\n", 383 | " Z1 = np.dot(W1, X) + b1\n", 384 | " A1 = relu(Z1)\n", 385 | " Z2 = np.dot(W2, A1) + b2\n", 386 | " A2 = relu(Z2)\n", 387 | " Z3 = np.dot(W3, A2) + b3\n", 388 | " A3 = sigmoid(Z3)\n", 389 | "\n", 390 | " # Cost\n", 391 | " logprobs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)\n", 392 | " cost = 1./m * np.sum(logprobs)\n", 393 | " \n", 394 | " cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)\n", 395 | " \n", 396 | " return cost, cache" 397 | ] 398 | }, 399 | { 400 | "cell_type": "markdown", 401 | "metadata": {}, 402 | "source": [ 403 | "Now, run backward propagation." 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 12, 409 | "metadata": { 410 | "collapsed": true 411 | }, 412 | "outputs": [], 413 | "source": [ 414 | "def backward_propagation_n(X, Y, cache):\n", 415 | " \"\"\"\n", 416 | " Implement the backward propagation presented in figure 2.\n", 417 | " \n", 418 | " Arguments:\n", 419 | " X -- input datapoint, of shape (input size, 1)\n", 420 | " Y -- true \"label\"\n", 421 | " cache -- cache output from forward_propagation_n()\n", 422 | " \n", 423 | " Returns:\n", 424 | " gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables.\n", 425 | " \"\"\"\n", 426 | " \n", 427 | " m = X.shape[1]\n", 428 | " (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache\n", 429 | " \n", 430 | " dZ3 = A3 - Y\n", 431 | " dW3 = 1./m * np.dot(dZ3, A2.T)\n", 432 | " db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)\n", 433 | " \n", 434 | " dA2 = np.dot(W3.T, dZ3)\n", 435 | " dZ2 = np.multiply(dA2, np.int64(A2 > 0))\n", 436 | " dW2 = 1./m * np.dot(dZ2, A1.T) * 2\n", 437 | " db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)\n", 438 | " \n", 439 | " dA1 = np.dot(W2.T, dZ2)\n", 440 | " dZ1 = np.multiply(dA1, np.int64(A1 > 0))\n", 441 | " dW1 = 1./m * np.dot(dZ1, X.T)\n", 442 | " db1 = 4./m * np.sum(dZ1, axis=1, keepdims = True)\n", 443 | " \n", 444 | " gradients = {\"dZ3\": dZ3, \"dW3\": dW3, \"db3\": db3,\n", 445 | " \"dA2\": dA2, \"dZ2\": dZ2, \"dW2\": dW2, \"db2\": db2,\n", 446 | " \"dA1\": dA1, \"dZ1\": dZ1, \"dW1\": dW1, \"db1\": db1}\n", 447 | " \n", 448 | " return gradients" 449 | ] 450 | }, 451 | { 452 | "cell_type": "markdown", 453 | "metadata": { 454 | "collapsed": true 455 | }, 456 | "source": [ 457 | "You obtained some results on the fraud detection test set but you are not 100% sure of your model. Nobody's perfect! Let's implement gradient checking to verify if your gradients are correct." 458 | ] 459 | }, 460 | { 461 | "cell_type": "markdown", 462 | "metadata": {}, 463 | "source": [ 464 | "**How does gradient checking work?**.\n", 465 | "\n", 466 | "As in 1) and 2), you want to compare \"gradapprox\" to the gradient computed by backpropagation. The formula is still:\n", 467 | "\n", 468 | "$$ \\frac{\\partial J}{\\partial \\theta} = \\lim_{\\varepsilon \\to 0} \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon} \\tag{1}$$\n", 469 | "\n", 470 | "However, $\\theta$ is not a scalar anymore. It is a dictionary called \"parameters\". We implemented a function \"`dictionary_to_vector()`\" for you. It converts the \"parameters\" dictionary into a vector called \"values\", obtained by reshaping all parameters (W1, b1, W2, b2, W3, b3) into vectors and concatenating them.\n", 471 | "\n", 472 | "The inverse function is \"`vector_to_dictionary`\" which outputs back the \"parameters\" dictionary.\n", 473 | "\n", 474 | "\n", 475 | "
**Figure 2** : **dictionary_to_vector() and vector_to_dictionary()**
You will need these functions in gradient_check_n()
\n", 476 | "\n", 477 | "We have also converted the \"gradients\" dictionary into a vector \"grad\" using gradients_to_vector(). You don't need to worry about that.\n", 478 | "\n", 479 | "**Exercise**: Implement gradient_check_n().\n", 480 | "\n", 481 | "**Instructions**: Here is pseudo-code that will help you implement the gradient check.\n", 482 | "\n", 483 | "For each i in num_parameters:\n", 484 | "- To compute `J_plus[i]`:\n", 485 | " 1. Set $\\theta^{+}$ to `np.copy(parameters_values)`\n", 486 | " 2. Set $\\theta^{+}_i$ to $\\theta^{+}_i + \\varepsilon$\n", 487 | " 3. Calculate $J^{+}_i$ using to `forward_propagation_n(x, y, vector_to_dictionary(`$\\theta^{+}$ `))`. \n", 488 | "- To compute `J_minus[i]`: do the same thing with $\\theta^{-}$\n", 489 | "- Compute $gradapprox[i] = \\frac{J^{+}_i - J^{-}_i}{2 \\varepsilon}$\n", 490 | "\n", 491 | "Thus, you get a vector gradapprox, where gradapprox[i] is an approximation of the gradient with respect to `parameter_values[i]`. You can now compare this gradapprox vector to the gradients vector from backpropagation. Just like for the 1D case (Steps 1', 2', 3'), compute: \n", 492 | "$$ difference = \\frac {\\| grad - gradapprox \\|_2}{\\| grad \\|_2 + \\| gradapprox \\|_2 } \\tag{3}$$" 493 | ] 494 | }, 495 | { 496 | "cell_type": "code", 497 | "execution_count": 27, 498 | "metadata": { 499 | "collapsed": true 500 | }, 501 | "outputs": [], 502 | "source": [ 503 | "# GRADED FUNCTION: gradient_check_n\n", 504 | "\n", 505 | "def gradient_check_n(parameters, gradients, X, Y, epsilon = 1e-7):\n", 506 | " \"\"\"\n", 507 | " Checks if backward_propagation_n computes correctly the gradient of the cost output by forward_propagation_n\n", 508 | " \n", 509 | " Arguments:\n", 510 | " parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n", 511 | " grad -- output of backward_propagation_n, contains gradients of the cost with respect to the parameters. \n", 512 | " x -- input datapoint, of shape (input size, 1)\n", 513 | " y -- true \"label\"\n", 514 | " epsilon -- tiny shift to the input to compute approximated gradient with formula(1)\n", 515 | " \n", 516 | " Returns:\n", 517 | " difference -- difference (2) between the approximated gradient and the backward propagation gradient\n", 518 | " \"\"\"\n", 519 | " \n", 520 | " # Set-up variables\n", 521 | " parameters_values, _ = dictionary_to_vector(parameters)\n", 522 | " grad = gradients_to_vector(gradients)\n", 523 | " num_parameters = parameters_values.shape[0]\n", 524 | " J_plus = np.zeros((num_parameters, 1))\n", 525 | " J_minus = np.zeros((num_parameters, 1))\n", 526 | " gradapprox = np.zeros((num_parameters, 1))\n", 527 | " \n", 528 | "# print(parameters_values.shape)\n", 529 | " # Compute gradapprox\n", 530 | " for i in range(num_parameters):\n", 531 | " \n", 532 | " # Compute J_plus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_plus[i]\".\n", 533 | " # \"_\" is used because the function you have to outputs two parameters but we only care about the first one\n", 534 | " ### START CODE HERE ### (approx. 3 lines)\n", 535 | " thetaplus = np.copy(parameters_values) # Step 1\n", 536 | " thetaplus[i] = thetaplus[i] + epsilon # Step 2\n", 537 | " J_plus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaplus)) # Step 3\n", 538 | " ### END CODE HERE ###\n", 539 | " \n", 540 | " # Compute J_minus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_minus[i]\".\n", 541 | " ### START CODE HERE ### (approx. 3 lines)\n", 542 | " thetaminus = np.copy(parameters_values) # Step 1\n", 543 | " thetaminus[i] = thetaminus[i] - epsilon # Step 2 \n", 544 | " J_minus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaminus)) # Step 3\n", 545 | " ### END CODE HERE ###\n", 546 | " \n", 547 | " # Compute gradapprox[i]\n", 548 | " ### START CODE HERE ### (approx. 1 line)\n", 549 | " gradapprox[i] = (J_plus[i] - J_minus[i]) / (2 * epsilon)\n", 550 | " ### END CODE HERE ###\n", 551 | " \n", 552 | " # Compare gradapprox to backward propagation gradients by computing difference.\n", 553 | " ### START CODE HERE ### (approx. 1 line)\n", 554 | " numerator = np.linalg.norm(grad - gradapprox) # Step 1'\n", 555 | " denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'\n", 556 | " difference = numerator / denominator \n", 557 | " ### END CODE HERE ###\n", 558 | "\n", 559 | " if difference > 1e-7:\n", 560 | " print (\"\\033[93m\" + \"There is a mistake in the backward propagation! difference = \" + str(difference) + \"\\033[0m\")\n", 561 | " else:\n", 562 | " print (\"\\033[92m\" + \"Your backward propagation works perfectly fine! difference = \" + str(difference) + \"\\033[0m\")\n", 563 | " \n", 564 | " return difference" 565 | ] 566 | }, 567 | { 568 | "cell_type": "code", 569 | "execution_count": 28, 570 | "metadata": { 571 | "scrolled": false 572 | }, 573 | "outputs": [ 574 | { 575 | "name": "stdout", 576 | "output_type": "stream", 577 | "text": [ 578 | "\u001b[93mThere is a mistake in the backward propagation! difference = 0.285093156781\u001b[0m\n" 579 | ] 580 | } 581 | ], 582 | "source": [ 583 | "X, Y, parameters = gradient_check_n_test_case()\n", 584 | "\n", 585 | "cost, cache = forward_propagation_n(X, Y, parameters)\n", 586 | "gradients = backward_propagation_n(X, Y, cache)\n", 587 | "difference = gradient_check_n(parameters, gradients, X, Y)" 588 | ] 589 | }, 590 | { 591 | "cell_type": "markdown", 592 | "metadata": {}, 593 | "source": [ 594 | "**Expected output**:\n", 595 | "\n", 596 | "\n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | "
** There is a mistake in the backward propagation!** difference = 0.285093156781
" 602 | ] 603 | }, 604 | { 605 | "cell_type": "markdown", 606 | "metadata": {}, 607 | "source": [ 608 | "It seems that there were errors in the `backward_propagation_n` code we gave you! Good that you've implemented the gradient check. Go back to `backward_propagation` and try to find/correct the errors *(Hint: check dW2 and db1)*. Rerun the gradient check when you think you've fixed it. Remember you'll need to re-execute the cell defining `backward_propagation_n()` if you modify the code. \n", 609 | "\n", 610 | "Can you get gradient check to declare your derivative computation correct? Even though this part of the assignment isn't graded, we strongly urge you to try to find the bug and re-run gradient check until you're convinced backprop is now correctly implemented. \n", 611 | "\n", 612 | "**Note** \n", 613 | "- Gradient Checking is slow! Approximating the gradient with $\\frac{\\partial J}{\\partial \\theta} \\approx \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon}$ is computationally costly. For this reason, we don't run gradient checking at every iteration during training. Just a few times to check if the gradient is correct. \n", 614 | "- Gradient Checking, at least as we've presented it, doesn't work with dropout. You would usually run the gradient check algorithm without dropout to make sure your backprop is correct, then add dropout. \n", 615 | "\n", 616 | "Congrats, you can be confident that your deep learning model for fraud detection is working correctly! You can even use this to convince your CEO. :) \n", 617 | "\n", 618 | "\n", 619 | "**What you should remember from this notebook**:\n", 620 | "- Gradient checking verifies closeness between the gradients from backpropagation and the numerical approximation of the gradient (computed using forward propagation).\n", 621 | "- Gradient checking is slow, so we don't run it in every iteration of training. You would usually run it only to make sure your code is correct, then turn it off and use backprop for the actual learning process. " 622 | ] 623 | } 624 | ], 625 | "metadata": { 626 | "coursera": { 627 | "course_slug": "deep-neural-network", 628 | "graded_item_id": "n6NBD", 629 | "launcher_item_id": "yfOsE" 630 | }, 631 | "kernelspec": { 632 | "display_name": "Python 3", 633 | "language": "python", 634 | "name": "python3" 635 | }, 636 | "language_info": { 637 | "codemirror_mode": { 638 | "name": "ipython", 639 | "version": 3 640 | }, 641 | "file_extension": ".py", 642 | "mimetype": "text/x-python", 643 | "name": "python", 644 | "nbconvert_exporter": "python", 645 | "pygments_lexer": "ipython3", 646 | "version": "3.6.0" 647 | } 648 | }, 649 | "nbformat": 4, 650 | "nbformat_minor": 1 651 | } 652 | -------------------------------------------------------------------------------- /Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Week 2 Programming Assignments/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Week 2 Programming Assignments/.DS_Store -------------------------------------------------------------------------------- /Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Week 3 Programming Assignments/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Improving Deep Neural Networks_Hyperparameter tuning, Regularization and Optimization/Week 3 Programming Assignments/.DS_Store -------------------------------------------------------------------------------- /Neural Networks and Deep Learning/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Neural Networks and Deep Learning/.DS_Store -------------------------------------------------------------------------------- /Neural Networks and Deep Learning/Certificate.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Neural Networks and Deep Learning/Certificate.pdf -------------------------------------------------------------------------------- /Neural Networks and Deep Learning/Deep-learning-notation.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Neural Networks and Deep Learning/Deep-learning-notation.pdf -------------------------------------------------------------------------------- /Neural Networks and Deep Learning/README.md: -------------------------------------------------------------------------------- 1 | # coursera-Deep-Learning-Specialization 2 | 3 | I will upload the slides and assignments gradually, and if you like this repository, please follow me and give me a star :) (slides are not available right now in the course page) 4 | 5 | 6 | 7 | This course is created by deeplearning.ai. Master Deep Learning, and Break into AI 8 | 9 | https://www.coursera.org/specializations/deep-learning 10 | 11 | 12 | cited from the homepage: 13 | 14 | If you want to break into AI, this Specialization will help you do so. Deep Learning is one of the most highly sought after skills in tech. We will help you become good at Deep Learning. 15 | 16 | In five courses, you will learn the foundations of Deep Learning, understand how to build neural networks, and learn how to lead successful machine learning projects. You will learn about Convolutional networks, RNNs, LSTM, Adam, Dropout, BatchNorm, Xavier/He initialization, and more. You will work on case studies from healthcare, autonomous driving, sign language reading, music generation, and natural language processing. You will master not only the theory, but also see how it is applied in industry. You will practice all these ideas in Python and in TensorFlow, which we will teach. 17 | 18 | You will also hear from many top leaders in Deep Learning, who will share with you their personal stories and give you career advice. 19 | 20 | AI is transforming multiple industries. After finishing this specialization, you will likely find creative ways to apply it to your work. 21 | 22 | We will help you master Deep Learning, understand how to apply it, and build a career in AI. 23 | -------------------------------------------------------------------------------- /Neural Networks and Deep Learning/Week 2 Programming Assignments/Python+Basics+With+Numpy_Practice.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Python Basics with Numpy (optional assignment)\n", 8 | "\n", 9 | "Welcome to your first assignment. This exercise gives you a brief introduction to Python. Even if you've used Python before, this will help familiarize you with functions we'll need. \n", 10 | "\n", 11 | "**Instructions:**\n", 12 | "- You will be using Python 3.\n", 13 | "- Avoid using for-loops and while-loops, unless you are explicitly told to do so.\n", 14 | "- Do not modify the (# GRADED FUNCTION [function name]) comment in some cells. Your work would not be graded if you change this. Each cell containing that comment should only contain one function.\n", 15 | "- After coding your function, run the cell right below it to check if your result is correct.\n", 16 | "\n", 17 | "**After this assignment you will:**\n", 18 | "- Be able to use iPython Notebooks\n", 19 | "- Be able to use numpy functions and numpy matrix/vector operations\n", 20 | "- Understand the concept of \"broadcasting\"\n", 21 | "- Be able to vectorize code\n", 22 | "\n", 23 | "Let's get started!" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## About iPython Notebooks ##\n", 31 | "\n", 32 | "iPython Notebooks are interactive coding environments embedded in a webpage. You will be using iPython notebooks in this class. You only need to write code between the ### START CODE HERE ### and ### END CODE HERE ### comments. After writing your code, you can run the cell by either pressing \"SHIFT\"+\"ENTER\" or by clicking on \"Run Cell\" (denoted by a play symbol) in the upper bar of the notebook. \n", 33 | "\n", 34 | "We will often specify \"(≈ X lines of code)\" in the comments to tell you about how much code you need to write. It is just a rough estimate, so don't feel bad if your code is longer or shorter.\n", 35 | "\n", 36 | "**Exercise**: Set test to `\"Hello World\"` in the cell below to print \"Hello World\" and run the two cells below." 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 1, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "### START CODE HERE ### (≈ 1 line of code)\n", 48 | "test = None\n", 49 | "### END CODE HERE ###" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 2, 55 | "metadata": { 56 | "collapsed": false 57 | }, 58 | "outputs": [ 59 | { 60 | "ename": "TypeError", 61 | "evalue": "Can't convert 'NoneType' object to str implicitly", 62 | "output_type": "error", 63 | "traceback": [ 64 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 65 | "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", 66 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\"test: \"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", 67 | "\u001b[0;31mTypeError\u001b[0m: Can't convert 'NoneType' object to str implicitly" 68 | ] 69 | } 70 | ], 71 | "source": [ 72 | "print (\"test: \" + test)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "**Expected output**:\n", 80 | "test: Hello World" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "\n", 88 | "**What you need to remember**:\n", 89 | "- Run your cells using SHIFT+ENTER (or \"Run cell\")\n", 90 | "- Write code in the designated areas using Python 3 only\n", 91 | "- Do not modify the code outside of the designated areas" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "## 1 - Building basic functions with numpy ##\n", 99 | "\n", 100 | "Numpy is the main package for scientific computing in Python. It is maintained by a large community (www.numpy.org). In this exercise you will learn several key numpy functions such as np.exp, np.log, and np.reshape. You will need to know how to use these functions for future assignments.\n", 101 | "\n", 102 | "### 1.1 - sigmoid function, np.exp() ###\n", 103 | "\n", 104 | "Before using np.exp(), you will use math.exp() to implement the sigmoid function. You will then see why np.exp() is preferable to math.exp().\n", 105 | "\n", 106 | "**Exercise**: Build a function that returns the sigmoid of a real number x. Use math.exp(x) for the exponential function.\n", 107 | "\n", 108 | "**Reminder**:\n", 109 | "$sigmoid(x) = \\frac{1}{1+e^{-x}}$ is sometimes also known as the logistic function. It is a non-linear function used not only in Machine Learning (Logistic Regression), but also in Deep Learning.\n", 110 | "\n", 111 | "\n", 112 | "\n", 113 | "To refer to a function belonging to a specific package you could call it using package_name.function(). Run the code below to see an example with math.exp()." 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": 9, 119 | "metadata": { 120 | "collapsed": true 121 | }, 122 | "outputs": [], 123 | "source": [ 124 | "# GRADED FUNCTION: basic_sigmoid\n", 125 | "\n", 126 | "import math\n", 127 | "import numpy as np\n", 128 | "\n", 129 | "def basic_sigmoid(x):\n", 130 | " \"\"\"\n", 131 | " Compute sigmoid of x.\n", 132 | "\n", 133 | " Arguments:\n", 134 | " x -- A scalar\n", 135 | "\n", 136 | " Return:\n", 137 | " s -- sigmoid(x)\n", 138 | " \"\"\"\n", 139 | " \n", 140 | " ### START CODE HERE ### (≈ 1 line of code)\n", 141 | " s = None\n", 142 | " \n", 143 | " s = 1./(1+np.exp(-x))\n", 144 | " \n", 145 | " ### END CODE HERE ###\n", 146 | " \n", 147 | " return s" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": 10, 153 | "metadata": { 154 | "collapsed": false 155 | }, 156 | "outputs": [ 157 | { 158 | "data": { 159 | "text/plain": [ 160 | "0.95257412682243336" 161 | ] 162 | }, 163 | "execution_count": 10, 164 | "metadata": {}, 165 | "output_type": "execute_result" 166 | } 167 | ], 168 | "source": [ 169 | "basic_sigmoid(3)" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "**Expected Output**: \n", 177 | "\n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | "\n", 183 | "
** basic_sigmoid(3) **0.9525741268224334
" 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": {}, 189 | "source": [ 190 | "Actually, we rarely use the \"math\" library in deep learning because the inputs of the functions are real numbers. In deep learning we mostly use matrices and vectors. This is why numpy is more useful. " 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": 11, 196 | "metadata": { 197 | "collapsed": false 198 | }, 199 | "outputs": [ 200 | { 201 | "data": { 202 | "text/plain": [ 203 | "array([ 0.73105858, 0.88079708, 0.95257413])" 204 | ] 205 | }, 206 | "execution_count": 11, 207 | "metadata": {}, 208 | "output_type": "execute_result" 209 | } 210 | ], 211 | "source": [ 212 | "### One reason why we use \"numpy\" instead of \"math\" in Deep Learning ###\n", 213 | "x = np.array([1, 2, 3])\n", 214 | "basic_sigmoid(x) # you will see this give an error when you run it, because x is a vector." 215 | ] 216 | }, 217 | { 218 | "cell_type": "markdown", 219 | "metadata": {}, 220 | "source": [ 221 | "In fact, if $ x = (x_1, x_2, ..., x_n)$ is a row vector then $np.exp(x)$ will apply the exponential function to every element of x. The output will thus be: $np.exp(x) = (e^{x_1}, e^{x_2}, ..., e^{x_n})$" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": 12, 227 | "metadata": { 228 | "collapsed": false 229 | }, 230 | "outputs": [ 231 | { 232 | "name": "stdout", 233 | "output_type": "stream", 234 | "text": [ 235 | "[ 2.71828183 7.3890561 20.08553692]\n" 236 | ] 237 | } 238 | ], 239 | "source": [ 240 | "import numpy as np\n", 241 | "\n", 242 | "# example of np.exp\n", 243 | "x = np.array([1, 2, 3])\n", 244 | "print(np.exp(x)) # result is (exp(1), exp(2), exp(3))" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "Furthermore, if x is a vector, then a Python operation such as $s = x + 3$ or $s = \\frac{1}{x}$ will output s as a vector of the same size as x." 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "metadata": { 258 | "collapsed": false 259 | }, 260 | "outputs": [], 261 | "source": [ 262 | "# example of vector operation\n", 263 | "x = np.array([1, 2, 3])\n", 264 | "print (x + 3)" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "Any time you need more info on a numpy function, we encourage you to look at [the official documentation](https://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.exp.html). \n", 272 | "\n", 273 | "You can also create a new cell in the notebook and write `np.exp?` (for example) to get quick access to the documentation.\n", 274 | "\n", 275 | "**Exercise**: Implement the sigmoid function using numpy. \n", 276 | "\n", 277 | "**Instructions**: x could now be either a real number, a vector, or a matrix. The data structures we use in numpy to represent these shapes (vectors, matrices...) are called numpy arrays. You don't need to know more for now.\n", 278 | "$$ \\text{For } x \\in \\mathbb{R}^n \\text{, } sigmoid(x) = sigmoid\\begin{pmatrix}\n", 279 | " x_1 \\\\\n", 280 | " x_2 \\\\\n", 281 | " ... \\\\\n", 282 | " x_n \\\\\n", 283 | "\\end{pmatrix} = \\begin{pmatrix}\n", 284 | " \\frac{1}{1+e^{-x_1}} \\\\\n", 285 | " \\frac{1}{1+e^{-x_2}} \\\\\n", 286 | " ... \\\\\n", 287 | " \\frac{1}{1+e^{-x_n}} \\\\\n", 288 | "\\end{pmatrix}\\tag{1} $$" 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": 13, 294 | "metadata": { 295 | "collapsed": false 296 | }, 297 | "outputs": [], 298 | "source": [ 299 | "# GRADED FUNCTION: sigmoid\n", 300 | "\n", 301 | "import numpy as np # this means you can access numpy functions by writing np.function() instead of numpy.function()\n", 302 | "\n", 303 | "def sigmoid(x):\n", 304 | " \"\"\"\n", 305 | " Compute the sigmoid of x\n", 306 | "\n", 307 | " Arguments:\n", 308 | " x -- A scalar or numpy array of any size\n", 309 | "\n", 310 | " Return:\n", 311 | " s -- sigmoid(x)\n", 312 | " \"\"\"\n", 313 | " \n", 314 | " ### START CODE HERE ### (≈ 1 line of code)\n", 315 | " s = None\n", 316 | " s = 1/(1+np.exp(-x))\n", 317 | " ### END CODE HERE ###\n", 318 | " \n", 319 | " return s" 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "execution_count": 14, 325 | "metadata": { 326 | "collapsed": false 327 | }, 328 | "outputs": [ 329 | { 330 | "data": { 331 | "text/plain": [ 332 | "array([ 0.73105858, 0.88079708, 0.95257413])" 333 | ] 334 | }, 335 | "execution_count": 14, 336 | "metadata": {}, 337 | "output_type": "execute_result" 338 | } 339 | ], 340 | "source": [ 341 | "x = np.array([1, 2, 3])\n", 342 | "sigmoid(x)" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "**Expected Output**: \n", 350 | "\n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | "
**sigmoid([1,2,3])** array([ 0.73105858, 0.88079708, 0.95257413])
\n" 356 | ] 357 | }, 358 | { 359 | "cell_type": "markdown", 360 | "metadata": {}, 361 | "source": [ 362 | "### 1.2 - Sigmoid gradient\n", 363 | "\n", 364 | "As you've seen in lecture, you will need to compute gradients to optimize loss functions using backpropagation. Let's code your first gradient function.\n", 365 | "\n", 366 | "**Exercise**: Implement the function sigmoid_grad() to compute the gradient of the sigmoid function with respect to its input x. The formula is: $$sigmoid\\_derivative(x) = \\sigma'(x) = \\sigma(x) (1 - \\sigma(x))\\tag{2}$$\n", 367 | "You often code this function in two steps:\n", 368 | "1. Set s to be the sigmoid of x. You might find your sigmoid(x) function useful.\n", 369 | "2. Compute $\\sigma'(x) = s(1-s)$" 370 | ] 371 | }, 372 | { 373 | "cell_type": "code", 374 | "execution_count": 15, 375 | "metadata": { 376 | "collapsed": false 377 | }, 378 | "outputs": [], 379 | "source": [ 380 | "# GRADED FUNCTION: sigmoid_derivative\n", 381 | "\n", 382 | "def sigmoid_derivative(x):\n", 383 | " \"\"\"\n", 384 | " Compute the gradient (also called the slope or derivative) of the sigmoid function with respect to its input x.\n", 385 | " You can store the output of the sigmoid function into variables and then use it to calculate the gradient.\n", 386 | " \n", 387 | " Arguments:\n", 388 | " x -- A scalar or numpy array\n", 389 | "\n", 390 | " Return:\n", 391 | " ds -- Your computed gradient.\n", 392 | " \"\"\"\n", 393 | " \n", 394 | " ### START CODE HERE ### (≈ 2 lines of code)\n", 395 | " s = None\n", 396 | " ds = None\n", 397 | " s = sigmoid(x)\n", 398 | " ds = s*(1-s)\n", 399 | " ### END CODE HERE ###\n", 400 | " \n", 401 | " return ds" 402 | ] 403 | }, 404 | { 405 | "cell_type": "code", 406 | "execution_count": 16, 407 | "metadata": { 408 | "collapsed": false 409 | }, 410 | "outputs": [ 411 | { 412 | "name": "stdout", 413 | "output_type": "stream", 414 | "text": [ 415 | "sigmoid_derivative(x) = [ 0.19661193 0.10499359 0.04517666]\n" 416 | ] 417 | } 418 | ], 419 | "source": [ 420 | "x = np.array([1, 2, 3])\n", 421 | "print (\"sigmoid_derivative(x) = \" + str(sigmoid_derivative(x)))" 422 | ] 423 | }, 424 | { 425 | "cell_type": "markdown", 426 | "metadata": {}, 427 | "source": [ 428 | "**Expected Output**: \n", 429 | "\n", 430 | "\n", 431 | "\n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | "
**sigmoid_derivative([1,2,3])** [ 0.19661193 0.10499359 0.04517666]
\n", 437 | "\n" 438 | ] 439 | }, 440 | { 441 | "cell_type": "markdown", 442 | "metadata": {}, 443 | "source": [ 444 | "### 1.3 - Reshaping arrays ###\n", 445 | "\n", 446 | "Two common numpy functions used in deep learning are [np.shape](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.shape.html) and [np.reshape()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html). \n", 447 | "- X.shape is used to get the shape (dimension) of a matrix/vector X. \n", 448 | "- X.reshape(...) is used to reshape X into some other dimension. \n", 449 | "\n", 450 | "For example, in computer science, an image is represented by a 3D array of shape $(length, height, depth = 3)$. However, when you read an image as the input of an algorithm you convert it to a vector of shape $(length*height*3, 1)$. In other words, you \"unroll\", or reshape, the 3D array into a 1D vector.\n", 451 | "\n", 452 | "\n", 453 | "\n", 454 | "**Exercise**: Implement `image2vector()` that takes an input of shape (length, height, 3) and returns a vector of shape (length\\*height\\*3, 1). For example, if you would like to reshape an array v of shape (a, b, c) into a vector of shape (a*b,c) you would do:\n", 455 | "``` python\n", 456 | "v = v.reshape((v.shape[0]*v.shape[1], v.shape[2])) # v.shape[0] = a ; v.shape[1] = b ; v.shape[2] = c\n", 457 | "```\n", 458 | "- Please don't hardcode the dimensions of image as a constant. Instead look up the quantities you need with `image.shape[0]`, etc. " 459 | ] 460 | }, 461 | { 462 | "cell_type": "code", 463 | "execution_count": 17, 464 | "metadata": { 465 | "collapsed": false 466 | }, 467 | "outputs": [], 468 | "source": [ 469 | "# GRADED FUNCTION: image2vector\n", 470 | "def image2vector(image):\n", 471 | " \"\"\"\n", 472 | " Argument:\n", 473 | " image -- a numpy array of shape (length, height, depth)\n", 474 | " \n", 475 | " Returns:\n", 476 | " v -- a vector of shape (length*height*depth, 1)\n", 477 | " \"\"\"\n", 478 | " \n", 479 | " ### START CODE HERE ### (≈ 1 line of code)\n", 480 | " v = None\n", 481 | " v = image.reshape((image.shape[0]*image.shape[1]*image.shape[2], 1))\n", 482 | " \n", 483 | " ### END CODE HERE ###\n", 484 | " \n", 485 | " return v" 486 | ] 487 | }, 488 | { 489 | "cell_type": "code", 490 | "execution_count": 18, 491 | "metadata": { 492 | "collapsed": false 493 | }, 494 | "outputs": [ 495 | { 496 | "name": "stdout", 497 | "output_type": "stream", 498 | "text": [ 499 | "image2vector(image) = [[ 0.67826139]\n", 500 | " [ 0.29380381]\n", 501 | " [ 0.90714982]\n", 502 | " [ 0.52835647]\n", 503 | " [ 0.4215251 ]\n", 504 | " [ 0.45017551]\n", 505 | " [ 0.92814219]\n", 506 | " [ 0.96677647]\n", 507 | " [ 0.85304703]\n", 508 | " [ 0.52351845]\n", 509 | " [ 0.19981397]\n", 510 | " [ 0.27417313]\n", 511 | " [ 0.60659855]\n", 512 | " [ 0.00533165]\n", 513 | " [ 0.10820313]\n", 514 | " [ 0.49978937]\n", 515 | " [ 0.34144279]\n", 516 | " [ 0.94630077]]\n" 517 | ] 518 | } 519 | ], 520 | "source": [ 521 | "# This is a 3 by 3 by 2 array, typically images will be (num_px_x, num_px_y,3) where 3 represents the RGB values\n", 522 | "image = np.array([[[ 0.67826139, 0.29380381],\n", 523 | " [ 0.90714982, 0.52835647],\n", 524 | " [ 0.4215251 , 0.45017551]],\n", 525 | "\n", 526 | " [[ 0.92814219, 0.96677647],\n", 527 | " [ 0.85304703, 0.52351845],\n", 528 | " [ 0.19981397, 0.27417313]],\n", 529 | "\n", 530 | " [[ 0.60659855, 0.00533165],\n", 531 | " [ 0.10820313, 0.49978937],\n", 532 | " [ 0.34144279, 0.94630077]]])\n", 533 | "\n", 534 | "print (\"image2vector(image) = \" + str(image2vector(image)))" 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": {}, 540 | "source": [ 541 | "**Expected Output**: \n", 542 | "\n", 543 | "\n", 544 | "\n", 545 | " \n", 546 | " \n", 547 | " \n", 565 | " \n", 566 | " \n", 567 | " \n", 568 | "
**image2vector(image)** [[ 0.67826139]\n", 548 | " [ 0.29380381]\n", 549 | " [ 0.90714982]\n", 550 | " [ 0.52835647]\n", 551 | " [ 0.4215251 ]\n", 552 | " [ 0.45017551]\n", 553 | " [ 0.92814219]\n", 554 | " [ 0.96677647]\n", 555 | " [ 0.85304703]\n", 556 | " [ 0.52351845]\n", 557 | " [ 0.19981397]\n", 558 | " [ 0.27417313]\n", 559 | " [ 0.60659855]\n", 560 | " [ 0.00533165]\n", 561 | " [ 0.10820313]\n", 562 | " [ 0.49978937]\n", 563 | " [ 0.34144279]\n", 564 | " [ 0.94630077]]
" 569 | ] 570 | }, 571 | { 572 | "cell_type": "markdown", 573 | "metadata": {}, 574 | "source": [ 575 | "### 1.4 - Normalizing rows\n", 576 | "\n", 577 | "Another common technique we use in Machine Learning and Deep Learning is to normalize our data. It often leads to a better performance because gradient descent converges faster after normalization. Here, by normalization we mean changing x to $ \\frac{x}{\\| x\\|} $ (dividing each row vector of x by its norm).\n", 578 | "\n", 579 | "For example, if $$x = \n", 580 | "\\begin{bmatrix}\n", 581 | " 0 & 3 & 4 \\\\\n", 582 | " 2 & 6 & 4 \\\\\n", 583 | "\\end{bmatrix}\\tag{3}$$ then $$\\| x\\| = np.linalg.norm(x, axis = 1, keepdims = True) = \\begin{bmatrix}\n", 584 | " 5 \\\\\n", 585 | " 8 \\\\\n", 586 | "\\end{bmatrix}\\tag{4} $$and $$ x\\_normalized = \\frac{x}{\\| x\\|} = \\begin{bmatrix}\n", 587 | " 0 & 3/5 & 4/5 \\\\\n", 588 | " 2/8 & 6/8 & 4/8 \\\\\n", 589 | "\\end{bmatrix}\\tag{5}$$ Note that you can divide matrices of different sizes and it works fine: this is called broadcasting and you're going to learn about it in part 5.\n", 590 | "\n", 591 | "\n", 592 | "**Exercise**: Implement normalizeRows() to normalize the rows of a matrix. After applying this function to an input matrix x, each row of x should be a vector of unit length (meaning length 1)." 593 | ] 594 | }, 595 | { 596 | "cell_type": "code", 597 | "execution_count": 19, 598 | "metadata": { 599 | "collapsed": false 600 | }, 601 | "outputs": [], 602 | "source": [ 603 | "# GRADED FUNCTION: normalizeRows\n", 604 | "\n", 605 | "def normalizeRows(x):\n", 606 | " \"\"\"\n", 607 | " Implement a function that normalizes each row of the matrix x (to have unit length).\n", 608 | " \n", 609 | " Argument:\n", 610 | " x -- A numpy matrix of shape (n, m)\n", 611 | " \n", 612 | " Returns:\n", 613 | " x -- The normalized (by row) numpy matrix. You are allowed to modify x.\n", 614 | " \"\"\"\n", 615 | " \n", 616 | " ### START CODE HERE ### (≈ 2 lines of code)\n", 617 | " # Compute x_norm as the norm 2 of x. Use np.linalg.norm(..., ord = 2, axis = ..., keepdims = True)\n", 618 | " x_norm = None\n", 619 | " x_norm = x / np.linalg.norm(x,axis=1,keepdims=True)\n", 620 | " # Divide x by its norm.\n", 621 | "# x = None\n", 622 | " x = x_norm\n", 623 | " ### END CODE HERE ###\n", 624 | "\n", 625 | " return x" 626 | ] 627 | }, 628 | { 629 | "cell_type": "code", 630 | "execution_count": 20, 631 | "metadata": { 632 | "collapsed": false 633 | }, 634 | "outputs": [ 635 | { 636 | "name": "stdout", 637 | "output_type": "stream", 638 | "text": [ 639 | "normalizeRows(x) = [[ 0. 0.6 0.8 ]\n", 640 | " [ 0.13736056 0.82416338 0.54944226]]\n" 641 | ] 642 | } 643 | ], 644 | "source": [ 645 | "x = np.array([\n", 646 | " [0, 3, 4],\n", 647 | " [1, 6, 4]])\n", 648 | "print(\"normalizeRows(x) = \" + str(normalizeRows(x)))" 649 | ] 650 | }, 651 | { 652 | "cell_type": "markdown", 653 | "metadata": {}, 654 | "source": [ 655 | "**Expected Output**: \n", 656 | "\n", 657 | "\n", 658 | "\n", 659 | " \n", 660 | " \n", 661 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | "
**normalizeRows(x)** [[ 0. 0.6 0.8 ]\n", 662 | " [ 0.13736056 0.82416338 0.54944226]]
" 667 | ] 668 | }, 669 | { 670 | "cell_type": "markdown", 671 | "metadata": {}, 672 | "source": [ 673 | "**Note**:\n", 674 | "In normalizeRows(), you can try to print the shapes of x_norm and x, and then rerun the assessment. You'll find out that they have different shapes. This is normal given that x_norm takes the norm of each row of x. So x_norm has the same number of rows but only 1 column. So how did it work when you divided x by x_norm? This is called broadcasting and we'll talk about it now! " 675 | ] 676 | }, 677 | { 678 | "cell_type": "markdown", 679 | "metadata": {}, 680 | "source": [ 681 | "### 1.5 - Broadcasting and the softmax function ####\n", 682 | "A very important concept to understand in numpy is \"broadcasting\". It is very useful for performing mathematical operations between arrays of different shapes. For the full details on broadcasting, you can read the official [broadcasting documentation](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)." 683 | ] 684 | }, 685 | { 686 | "cell_type": "markdown", 687 | "metadata": {}, 688 | "source": [ 689 | "**Exercise**: Implement a softmax function using numpy. You can think of softmax as a normalizing function used when your algorithm needs to classify two or more classes. You will learn more about softmax in the second course of this specialization.\n", 690 | "\n", 691 | "**Instructions**:\n", 692 | "- $ \\text{for } x \\in \\mathbb{R}^{1\\times n} \\text{, } softmax(x) = softmax(\\begin{bmatrix}\n", 693 | " x_1 &&\n", 694 | " x_2 &&\n", 695 | " ... &&\n", 696 | " x_n \n", 697 | "\\end{bmatrix}) = \\begin{bmatrix}\n", 698 | " \\frac{e^{x_1}}{\\sum_{j}e^{x_j}} &&\n", 699 | " \\frac{e^{x_2}}{\\sum_{j}e^{x_j}} &&\n", 700 | " ... &&\n", 701 | " \\frac{e^{x_n}}{\\sum_{j}e^{x_j}} \n", 702 | "\\end{bmatrix} $ \n", 703 | "\n", 704 | "- $\\text{for a matrix } x \\in \\mathbb{R}^{m \\times n} \\text{, $x_{ij}$ maps to the element in the $i^{th}$ row and $j^{th}$ column of $x$, thus we have: }$ $$softmax(x) = softmax\\begin{bmatrix}\n", 705 | " x_{11} & x_{12} & x_{13} & \\dots & x_{1n} \\\\\n", 706 | " x_{21} & x_{22} & x_{23} & \\dots & x_{2n} \\\\\n", 707 | " \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", 708 | " x_{m1} & x_{m2} & x_{m3} & \\dots & x_{mn}\n", 709 | "\\end{bmatrix} = \\begin{bmatrix}\n", 710 | " \\frac{e^{x_{11}}}{\\sum_{j}e^{x_{1j}}} & \\frac{e^{x_{12}}}{\\sum_{j}e^{x_{1j}}} & \\frac{e^{x_{13}}}{\\sum_{j}e^{x_{1j}}} & \\dots & \\frac{e^{x_{1n}}}{\\sum_{j}e^{x_{1j}}} \\\\\n", 711 | " \\frac{e^{x_{21}}}{\\sum_{j}e^{x_{2j}}} & \\frac{e^{x_{22}}}{\\sum_{j}e^{x_{2j}}} & \\frac{e^{x_{23}}}{\\sum_{j}e^{x_{2j}}} & \\dots & \\frac{e^{x_{2n}}}{\\sum_{j}e^{x_{2j}}} \\\\\n", 712 | " \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", 713 | " \\frac{e^{x_{m1}}}{\\sum_{j}e^{x_{mj}}} & \\frac{e^{x_{m2}}}{\\sum_{j}e^{x_{mj}}} & \\frac{e^{x_{m3}}}{\\sum_{j}e^{x_{mj}}} & \\dots & \\frac{e^{x_{mn}}}{\\sum_{j}e^{x_{mj}}}\n", 714 | "\\end{bmatrix} = \\begin{pmatrix}\n", 715 | " softmax\\text{(first row of x)} \\\\\n", 716 | " softmax\\text{(second row of x)} \\\\\n", 717 | " ... \\\\\n", 718 | " softmax\\text{(last row of x)} \\\\\n", 719 | "\\end{pmatrix} $$" 720 | ] 721 | }, 722 | { 723 | "cell_type": "code", 724 | "execution_count": 32, 725 | "metadata": { 726 | "collapsed": false 727 | }, 728 | "outputs": [], 729 | "source": [ 730 | "# GRADED FUNCTION: softmax\n", 731 | "\n", 732 | "def softmax(x):\n", 733 | " \"\"\"Calculates the softmax for each row of the input x.\n", 734 | "\n", 735 | " Your code should work for a row vector and also for matrices of shape (n, m).\n", 736 | "\n", 737 | " Argument:\n", 738 | " x -- A numpy matrix of shape (n,m)\n", 739 | "\n", 740 | " Returns:\n", 741 | " s -- A numpy matrix equal to the softmax of x, of shape (n,m)\n", 742 | " \"\"\"\n", 743 | " \n", 744 | " ### START CODE HERE ### (≈ 3 lines of code)\n", 745 | " # Apply exp() element-wise to x. Use np.exp(...).\n", 746 | " x_exp = None\n", 747 | " x_exp = np.exp(x)\n", 748 | " # Create a vector x_sum that sums each row of x_exp. Use np.sum(..., axis = 1, keepdims = True).\n", 749 | " x_sum = None\n", 750 | " x_sum = np.sum(x_exp, axis = 1, keepdims = True)\n", 751 | " # Compute softmax(x) by dividing x_exp by x_sum. It should automatically use numpy broadcasting.\n", 752 | " s = None\n", 753 | " s = x_exp / x_sum\n", 754 | " ### END CODE HERE ###\n", 755 | " \n", 756 | " return s" 757 | ] 758 | }, 759 | { 760 | "cell_type": "code", 761 | "execution_count": 33, 762 | "metadata": { 763 | "collapsed": false 764 | }, 765 | "outputs": [ 766 | { 767 | "name": "stdout", 768 | "output_type": "stream", 769 | "text": [ 770 | "softmax(x) = [[ 9.80897665e-01 8.94462891e-04 1.79657674e-02 1.21052389e-04\n", 771 | " 1.21052389e-04]\n", 772 | " [ 8.78679856e-01 1.18916387e-01 8.01252314e-04 8.01252314e-04\n", 773 | " 8.01252314e-04]]\n" 774 | ] 775 | } 776 | ], 777 | "source": [ 778 | "x = np.array([\n", 779 | " [9, 2, 5, 0, 0],\n", 780 | " [7, 5, 0, 0 ,0]])\n", 781 | "print(\"softmax(x) = \" + str(softmax(x)))" 782 | ] 783 | }, 784 | { 785 | "cell_type": "markdown", 786 | "metadata": {}, 787 | "source": [ 788 | "**Expected Output**:\n", 789 | "\n", 790 | "\n", 791 | "\n", 792 | " \n", 793 | " \n", 794 | " \n", 798 | " \n", 799 | "
**softmax(x)** [[ 9.80897665e-01 8.94462891e-04 1.79657674e-02 1.21052389e-04\n", 795 | " 1.21052389e-04]\n", 796 | " [ 8.78679856e-01 1.18916387e-01 8.01252314e-04 8.01252314e-04\n", 797 | " 8.01252314e-04]]
\n" 800 | ] 801 | }, 802 | { 803 | "cell_type": "markdown", 804 | "metadata": {}, 805 | "source": [ 806 | "**Note**:\n", 807 | "- If you print the shapes of x_exp, x_sum and s above and rerun the assessment cell, you will see that x_sum is of shape (2,1) while x_exp and s are of shape (2,5). **x_exp/x_sum** works due to python broadcasting.\n", 808 | "\n", 809 | "Congratulations! You now have a pretty good understanding of python numpy and have implemented a few useful functions that you will be using in deep learning." 810 | ] 811 | }, 812 | { 813 | "cell_type": "markdown", 814 | "metadata": {}, 815 | "source": [ 816 | "\n", 817 | "**What you need to remember:**\n", 818 | "- np.exp(x) works for any np.array x and applies the exponential function to every coordinate\n", 819 | "- the sigmoid function and its gradient\n", 820 | "- image2vector is commonly used in deep learning\n", 821 | "- np.reshape is widely used. In the future, you'll see that keeping your matrix/vector dimensions straight will go toward eliminating a lot of bugs. \n", 822 | "- numpy has efficient built-in functions\n", 823 | "- broadcasting is extremely useful" 824 | ] 825 | }, 826 | { 827 | "cell_type": "markdown", 828 | "metadata": { 829 | "collapsed": true 830 | }, 831 | "source": [ 832 | "## 2) Vectorization" 833 | ] 834 | }, 835 | { 836 | "cell_type": "markdown", 837 | "metadata": {}, 838 | "source": [ 839 | "\n", 840 | "In deep learning, you deal with very large datasets. Hence, a non-computationally-optimal function can become a huge bottleneck in your algorithm and can result in a model that takes ages to run. To make sure that your code is computationally efficient, you will use vectorization. For example, try to tell the difference between the following implementations of the dot/outer/elementwise product." 841 | ] 842 | }, 843 | { 844 | "cell_type": "code", 845 | "execution_count": 34, 846 | "metadata": { 847 | "collapsed": false 848 | }, 849 | "outputs": [ 850 | { 851 | "name": "stdout", 852 | "output_type": "stream", 853 | "text": [ 854 | "dot = 278\n", 855 | " ----- Computation time = 0.16770200000015834ms\n", 856 | "outer = [[ 81. 18. 18. 81. 0. 81. 18. 45. 0. 0. 81. 18. 45. 0.\n", 857 | " 0.]\n", 858 | " [ 18. 4. 4. 18. 0. 18. 4. 10. 0. 0. 18. 4. 10. 0.\n", 859 | " 0.]\n", 860 | " [ 45. 10. 10. 45. 0. 45. 10. 25. 0. 0. 45. 10. 25. 0.\n", 861 | " 0.]\n", 862 | " [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 863 | " 0.]\n", 864 | " [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 865 | " 0.]\n", 866 | " [ 63. 14. 14. 63. 0. 63. 14. 35. 0. 0. 63. 14. 35. 0.\n", 867 | " 0.]\n", 868 | " [ 45. 10. 10. 45. 0. 45. 10. 25. 0. 0. 45. 10. 25. 0.\n", 869 | " 0.]\n", 870 | " [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 871 | " 0.]\n", 872 | " [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 873 | " 0.]\n", 874 | " [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 875 | " 0.]\n", 876 | " [ 81. 18. 18. 81. 0. 81. 18. 45. 0. 0. 81. 18. 45. 0.\n", 877 | " 0.]\n", 878 | " [ 18. 4. 4. 18. 0. 18. 4. 10. 0. 0. 18. 4. 10. 0.\n", 879 | " 0.]\n", 880 | " [ 45. 10. 10. 45. 0. 45. 10. 25. 0. 0. 45. 10. 25. 0.\n", 881 | " 0.]\n", 882 | " [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 883 | " 0.]\n", 884 | " [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n", 885 | " 0.]]\n", 886 | " ----- Computation time = 0.33875799999982803ms\n", 887 | "elementwise multiplication = [ 81. 4. 10. 0. 0. 63. 10. 0. 0. 0. 81. 4. 25. 0. 0.]\n", 888 | " ----- Computation time = 0.17767200000018413ms\n", 889 | "gdot = [ 25.25047968 36.53278411 24.65618131]\n", 890 | " ----- Computation time = 0.2817469999998323ms\n" 891 | ] 892 | } 893 | ], 894 | "source": [ 895 | "import time\n", 896 | "\n", 897 | "x1 = [9, 2, 5, 0, 0, 7, 5, 0, 0, 0, 9, 2, 5, 0, 0]\n", 898 | "x2 = [9, 2, 2, 9, 0, 9, 2, 5, 0, 0, 9, 2, 5, 0, 0]\n", 899 | "\n", 900 | "### CLASSIC DOT PRODUCT OF VECTORS IMPLEMENTATION ###\n", 901 | "tic = time.process_time()\n", 902 | "dot = 0\n", 903 | "for i in range(len(x1)):\n", 904 | " dot+= x1[i]*x2[i]\n", 905 | "toc = time.process_time()\n", 906 | "print (\"dot = \" + str(dot) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")\n", 907 | "\n", 908 | "### CLASSIC OUTER PRODUCT IMPLEMENTATION ###\n", 909 | "tic = time.process_time()\n", 910 | "outer = np.zeros((len(x1),len(x2))) # we create a len(x1)*len(x2) matrix with only zeros\n", 911 | "for i in range(len(x1)):\n", 912 | " for j in range(len(x2)):\n", 913 | " outer[i,j] = x1[i]*x2[j]\n", 914 | "toc = time.process_time()\n", 915 | "print (\"outer = \" + str(outer) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")\n", 916 | "\n", 917 | "### CLASSIC ELEMENTWISE IMPLEMENTATION ###\n", 918 | "tic = time.process_time()\n", 919 | "mul = np.zeros(len(x1))\n", 920 | "for i in range(len(x1)):\n", 921 | " mul[i] = x1[i]*x2[i]\n", 922 | "toc = time.process_time()\n", 923 | "print (\"elementwise multiplication = \" + str(mul) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")\n", 924 | "\n", 925 | "### CLASSIC GENERAL DOT PRODUCT IMPLEMENTATION ###\n", 926 | "W = np.random.rand(3,len(x1)) # Random 3*len(x1) numpy array\n", 927 | "tic = time.process_time()\n", 928 | "gdot = np.zeros(W.shape[0])\n", 929 | "for i in range(W.shape[0]):\n", 930 | " for j in range(len(x1)):\n", 931 | " gdot[i] += W[i,j]*x1[j]\n", 932 | "toc = time.process_time()\n", 933 | "print (\"gdot = \" + str(gdot) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")" 934 | ] 935 | }, 936 | { 937 | "cell_type": "code", 938 | "execution_count": 35, 939 | "metadata": { 940 | "collapsed": false 941 | }, 942 | "outputs": [ 943 | { 944 | "name": "stdout", 945 | "output_type": "stream", 946 | "text": [ 947 | "dot = 278\n", 948 | " ----- Computation time = 0.1830069999999573ms\n", 949 | "outer = [[81 18 18 81 0 81 18 45 0 0 81 18 45 0 0]\n", 950 | " [18 4 4 18 0 18 4 10 0 0 18 4 10 0 0]\n", 951 | " [45 10 10 45 0 45 10 25 0 0 45 10 25 0 0]\n", 952 | " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", 953 | " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", 954 | " [63 14 14 63 0 63 14 35 0 0 63 14 35 0 0]\n", 955 | " [45 10 10 45 0 45 10 25 0 0 45 10 25 0 0]\n", 956 | " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", 957 | " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", 958 | " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", 959 | " [81 18 18 81 0 81 18 45 0 0 81 18 45 0 0]\n", 960 | " [18 4 4 18 0 18 4 10 0 0 18 4 10 0 0]\n", 961 | " [45 10 10 45 0 45 10 25 0 0 45 10 25 0 0]\n", 962 | " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", 963 | " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]\n", 964 | " ----- Computation time = 0.14421799999997376ms\n", 965 | "elementwise multiplication = [81 4 10 0 0 63 10 0 0 0 81 4 25 0 0]\n", 966 | " ----- Computation time = 0.11799799999989702ms\n", 967 | "gdot = [ 25.25047968 36.53278411 24.65618131]\n", 968 | " ----- Computation time = 0.1730219999998006ms\n" 969 | ] 970 | } 971 | ], 972 | "source": [ 973 | "x1 = [9, 2, 5, 0, 0, 7, 5, 0, 0, 0, 9, 2, 5, 0, 0]\n", 974 | "x2 = [9, 2, 2, 9, 0, 9, 2, 5, 0, 0, 9, 2, 5, 0, 0]\n", 975 | "\n", 976 | "### VECTORIZED DOT PRODUCT OF VECTORS ###\n", 977 | "tic = time.process_time()\n", 978 | "dot = np.dot(x1,x2)\n", 979 | "toc = time.process_time()\n", 980 | "print (\"dot = \" + str(dot) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")\n", 981 | "\n", 982 | "### VECTORIZED OUTER PRODUCT ###\n", 983 | "tic = time.process_time()\n", 984 | "outer = np.outer(x1,x2)\n", 985 | "toc = time.process_time()\n", 986 | "print (\"outer = \" + str(outer) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")\n", 987 | "\n", 988 | "### VECTORIZED ELEMENTWISE MULTIPLICATION ###\n", 989 | "tic = time.process_time()\n", 990 | "mul = np.multiply(x1,x2)\n", 991 | "toc = time.process_time()\n", 992 | "print (\"elementwise multiplication = \" + str(mul) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")\n", 993 | "\n", 994 | "### VECTORIZED GENERAL DOT PRODUCT ###\n", 995 | "tic = time.process_time()\n", 996 | "dot = np.dot(W,x1)\n", 997 | "toc = time.process_time()\n", 998 | "print (\"gdot = \" + str(dot) + \"\\n ----- Computation time = \" + str(1000*(toc - tic)) + \"ms\")" 999 | ] 1000 | }, 1001 | { 1002 | "cell_type": "markdown", 1003 | "metadata": {}, 1004 | "source": [ 1005 | "As you may have noticed, the vectorized implementation is much cleaner and more efficient. For bigger vectors/matrices, the differences in running time become even bigger. \n", 1006 | "\n", 1007 | "**Note** that `np.dot()` performs a matrix-matrix or matrix-vector multiplication. This is different from `np.multiply()` and the `*` operator (which is equivalent to `.*` in Matlab/Octave), which performs an element-wise multiplication." 1008 | ] 1009 | }, 1010 | { 1011 | "cell_type": "markdown", 1012 | "metadata": {}, 1013 | "source": [ 1014 | "### 2.1 Implement the L1 and L2 loss functions\n", 1015 | "\n", 1016 | "**Exercise**: Implement the numpy vectorized version of the L1 loss. You may find the function abs(x) (absolute value of x) useful.\n", 1017 | "\n", 1018 | "**Reminder**:\n", 1019 | "- The loss is used to evaluate the performance of your model. The bigger your loss is, the more different your predictions ($ \\hat{y} $) are from the true values ($y$). In deep learning, you use optimization algorithms like Gradient Descent to train your model and to minimize the cost.\n", 1020 | "- L1 loss is defined as:\n", 1021 | "$$\\begin{align*} & L_1(\\hat{y}, y) = \\sum_{i=0}^m|y^{(i)} - \\hat{y}^{(i)}| \\end{align*}\\tag{6}$$" 1022 | ] 1023 | }, 1024 | { 1025 | "cell_type": "code", 1026 | "execution_count": 36, 1027 | "metadata": { 1028 | "collapsed": false 1029 | }, 1030 | "outputs": [], 1031 | "source": [ 1032 | "# GRADED FUNCTION: L1\n", 1033 | "\n", 1034 | "def L1(yhat, y):\n", 1035 | " \"\"\"\n", 1036 | " Arguments:\n", 1037 | " yhat -- vector of size m (predicted labels)\n", 1038 | " y -- vector of size m (true labels)\n", 1039 | " \n", 1040 | " Returns:\n", 1041 | " loss -- the value of the L1 loss function defined above\n", 1042 | " \"\"\"\n", 1043 | " \n", 1044 | " ### START CODE HERE ### (≈ 1 line of code)\n", 1045 | " loss = None\n", 1046 | " loss = sum(abs(yhat - y))\n", 1047 | " ### END CODE HERE ###\n", 1048 | " \n", 1049 | " return loss" 1050 | ] 1051 | }, 1052 | { 1053 | "cell_type": "code", 1054 | "execution_count": 37, 1055 | "metadata": { 1056 | "collapsed": false 1057 | }, 1058 | "outputs": [ 1059 | { 1060 | "name": "stdout", 1061 | "output_type": "stream", 1062 | "text": [ 1063 | "L1 = 1.1\n" 1064 | ] 1065 | } 1066 | ], 1067 | "source": [ 1068 | "yhat = np.array([.9, 0.2, 0.1, .4, .9])\n", 1069 | "y = np.array([1, 0, 0, 1, 1])\n", 1070 | "print(\"L1 = \" + str(L1(yhat,y)))" 1071 | ] 1072 | }, 1073 | { 1074 | "cell_type": "markdown", 1075 | "metadata": {}, 1076 | "source": [ 1077 | "**Expected Output**:\n", 1078 | "\n", 1079 | "\n", 1080 | "\n", 1081 | " \n", 1082 | " \n", 1083 | " \n", 1084 | " \n", 1085 | "
**L1** 1.1
\n" 1086 | ] 1087 | }, 1088 | { 1089 | "cell_type": "markdown", 1090 | "metadata": {}, 1091 | "source": [ 1092 | "**Exercise**: Implement the numpy vectorized version of the L2 loss. There are several way of implementing the L2 loss but you may find the function np.dot() useful. As a reminder, if $x = [x_1, x_2, ..., x_n]$, then `np.dot(x,x)` = $\\sum_{j=0}^n x_j^{2}$. \n", 1093 | "\n", 1094 | "- L2 loss is defined as $$\\begin{align*} & L_2(\\hat{y},y) = \\sum_{i=0}^m(y^{(i)} - \\hat{y}^{(i)})^2 \\end{align*}\\tag{7}$$" 1095 | ] 1096 | }, 1097 | { 1098 | "cell_type": "code", 1099 | "execution_count": 40, 1100 | "metadata": { 1101 | "collapsed": false 1102 | }, 1103 | "outputs": [], 1104 | "source": [ 1105 | "# GRADED FUNCTION: L2\n", 1106 | "\n", 1107 | "def L2(yhat, y):\n", 1108 | " \"\"\"\n", 1109 | " Arguments:\n", 1110 | " yhat -- vector of size m (predicted labels)\n", 1111 | " y -- vector of size m (true labels)\n", 1112 | " \n", 1113 | " Returns:\n", 1114 | " loss -- the value of the L2 loss function defined above\n", 1115 | " \"\"\"\n", 1116 | " \n", 1117 | " ### START CODE HERE ### (≈ 1 line of code)\n", 1118 | " loss = None\n", 1119 | " loss = np.dot(yhat - y, yhat - y)\n", 1120 | " ### END CODE HERE ###\n", 1121 | " \n", 1122 | " return loss" 1123 | ] 1124 | }, 1125 | { 1126 | "cell_type": "code", 1127 | "execution_count": 41, 1128 | "metadata": { 1129 | "collapsed": false 1130 | }, 1131 | "outputs": [ 1132 | { 1133 | "name": "stdout", 1134 | "output_type": "stream", 1135 | "text": [ 1136 | "L2 = 0.43\n" 1137 | ] 1138 | } 1139 | ], 1140 | "source": [ 1141 | "yhat = np.array([.9, 0.2, 0.1, .4, .9])\n", 1142 | "y = np.array([1, 0, 0, 1, 1])\n", 1143 | "print(\"L2 = \" + str(L2(yhat,y)))" 1144 | ] 1145 | }, 1146 | { 1147 | "cell_type": "markdown", 1148 | "metadata": {}, 1149 | "source": [ 1150 | "**Expected Output**: \n", 1151 | "\n", 1152 | " \n", 1153 | " \n", 1154 | " \n", 1155 | " \n", 1156 | "
**L2** 0.43
" 1157 | ] 1158 | }, 1159 | { 1160 | "cell_type": "markdown", 1161 | "metadata": {}, 1162 | "source": [ 1163 | "Congratulations on completing this assignment. We hope that this little warm-up exercise helps you in the future assignments, which will be more exciting and interesting!" 1164 | ] 1165 | }, 1166 | { 1167 | "cell_type": "markdown", 1168 | "metadata": {}, 1169 | "source": [ 1170 | "\n", 1171 | "**What to remember:**\n", 1172 | "- Vectorization is very important in deep learning. It provides computational efficiency and clarity.\n", 1173 | "- You have reviewed the L1 and L2 loss.\n", 1174 | "- You are familiar with many numpy functions such as np.sum, np.dot, np.multiply, np.maximum, etc..." 1175 | ] 1176 | } 1177 | ], 1178 | "metadata": { 1179 | "coursera": { 1180 | "course_slug": "neural-networks-deep-learning", 1181 | "graded_item_id": "XHpfv", 1182 | "launcher_item_id": "Zh0CU" 1183 | }, 1184 | "kernelspec": { 1185 | "display_name": "Python 3", 1186 | "language": "python", 1187 | "name": "python3" 1188 | }, 1189 | "language_info": { 1190 | "codemirror_mode": { 1191 | "name": "ipython", 1192 | "version": 3 1193 | }, 1194 | "file_extension": ".py", 1195 | "mimetype": "text/x-python", 1196 | "name": "python", 1197 | "nbconvert_exporter": "python", 1198 | "pygments_lexer": "ipython3", 1199 | "version": "3.5.2" 1200 | } 1201 | }, 1202 | "nbformat": 4, 1203 | "nbformat_minor": 2 1204 | } 1205 | -------------------------------------------------------------------------------- /Neural Networks and Deep Learning/Week 3 Programming Assignment/test: -------------------------------------------------------------------------------- 1 | sad 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # coursera-Deep-Learning-Specialization 2 | 3 | Updated Part 4: Convolutional Neural Networks's programme assignments 4 | 5 | I will upload the slides and assignments gradually, and if you like this repository, please follow me and give me a star :) (slides are not available right now in the course page). The third course Structuring Machine Learning Projects focuses on the systematic method to improve the performance of Neural Network, while with no programming assignments. 6 | 7 | This course is created by deeplearning.ai. Master Deep Learning, and Break into AI 8 | 9 | https://www.coursera.org/specializations/deep-learning 10 | 11 | cited from the homepage: 12 | 13 | If you want to break into AI, this Specialization will help you do so. Deep Learning is one of the most highly sought after skills in tech. We will help you become good at Deep Learning. 14 | 15 | In five courses, you will learn the foundations of Deep Learning, understand how to build neural networks, and learn how to lead successful machine learning projects. You will learn about Convolutional networks, RNNs, LSTM, Adam, Dropout, BatchNorm, Xavier/He initialization, and more. You will work on case studies from healthcare, autonomous driving, sign language reading, music generation, and natural language processing. You will master not only the theory, but also see how it is applied in industry. You will practice all these ideas in Python and in TensorFlow, which we will teach. 16 | 17 | You will also hear from many top leaders in Deep Learning, who will share with you their personal stories and give you career advice. 18 | 19 | AI is transforming multiple industries. After finishing this specialization, you will likely find creative ways to apply it to your work. 20 | 21 | We will help you master Deep Learning, understand how to apply it, and build a career in AI. 22 | -------------------------------------------------------------------------------- /Structuring Machine Learning Projects_Certificate.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Qian-Han/coursera-Deep-Learning-Specialization/86cc8001afa4329d5389620c0e619ffb5d975147/Structuring Machine Learning Projects_Certificate.pdf --------------------------------------------------------------------------------