├── 1. Neural Networks and Deep Learning ├── Building your Deep Neural Network - Step by Step v5.ipynb ├── Deep Neural Network - Application v3.ipynb ├── Logistic Regression with a Neural Network mindset v3.ipynb ├── Planar data classification with one hidden layer v4.ipynb ├── Python Basics With Numpy v3.ipynb ├── README.md └── images │ ├── 2layerNN.png │ ├── 2layerNN_kiank.png │ ├── GAC.jpeg │ ├── GYZ_066.jpeg │ ├── GYZ_067.jpeg │ ├── LlayerNN_kiank.png │ ├── LogReg_kiank.png │ ├── NlayerNN.png │ ├── Sigmoid.png │ ├── backpass.png │ ├── backprop.png │ ├── backprop_kiank.png │ ├── cat_in_iran.jpg │ ├── classification_kiank.png │ ├── final outline.png │ ├── final_outline.png │ ├── gargouille.jpg │ ├── grad_summary.png │ ├── image1.png │ ├── image2.png │ ├── image2vector.png │ ├── image2vector_kiank.png │ ├── imvector.png │ ├── imvectorkiank.png │ ├── la_defense.jpg │ ├── linearback_kiank.png │ ├── mn_backward.png │ ├── model_architecture2.png │ ├── model_architecture_kiank.png │ ├── my_image.jpg │ ├── my_image2.jpg │ ├── n_model_backward.png │ ├── nm_backward.png │ ├── relu.png │ ├── sgd.gif │ ├── sgd_bad.gif │ └── structure.png ├── 2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization ├── Gradient Checking v1.ipynb ├── Initialization.ipynb ├── Optimization methods.ipynb ├── README.md ├── Regularization.ipynb ├── Tensorflow Tutorial.ipynb └── images │ ├── 1Dgrad_kiank.png │ ├── Momentum.png │ ├── NDgrad_kiank.png │ ├── cost.jpg │ ├── dictionary_to_vector.png │ ├── dropout1_kiank.mp4 │ ├── dropout2_kiank.mp4 │ ├── field_kiank.png │ ├── handbackward_kiank.png │ ├── handforward_kiank.png │ ├── hands.png │ ├── kiank_minibatch.png │ ├── kiank_partition.png │ ├── kiank_sgd.png │ ├── kiank_shuffle.png │ ├── onehot.png │ ├── opt1.gif │ ├── opt2.gif │ ├── opt_momentum.png │ └── thumbs_up.jpg ├── 3. Structuring Machine Learning Projects ├── README.md ├── Week 1 Quiz - Bird recognition in the city of Peacetopia (case study).md └── Week 2 Quiz - Autonomous driving (case study).md ├── 4. Convolutional Neural Networks ├── Art Generation with Neural Style Transfer - v2.ipynb ├── Autonomous driving application - Car detection - v1.ipynb ├── Convolution model - Application - v1.ipynb ├── Convolution model - Step by Step - v2.ipynb ├── Face Recognition for the Happy House - v3.ipynb ├── Keras - Tutorial - Happy House v2.ipynb ├── README.md ├── Residual Networks - v2.ipynb ├── images │ ├── Convolution_schematic.gif │ ├── GYZ_066.jpeg │ ├── GYZ_067.jpeg │ ├── NST_GM.png │ ├── NST_LOSS.png │ ├── PAD.png │ ├── SIGNS.png │ ├── a_pool.png │ ├── andrew.jpg │ ├── arnaud.jpg │ ├── ave-pool.png │ ├── ave_pool1.png │ ├── average_pool.png │ ├── benoit.jpg │ ├── bertrand.jpg │ ├── camera_0.jpg │ ├── camera_1.jpg │ ├── camera_2.jpg │ ├── camera_3.jpg │ ├── camera_4.jpg │ ├── camera_5.jpg │ ├── camp-nou.jpg │ ├── cat.jpg │ ├── circle_abstract.png │ ├── claude-monet.jpg │ ├── content.jpeg │ ├── content300.jpg │ ├── content_plus_style.png │ ├── conv.png │ ├── conv1.png │ ├── conv_kiank.mp4 │ ├── conv_nn.png │ ├── convblock_kiank.png │ ├── dan.jpg │ ├── danielle.png │ ├── distance_kiank.png │ ├── distance_matrix.png │ ├── drop-of-water.jpg │ ├── f_x.png │ ├── felix.jpg │ ├── gram.png │ ├── happy-house.jpg │ ├── hidden_layers.png │ ├── house-members.png │ ├── idblock2_kiank.png │ ├── idblock3_kiank.png │ ├── inception_block1a.png │ ├── kevin.jpg │ ├── kian.jpg │ ├── louvre.jpg │ ├── louvre_generated.png │ ├── louvre_small.jpg │ ├── max_pool.png │ ├── max_pool1.png │ ├── model.png │ ├── monet.jpg │ ├── monet_800600.jpg │ ├── my_image.jpg │ ├── pasargad_kashi.png │ ├── persian_cat.jpg │ ├── persian_cat_content.jpg │ ├── perspolis_vangogh.png │ ├── pixel_comparison.png │ ├── reshape_loss.png │ ├── resnet_kiank.png │ ├── result.png │ ├── sandstone.jpg │ ├── sebastiano.jpg │ ├── signs_data_kiank.png │ ├── skip_connection_kiank.png │ ├── stone_style.jpg │ ├── style300.jpg │ ├── thumbs_up.jpg │ ├── tian.jpg │ ├── triplet_comparison.png │ ├── vanishing_grad_kiank.png │ ├── vert_horiz_kiank.png │ └── younes.jpg └── nb_images │ ├── anchor_map.png │ ├── architecture.png │ ├── box_label.png │ ├── driveai.png │ ├── flatten.png │ ├── index.md │ ├── iou.png │ ├── non-max-suppression.png │ ├── pred_video_compressed2.mp4 │ ├── proba_map.png │ ├── probability_extraction.png │ └── road_video_compressed2.mp4 ├── 5. Sequence Models ├── Building a Recurrent Neural Network - Step by Step - v3.ipynb ├── Dinosaurus Island -- Character level language model final - v3.ipynb ├── Emojify+-+v2.ipynb ├── Improvise a Jazz Solo with an LSTM Network - v3.ipynb ├── Neural machine translation with attention - v4.ipynb ├── Operations on word vectors - v2.ipynb ├── README.md ├── Trigger word detection - v1.ipynb └── images │ ├── 1-hot-vector.png │ ├── LSTM.png │ ├── LSTM_rnn.png │ ├── RNN.png │ ├── attn_mechanism.png │ ├── attn_model.png │ ├── clip.png │ ├── cosine_sim.png │ ├── data_set.png │ ├── dataset_kiank.png │ ├── date_attention.png │ ├── date_attention2.png │ ├── dino.jpg │ ├── dinos3.png │ ├── emb_kiank.png │ ├── embedding1.png │ ├── emo_model.png │ ├── emojifier-v2.png │ ├── emojifierv1.png │ ├── emojify_model.png │ ├── emojiss.png │ ├── equalize10.png │ ├── image_1.png │ ├── initial_state.png │ ├── jazz.jpg │ ├── label_diagram.png │ ├── lookup.png │ ├── mangosaurus.jpeg │ ├── model.png │ ├── music_gen.png │ ├── music_generation.png │ ├── neutral.png │ ├── neutralize.png │ ├── neutralize_kiank.png │ ├── ones_reference.png │ ├── poorly_trained_model.png │ ├── rnn.png │ ├── rnn_cell_backprop.png │ ├── rnn_minusculas.png │ ├── rnn_step_forward.png │ ├── shakespeare.jpg │ ├── sound.png │ ├── spectrogram.png │ ├── table.png │ ├── tmp │ ├── train_label.png │ ├── train_reference.png │ └── woebot.png ├── LICENSE ├── README.md └── prediction_video_compressed.mp4 /1. Neural Networks and Deep Learning/README.md: -------------------------------------------------------------------------------- 1 | ## Objectives: 2 | - Be able to explain the major trends driving the rise of deep learning, and understand where and how it is applied today. 3 | - Learn to set up a machine learning problem with a neural network mindset. Learn to use vectorization to speed up your models. 4 | - Learn to build a neural network with one hidden layer, using forward propagation and backpropagation. 5 | - Understand the key computations underlying deep learning, use them to build and train deep neural networks, and apply it to computer vision. 6 | 7 | ## Week 2: 8 | - Python Basics With Numpy v3.ipynb 9 | - Logistic Regression with a Neural Network mindset v3.ipynb 10 | 11 | ## Week 3: 12 | - Planar data classification with one hidden layer v4.ipynb 13 | 14 | ## Week 4: 15 | - Building your Deep Neural Network - Step by Step v5.ipynb 16 | - Deep Neural Network - Application v3.ipynb 17 | 18 | Every image required to view correctly the Jupyter Notebooks are stored in `images` folder. 19 | 20 | 21 | -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/2layerNN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/2layerNN.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/2layerNN_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/2layerNN_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/GAC.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/GAC.jpeg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/GYZ_066.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/GYZ_066.jpeg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/GYZ_067.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/GYZ_067.jpeg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/LlayerNN_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/LlayerNN_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/LogReg_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/LogReg_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/NlayerNN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/NlayerNN.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/Sigmoid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/Sigmoid.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/backpass.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/backpass.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/backprop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/backprop.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/backprop_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/backprop_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/cat_in_iran.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/cat_in_iran.jpg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/classification_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/classification_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/final outline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/final outline.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/final_outline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/final_outline.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/gargouille.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/gargouille.jpg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/grad_summary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/grad_summary.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/image1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/image1.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/image2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/image2.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/image2vector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/image2vector.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/image2vector_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/image2vector_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/imvector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/imvector.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/imvectorkiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/imvectorkiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/la_defense.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/la_defense.jpg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/linearback_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/linearback_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/mn_backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/mn_backward.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/model_architecture2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/model_architecture2.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/model_architecture_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/model_architecture_kiank.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/my_image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/my_image.jpg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/my_image2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/my_image2.jpg -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/n_model_backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/n_model_backward.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/nm_backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/nm_backward.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/relu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/relu.png -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/sgd.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/sgd.gif -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/sgd_bad.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/sgd_bad.gif -------------------------------------------------------------------------------- /1. Neural Networks and Deep Learning/images/structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/1. Neural Networks and Deep Learning/images/structure.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/Gradient Checking v1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Gradient Checking\n", 8 | "\n", 9 | "Welcome to the final assignment for this week! In this assignment you will learn to implement and use gradient checking. \n", 10 | "\n", 11 | "You are part of a team working to make mobile payments available globally, and are asked to build a deep learning model to detect fraud--whenever someone makes a payment, you want to see if the payment might be fraudulent, such as if the user's account has been taken over by a hacker. \n", 12 | "\n", 13 | "But backpropagation is quite challenging to implement, and sometimes has bugs. Because this is a mission-critical application, your company's CEO wants to be really certain that your implementation of backpropagation is correct. Your CEO says, \"Give me a proof that your backpropagation is actually working!\" To give this reassurance, you are going to use \"gradient checking\".\n", 14 | "\n", 15 | "Let's do it!" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 5, 21 | "metadata": { 22 | "collapsed": true 23 | }, 24 | "outputs": [], 25 | "source": [ 26 | "# Packages\n", 27 | "import numpy as np\n", 28 | "from testCases import *\n", 29 | "from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "## 1) How does gradient checking work?\n", 37 | "\n", 38 | "Backpropagation computes the gradients $\\frac{\\partial J}{\\partial \\theta}$, where $\\theta$ denotes the parameters of the model. $J$ is computed using forward propagation and your loss function.\n", 39 | "\n", 40 | "Because forward propagation is relatively easy to implement, you're confident you got that right, and so you're almost 100% sure that you're computing the cost $J$ correctly. Thus, you can use your code for computing $J$ to verify the code for computing $\\frac{\\partial J}{\\partial \\theta}$. \n", 41 | "\n", 42 | "Let's look back at the definition of a derivative (or gradient):\n", 43 | "$$ \\frac{\\partial J}{\\partial \\theta} = \\lim_{\\varepsilon \\to 0} \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon} \\tag{1}$$\n", 44 | "\n", 45 | "If you're not familiar with the \"$\\displaystyle \\lim_{\\varepsilon \\to 0}$\" notation, it's just a way of saying \"when $\\varepsilon$ is really really small.\"\n", 46 | "\n", 47 | "We know the following:\n", 48 | "\n", 49 | "- $\\frac{\\partial J}{\\partial \\theta}$ is what you want to make sure you're computing correctly. \n", 50 | "- You can compute $J(\\theta + \\varepsilon)$ and $J(\\theta - \\varepsilon)$ (in the case that $\\theta$ is a real number), since you're confident your implementation for $J$ is correct. \n", 51 | "\n", 52 | "Lets use equation (1) and a small value for $\\varepsilon$ to convince your CEO that your code for computing $\\frac{\\partial J}{\\partial \\theta}$ is correct!" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "## 2) 1-dimensional gradient checking\n", 60 | "\n", 61 | "Consider a 1D linear function $J(\\theta) = \\theta x$. The model contains only a single real-valued parameter $\\theta$, and takes $x$ as input.\n", 62 | "\n", 63 | "You will implement code to compute $J(.)$ and its derivative $\\frac{\\partial J}{\\partial \\theta}$. You will then use gradient checking to make sure your derivative computation for $J$ is correct. \n", 64 | "\n", 65 | "\n", 66 | "
**Figure 1** : **1D linear model**
\n", 67 | "\n", 68 | "The diagram above shows the key computation steps: First start with $x$, then evaluate the function $J(x)$ (\"forward propagation\"). Then compute the derivative $\\frac{\\partial J}{\\partial \\theta}$ (\"backward propagation\"). \n", 69 | "\n", 70 | "**Exercise**: implement \"forward propagation\" and \"backward propagation\" for this simple function. I.e., compute both $J(.)$ (\"forward propagation\") and its derivative with respect to $\\theta$ (\"backward propagation\"), in two separate functions. " 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 6, 76 | "metadata": { 77 | "collapsed": true 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "# GRADED FUNCTION: forward_propagation\n", 82 | "\n", 83 | "def forward_propagation(x, theta):\n", 84 | " \"\"\"\n", 85 | " Implement the linear forward propagation (compute J) presented in Figure 1 (J(theta) = theta * x)\n", 86 | " \n", 87 | " Arguments:\n", 88 | " x -- a real-valued input\n", 89 | " theta -- our parameter, a real number as well\n", 90 | " \n", 91 | " Returns:\n", 92 | " J -- the value of function J, computed using the formula J(theta) = theta * x\n", 93 | " \"\"\"\n", 94 | " \n", 95 | " ### START CODE HERE ### (approx. 1 line)\n", 96 | " J = theta*x\n", 97 | " ### END CODE HERE ###\n", 98 | " \n", 99 | " return J" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 7, 105 | "metadata": {}, 106 | "outputs": [ 107 | { 108 | "name": "stdout", 109 | "output_type": "stream", 110 | "text": [ 111 | "J = 8\n" 112 | ] 113 | } 114 | ], 115 | "source": [ 116 | "x, theta = 2, 4\n", 117 | "J = forward_propagation(x, theta)\n", 118 | "print (\"J = \" + str(J))" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "**Expected Output**:\n", 126 | "\n", 127 | "\n", 128 | " \n", 129 | " \n", 130 | " \n", 131 | " \n", 132 | "
** J ** 8
" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "**Exercise**: Now, implement the backward propagation step (derivative computation) of Figure 1. That is, compute the derivative of $J(\\theta) = \\theta x$ with respect to $\\theta$. To save you from doing the calculus, you should get $dtheta = \\frac { \\partial J }{ \\partial \\theta} = x$." 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 8, 145 | "metadata": { 146 | "collapsed": true 147 | }, 148 | "outputs": [], 149 | "source": [ 150 | "# GRADED FUNCTION: backward_propagation\n", 151 | "\n", 152 | "def backward_propagation(x, theta):\n", 153 | " \"\"\"\n", 154 | " Computes the derivative of J with respect to theta (see Figure 1).\n", 155 | " \n", 156 | " Arguments:\n", 157 | " x -- a real-valued input\n", 158 | " theta -- our parameter, a real number as well\n", 159 | " \n", 160 | " Returns:\n", 161 | " dtheta -- the gradient of the cost with respect to theta\n", 162 | " \"\"\"\n", 163 | " \n", 164 | " ### START CODE HERE ### (approx. 1 line)\n", 165 | " dtheta = x\n", 166 | " ### END CODE HERE ###\n", 167 | " \n", 168 | " return dtheta" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": 9, 174 | "metadata": { 175 | "scrolled": true 176 | }, 177 | "outputs": [ 178 | { 179 | "name": "stdout", 180 | "output_type": "stream", 181 | "text": [ 182 | "dtheta = 2\n" 183 | ] 184 | } 185 | ], 186 | "source": [ 187 | "x, theta = 2, 4\n", 188 | "dtheta = backward_propagation(x, theta)\n", 189 | "print (\"dtheta = \" + str(dtheta))" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "**Expected Output**:\n", 197 | "\n", 198 | "\n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | " \n", 203 | "
** dtheta ** 2
" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "**Exercise**: To show that the `backward_propagation()` function is correctly computing the gradient $\\frac{\\partial J}{\\partial \\theta}$, let's implement gradient checking.\n", 211 | "\n", 212 | "**Instructions**:\n", 213 | "- First compute \"gradapprox\" using the formula above (1) and a small value of $\\varepsilon$. Here are the Steps to follow:\n", 214 | " 1. $\\theta^{+} = \\theta + \\varepsilon$\n", 215 | " 2. $\\theta^{-} = \\theta - \\varepsilon$\n", 216 | " 3. $J^{+} = J(\\theta^{+})$\n", 217 | " 4. $J^{-} = J(\\theta^{-})$\n", 218 | " 5. $gradapprox = \\frac{J^{+} - J^{-}}{2 \\varepsilon}$\n", 219 | "- Then compute the gradient using backward propagation, and store the result in a variable \"grad\"\n", 220 | "- Finally, compute the relative difference between \"gradapprox\" and the \"grad\" using the following formula:\n", 221 | "$$ difference = \\frac {\\mid\\mid grad - gradapprox \\mid\\mid_2}{\\mid\\mid grad \\mid\\mid_2 + \\mid\\mid gradapprox \\mid\\mid_2} \\tag{2}$$\n", 222 | "You will need 3 Steps to compute this formula:\n", 223 | " - 1'. compute the numerator using np.linalg.norm(...)\n", 224 | " - 2'. compute the denominator. You will need to call np.linalg.norm(...) twice.\n", 225 | " - 3'. divide them.\n", 226 | "- If this difference is small (say less than $10^{-7}$), you can be quite confident that you have computed your gradient correctly. Otherwise, there may be a mistake in the gradient computation. \n" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 10, 232 | "metadata": { 233 | "collapsed": true 234 | }, 235 | "outputs": [], 236 | "source": [ 237 | "# GRADED FUNCTION: gradient_check\n", 238 | "\n", 239 | "def gradient_check(x, theta, epsilon = 1e-7):\n", 240 | " \"\"\"\n", 241 | " Implement the backward propagation presented in Figure 1.\n", 242 | " \n", 243 | " Arguments:\n", 244 | " x -- a real-valued input\n", 245 | " theta -- our parameter, a real number as well\n", 246 | " epsilon -- tiny shift to the input to compute approximated gradient with formula(1)\n", 247 | " \n", 248 | " Returns:\n", 249 | " difference -- difference (2) between the approximated gradient and the backward propagation gradient\n", 250 | " \"\"\"\n", 251 | " \n", 252 | " # Compute gradapprox using left side of formula (1). epsilon is small enough, you don't need to worry about the limit.\n", 253 | " ### START CODE HERE ### (approx. 5 lines)\n", 254 | " thetaplus = theta + epsilon # Step 1\n", 255 | " thetaminus = theta - epsilon # Step 2\n", 256 | " J_plus = forward_propagation(x,thetaplus) # Step 3\n", 257 | " J_minus = forward_propagation(x,thetaminus) # Step 4\n", 258 | " gradapprox = (J_plus - J_minus)/(2*epsilon) # Step 5\n", 259 | " ### END CODE HERE ###\n", 260 | " \n", 261 | " # Check if gradapprox is close enough to the output of backward_propagation()\n", 262 | " ### START CODE HERE ### (approx. 1 line)\n", 263 | " grad = backward_propagation(x, theta)\n", 264 | " ### END CODE HERE ###\n", 265 | " \n", 266 | " ### START CODE HERE ### (approx. 1 line)\n", 267 | " numerator = np.linalg.norm(grad-gradapprox) # Step 1'\n", 268 | " denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'\n", 269 | " difference = numerator / denominator # Step 3'\n", 270 | " ### END CODE HERE ###\n", 271 | " \n", 272 | " if difference < 1e-7:\n", 273 | " print (\"The gradient is correct!\")\n", 274 | " else:\n", 275 | " print (\"The gradient is wrong!\")\n", 276 | " \n", 277 | " return difference" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 11, 283 | "metadata": { 284 | "scrolled": true 285 | }, 286 | "outputs": [ 287 | { 288 | "name": "stdout", 289 | "output_type": "stream", 290 | "text": [ 291 | "The gradient is correct!\n", 292 | "difference = 2.91933588329e-10\n" 293 | ] 294 | } 295 | ], 296 | "source": [ 297 | "x, theta = 2, 4\n", 298 | "difference = gradient_check(x, theta)\n", 299 | "print(\"difference = \" + str(difference))" 300 | ] 301 | }, 302 | { 303 | "cell_type": "markdown", 304 | "metadata": {}, 305 | "source": [ 306 | "**Expected Output**:\n", 307 | "The gradient is correct!\n", 308 | "\n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | "
** difference ** 2.9193358103083e-10
" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": {}, 319 | "source": [ 320 | "Congrats, the difference is smaller than the $10^{-7}$ threshold. So you can have high confidence that you've correctly computed the gradient in `backward_propagation()`. \n", 321 | "\n", 322 | "Now, in the more general case, your cost function $J$ has more than a single 1D input. When you are training a neural network, $\\theta$ actually consists of multiple matrices $W^{[l]}$ and biases $b^{[l]}$! It is important to know how to do a gradient check with higher-dimensional inputs. Let's do it!" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "## 3) N-dimensional gradient checking" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "metadata": { 335 | "collapsed": true 336 | }, 337 | "source": [ 338 | "The following figure describes the forward and backward propagation of your fraud detection model.\n", 339 | "\n", 340 | "\n", 341 | "
**Figure 2** : **deep neural network**
*LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID*
\n", 342 | "\n", 343 | "Let's look at your implementations for forward propagation and backward propagation. " 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 12, 349 | "metadata": { 350 | "collapsed": true 351 | }, 352 | "outputs": [], 353 | "source": [ 354 | "def forward_propagation_n(X, Y, parameters):\n", 355 | " \"\"\"\n", 356 | " Implements the forward propagation (and computes the cost) presented in Figure 3.\n", 357 | " \n", 358 | " Arguments:\n", 359 | " X -- training set for m examples\n", 360 | " Y -- labels for m examples \n", 361 | " parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n", 362 | " W1 -- weight matrix of shape (5, 4)\n", 363 | " b1 -- bias vector of shape (5, 1)\n", 364 | " W2 -- weight matrix of shape (3, 5)\n", 365 | " b2 -- bias vector of shape (3, 1)\n", 366 | " W3 -- weight matrix of shape (1, 3)\n", 367 | " b3 -- bias vector of shape (1, 1)\n", 368 | " \n", 369 | " Returns:\n", 370 | " cost -- the cost function (logistic cost for one example)\n", 371 | " \"\"\"\n", 372 | " \n", 373 | " # retrieve parameters\n", 374 | " m = X.shape[1]\n", 375 | " W1 = parameters[\"W1\"]\n", 376 | " b1 = parameters[\"b1\"]\n", 377 | " W2 = parameters[\"W2\"]\n", 378 | " b2 = parameters[\"b2\"]\n", 379 | " W3 = parameters[\"W3\"]\n", 380 | " b3 = parameters[\"b3\"]\n", 381 | "\n", 382 | " # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID\n", 383 | " Z1 = np.dot(W1, X) + b1\n", 384 | " A1 = relu(Z1)\n", 385 | " Z2 = np.dot(W2, A1) + b2\n", 386 | " A2 = relu(Z2)\n", 387 | " Z3 = np.dot(W3, A2) + b3\n", 388 | " A3 = sigmoid(Z3)\n", 389 | "\n", 390 | " # Cost\n", 391 | " logprobs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)\n", 392 | " cost = 1./m * np.sum(logprobs)\n", 393 | " \n", 394 | " cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)\n", 395 | " \n", 396 | " return cost, cache" 397 | ] 398 | }, 399 | { 400 | "cell_type": "markdown", 401 | "metadata": {}, 402 | "source": [ 403 | "Now, run backward propagation." 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 13, 409 | "metadata": { 410 | "collapsed": true 411 | }, 412 | "outputs": [], 413 | "source": [ 414 | "def backward_propagation_n(X, Y, cache):\n", 415 | " \"\"\"\n", 416 | " Implement the backward propagation presented in figure 2.\n", 417 | " \n", 418 | " Arguments:\n", 419 | " X -- input datapoint, of shape (input size, 1)\n", 420 | " Y -- true \"label\"\n", 421 | " cache -- cache output from forward_propagation_n()\n", 422 | " \n", 423 | " Returns:\n", 424 | " gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables.\n", 425 | " \"\"\"\n", 426 | " \n", 427 | " m = X.shape[1]\n", 428 | " (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache\n", 429 | " \n", 430 | " dZ3 = A3 - Y\n", 431 | " dW3 = 1./m * np.dot(dZ3, A2.T)\n", 432 | " db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)\n", 433 | " \n", 434 | " dA2 = np.dot(W3.T, dZ3)\n", 435 | " dZ2 = np.multiply(dA2, np.int64(A2 > 0))\n", 436 | " dW2 = 1./m * np.dot(dZ2, A1.T) * 2\n", 437 | " db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)\n", 438 | " \n", 439 | " dA1 = np.dot(W2.T, dZ2)\n", 440 | " dZ1 = np.multiply(dA1, np.int64(A1 > 0))\n", 441 | " dW1 = 1./m * np.dot(dZ1, X.T)\n", 442 | " db1 = 4./m * np.sum(dZ1, axis=1, keepdims = True)\n", 443 | " \n", 444 | " gradients = {\"dZ3\": dZ3, \"dW3\": dW3, \"db3\": db3,\n", 445 | " \"dA2\": dA2, \"dZ2\": dZ2, \"dW2\": dW2, \"db2\": db2,\n", 446 | " \"dA1\": dA1, \"dZ1\": dZ1, \"dW1\": dW1, \"db1\": db1}\n", 447 | " \n", 448 | " return gradients" 449 | ] 450 | }, 451 | { 452 | "cell_type": "markdown", 453 | "metadata": { 454 | "collapsed": true 455 | }, 456 | "source": [ 457 | "You obtained some results on the fraud detection test set but you are not 100% sure of your model. Nobody's perfect! Let's implement gradient checking to verify if your gradients are correct." 458 | ] 459 | }, 460 | { 461 | "cell_type": "markdown", 462 | "metadata": {}, 463 | "source": [ 464 | "**How does gradient checking work?**.\n", 465 | "\n", 466 | "As in 1) and 2), you want to compare \"gradapprox\" to the gradient computed by backpropagation. The formula is still:\n", 467 | "\n", 468 | "$$ \\frac{\\partial J}{\\partial \\theta} = \\lim_{\\varepsilon \\to 0} \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon} \\tag{1}$$\n", 469 | "\n", 470 | "However, $\\theta$ is not a scalar anymore. It is a dictionary called \"parameters\". We implemented a function \"`dictionary_to_vector()`\" for you. It converts the \"parameters\" dictionary into a vector called \"values\", obtained by reshaping all parameters (W1, b1, W2, b2, W3, b3) into vectors and concatenating them.\n", 471 | "\n", 472 | "The inverse function is \"`vector_to_dictionary`\" which outputs back the \"parameters\" dictionary.\n", 473 | "\n", 474 | "\n", 475 | "
**Figure 2** : **dictionary_to_vector() and vector_to_dictionary()**
You will need these functions in gradient_check_n()
\n", 476 | "\n", 477 | "We have also converted the \"gradients\" dictionary into a vector \"grad\" using gradients_to_vector(). You don't need to worry about that.\n", 478 | "\n", 479 | "**Exercise**: Implement gradient_check_n().\n", 480 | "\n", 481 | "**Instructions**: Here is pseudo-code that will help you implement the gradient check.\n", 482 | "\n", 483 | "For each i in num_parameters:\n", 484 | "- To compute `J_plus[i]`:\n", 485 | " 1. Set $\\theta^{+}$ to `np.copy(parameters_values)`\n", 486 | " 2. Set $\\theta^{+}_i$ to $\\theta^{+}_i + \\varepsilon$\n", 487 | " 3. Calculate $J^{+}_i$ using to `forward_propagation_n(x, y, vector_to_dictionary(`$\\theta^{+}$ `))`. \n", 488 | "- To compute `J_minus[i]`: do the same thing with $\\theta^{-}$\n", 489 | "- Compute $gradapprox[i] = \\frac{J^{+}_i - J^{-}_i}{2 \\varepsilon}$\n", 490 | "\n", 491 | "Thus, you get a vector gradapprox, where gradapprox[i] is an approximation of the gradient with respect to `parameter_values[i]`. You can now compare this gradapprox vector to the gradients vector from backpropagation. Just like for the 1D case (Steps 1', 2', 3'), compute: \n", 492 | "$$ difference = \\frac {\\| grad - gradapprox \\|_2}{\\| grad \\|_2 + \\| gradapprox \\|_2 } \\tag{3}$$" 493 | ] 494 | }, 495 | { 496 | "cell_type": "code", 497 | "execution_count": 31, 498 | "metadata": {}, 499 | "outputs": [], 500 | "source": [ 501 | "# GRADED FUNCTION: gradient_check_n\n", 502 | "\n", 503 | "def gradient_check_n(parameters, gradients, X, Y, epsilon = 1e-7):\n", 504 | " \"\"\"\n", 505 | " Checks if backward_propagation_n computes correctly the gradient of the cost output by forward_propagation_n\n", 506 | " \n", 507 | " Arguments:\n", 508 | " parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n", 509 | " grad -- output of backward_propagation_n, contains gradients of the cost with respect to the parameters. \n", 510 | " x -- input datapoint, of shape (input size, 1)\n", 511 | " y -- true \"label\"\n", 512 | " epsilon -- tiny shift to the input to compute approximated gradient with formula(1)\n", 513 | " \n", 514 | " Returns:\n", 515 | " difference -- difference (2) between the approximated gradient and the backward propagation gradient\n", 516 | " \"\"\"\n", 517 | " \n", 518 | " # Set-up variables\n", 519 | " parameters_values, _ = dictionary_to_vector(parameters)\n", 520 | " grad = gradients_to_vector(gradients)\n", 521 | " num_parameters = parameters_values.shape[0]\n", 522 | " J_plus = np.zeros((num_parameters, 1))\n", 523 | " J_minus = np.zeros((num_parameters, 1))\n", 524 | " gradapprox = np.zeros((num_parameters, 1))\n", 525 | " \n", 526 | " # Compute gradapprox\n", 527 | " for i in range(num_parameters):\n", 528 | " \n", 529 | " # Compute J_plus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_plus[i]\".\n", 530 | " # \"_\" is used because the function you have to outputs two parameters but we only care about the first one\n", 531 | " ### START CODE HERE ### (approx. 3 lines)\n", 532 | " thetaplus = np.copy(parameters_values) # Step 1\n", 533 | " thetaplus[i][0] = thetaplus[i][0]+epsilon # Step 2\n", 534 | " J_plus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaplus)) # Step 3\n", 535 | " ### END CODE HERE ###\n", 536 | " \n", 537 | " # Compute J_minus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_minus[i]\".\n", 538 | " ### START CODE HERE ### (approx. 3 lines)\n", 539 | " thetaminus = np.copy(parameters_values) # Step 1\n", 540 | " thetaminus[i][0] = thetaminus[i][0]-epsilon # Step 2 \n", 541 | " J_minus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaminus)) # Step 3\n", 542 | " ### END CODE HERE ###\n", 543 | " \n", 544 | " # Compute gradapprox[i]\n", 545 | " ### START CODE HERE ### (approx. 1 line)\n", 546 | " gradapprox[i] = (J_plus[i]- J_minus[i])/(2*epsilon)\n", 547 | " ### END CODE HERE ###\n", 548 | " \n", 549 | " # Compare gradapprox to backward propagation gradients by computing difference.\n", 550 | " ### START CODE HERE ### (approx. 1 line)\n", 551 | " numerator = np.linalg.norm(grad-gradapprox) # Step 1'\n", 552 | " denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'\n", 553 | " difference = numerator / denominator # Step 3'\n", 554 | " ### END CODE HERE ###\n", 555 | "\n", 556 | " if difference > 2e-7:\n", 557 | " print (\"\\033[93m\" + \"There is a mistake in the backward propagation! difference = \" + str(difference) + \"\\033[0m\")\n", 558 | " else:\n", 559 | " print (\"\\033[92m\" + \"Your backward propagation works perfectly fine! difference = \" + str(difference) + \"\\033[0m\")\n", 560 | " \n", 561 | " return difference" 562 | ] 563 | }, 564 | { 565 | "cell_type": "code", 566 | "execution_count": 32, 567 | "metadata": { 568 | "scrolled": false 569 | }, 570 | "outputs": [ 571 | { 572 | "name": "stdout", 573 | "output_type": "stream", 574 | "text": [ 575 | "\u001b[93mThere is a mistake in the backward propagation! difference = 0.285093156781\u001b[0m\n" 576 | ] 577 | } 578 | ], 579 | "source": [ 580 | "X, Y, parameters = gradient_check_n_test_case()\n", 581 | "\n", 582 | "cost, cache = forward_propagation_n(X, Y, parameters)\n", 583 | "gradients = backward_propagation_n(X, Y, cache)\n", 584 | "difference = gradient_check_n(parameters, gradients, X, Y)" 585 | ] 586 | }, 587 | { 588 | "cell_type": "markdown", 589 | "metadata": {}, 590 | "source": [ 591 | "**Expected output**:\n", 592 | "\n", 593 | "\n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | "
** There is a mistake in the backward propagation!** difference = 0.285093156781
" 599 | ] 600 | }, 601 | { 602 | "cell_type": "markdown", 603 | "metadata": {}, 604 | "source": [ 605 | "It seems that there were errors in the `backward_propagation_n` code we gave you! Good that you've implemented the gradient check. Go back to `backward_propagation` and try to find/correct the errors *(Hint: check dW2 and db1)*. Rerun the gradient check when you think you've fixed it. Remember you'll need to re-execute the cell defining `backward_propagation_n()` if you modify the code. \n", 606 | "\n", 607 | "Can you get gradient check to declare your derivative computation correct? Even though this part of the assignment isn't graded, we strongly urge you to try to find the bug and re-run gradient check until you're convinced backprop is now correctly implemented. \n", 608 | "\n", 609 | "**Note** \n", 610 | "- Gradient Checking is slow! Approximating the gradient with $\\frac{\\partial J}{\\partial \\theta} \\approx \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon}$ is computationally costly. For this reason, we don't run gradient checking at every iteration during training. Just a few times to check if the gradient is correct. \n", 611 | "- Gradient Checking, at least as we've presented it, doesn't work with dropout. You would usually run the gradient check algorithm without dropout to make sure your backprop is correct, then add dropout. \n", 612 | "\n", 613 | "Congrats, you can be confident that your deep learning model for fraud detection is working correctly! You can even use this to convince your CEO. :) \n", 614 | "\n", 615 | "\n", 616 | "**What you should remember from this notebook**:\n", 617 | "- Gradient checking verifies closeness between the gradients from backpropagation and the numerical approximation of the gradient (computed using forward propagation).\n", 618 | "- Gradient checking is slow, so we don't run it in every iteration of training. You would usually run it only to make sure your code is correct, then turn it off and use backprop for the actual learning process. " 619 | ] 620 | }, 621 | { 622 | "cell_type": "code", 623 | "execution_count": null, 624 | "metadata": { 625 | "collapsed": true 626 | }, 627 | "outputs": [], 628 | "source": [ 629 | "!tar cvfz notebook.tar.gz *" 630 | ] 631 | }, 632 | { 633 | "cell_type": "code", 634 | "execution_count": null, 635 | "metadata": { 636 | "collapsed": true 637 | }, 638 | "outputs": [], 639 | "source": [ 640 | "!tar cvfz notebook.tar.gz *" 641 | ] 642 | }, 643 | { 644 | "cell_type": "code", 645 | "execution_count": null, 646 | "metadata": { 647 | "collapsed": true 648 | }, 649 | "outputs": [], 650 | "source": [] 651 | } 652 | ], 653 | "metadata": { 654 | "coursera": { 655 | "course_slug": "deep-neural-network", 656 | "graded_item_id": "n6NBD", 657 | "launcher_item_id": "yfOsE" 658 | }, 659 | "kernelspec": { 660 | "display_name": "Python 3", 661 | "language": "python", 662 | "name": "python3" 663 | }, 664 | "language_info": { 665 | "codemirror_mode": { 666 | "name": "ipython", 667 | "version": 3 668 | }, 669 | "file_extension": ".py", 670 | "mimetype": "text/x-python", 671 | "name": "python", 672 | "nbconvert_exporter": "python", 673 | "pygments_lexer": "ipython3", 674 | "version": "3.6.0" 675 | } 676 | }, 677 | "nbformat": 4, 678 | "nbformat_minor": 1 679 | } 680 | -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/README.md: -------------------------------------------------------------------------------- 1 | ## Objectives: 2 | - Recall that different types of initializations lead to different results 3 | - Recognize the importance of initialization in complex neural networks. 4 | - Recognize the difference between train/dev/test sets 5 | - Diagnose the bias and variance issues in your model 6 | - Learn when and how to use regularization methods such as dropout or L2 regularization. 7 | - Understand experimental issues in deep learning such as Vanishing or Exploding gradients and learn how to deal with them 8 | - Use gradient checking to verify the correctness of your backpropagation implementation 9 | - Remember different optimization methods such as (Stochastic) Gradient Descent, Momentum, RMSProp and Adam 10 | - Use random minibatches to accelerate the convergence and improve the optimization 11 | - Know the benefits of learning rate decay and apply it to your optimization 12 | - Master the process of hyperparameter tuning 13 | 14 | ## Week 1: 15 | - Initialization.ipynb 16 | - Regularization.ipynb 17 | - Gradient Checking v1.ipynb 18 | 19 | ## Week 2: 20 | - Optimization methods.ipynb 21 | 22 | ## Week 3: 23 | - Tensorflow Tutorial.ipynb 24 | -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/1Dgrad_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/1Dgrad_kiank.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/Momentum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/Momentum.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/NDgrad_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/NDgrad_kiank.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/cost.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/cost.jpg -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/dictionary_to_vector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/dictionary_to_vector.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/dropout1_kiank.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/dropout1_kiank.mp4 -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/dropout2_kiank.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/dropout2_kiank.mp4 -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/field_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/field_kiank.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/handbackward_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/handbackward_kiank.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/handforward_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/handforward_kiank.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/hands.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/hands.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_minibatch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_minibatch.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_partition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_partition.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_sgd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_sgd.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_shuffle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/kiank_shuffle.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/onehot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/onehot.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/opt1.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/opt1.gif -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/opt2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/opt2.gif -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/opt_momentum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/opt_momentum.png -------------------------------------------------------------------------------- /2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/thumbs_up.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/2. Improving Deep Neural Networks- Hyperparameter tuning- Regularization and Optimization/images/thumbs_up.jpg -------------------------------------------------------------------------------- /3. Structuring Machine Learning Projects/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Objectives 3 | - Understand why Machine Learning strategy is important 4 | - Apply satisficing and optimizing metrics to set up your goal for ML projects 5 | - Choose a correct train/dev/test split of your dataset 6 | - Understand how to define human-level performance 7 | - Use human-level perform to define your key priorities in ML projects 8 | - Take the correct ML Strategic decision based on observations of performances and dataset 9 | - Understand what multi-task learning and transfer learning are 10 | - Recognize bias, variance and data-mismatch by looking at the performances of your algorithm on train/dev/test sets 11 | -------------------------------------------------------------------------------- /3. Structuring Machine Learning Projects/Week 1 Quiz - Bird recognition in the city of Peacetopia (case study).md: -------------------------------------------------------------------------------- 1 | ## Week 1 Quiz - Bird recognition in the city of Peacetopia (case study) 2 | 3 | 1. Having three evaluation metrics makes it harder for you to quickly choose between two different algorithms, and will slow down the speed with which your team can iterate. True/False? 4 | 5 | - [x] True 6 | - [ ] False 7 | 8 | 2. If you had the three following models, which one would you choose? 9 | 10 | - Test Accuracy 98% 11 | - Runtime 9 sec 12 | - Memory size 9MB 13 | 14 | 3. Based on the city’s requests, which of the following would you say is true? 15 | 16 | - [x] Accuracy is an optimizing metric; running time and memory size are a satisficing metrics. 17 | - [ ] Accuracy is a satisficing metric; running time and memory size are an optimizing metric. 18 | - [ ] Accuracy, running time and memory size are all optimizing metrics because you want to do well on all three. 19 | - [ ] Accuracy, running time and memory size are all satisficing metrics because you have to do sufficiently well on all three for your system to be acceptable. 20 | 21 | 4. Before implementing your algorithm, you need to split your data into train/dev/test sets. Which of these do you think is the best choice? 22 | 23 | - Train 9,500,000 24 | - Dev 250,000 25 | - Test 250,000 26 | 27 | 5. After setting up your train/dev/test sets, the City Council comes across another 1,000,000 images, called the “citizens’ data”. Apparently the citizens of Peacetopia are so scared of birds that they volunteered to take pictures of the sky and label them, thus contributing these additional 1,000,000 images. These images are different from the distribution of images the City Council had originally given you, but you think it could help your algorithm. 28 | 29 | You should not add the citizens’ data to the training set, because this will cause the training and dev/test set distributions to become different, thus hurting dev and test set performance. True/False? 30 | 31 | - [ ] True 32 | - [x] False 33 | 34 | 6. One member of the City Council knows a little about machine learning, and thinks you should add the 1,000,000 citizens’ data images to the test set. You object because: 35 | 36 | - The test set no longer reflects the distribution of data (security cameras) you most care about. 37 | - This would cause the dev and test set distributions to become different. This is a bad idea because you’re not aiming where you want to hit. 38 | 39 | 7. You train a system, and its errors are as follows (error = 100%-Accuracy): 40 | 41 | - Training set error 4.0% 42 | - Dev set error 4.5% 43 | 44 | This suggests that one good avenue for improving performance is to train a bigger network so as to drive down the 4.0% training error. Do you agree? 45 | 46 | - No, because there is insufficient information to tell. 47 | 48 | 8. You ask a few people to label the dataset so as to find out what is human-level performance. You find the following levels of accuracy: 49 | 50 | - Bird watching expert #1 0.3% error 51 | - Bird watching expert #2 0.5% error 52 | - Normal person #1 (not a bird watching expert) 1.0% error 53 | - Normal person #2 (not a bird watching expert) 1.2% error 54 | 55 | If your goal is to have “human-level performance” be a proxy (or estimate) for Bayes error, how would you define “human-level performance”? 56 | 57 | - 0.3% (accuracy of expert #1) 58 | 59 | 9. Which of the following statements do you agree with? 60 | 61 | - A learning algorithm’s performance can be better human-level performance but it can never be better than Bayes error. 62 | 63 | 10. You find that a team of ornithologists debating and discussing an image gets an even better 0.1% performance, so you define that as “human-level performance.” After working further on your algorithm, you end up with the following: 64 | 65 | - Human-level performance 0.1% 66 | - Training set error 2.0% 67 | - Dev set error 2.1% 68 | 69 | Based on the evidence you have, which two of the following four options seem the most promising to try? (Check two options.) 70 | 71 | - Try decreasing regularization. 72 | - Train a bigger model to try to do better on the training set. 73 | 74 | 11. You also evaluate your model on the test set, and find the following: 75 | 76 | - Human-level performance 0.1% 77 | - Training set error 2.0% 78 | - Dev set error 2.1% 79 | - Test set error 7.0% 80 | 81 | What does this mean? (Check the two best options.) 82 | 83 | - You should try to get a bigger dev set. 84 | - You have overfit to the dev set. 85 | 86 | 12. After working on this project for a year, you finally achieve: 87 | 88 | - Human-level performance 0.10% 89 | - Training set error 0.05% 90 | - Dev set error 0.05% 91 | 92 | What can you conclude? (Check all that apply.) 93 | 94 | - It is now harder to measure avoidable bias, thus progress will be slower going forward. 95 | - If the test set is big enough for the 0,05% error estimate to be accurate, this implies Bayes error is ≤0.05 96 | 97 | 13. It turns out Peacetopia has hired one of your competitors to build a system as well. Your system and your competitor both deliver systems with about the same running time and memory size. However, your system has higher accuracy! However, when Peacetopia tries out your and your competitor’s systems, they conclude they actually like your competitor’s system better, because even though you have higher overall accuracy, you have more false negatives (failing to raise an alarm when a bird is in the air). What should you do? 98 | 99 | - Rethink the appropriate metric for this task, and ask your team to tune to the new metric. 100 | 101 | 14. You’ve handily beaten your competitor, and your system is now deployed in Peacetopia and is protecting the citizens from birds! But over the last few months, a new species of bird has been slowly migrating into the area, so the performance of your system slowly degrades because your data is being tested on a new type of data. 102 | 103 | - Use the data you have to define a new evaluation metric (using a new dev/test set) taking into account the new species, and use that to drive further progress for your team. 104 | 105 | 15. The City Council thinks that having more Cats in the city would help scare off birds. They are so happy with your work on the Bird detector that they also hire you to build a Cat detector. (Wow Cat detectors are just incredibly useful aren’t they.) Because of years of working on Cat detectors, you have such a huge dataset of 100,000,000 cat images that training on this data takes about two weeks. Which of the statements do you agree with? (Check all that agree.) 106 | 107 | - If 100,000,000 examples is enough to build a good enough Cat detector, you might be better of training with just 10,000,000 examples to gain a ≈10x improvement in how quickly you can run experiments, even if each model performs a bit worse because it’s trained on less data. 108 | - Buying faster computers could speed up your teams’ iteration speed and thus your team’s productivity. 109 | - Needing two weeks to train will limit the speed at which you can iterate. 110 | -------------------------------------------------------------------------------- /3. Structuring Machine Learning Projects/Week 2 Quiz - Autonomous driving (case study).md: -------------------------------------------------------------------------------- 1 | ## Week 2 Quiz - Autonomous driving (case study) 2 | 3 | 1. You are just getting started on this project. What is the first thing you do? Assume each of the steps below would take about an equal amount of time (a few days). 4 | 5 | - Spend a few days training a basic model and see what mistakes it makes. 6 | 7 | > As discussed in lecture, applied ML is a highly iterative process. If you train a basic model and carry out error analysis (see what mistakes it makes) it will help point you in more promising directions. 8 | 9 | 2. Your goal is to detect road signs (stop sign, pedestrian crossing sign, construction ahead sign) and traffic signals (red and green lights) in images. The goal is to recognize which of these objects appear in each image. You plan to use a deep neural network with ReLU units in the hidden layers. 10 | 11 | For the output layer, a softmax activation would be a good choice for the output layer because this is a multi-task learning problem. True/False? 12 | 13 | - [ ] True 14 | - [x] False 15 | 16 | > Softmax would be a good choice if one and only one of the possibilities (stop sign, speed bump, pedestrian crossing, green light and red light) was present in each image. 17 | 18 | 3. You are carrying out error analysis and counting up what errors the algorithm makes. Which of these datasets do you think you should manually go through and carefully examine, one image at a time? 19 | 20 | - [ ] 10,000 randomly chosen images 21 | - [ ] 500 randomly chosen images 22 | - [x] 500 images on which the algorithm made a mistake 23 | - [ ] 10,000 images on which the algorithm made a mistake 24 | 25 | 4. After working on the data for several weeks, your team ends up with the following data: 26 | 27 | - 100,000 labeled images taken using the front-facing camera of your car. 28 | - 900,000 labeled images of roads downloaded from the internet. 29 | 30 | Each image’s labels precisely indicate the presence of any specific road signs and traffic signals or combinations of them. For example, y(i) = [1 0 0 1 0] means the image contains a stop sign and a red traffic light. 31 | Because this is a multi-task learning problem, you need to have all your y(i) vectors fully labeled. If one example is equal to [0 ? 1 1 ?] then the learning algorithm will not be able to use that example. True/False? 32 | 33 | - [ ] True 34 | - [x] False 35 | 36 | > As seen in the lecture on multi-task learning, you can compute the cost such that it is not influenced by the fact that some entries haven’t been labeled. 37 | 38 | 5. The distribution of data you care about contains images from your car’s front-facing camera; which comes from a different distribution than the images you were able to find and download off the internet. How should you split the dataset into train/dev/test sets? 39 | 40 | - [ ] Mix all the 100,000 images with the 900,000 images you found online. Shuffle everything. Split the 1,000,000 images dataset into 600,000 for the training set, 200,000 for the dev set and 200,000 for the test set. 41 | - [ ] Mix all the 100,000 images with the 900,000 images you found online. Shuffle everything. Split the 1,000,000 images dataset into 980,000 for the training set, 10,000 for the dev set and 10,000 for the test set. 42 | - [x] Choose the training set to be the 900,000 images from the internet along with 80,000 images from your car’s front-facing camera. The 20,000 remaining images will be split equally in dev and test sets. 43 | - [ ] Choose the training set to be the 900,000 images from the internet along with 20,000 images from your car’s front-facing camera. The 80,000 remaining images will be split equally in dev and test sets. 44 | > As seen in lecture, it is important that your dev and test set have the closest possible distribution to “real”-data. It is also important for the training set to contain enough “real”-data to avoid having a data-mismatch problem. 45 | 46 | 6. Assume you’ve finally chosen the following split between of the data: 47 | 48 | - Training 940,000 images randomly picked from (900,000 internet images + 60,000 car’s front-facing camera images) 8.8% 49 | - Training-Dev 20,000 images randomly picked from (900,000 internet images + 60,000 car’s front-facing camera images) 9.1% 50 | - Dev 20,000 images from your car’s front-facing camera 14.3% 51 | - Test 20,000 images from the car’s front-facing camera 14.8% 52 | 53 | You also know that human-level error on the road sign and traffic signals classification task is around 0.5%. Which of the following are True? (Check all that apply). 54 | 55 | - You have a large avoidable-bias problem because your training error is quite a bit higher than the human-level error. 56 | - You have a large data-mismatch problem because your model does a lot better on the training-dev set than on the dev set. 57 | 58 | 7. Based on table from the previous question, a friend thinks that the training data distribution is much easier than the dev/test distribution. What do you think? 59 | 60 | - There’s insufficient information to tell if your friend is right or wrong. 61 | 62 | > The algorithm does better on the distribution of data it trained on. But you don’t know if it’s because it trained on that no distribution or if it really is easier. To get a better sense, measure human-level error separately on both distributions. 63 | 64 | 8. You decide to focus on the dev set and check by hand what are the errors due to. Here is a table summarizing your discoveries: 65 | 66 | - Overall dev set error 14.3% 67 | - Errors due to incorrectly labeled data 4.1% 68 | - Errors due to foggy pictures 8.0% 69 | - Errors due to rain drops stuck on your car’s front-facing camera 2.2% 70 | - Errors due to other causes 1.0% 71 | 72 | In this table, 4.1%, 8.0%, etc.are a fraction of the total dev set (not just examples your algorithm mislabeled). I.e. about 8.0/14.3 = 56% of your errors are due to foggy pictures. 73 | 74 | The results from this analysis implies that the team’s highest priority should be to bring more foggy pictures into the training set so as to address the 8.0% of errors in that category. True/False? 75 | 76 | - [x] False because this would depend on how easy it is to add this data and how much you think your team thinks it’ll help. 77 | - [ ] True because it is the largest category of errors. As discussed in lecture, we should prioritize the largest category of error to avoid wasting the team’s time. 78 | - [ ] True because it is greater than the other error categories added together (8.0 > 4.1+2.2+1.0). 79 | - [ ] False because data augmentation (synthesizing foggy images by clean/non-foggy images) is more efficient. 80 | 81 | 9. You can buy a specially designed windshield wiper that help wipe off some of the raindrops on the front-facing camera. Based on the table from the previous question, which of the following statements do you agree with? 82 | 83 | - 2.2% would be a reasonable estimate of the maximum amount this windshield wiper could improve performance. 84 | 85 | > You will probably not improve performance by more than 2.2% by solving the raindrops problem. If your dataset was infinitely big, 2.2% would be a perfect estimate of the improvement you can achieve by purchasing a specially designed windshield wiper that removes the raindrops. 86 | 87 | 10. You decide to use data augmentation to address foggy images. You find 1,000 pictures of fog off the internet, and “add” them to clean images to synthesize foggy days, like this: 88 | 89 | Which of the following statements do you agree with? (Check all that apply.) 90 | 91 | - So long as the synthesized fog looks realistic to the human eye, you can be confident that the synthesized data is accurately capturing the distribution of real foggy images, since human vision is very accurate for the problem you’re solving. 92 | 93 | > If the synthesized images look realistic, then the model will just see them as if you had added useful data to identify road signs and traffic signals in a foggy weather. I will very likely help. 94 | 95 | 11. After working further on the problem, you’ve decided to correct the incorrectly labeled data on the dev set. Which of these statements do you agree with? (Check all that apply). 96 | 97 | - You should not correct incorrectly labeled data in the training set as well so as to avoid your training set now being even more different from your dev set. 98 | 99 | > Deep learning algorithms are quite robust to having slightly different train and dev distributions. 100 | 101 | - You should also correct the incorrectly labeled data in the test set, so that the dev and test sets continue to come from the same distribution 102 | 103 | > Because you want to make sure that your dev and test data come from the same distribution for your algorithm to make your team’s iterative development process is efficient. 104 | 105 | 12. So far your algorithm only recognizes red and green traffic lights. One of your colleagues in the startup is starting to work on recognizing a yellow traffic light. (Some countries call it an orange light rather than a yellow light; we’ll use the US convention of calling it yellow.) Images containing yellow lights are quite rare, and she doesn’t have enough data to build a good model. She hopes you can help her out using transfer learning. 106 | 107 | What do you tell your colleague? 108 | 109 | - She should try using weights pre-trained on your dataset, and fine-tuning further with the yellow-light dataset. 110 | 111 | > You have trained your model on a huge dataset, and she has a small dataset. Although your labels are different, the parameters of your model have been trained to recognize many characteristics of road and traffic images which will be useful for her problem. This is a perfect case for transfer learning, she can start with a model with the same architecture as yours, change what is after the last hidden layer and initialize it with your trained parameters. 112 | 113 | 13. Another colleague wants to use microphones placed outside the car to better hear if there’re other vehicles around you. For example, if there is a police vehicle behind you, you would be able to hear their siren. However, they don’t have much to train this audio system. How can you help? 114 | 115 | - Neither transfer learning nor multi-task learning seems promising. 116 | 117 | > The problem he is trying to solve is quite different from yours. The different dataset structures make it probably impossible to use transfer learning or multi-task learning. 118 | 119 | 14. To recognize red and green lights, you have been using this approach: 120 | 121 | - (A) Input an image (x) to a neural network and have it directly learn a mapping to make a prediction as to whether there’s a red light and/or green light (y). 122 | 123 | A teammate proposes a different, two-step approach: 124 | 125 | - (B) In this two-step approach, you would first (i) detect the traffic light in the image (if any), then (ii) determine the color of the illuminated lamp in the traffic light. 126 | Between these two, Approach B is more of an end-to-end approach because it has distinct steps for the input end and the output end. True/False? 127 | 128 | 129 | - [ ] True 130 | - [x] False 131 | 132 | > (A) is an end-to-end approach as it maps directly the input (x) to the output (y). 133 | 134 | 15. Approach A (in the question above) tends to be more promising than approach B if you have a ________ (fill in the blank). 135 | 136 | - [x] Large training set 137 | - [ ] Multi-task learning problem. 138 | - [ ] Large bias problem. 139 | - [ ] Problem with a high Bayes error. 140 | 141 | > In many fields, it has been observed that end-to-end learning works better in practice, but requires a large amount of data. 142 | -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/Face Recognition for the Happy House - v3.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Face Recognition for the Happy House\n", 8 | "\n", 9 | "Welcome to the first assignment of week 4! Here you will build a face recognition system. Many of the ideas presented here are from [FaceNet](https://arxiv.org/pdf/1503.03832.pdf). In lecture, we also talked about [DeepFace](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf). \n", 10 | "\n", 11 | "Face recognition problems commonly fall into two categories: \n", 12 | "\n", 13 | "- **Face Verification** - \"is this the claimed person?\". For example, at some airports, you can pass through customs by letting a system scan your passport and then verifying that you (the person carrying the passport) are the correct person. A mobile phone that unlocks using your face is also using face verification. This is a 1:1 matching problem. \n", 14 | "- **Face Recognition** - \"who is this person?\". For example, the video lecture showed a face recognition video (https://www.youtube.com/watch?v=wr4rx0Spihs) of Baidu employees entering the office without needing to otherwise identify themselves. This is a 1:K matching problem. \n", 15 | "\n", 16 | "FaceNet learns a neural network that encodes a face image into a vector of 128 numbers. By comparing two such vectors, you can then determine if two pictures are of the same person.\n", 17 | " \n", 18 | "**In this assignment, you will:**\n", 19 | "- Implement the triplet loss function\n", 20 | "- Use a pretrained model to map face images into 128-dimensional encodings\n", 21 | "- Use these encodings to perform face verification and face recognition\n", 22 | "\n", 23 | "In this exercise, we will be using a pre-trained model which represents ConvNet activations using a \"channels first\" convention, as opposed to the \"channels last\" convention used in lecture and previous programming assignments. In other words, a batch of images will be of shape $(m, n_C, n_H, n_W)$ instead of $(m, n_H, n_W, n_C)$. Both of these conventions have a reasonable amount of traction among open-source implementations; there isn't a uniform standard yet within the deep learning community. \n", 24 | "\n", 25 | "Let's load the required packages. \n" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 3, 31 | "metadata": {}, 32 | "outputs": [ 33 | { 34 | "name": "stderr", 35 | "output_type": "stream", 36 | "text": [ 37 | "Using TensorFlow backend.\n" 38 | ] 39 | } 40 | ], 41 | "source": [ 42 | "from keras.models import Sequential\n", 43 | "from keras.layers import Conv2D, ZeroPadding2D, Activation, Input, concatenate\n", 44 | "from keras.models import Model\n", 45 | "from keras.layers.normalization import BatchNormalization\n", 46 | "from keras.layers.pooling import MaxPooling2D, AveragePooling2D\n", 47 | "from keras.layers.merge import Concatenate\n", 48 | "from keras.layers.core import Lambda, Flatten, Dense\n", 49 | "from keras.initializers import glorot_uniform\n", 50 | "from keras.engine.topology import Layer\n", 51 | "from keras import backend as K\n", 52 | "K.set_image_data_format('channels_first')\n", 53 | "import cv2\n", 54 | "import os\n", 55 | "import numpy as np\n", 56 | "from numpy import genfromtxt\n", 57 | "import pandas as pd\n", 58 | "import tensorflow as tf\n", 59 | "from fr_utils import *\n", 60 | "from inception_blocks_v2 import *\n", 61 | "\n", 62 | "%matplotlib inline\n", 63 | "%load_ext autoreload\n", 64 | "%autoreload 2\n", 65 | "\n", 66 | "np.set_printoptions(threshold=np.nan)" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "## 0 - Naive Face Verification\n", 74 | "\n", 75 | "In Face Verification, you're given two images and you have to tell if they are of the same person. The simplest way to do this is to compare the two images pixel-by-pixel. If the distance between the raw images are less than a chosen threshold, it may be the same person! \n", 76 | "\n", 77 | "\n", 78 | "
**Figure 1**
" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": { 84 | "collapsed": true 85 | }, 86 | "source": [ 87 | "Of course, this algorithm performs really poorly, since the pixel values change dramatically due to variations in lighting, orientation of the person's face, even minor changes in head position, and so on. \n", 88 | "\n", 89 | "You'll see that rather than using the raw image, you can learn an encoding $f(img)$ so that element-wise comparisons of this encoding gives more accurate judgements as to whether two pictures are of the same person." 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": {}, 95 | "source": [ 96 | "## 1 - Encoding face images into a 128-dimensional vector \n", 97 | "\n", 98 | "### 1.1 - Using an ConvNet to compute encodings\n", 99 | "\n", 100 | "The FaceNet model takes a lot of data and a long time to train. So following common practice in applied deep learning settings, let's just load weights that someone else has already trained. The network architecture follows the Inception model from [Szegedy *et al.*](https://arxiv.org/abs/1409.4842). We have provided an inception network implementation. You can look in the file `inception_blocks.py` to see how it is implemented (do so by going to \"File->Open...\" at the top of the Jupyter notebook). \n" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "The key things you need to know are:\n", 108 | "\n", 109 | "- This network uses 96x96 dimensional RGB images as its input. Specifically, inputs a face image (or batch of $m$ face images) as a tensor of shape $(m, n_C, n_H, n_W) = (m, 3, 96, 96)$ \n", 110 | "- It outputs a matrix of shape $(m, 128)$ that encodes each input face image into a 128-dimensional vector\n", 111 | "\n", 112 | "Run the cell below to create the model for face images." 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 4, 118 | "metadata": { 119 | "collapsed": true 120 | }, 121 | "outputs": [], 122 | "source": [ 123 | "FRmodel = faceRecoModel(input_shape=(3, 96, 96))" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 5, 129 | "metadata": {}, 130 | "outputs": [ 131 | { 132 | "name": "stdout", 133 | "output_type": "stream", 134 | "text": [ 135 | "Total Params: 3743280\n" 136 | ] 137 | } 138 | ], 139 | "source": [ 140 | "print(\"Total Params:\", FRmodel.count_params())" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "** Expected Output **\n", 148 | "\n", 149 | "
\n", 150 | "Total Params: 3743280\n", 151 | "
\n", 152 | "
\n" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "By using a 128-neuron fully connected layer as its last layer, the model ensures that the output is an encoding vector of size 128. You then use the encodings the compare two face images as follows:\n", 160 | "\n", 161 | "\n", 162 | "
**Figure 2**:
By computing a distance between two encodings and thresholding, you can determine if the two pictures represent the same person
\n", 163 | "\n", 164 | "So, an encoding is a good one if: \n", 165 | "- The encodings of two images of the same person are quite similar to each other \n", 166 | "- The encodings of two images of different persons are very different\n", 167 | "\n", 168 | "The triplet loss function formalizes this, and tries to \"push\" the encodings of two images of the same person (Anchor and Positive) closer together, while \"pulling\" the encodings of two images of different persons (Anchor, Negative) further apart. \n", 169 | "\n", 170 | "\n", 171 | "
\n", 172 | "
**Figure 3**:
In the next part, we will call the pictures from left to right: Anchor (A), Positive (P), Negative (N)
" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "\n", 180 | "\n", 181 | "### 1.2 - The Triplet Loss\n", 182 | "\n", 183 | "For an image $x$, we denote its encoding $f(x)$, where $f$ is the function computed by the neural network.\n", 184 | "\n", 185 | "\n", 186 | "\n", 187 | "\n", 190 | "\n", 191 | "Training will use triplets of images $(A, P, N)$: \n", 192 | "\n", 193 | "- A is an \"Anchor\" image--a picture of a person. \n", 194 | "- P is a \"Positive\" image--a picture of the same person as the Anchor image.\n", 195 | "- N is a \"Negative\" image--a picture of a different person than the Anchor image.\n", 196 | "\n", 197 | "These triplets are picked from our training dataset. We will write $(A^{(i)}, P^{(i)}, N^{(i)})$ to denote the $i$-th training example. \n", 198 | "\n", 199 | "You'd like to make sure that an image $A^{(i)}$ of an individual is closer to the Positive $P^{(i)}$ than to the Negative image $N^{(i)}$) by at least a margin $\\alpha$:\n", 200 | "\n", 201 | "$$\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 + \\alpha < \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$$\n", 202 | "\n", 203 | "You would thus like to minimize the following \"triplet cost\":\n", 204 | "\n", 205 | "$$\\mathcal{J} = \\sum^{m}_{i=1} \\large[ \\small \\underbrace{\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2}_\\text{(1)} - \\underbrace{\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2}_\\text{(2)} + \\alpha \\large ] \\small_+ \\tag{3}$$\n", 206 | "\n", 207 | "Here, we are using the notation \"$[z]_+$\" to denote $max(z,0)$. \n", 208 | "\n", 209 | "Notes:\n", 210 | "- The term (1) is the squared distance between the anchor \"A\" and the positive \"P\" for a given triplet; you want this to be small. \n", 211 | "- The term (2) is the squared distance between the anchor \"A\" and the negative \"N\" for a given triplet, you want this to be relatively large, so it thus makes sense to have a minus sign preceding it. \n", 212 | "- $\\alpha$ is called the margin. It is a hyperparameter that you should pick manually. We will use $\\alpha = 0.2$. \n", 213 | "\n", 214 | "Most implementations also normalize the encoding vectors to have norm equal one (i.e., $\\mid \\mid f(img)\\mid \\mid_2$=1); you won't have to worry about that here.\n", 215 | "\n", 216 | "**Exercise**: Implement the triplet loss as defined by formula (3). Here are the 4 steps:\n", 217 | "1. Compute the distance between the encodings of \"anchor\" and \"positive\": $\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2$\n", 218 | "2. Compute the distance between the encodings of \"anchor\" and \"negative\": $\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$\n", 219 | "3. Compute the formula per training example: $ \\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid - \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2 + \\alpha$\n", 220 | "3. Compute the full formula by taking the max with zero and summing over the training examples:\n", 221 | "$$\\mathcal{J} = \\sum^{m}_{i=1} \\large[ \\small \\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 - \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2+ \\alpha \\large ] \\small_+ \\tag{3}$$\n", 222 | "\n", 223 | "Useful functions: `tf.reduce_sum()`, `tf.square()`, `tf.subtract()`, `tf.add()`, `tf.maximum()`.\n", 224 | "For steps 1 and 2, you will need to sum over the entries of $\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2$ and $\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$ while for step 4 you will need to sum over the training examples." 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": 8, 230 | "metadata": { 231 | "collapsed": true 232 | }, 233 | "outputs": [], 234 | "source": [ 235 | "# GRADED FUNCTION: triplet_loss\n", 236 | "\n", 237 | "def triplet_loss(y_true, y_pred, alpha = 0.2):\n", 238 | " \"\"\"\n", 239 | " Implementation of the triplet loss as defined by formula (3)\n", 240 | " \n", 241 | " Arguments:\n", 242 | " y_true -- true labels, required when you define a loss in Keras, you don't need it in this function.\n", 243 | " y_pred -- python list containing three objects:\n", 244 | " anchor -- the encodings for the anchor images, of shape (None, 128)\n", 245 | " positive -- the encodings for the positive images, of shape (None, 128)\n", 246 | " negative -- the encodings for the negative images, of shape (None, 128)\n", 247 | " \n", 248 | " Returns:\n", 249 | " loss -- real number, value of the loss\n", 250 | " \"\"\"\n", 251 | " \n", 252 | " anchor, positive, negative = y_pred[0], y_pred[1], y_pred[2]\n", 253 | " \n", 254 | " ### START CODE HERE ### (≈ 4 lines)\n", 255 | " # Step 1: Compute the (encoding) distance between the anchor and the positive, you will need to sum over axis=-1\n", 256 | " pos_dist = tf.reduce_sum(tf.square(tf.subtract(anchor,positive)),axis=-1)\n", 257 | " # Step 2: Compute the (encoding) distance between the anchor and the negative, you will need to sum over axis=-1\n", 258 | " neg_dist = tf.reduce_sum(tf.square(tf.subtract(anchor,negative)),axis=-1)\n", 259 | " # Step 3: subtract the two previous distances and add alpha.\n", 260 | " basic_loss = tf.add(tf.subtract(pos_dist,neg_dist),alpha)\n", 261 | " # Step 4: Take the maximum of basic_loss and 0.0. Sum over the training examples.\n", 262 | " loss = tf.reduce_sum(tf.maximum(basic_loss,0.0))\n", 263 | " ### END CODE HERE ###\n", 264 | " \n", 265 | " return loss" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": 9, 271 | "metadata": {}, 272 | "outputs": [ 273 | { 274 | "name": "stdout", 275 | "output_type": "stream", 276 | "text": [ 277 | "loss = 528.143\n" 278 | ] 279 | } 280 | ], 281 | "source": [ 282 | "with tf.Session() as test:\n", 283 | " tf.set_random_seed(1)\n", 284 | " y_true = (None, None, None)\n", 285 | " y_pred = (tf.random_normal([3, 128], mean=6, stddev=0.1, seed = 1),\n", 286 | " tf.random_normal([3, 128], mean=1, stddev=1, seed = 1),\n", 287 | " tf.random_normal([3, 128], mean=3, stddev=4, seed = 1))\n", 288 | " loss = triplet_loss(y_true, y_pred)\n", 289 | " \n", 290 | " print(\"loss = \" + str(loss.eval()))" 291 | ] 292 | }, 293 | { 294 | "cell_type": "markdown", 295 | "metadata": {}, 296 | "source": [ 297 | "**Expected Output**:\n", 298 | "\n", 299 | "\n", 300 | " \n", 301 | " \n", 304 | " \n", 307 | " \n", 308 | "\n", 309 | "
\n", 302 | " **loss**\n", 303 | " \n", 305 | " 528.143\n", 306 | "
" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "## 2 - Loading the trained model\n", 317 | "\n", 318 | "FaceNet is trained by minimizing the triplet loss. But since training requires a lot of data and a lot of computation, we won't train it from scratch here. Instead, we load a previously trained model. Load a model using the following cell; this might take a couple of minutes to run. " 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 10, 324 | "metadata": { 325 | "collapsed": true 326 | }, 327 | "outputs": [], 328 | "source": [ 329 | "FRmodel.compile(optimizer = 'adam', loss = triplet_loss, metrics = ['accuracy'])\n", 330 | "load_weights_from_FaceNet(FRmodel)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "Here're some examples of distances between the encodings between three individuals:\n", 338 | "\n", 339 | "\n", 340 | "
\n", 341 | "
**Figure 4**:
Example of distance outputs between three individuals' encodings
\n", 342 | "\n", 343 | "Let's now use this model to perform face verification and face recognition! " 344 | ] 345 | }, 346 | { 347 | "cell_type": "markdown", 348 | "metadata": {}, 349 | "source": [ 350 | "## 3 - Applying the model" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "Back to the Happy House! Residents are living blissfully since you implemented happiness recognition for the house in an earlier assignment. \n", 358 | "\n", 359 | "However, several issues keep coming up: The Happy House became so happy that every happy person in the neighborhood is coming to hang out in your living room. It is getting really crowded, which is having a negative impact on the residents of the house. All these random happy people are also eating all your food. \n", 360 | "\n", 361 | "So, you decide to change the door entry policy, and not just let random happy people enter anymore, even if they are happy! Instead, you'd like to build a **Face verification** system so as to only let people from a specified list come in. To get admitted, each person has to swipe an ID card (identification card) to identify themselves at the door. The face recognition system then checks that they are who they claim to be. " 362 | ] 363 | }, 364 | { 365 | "cell_type": "markdown", 366 | "metadata": {}, 367 | "source": [ 368 | "### 3.1 - Face Verification\n", 369 | "\n", 370 | "Let's build a database containing one encoding vector for each person allowed to enter the happy house. To generate the encoding we use `img_to_encoding(image_path, model)` which basically runs the forward propagation of the model on the specified image. \n", 371 | "\n", 372 | "Run the following code to build the database (represented as a python dictionary). This database maps each person's name to a 128-dimensional encoding of their face." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": 11, 378 | "metadata": { 379 | "collapsed": true 380 | }, 381 | "outputs": [], 382 | "source": [ 383 | "database = {}\n", 384 | "database[\"danielle\"] = img_to_encoding(\"images/danielle.png\", FRmodel)\n", 385 | "database[\"younes\"] = img_to_encoding(\"images/younes.jpg\", FRmodel)\n", 386 | "database[\"tian\"] = img_to_encoding(\"images/tian.jpg\", FRmodel)\n", 387 | "database[\"andrew\"] = img_to_encoding(\"images/andrew.jpg\", FRmodel)\n", 388 | "database[\"kian\"] = img_to_encoding(\"images/kian.jpg\", FRmodel)\n", 389 | "database[\"dan\"] = img_to_encoding(\"images/dan.jpg\", FRmodel)\n", 390 | "database[\"sebastiano\"] = img_to_encoding(\"images/sebastiano.jpg\", FRmodel)\n", 391 | "database[\"bertrand\"] = img_to_encoding(\"images/bertrand.jpg\", FRmodel)\n", 392 | "database[\"kevin\"] = img_to_encoding(\"images/kevin.jpg\", FRmodel)\n", 393 | "database[\"felix\"] = img_to_encoding(\"images/felix.jpg\", FRmodel)\n", 394 | "database[\"benoit\"] = img_to_encoding(\"images/benoit.jpg\", FRmodel)\n", 395 | "database[\"arnaud\"] = img_to_encoding(\"images/arnaud.jpg\", FRmodel)" 396 | ] 397 | }, 398 | { 399 | "cell_type": "markdown", 400 | "metadata": {}, 401 | "source": [ 402 | "Now, when someone shows up at your front door and swipes their ID card (thus giving you their name), you can look up their encoding in the database, and use it to check if the person standing at the front door matches the name on the ID.\n", 403 | "\n", 404 | "**Exercise**: Implement the verify() function which checks if the front-door camera picture (`image_path`) is actually the person called \"identity\". You will have to go through the following steps:\n", 405 | "1. Compute the encoding of the image from image_path\n", 406 | "2. Compute the distance about this encoding and the encoding of the identity image stored in the database\n", 407 | "3. Open the door if the distance is less than 0.7, else do not open.\n", 408 | "\n", 409 | "As presented above, you should use the L2 distance (np.linalg.norm). (Note: In this implementation, compare the L2 distance, not the square of the L2 distance, to the threshold 0.7.) " 410 | ] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "execution_count": 14, 415 | "metadata": { 416 | "collapsed": true 417 | }, 418 | "outputs": [], 419 | "source": [ 420 | "# GRADED FUNCTION: verify\n", 421 | "\n", 422 | "def verify(image_path, identity, database, model):\n", 423 | " \"\"\"\n", 424 | " Function that verifies if the person on the \"image_path\" image is \"identity\".\n", 425 | " \n", 426 | " Arguments:\n", 427 | " image_path -- path to an image\n", 428 | " identity -- string, name of the person you'd like to verify the identity. Has to be a resident of the Happy house.\n", 429 | " database -- python dictionary mapping names of allowed people's names (strings) to their encodings (vectors).\n", 430 | " model -- your Inception model instance in Keras\n", 431 | " \n", 432 | " Returns:\n", 433 | " dist -- distance between the image_path and the image of \"identity\" in the database.\n", 434 | " door_open -- True, if the door should open. False otherwise.\n", 435 | " \"\"\"\n", 436 | " \n", 437 | " ### START CODE HERE ###\n", 438 | " \n", 439 | " # Step 1: Compute the encoding for the image. Use img_to_encoding() see example above. (≈ 1 line)\n", 440 | " encoding = img_to_encoding(image_path, model)\n", 441 | " \n", 442 | " # Step 2: Compute distance with identity's image (≈ 1 line)\n", 443 | " # the L² norm of the difference between two vectors is equivalent to the Euclidean distance between the two points\n", 444 | " dist = np.linalg.norm(encoding - database[identity])\n", 445 | " \n", 446 | " # Step 3: Open the door if dist < 0.7, else don't open (≈ 3 lines)\n", 447 | " if dist<0.7:\n", 448 | " print(\"It's \" + str(identity) + \", welcome home!\")\n", 449 | " door_open = True\n", 450 | " else:\n", 451 | " print(\"It's not \" + str(identity) + \", please go away\")\n", 452 | " door_open = False\n", 453 | " \n", 454 | " ### END CODE HERE ###\n", 455 | " \n", 456 | " return dist, door_open" 457 | ] 458 | }, 459 | { 460 | "cell_type": "markdown", 461 | "metadata": {}, 462 | "source": [ 463 | "Younes is trying to enter the Happy House and the camera takes a picture of him (\"images/camera_0.jpg\"). Let's run your verification algorithm on this picture:\n", 464 | "\n", 465 | "" 466 | ] 467 | }, 468 | { 469 | "cell_type": "code", 470 | "execution_count": 15, 471 | "metadata": {}, 472 | "outputs": [ 473 | { 474 | "name": "stdout", 475 | "output_type": "stream", 476 | "text": [ 477 | "It's younes, welcome home!\n" 478 | ] 479 | }, 480 | { 481 | "data": { 482 | "text/plain": [ 483 | "(0.65939283, True)" 484 | ] 485 | }, 486 | "execution_count": 15, 487 | "metadata": {}, 488 | "output_type": "execute_result" 489 | } 490 | ], 491 | "source": [ 492 | "verify(\"images/camera_0.jpg\", \"younes\", database, FRmodel)" 493 | ] 494 | }, 495 | { 496 | "cell_type": "markdown", 497 | "metadata": { 498 | "collapsed": true 499 | }, 500 | "source": [ 501 | "**Expected Output**:\n", 502 | "\n", 503 | "\n", 504 | " \n", 505 | " \n", 508 | " \n", 511 | " \n", 512 | "\n", 513 | "
\n", 506 | " **It's younes, welcome home!**\n", 507 | " \n", 509 | " (0.65939283, True)\n", 510 | "
" 514 | ] 515 | }, 516 | { 517 | "cell_type": "markdown", 518 | "metadata": { 519 | "collapsed": true 520 | }, 521 | "source": [ 522 | "Benoit, who broke the aquarium last weekend, has been banned from the house and removed from the database. He stole Kian's ID card and came back to the house to try to present himself as Kian. The front-door camera took a picture of Benoit (\"images/camera_2.jpg). Let's run the verification algorithm to check if benoit can enter.\n", 523 | "" 524 | ] 525 | }, 526 | { 527 | "cell_type": "code", 528 | "execution_count": 16, 529 | "metadata": {}, 530 | "outputs": [ 531 | { 532 | "name": "stdout", 533 | "output_type": "stream", 534 | "text": [ 535 | "It's not kian, please go away\n" 536 | ] 537 | }, 538 | { 539 | "data": { 540 | "text/plain": [ 541 | "(0.86224014, False)" 542 | ] 543 | }, 544 | "execution_count": 16, 545 | "metadata": {}, 546 | "output_type": "execute_result" 547 | } 548 | ], 549 | "source": [ 550 | "verify(\"images/camera_2.jpg\", \"kian\", database, FRmodel)" 551 | ] 552 | }, 553 | { 554 | "cell_type": "markdown", 555 | "metadata": {}, 556 | "source": [ 557 | "**Expected Output**:\n", 558 | "\n", 559 | "\n", 560 | " \n", 561 | " \n", 564 | " \n", 567 | " \n", 568 | "\n", 569 | "
\n", 562 | " **It's not kian, please go away**\n", 563 | " \n", 565 | " (0.86224014, False)\n", 566 | "
" 570 | ] 571 | }, 572 | { 573 | "cell_type": "markdown", 574 | "metadata": {}, 575 | "source": [ 576 | "### 3.2 - Face Recognition\n", 577 | "\n", 578 | "Your face verification system is mostly working well. But since Kian got his ID card stolen, when he came back to the house that evening he couldn't get in! \n", 579 | "\n", 580 | "To reduce such shenanigans, you'd like to change your face verification system to a face recognition system. This way, no one has to carry an ID card anymore. An authorized person can just walk up to the house, and the front door will unlock for them! \n", 581 | "\n", 582 | "You'll implement a face recognition system that takes as input an image, and figures out if it is one of the authorized persons (and if so, who). Unlike the previous face verification system, we will no longer get a person's name as another input. \n", 583 | "\n", 584 | "**Exercise**: Implement `who_is_it()`. You will have to go through the following steps:\n", 585 | "1. Compute the target encoding of the image from image_path\n", 586 | "2. Find the encoding from the database that has smallest distance with the target encoding. \n", 587 | " - Initialize the `min_dist` variable to a large enough number (100). It will help you keep track of what is the closest encoding to the input's encoding.\n", 588 | " - Loop over the database dictionary's names and encodings. To loop use `for (name, db_enc) in database.items()`.\n", 589 | " - Compute L2 distance between the target \"encoding\" and the current \"encoding\" from the database.\n", 590 | " - If this distance is less than the min_dist, then set min_dist to dist, and identity to name." 591 | ] 592 | }, 593 | { 594 | "cell_type": "code", 595 | "execution_count": 21, 596 | "metadata": { 597 | "collapsed": true 598 | }, 599 | "outputs": [], 600 | "source": [ 601 | "# GRADED FUNCTION: who_is_it\n", 602 | "\n", 603 | "def who_is_it(image_path, database, model):\n", 604 | " \"\"\"\n", 605 | " Implements face recognition for the happy house by finding who is the person on the image_path image.\n", 606 | " \n", 607 | " Arguments:\n", 608 | " image_path -- path to an image\n", 609 | " database -- database containing image encodings along with the name of the person on the image\n", 610 | " model -- your Inception model instance in Keras\n", 611 | " \n", 612 | " Returns:\n", 613 | " min_dist -- the minimum distance between image_path encoding and the encodings from the database\n", 614 | " identity -- string, the name prediction for the person on image_path\n", 615 | " \"\"\"\n", 616 | " \n", 617 | " ### START CODE HERE ### \n", 618 | " \n", 619 | " ## Step 1: Compute the target \"encoding\" for the image. Use img_to_encoding() see example above. ## (≈ 1 line)\n", 620 | " encoding = img_to_encoding(image_path, model)\n", 621 | " \n", 622 | " ## Step 2: Find the closest encoding ##\n", 623 | " \n", 624 | " # Initialize \"min_dist\" to a large value, say 100 (≈1 line)\n", 625 | " min_dist = 100\n", 626 | " \n", 627 | " # Loop over the database dictionary's names and encodings.\n", 628 | " for (name, db_enc) in database.items():\n", 629 | " \n", 630 | " # Compute L2 distance between the target \"encoding\" and the current \"emb\" from the database. (≈ 1 line)\n", 631 | " dist = np.linalg.norm(encoding - db_enc)\n", 632 | "\n", 633 | " # If this distance is less than the min_dist, then set min_dist to dist, and identity to name. (≈ 3 lines)\n", 634 | " if dist < min_dist:\n", 635 | " min_dist = dist\n", 636 | " identity = name \n", 637 | "\n", 638 | " ### END CODE HERE ###\n", 639 | " \n", 640 | " if min_dist > 0.7:\n", 641 | " print(\"Not in the database.\")\n", 642 | " else:\n", 643 | " print (\"it's \" + str(identity) + \", the distance is \" + str(min_dist))\n", 644 | " \n", 645 | " return min_dist, identity" 646 | ] 647 | }, 648 | { 649 | "cell_type": "markdown", 650 | "metadata": {}, 651 | "source": [ 652 | "Younes is at the front-door and the camera takes a picture of him (\"images/camera_0.jpg\"). Let's see if your who_it_is() algorithm identifies Younes. " 653 | ] 654 | }, 655 | { 656 | "cell_type": "code", 657 | "execution_count": 22, 658 | "metadata": { 659 | "scrolled": false 660 | }, 661 | "outputs": [ 662 | { 663 | "name": "stdout", 664 | "output_type": "stream", 665 | "text": [ 666 | "it's younes, the distance is 0.659393\n" 667 | ] 668 | }, 669 | { 670 | "data": { 671 | "text/plain": [ 672 | "(0.65939283, 'younes')" 673 | ] 674 | }, 675 | "execution_count": 22, 676 | "metadata": {}, 677 | "output_type": "execute_result" 678 | } 679 | ], 680 | "source": [ 681 | "who_is_it(\"images/camera_0.jpg\", database, FRmodel)" 682 | ] 683 | }, 684 | { 685 | "cell_type": "markdown", 686 | "metadata": {}, 687 | "source": [ 688 | "**Expected Output**:\n", 689 | "\n", 690 | "\n", 691 | " \n", 692 | " \n", 695 | " \n", 698 | " \n", 699 | "\n", 700 | "
\n", 693 | " **it's younes, the distance is 0.659393**\n", 694 | " \n", 696 | " (0.65939283, 'younes')\n", 697 | "
" 701 | ] 702 | }, 703 | { 704 | "cell_type": "markdown", 705 | "metadata": {}, 706 | "source": [ 707 | "You can change \"`camera_0.jpg`\" (picture of younes) to \"`camera_1.jpg`\" (picture of bertrand) and see the result." 708 | ] 709 | }, 710 | { 711 | "cell_type": "markdown", 712 | "metadata": {}, 713 | "source": [ 714 | "Your Happy House is running well. It only lets in authorized persons, and people don't need to carry an ID card around anymore! \n", 715 | "\n", 716 | "You've now seen how a state-of-the-art face recognition system works.\n", 717 | "\n", 718 | "Although we won't implement it here, here're some ways to further improve the algorithm:\n", 719 | "- Put more images of each person (under different lighting conditions, taken on different days, etc.) into the database. Then given a new image, compare the new face to multiple pictures of the person. This would increae accuracy.\n", 720 | "- Crop the images to just contain the face, and less of the \"border\" region around the face. This preprocessing removes some of the irrelevant pixels around the face, and also makes the algorithm more robust.\n" 721 | ] 722 | }, 723 | { 724 | "cell_type": "markdown", 725 | "metadata": {}, 726 | "source": [ 727 | "\n", 728 | "**What you should remember**:\n", 729 | "- Face verification solves an easier 1:1 matching problem; face recognition addresses a harder 1:K matching problem. \n", 730 | "- The triplet loss is an effective loss function for training a neural network to learn an encoding of a face image.\n", 731 | "- The same encoding can be used for verification and recognition. Measuring distances between two images' encodings allows you to determine whether they are pictures of the same person. " 732 | ] 733 | }, 734 | { 735 | "cell_type": "markdown", 736 | "metadata": {}, 737 | "source": [ 738 | "Congrats on finishing this assignment! \n" 739 | ] 740 | }, 741 | { 742 | "cell_type": "markdown", 743 | "metadata": {}, 744 | "source": [ 745 | "### References:\n", 746 | "\n", 747 | "- Florian Schroff, Dmitry Kalenichenko, James Philbin (2015). [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/pdf/1503.03832.pdf)\n", 748 | "- Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, Lior Wolf (2014). [DeepFace: Closing the gap to human-level performance in face verification](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf) \n", 749 | "- The pretrained model we use is inspired by Victor Sy Wang's implementation and was loaded using his code: https://github.com/iwantooxxoox/Keras-OpenFace.\n", 750 | "- Our implementation also took a lot of inspiration from the official FaceNet github repository: https://github.com/davidsandberg/facenet \n" 751 | ] 752 | } 753 | ], 754 | "metadata": { 755 | "coursera": { 756 | "course_slug": "convolutional-neural-networks", 757 | "graded_item_id": "IaknP", 758 | "launcher_item_id": "5UMr4" 759 | }, 760 | "kernelspec": { 761 | "display_name": "Python 3", 762 | "language": "python", 763 | "name": "python3" 764 | }, 765 | "language_info": { 766 | "codemirror_mode": { 767 | "name": "ipython", 768 | "version": 3 769 | }, 770 | "file_extension": ".py", 771 | "mimetype": "text/x-python", 772 | "name": "python", 773 | "nbconvert_exporter": "python", 774 | "pygments_lexer": "ipython3", 775 | "version": "3.6.0" 776 | } 777 | }, 778 | "nbformat": 4, 779 | "nbformat_minor": 2 780 | } 781 | -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/README.md: -------------------------------------------------------------------------------- 1 | ## Objectives: 2 | - Learn to implement the foundational layers of CNNs (pooling, convolutions) and to stack them properly in a deep network to solve multi-class image classification problems. 3 | - Learn about the practical tricks and methods used in deep CNNs straight from the research papers. 4 | - Learn how to apply your knowledge of CNNs to one of the toughest but hottest field of computer vision: Object detection. 5 | - Discover how CNNs can be applied to multiple fields, including art generation and face recognition. Implement your own algorithm to generate art and recognize faces! 6 | 7 | ## Week 1: 8 | - Convolution model - Step by Step - v2.ipynb 9 | - Convolution model - Application - v1.ipynb 10 | 11 | ## Week 2: 12 | - Keras - Tutorial - Happy House v2.ipynb 13 | - Residual Networks - v2.ipynb 14 | 15 | ## Week 3: 16 | - Autonomous driving application - Car detection - v1.ipynb 17 | 18 | ## Week 4: 19 | - Face Recognition for the Happy House - v3.ipynb 20 | - Art Generation with Neural Style Transfer - v2.ipynb 21 | 22 | -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/Convolution_schematic.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/Convolution_schematic.gif -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/GYZ_066.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/GYZ_066.jpeg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/GYZ_067.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/GYZ_067.jpeg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/NST_GM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/NST_GM.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/NST_LOSS.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/NST_LOSS.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/PAD.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/PAD.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/SIGNS.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/SIGNS.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/a_pool.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/a_pool.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/andrew.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/andrew.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/arnaud.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/arnaud.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/ave-pool.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/ave-pool.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/ave_pool1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/ave_pool1.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/average_pool.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/average_pool.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/benoit.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/benoit.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/bertrand.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/bertrand.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/camera_0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/camera_0.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/camera_1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/camera_1.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/camera_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/camera_2.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/camera_3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/camera_3.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/camera_4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/camera_4.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/camera_5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/camera_5.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/camp-nou.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/camp-nou.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/cat.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/cat.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/circle_abstract.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/circle_abstract.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/claude-monet.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/claude-monet.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/content.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/content.jpeg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/content300.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/content300.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/content_plus_style.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/content_plus_style.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/conv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/conv.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/conv1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/conv1.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/conv_kiank.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/conv_kiank.mp4 -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/conv_nn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/conv_nn.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/convblock_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/convblock_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/dan.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/dan.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/danielle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/danielle.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/distance_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/distance_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/distance_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/distance_matrix.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/drop-of-water.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/drop-of-water.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/f_x.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/f_x.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/felix.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/felix.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/gram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/gram.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/happy-house.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/happy-house.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/hidden_layers.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/hidden_layers.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/house-members.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/house-members.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/idblock2_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/idblock2_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/idblock3_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/idblock3_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/inception_block1a.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/inception_block1a.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/kevin.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/kevin.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/kian.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/kian.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/louvre.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/louvre.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/louvre_generated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/louvre_generated.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/louvre_small.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/louvre_small.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/max_pool.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/max_pool.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/max_pool1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/max_pool1.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/model.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/monet.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/monet.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/monet_800600.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/monet_800600.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/my_image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/my_image.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/pasargad_kashi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/pasargad_kashi.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/persian_cat.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/persian_cat.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/persian_cat_content.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/persian_cat_content.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/perspolis_vangogh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/perspolis_vangogh.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/pixel_comparison.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/pixel_comparison.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/reshape_loss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/reshape_loss.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/resnet_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/resnet_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/result.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/sandstone.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/sandstone.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/sebastiano.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/sebastiano.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/signs_data_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/signs_data_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/skip_connection_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/skip_connection_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/stone_style.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/stone_style.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/style300.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/style300.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/thumbs_up.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/thumbs_up.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/tian.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/tian.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/triplet_comparison.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/triplet_comparison.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/vanishing_grad_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/vanishing_grad_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/vert_horiz_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/vert_horiz_kiank.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/images/younes.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/images/younes.jpg -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/anchor_map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/anchor_map.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/architecture.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/box_label.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/box_label.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/driveai.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/driveai.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/flatten.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/flatten.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/index.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/iou.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/iou.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/non-max-suppression.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/non-max-suppression.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/pred_video_compressed2.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/pred_video_compressed2.mp4 -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/proba_map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/proba_map.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/probability_extraction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/probability_extraction.png -------------------------------------------------------------------------------- /4. Convolutional Neural Networks/nb_images/road_video_compressed2.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/4. Convolutional Neural Networks/nb_images/road_video_compressed2.mp4 -------------------------------------------------------------------------------- /5. Sequence Models/Operations on word vectors - v2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Operations on word vectors\n", 8 | "\n", 9 | "Welcome to your first assignment of this week! \n", 10 | "\n", 11 | "Because word embeddings are very computionally expensive to train, most ML practitioners will load a pre-trained set of embeddings. \n", 12 | "\n", 13 | "**After this assignment you will be able to:**\n", 14 | "\n", 15 | "- Load pre-trained word vectors, and measure similarity using cosine similarity\n", 16 | "- Use word embeddings to solve word analogy problems such as Man is to Woman as King is to ______. \n", 17 | "- Modify word embeddings to reduce their gender bias \n", 18 | "\n", 19 | "Let's get started! Run the following cell to load the packages you will need." 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 1, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stderr", 29 | "output_type": "stream", 30 | "text": [ 31 | "Using TensorFlow backend.\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "import numpy as np\n", 37 | "from w2v_utils import *" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "Next, lets load the word vectors. For this assignment, we will use 50-dimensional GloVe vectors to represent words. Run the following cell to load the `word_to_vec_map`. " 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 2, 50 | "metadata": { 51 | "collapsed": true 52 | }, 53 | "outputs": [], 54 | "source": [ 55 | "words, word_to_vec_map = read_glove_vecs('data/glove.6B.50d.txt')" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "You've loaded:\n", 63 | "- `words`: set of words in the vocabulary.\n", 64 | "- `word_to_vec_map`: dictionary mapping words to their GloVe vector representation.\n", 65 | "\n", 66 | "You've seen that one-hot vectors do not do a good job cpaturing what words are similar. GloVe vectors provide much more useful information about the meaning of individual words. Lets now see how you can use GloVe vectors to decide how similar two words are. \n", 67 | "\n" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "# 1 - Cosine similarity\n", 75 | "\n", 76 | "To measure how similar two words are, we need a way to measure the degree of similarity between two embedding vectors for the two words. Given two vectors $u$ and $v$, cosine similarity is defined as follows: \n", 77 | "\n", 78 | "$$\\text{CosineSimilarity(u, v)} = \\frac {u . v} {||u||_2 ||v||_2} = cos(\\theta) \\tag{1}$$\n", 79 | "\n", 80 | "where $u.v$ is the dot product (or inner product) of two vectors, $||u||_2$ is the norm (or length) of the vector $u$, and $\\theta$ is the angle between $u$ and $v$. This similarity depends on the angle between $u$ and $v$. If $u$ and $v$ are very similar, their cosine similarity will be close to 1; if they are dissimilar, the cosine similarity will take a smaller value. \n", 81 | "\n", 82 | "\n", 83 | "
**Figure 1**: The cosine of the angle between two vectors is a measure of how similar they are
\n", 84 | "\n", 85 | "**Exercise**: Implement the function `cosine_similarity()` to evaluate similarity between word vectors.\n", 86 | "\n", 87 | "**Reminder**: The norm of $u$ is defined as $ ||u||_2 = \\sqrt{\\sum_{i=1}^{n} u_i^2}$" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 3, 93 | "metadata": { 94 | "collapsed": true 95 | }, 96 | "outputs": [], 97 | "source": [ 98 | "# GRADED FUNCTION: cosine_similarity\n", 99 | "\n", 100 | "def cosine_similarity(u, v):\n", 101 | " \"\"\"\n", 102 | " Cosine similarity reflects the degree of similariy between u and v\n", 103 | " \n", 104 | " Arguments:\n", 105 | " u -- a word vector of shape (n,) \n", 106 | " v -- a word vector of shape (n,)\n", 107 | "\n", 108 | " Returns:\n", 109 | " cosine_similarity -- the cosine similarity between u and v defined by the formula above.\n", 110 | " \"\"\"\n", 111 | " \n", 112 | " distance = 0.0\n", 113 | " \n", 114 | " ### START CODE HERE ###\n", 115 | " # Compute the dot product between u and v (≈1 line)\n", 116 | " dot = np.dot(u,v)\n", 117 | " # Compute the L2 norm of u (≈1 line)\n", 118 | " norm_u = np.linalg.norm(u)\n", 119 | " \n", 120 | " # Compute the L2 norm of v (≈1 line)\n", 121 | " norm_v = np.linalg.norm(v)\n", 122 | " # Compute the cosine similarity defined by formula (1) (≈1 line)\n", 123 | " cosine_similarity = dot/(norm_u*norm_v)\n", 124 | " ### END CODE HERE ###\n", 125 | " \n", 126 | " return cosine_similarity" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 4, 132 | "metadata": {}, 133 | "outputs": [ 134 | { 135 | "name": "stdout", 136 | "output_type": "stream", 137 | "text": [ 138 | "cosine_similarity(father, mother) = 0.890903844289\n", 139 | "cosine_similarity(ball, crocodile) = 0.274392462614\n", 140 | "cosine_similarity(france - paris, rome - italy) = -0.675147930817\n" 141 | ] 142 | } 143 | ], 144 | "source": [ 145 | "father = word_to_vec_map[\"father\"]\n", 146 | "mother = word_to_vec_map[\"mother\"]\n", 147 | "ball = word_to_vec_map[\"ball\"]\n", 148 | "crocodile = word_to_vec_map[\"crocodile\"]\n", 149 | "france = word_to_vec_map[\"france\"]\n", 150 | "italy = word_to_vec_map[\"italy\"]\n", 151 | "paris = word_to_vec_map[\"paris\"]\n", 152 | "rome = word_to_vec_map[\"rome\"]\n", 153 | "\n", 154 | "print(\"cosine_similarity(father, mother) = \", cosine_similarity(father, mother))\n", 155 | "print(\"cosine_similarity(ball, crocodile) = \",cosine_similarity(ball, crocodile))\n", 156 | "print(\"cosine_similarity(france - paris, rome - italy) = \",cosine_similarity(france - paris, rome - italy))" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": {}, 162 | "source": [ 163 | "**Expected Output**:\n", 164 | "\n", 165 | "\n", 166 | " \n", 167 | " \n", 170 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 178 | " \n", 181 | " \n", 182 | " \n", 183 | " \n", 186 | " \n", 189 | " \n", 190 | "
\n", 168 | " **cosine_similarity(father, mother)** =\n", 169 | " \n", 171 | " 0.890903844289\n", 172 | "
\n", 176 | " **cosine_similarity(ball, crocodile)** =\n", 177 | " \n", 179 | " 0.274392462614\n", 180 | "
\n", 184 | " **cosine_similarity(france - paris, rome - italy)** =\n", 185 | " \n", 187 | " -0.675147930817\n", 188 | "
" 191 | ] 192 | }, 193 | { 194 | "cell_type": "markdown", 195 | "metadata": {}, 196 | "source": [ 197 | "After you get the correct expected output, please feel free to modify the inputs and measure the cosine similarity between other pairs of words! Playing around the cosine similarity of other inputs will give you a better sense of how word vectors behave. " 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": {}, 203 | "source": [ 204 | "## 2 - Word analogy task\n", 205 | "\n", 206 | "In the word analogy task, we complete the sentence \"*a* is to *b* as *c* is to **____**\". An example is '*man* is to *woman* as *king* is to *queen*' . In detail, we are trying to find a word *d*, such that the associated word vectors $e_a, e_b, e_c, e_d$ are related in the following manner: $e_b - e_a \\approx e_d - e_c$. We will measure the similarity between $e_b - e_a$ and $e_d - e_c$ using cosine similarity. \n", 207 | "\n", 208 | "**Exercise**: Complete the code below to be able to perform word analogies!" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 17, 214 | "metadata": { 215 | "collapsed": true 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "# GRADED FUNCTION: complete_analogy\n", 220 | "\n", 221 | "def complete_analogy(word_a, word_b, word_c, word_to_vec_map):\n", 222 | " \"\"\"\n", 223 | " Performs the word analogy task as explained above: a is to b as c is to ____. \n", 224 | " \n", 225 | " Arguments:\n", 226 | " word_a -- a word, string\n", 227 | " word_b -- a word, string\n", 228 | " word_c -- a word, string\n", 229 | " word_to_vec_map -- dictionary that maps words to their corresponding vectors. \n", 230 | " \n", 231 | " Returns:\n", 232 | " best_word -- the word such that v_b - v_a is close to v_best_word - v_c, as measured by cosine similarity\n", 233 | " \"\"\"\n", 234 | " \n", 235 | " # convert words to lower case\n", 236 | " word_a, word_b, word_c = word_a.lower(), word_b.lower(), word_c.lower()\n", 237 | " \n", 238 | " ### START CODE HERE ###\n", 239 | " # Get the word embeddings v_a, v_b and v_c (≈1-3 lines)\n", 240 | " e_a = word_to_vec_map.get(word_a)\n", 241 | " e_b = word_to_vec_map.get(word_b)\n", 242 | " e_c = word_to_vec_map.get(word_c)\n", 243 | " # e_a, e_b, e_c = word_to_vec_map[word_a], word_to_vec_map[word_b], word_to_vec_map[word_c]\n", 244 | " ### END CODE HERE ###\n", 245 | " \n", 246 | " words = word_to_vec_map.keys()\n", 247 | " max_cosine_sim = -100 # Initialize max_cosine_sim to a large negative number\n", 248 | " best_word = None # Initialize best_word with None, it will help keep track of the word to output\n", 249 | "\n", 250 | " # loop over the whole word vector set\n", 251 | " for w in words: \n", 252 | " # to avoid best_word being one of the input words, pass on them.\n", 253 | " if w in [word_a, word_b, word_c] :\n", 254 | " continue\n", 255 | " \n", 256 | " ### START CODE HERE ###\n", 257 | " # Compute cosine similarity between the vector (e_b - e_a) and the vector ((w's vector representation) - e_c) (≈1 line)\n", 258 | " cosine_sim = cosine_similarity(np.subtract(e_b,e_a), np.subtract(word_to_vec_map.get(w),e_c))\n", 259 | " \n", 260 | " # If the cosine_sim is more than the max_cosine_sim seen so far,\n", 261 | " # then: set the new max_cosine_sim to the current cosine_sim and the best_word to the current word (≈3 lines)\n", 262 | " if cosine_sim > max_cosine_sim:\n", 263 | " max_cosine_sim = cosine_sim\n", 264 | " best_word = w\n", 265 | " ### END CODE HERE ###\n", 266 | " \n", 267 | " return best_word" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "Run the cell below to test your code, this may take 1-2 minutes." 275 | ] 276 | }, 277 | { 278 | "cell_type": "code", 279 | "execution_count": 18, 280 | "metadata": {}, 281 | "outputs": [ 282 | { 283 | "name": "stdout", 284 | "output_type": "stream", 285 | "text": [ 286 | "italy -> italian :: spain -> spanish\n", 287 | "india -> delhi :: japan -> tokyo\n", 288 | "man -> woman :: boy -> girl\n", 289 | "small -> smaller :: large -> larger\n" 290 | ] 291 | } 292 | ], 293 | "source": [ 294 | "triads_to_try = [('italy', 'italian', 'spain'), ('india', 'delhi', 'japan'), ('man', 'woman', 'boy'), ('small', 'smaller', 'large')]\n", 295 | "for triad in triads_to_try:\n", 296 | " print ('{} -> {} :: {} -> {}'.format( *triad, complete_analogy(*triad,word_to_vec_map)))" 297 | ] 298 | }, 299 | { 300 | "cell_type": "markdown", 301 | "metadata": {}, 302 | "source": [ 303 | "**Expected Output**:\n", 304 | "\n", 305 | "\n", 306 | " \n", 307 | " \n", 310 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 318 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 326 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 334 | " \n", 337 | " \n", 338 | "
\n", 308 | " **italy -> italian** ::\n", 309 | " \n", 311 | " spain -> spanish\n", 312 | "
\n", 316 | " **india -> delhi** ::\n", 317 | " \n", 319 | " japan -> tokyo\n", 320 | "
\n", 324 | " **man -> woman ** ::\n", 325 | " \n", 327 | " boy -> girl\n", 328 | "
\n", 332 | " **small -> smaller ** ::\n", 333 | " \n", 335 | " large -> larger\n", 336 | "
" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": {}, 344 | "source": [ 345 | "Once you get the correct expected output, please feel free to modify the input cells above to test your own analogies. Try to find some other analogy pairs that do work, but also find some where the algorithm doesn't give the right answer: For example, you can try small->smaller as big->?. " 346 | ] 347 | }, 348 | { 349 | "cell_type": "markdown", 350 | "metadata": {}, 351 | "source": [ 352 | "### Congratulations!\n", 353 | "\n", 354 | "You've come to the end of this assignment. Here are the main points you should remember:\n", 355 | "\n", 356 | "- Cosine similarity a good way to compare similarity between pairs of word vectors. (Though L2 distance works too.) \n", 357 | "- For NLP applications, using a pre-trained set of word vectors from the internet is often a good way to get started. \n", 358 | "\n", 359 | "Even though you have finished the graded portions, we recommend you take a look too at the rest of this notebook. \n", 360 | "\n", 361 | "Congratulations on finishing the graded portions of this notebook! \n" 362 | ] 363 | }, 364 | { 365 | "cell_type": "markdown", 366 | "metadata": {}, 367 | "source": [ 368 | "## 3 - Debiasing word vectors (OPTIONAL/UNGRADED) " 369 | ] 370 | }, 371 | { 372 | "cell_type": "markdown", 373 | "metadata": {}, 374 | "source": [ 375 | "In the following exercise, you will examine gender biases that can be reflected in a word embedding, and explore algorithms for reducing the bias. In addition to learning about the topic of debiasing, this exercise will also help hone your intuition about what word vectors are doing. This section involves a bit of linear algebra, though you can probably complete it even without being expert in linear algebra, and we encourage you to give it a shot. This portion of the notebook is optional and is not graded. \n", 376 | "\n", 377 | "Lets first see how the GloVe word embeddings relate to gender. You will first compute a vector $g = e_{woman}-e_{man}$, where $e_{woman}$ represents the word vector corresponding to the word *woman*, and $e_{man}$ corresponds to the word vector corresponding to the word *man*. The resulting vector $g$ roughly encodes the concept of \"gender\". (You might get a more accurate representation if you compute $g_1 = e_{mother}-e_{father}$, $g_2 = e_{girl}-e_{boy}$, etc. and average over them. But just using $e_{woman}-e_{man}$ will give good enough results for now.) \n" 378 | ] 379 | }, 380 | { 381 | "cell_type": "code", 382 | "execution_count": 19, 383 | "metadata": {}, 384 | "outputs": [ 385 | { 386 | "name": "stdout", 387 | "output_type": "stream", 388 | "text": [ 389 | "[-0.087144 0.2182 -0.40986 -0.03922 -0.1032 0.94165\n", 390 | " -0.06042 0.32988 0.46144 -0.35962 0.31102 -0.86824\n", 391 | " 0.96006 0.01073 0.24337 0.08193 -1.02722 -0.21122\n", 392 | " 0.695044 -0.00222 0.29106 0.5053 -0.099454 0.40445\n", 393 | " 0.30181 0.1355 -0.0606 -0.07131 -0.19245 -0.06115\n", 394 | " -0.3204 0.07165 -0.13337 -0.25068714 -0.14293 -0.224957\n", 395 | " -0.149 0.048882 0.12191 -0.27362 -0.165476 -0.20426\n", 396 | " 0.54376 -0.271425 -0.10245 -0.32108 0.2516 -0.33455\n", 397 | " -0.04371 0.01258 ]\n" 398 | ] 399 | } 400 | ], 401 | "source": [ 402 | "g = word_to_vec_map['woman'] - word_to_vec_map['man']\n", 403 | "print(g)" 404 | ] 405 | }, 406 | { 407 | "cell_type": "markdown", 408 | "metadata": {}, 409 | "source": [ 410 | "Now, you will consider the cosine similarity of different words with $g$. Consider what a positive value of similarity means vs a negative cosine similarity. " 411 | ] 412 | }, 413 | { 414 | "cell_type": "code", 415 | "execution_count": 20, 416 | "metadata": { 417 | "scrolled": false 418 | }, 419 | "outputs": [ 420 | { 421 | "name": "stdout", 422 | "output_type": "stream", 423 | "text": [ 424 | "List of names and their similarities with constructed vector:\n", 425 | "john -0.23163356146\n", 426 | "marie 0.315597935396\n", 427 | "sophie 0.318687898594\n", 428 | "ronaldo -0.312447968503\n", 429 | "priya 0.17632041839\n", 430 | "rahul -0.169154710392\n", 431 | "danielle 0.243932992163\n", 432 | "reza -0.079304296722\n", 433 | "katy 0.283106865957\n", 434 | "yasmin 0.233138577679\n" 435 | ] 436 | } 437 | ], 438 | "source": [ 439 | "print ('List of names and their similarities with constructed vector:')\n", 440 | "\n", 441 | "# girls and boys name\n", 442 | "name_list = ['john', 'marie', 'sophie', 'ronaldo', 'priya', 'rahul', 'danielle', 'reza', 'katy', 'yasmin']\n", 443 | "\n", 444 | "for w in name_list:\n", 445 | " print (w, cosine_similarity(word_to_vec_map[w], g))" 446 | ] 447 | }, 448 | { 449 | "cell_type": "markdown", 450 | "metadata": {}, 451 | "source": [ 452 | "As you can see, female first names tend to have a positive cosine similarity with our constructed vector $g$, while male first names tend to have a negative cosine similarity. This is not suprising, and the result seems acceptable. \n", 453 | "\n", 454 | "But let's try with some other words." 455 | ] 456 | }, 457 | { 458 | "cell_type": "code", 459 | "execution_count": 21, 460 | "metadata": {}, 461 | "outputs": [ 462 | { 463 | "name": "stdout", 464 | "output_type": "stream", 465 | "text": [ 466 | "Other words and their similarities:\n", 467 | "lipstick 0.276919162564\n", 468 | "guns -0.18884855679\n", 469 | "science -0.0608290654093\n", 470 | "arts 0.00818931238588\n", 471 | "literature 0.0647250443346\n", 472 | "warrior -0.209201646411\n", 473 | "doctor 0.118952894109\n", 474 | "tree -0.0708939917548\n", 475 | "receptionist 0.330779417506\n", 476 | "technology -0.131937324476\n", 477 | "fashion 0.0356389462577\n", 478 | "teacher 0.179209234318\n", 479 | "engineer -0.0803928049452\n", 480 | "pilot 0.00107644989919\n", 481 | "computer -0.103303588739\n", 482 | "singer 0.185005181365\n" 483 | ] 484 | } 485 | ], 486 | "source": [ 487 | "print('Other words and their similarities:')\n", 488 | "word_list = ['lipstick', 'guns', 'science', 'arts', 'literature', 'warrior','doctor', 'tree', 'receptionist', \n", 489 | " 'technology', 'fashion', 'teacher', 'engineer', 'pilot', 'computer', 'singer']\n", 490 | "for w in word_list:\n", 491 | " print (w, cosine_similarity(word_to_vec_map[w], g))" 492 | ] 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "metadata": {}, 497 | "source": [ 498 | "Do you notice anything surprising? It is astonishing how these results reflect certain unhealthy gender stereotypes. For example, \"computer\" is closer to \"man\" while \"literature\" is closer to \"woman\". Ouch! \n", 499 | "\n", 500 | "We'll see below how to reduce the bias of these vectors, using an algorithm due to [Boliukbasi et al., 2016](https://arxiv.org/abs/1607.06520). Note that some word pairs such as \"actor\"/\"actress\" or \"grandmother\"/\"grandfather\" should remain gender specific, while other words such as \"receptionist\" or \"technology\" should be neutralized, i.e. not be gender-related. You will have to treat these two type of words differently when debiasing.\n", 501 | "\n", 502 | "### 3.1 - Neutralize bias for non-gender specific words \n", 503 | "\n", 504 | "The figure below should help you visualize what neutralizing does. If you're using a 50-dimensional word embedding, the 50 dimensional space can be split into two parts: The bias-direction $g$, and the remaining 49 dimensions, which we'll call $g_{\\perp}$. In linear algebra, we say that the 49 dimensional $g_{\\perp}$ is perpendicular (or \"othogonal\") to $g$, meaning it is at 90 degrees to $g$. The neutralization step takes a vector such as $e_{receptionist}$ and zeros out the component in the direction of $g$, giving us $e_{receptionist}^{debiased}$. \n", 505 | "\n", 506 | "Even though $g_{\\perp}$ is 49 dimensional, given the limitations of what we can draw on a screen, we illustrate it using a 1 dimensional axis below. \n", 507 | "\n", 508 | "\n", 509 | "
**Figure 2**: The word vector for \"receptionist\" represented before and after applying the neutralize operation.
\n", 510 | "\n", 511 | "**Exercise**: Implement `neutralize()` to remove the bias of words such as \"receptionist\" or \"scientist\". Given an input embedding $e$, you can use the following formulas to compute $e^{debiased}$: \n", 512 | "\n", 513 | "$$e^{bias\\_component} = \\frac{e \\cdot g}{||g||_2^2} * g\\tag{2}$$\n", 514 | "$$e^{debiased} = e - e^{bias\\_component}\\tag{3}$$\n", 515 | "\n", 516 | "If you are an expert in linear algebra, you may recognize $e^{bias\\_component}$ as the projection of $e$ onto the direction $g$. If you're not an expert in linear algebra, don't worry about this.\n", 517 | "\n", 518 | " " 523 | ] 524 | }, 525 | { 526 | "cell_type": "code", 527 | "execution_count": 24, 528 | "metadata": { 529 | "collapsed": true 530 | }, 531 | "outputs": [], 532 | "source": [ 533 | "def neutralize(word, g, word_to_vec_map):\n", 534 | " \"\"\"\n", 535 | " Removes the bias of \"word\" by projecting it on the space orthogonal to the bias axis. \n", 536 | " This function ensures that gender neutral words are zero in the gender subspace.\n", 537 | " \n", 538 | " Arguments:\n", 539 | " word -- string indicating the word to debias\n", 540 | " g -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)\n", 541 | " word_to_vec_map -- dictionary mapping words to their corresponding vectors.\n", 542 | " \n", 543 | " Returns:\n", 544 | " e_debiased -- neutralized word vector representation of the input \"word\"\n", 545 | " \"\"\"\n", 546 | " \n", 547 | " ### START CODE HERE ###\n", 548 | " # Select word vector representation of \"word\". Use word_to_vec_map. (≈ 1 line)\n", 549 | " e = word_to_vec_map[word]\n", 550 | " \n", 551 | " # Compute e_biascomponent using the formula give above. (≈ 1 line)\n", 552 | " e_biascomponent = (np.dot(e,g)/np.linalg.norm(g)**2)*g\n", 553 | " \n", 554 | " # Neutralize e by substracting e_biascomponent from it \n", 555 | " # e_debiased should be equal to its orthogonal projection. (≈ 1 line)\n", 556 | " e_debiased = e-e_biascomponent\n", 557 | " ### END CODE HERE ###\n", 558 | " \n", 559 | " return e_debiased" 560 | ] 561 | }, 562 | { 563 | "cell_type": "code", 564 | "execution_count": 25, 565 | "metadata": {}, 566 | "outputs": [ 567 | { 568 | "name": "stdout", 569 | "output_type": "stream", 570 | "text": [ 571 | "cosine similarity between receptionist and g, before neutralizing: 0.330779417506\n", 572 | "cosine similarity between receptionist and g, after neutralizing: -3.26732746085e-17\n" 573 | ] 574 | } 575 | ], 576 | "source": [ 577 | "e = \"receptionist\"\n", 578 | "print(\"cosine similarity between \" + e + \" and g, before neutralizing: \", cosine_similarity(word_to_vec_map[\"receptionist\"], g))\n", 579 | "\n", 580 | "e_debiased = neutralize(\"receptionist\", g, word_to_vec_map)\n", 581 | "print(\"cosine similarity between \" + e + \" and g, after neutralizing: \", cosine_similarity(e_debiased, g))" 582 | ] 583 | }, 584 | { 585 | "cell_type": "markdown", 586 | "metadata": {}, 587 | "source": [ 588 | "**Expected Output**: The second result is essentially 0, up to numerical roundof (on the order of $10^{-17}$).\n", 589 | "\n", 590 | "\n", 591 | "\n", 592 | " \n", 593 | " \n", 596 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 604 | " \n", 607 | "
\n", 594 | " **cosine similarity between receptionist and g, before neutralizing:** :\n", 595 | " \n", 597 | " 0.330779417506\n", 598 | "
\n", 602 | " **cosine similarity between receptionist and g, after neutralizing:** :\n", 603 | " \n", 605 | " -3.26732746085e-17\n", 606 | "
" 608 | ] 609 | }, 610 | { 611 | "cell_type": "markdown", 612 | "metadata": {}, 613 | "source": [ 614 | "### 3.2 - Equalization algorithm for gender-specific words\n", 615 | "\n", 616 | "Next, lets see how debiasing can also be applied to word pairs such as \"actress\" and \"actor.\" Equalization is applied to pairs of words that you might want to have differ only through the gender property. As a concrete example, suppose that \"actress\" is closer to \"babysit\" than \"actor.\" By applying neutralizing to \"babysit\" we can reduce the gender-stereotype associated with babysitting. But this still does not guarantee that \"actor\" and \"actress\" are equidistant from \"babysit.\" The equalization algorithm takes care of this. \n", 617 | "\n", 618 | "The key idea behind equalization is to make sure that a particular pair of words are equi-distant from the 49-dimensional $g_\\perp$. The equalization step also ensures that the two equalized steps are now the same distance from $e_{receptionist}^{debiased}$, or from any other work that has been neutralized. In pictures, this is how equalization works: \n", 619 | "\n", 620 | "\n", 621 | "\n", 622 | "\n", 623 | "The derivation of the linear algebra to do this is a bit more complex. (See Bolukbasi et al., 2016 for details.) But the key equations are: \n", 624 | "\n", 625 | "$$ \\mu = \\frac{e_{w1} + e_{w2}}{2}\\tag{4}$$ \n", 626 | "\n", 627 | "$$ \\mu_{B} = \\frac {\\mu \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n", 628 | "\\tag{5}$$ \n", 629 | "\n", 630 | "$$\\mu_{\\perp} = \\mu - \\mu_{B} \\tag{6}$$\n", 631 | "\n", 632 | "$$ e_{w1B} = \\frac {e_{w1} \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n", 633 | "\\tag{7}$$ \n", 634 | "$$ e_{w2B} = \\frac {e_{w2} \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n", 635 | "\\tag{8}$$\n", 636 | "\n", 637 | "\n", 638 | "$$e_{w1B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w1B}} - \\mu_B} {|(e_{w1} - \\mu_{\\perp}) - \\mu_B)|} \\tag{9}$$\n", 639 | "\n", 640 | "\n", 641 | "$$e_{w2B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w2B}} - \\mu_B} {|(e_{w2} - \\mu_{\\perp}) - \\mu_B)|} \\tag{10}$$\n", 642 | "\n", 643 | "$$e_1 = e_{w1B}^{corrected} + \\mu_{\\perp} \\tag{11}$$\n", 644 | "$$e_2 = e_{w2B}^{corrected} + \\mu_{\\perp} \\tag{12}$$\n", 645 | "\n", 646 | "\n", 647 | "**Exercise**: Implement the function below. Use the equations above to get the final equalized version of the pair of words. Good luck!" 648 | ] 649 | }, 650 | { 651 | "cell_type": "code", 652 | "execution_count": 31, 653 | "metadata": { 654 | "collapsed": true 655 | }, 656 | "outputs": [], 657 | "source": [ 658 | "def equalize(pair, bias_axis, word_to_vec_map):\n", 659 | " \"\"\"\n", 660 | " Debias gender specific words by following the equalize method described in the figure above.\n", 661 | " \n", 662 | " Arguments:\n", 663 | " pair -- pair of strings of gender specific words to debias, e.g. (\"actress\", \"actor\") \n", 664 | " bias_axis -- numpy-array of shape (50,), vector corresponding to the bias axis, e.g. gender\n", 665 | " word_to_vec_map -- dictionary mapping words to their corresponding vectors\n", 666 | " \n", 667 | " Returns\n", 668 | " e_1 -- word vector corresponding to the first word\n", 669 | " e_2 -- word vector corresponding to the second word\n", 670 | " \"\"\"\n", 671 | " \n", 672 | " ### START CODE HERE ###\n", 673 | " # Step 1: Select word vector representation of \"word\". Use word_to_vec_map. (≈ 2 lines)\n", 674 | " w1, w2 = pair[0],pair[1]\n", 675 | " e_w1, e_w2 = word_to_vec_map[w1],word_to_vec_map[w2]\n", 676 | " \n", 677 | " # Step 2: Compute the mean of e_w1 and e_w2 (≈ 1 line)\n", 678 | " mu = (e_w1 + e_w2)/2\n", 679 | "\n", 680 | " # Step 3: Compute the projections of mu over the bias axis and the orthogonal axis (≈ 2 lines)\n", 681 | " mu_B = (np.dot(mu,bias_axis)/np.linalg.norm(bias_axis)**2)*bias_axis\n", 682 | " mu_orth = mu-mu_B\n", 683 | "\n", 684 | " # Step 4: Use equations (7) and (8) to compute e_w1B and e_w2B (≈2 lines)\n", 685 | " e_w1B = (np.dot(e_w1,bias_axis)/np.linalg.norm(bias_axis)**2)*bias_axis\n", 686 | " e_w2B = (np.dot(e_w2,bias_axis)/np.linalg.norm(bias_axis)**2)*bias_axis\n", 687 | " \n", 688 | " # Step 5: Adjust the Bias part of e_w1B and e_w2B using the formulas (9) and (10) given above (≈2 lines)\n", 689 | " corrected_e_w1B = np.sqrt(np.abs(1-np.linalg.norm(mu_orth)**2))*((e_w1B - mu_B)/np.abs((e_w1-mu_orth)-mu_B))\n", 690 | " corrected_e_w2B = np.sqrt(np.abs(1-np.linalg.norm(mu_orth)**2))*((e_w2B - mu_B)/np.abs((e_w2-mu_orth)-mu_B))\n", 691 | "\n", 692 | " # Step 6: Debias by equalizing e1 and e2 to the sum of their corrected projections (≈2 lines)\n", 693 | " e1 = corrected_e_w1B + mu_orth\n", 694 | " e2 = corrected_e_w2B + mu_orth\n", 695 | " \n", 696 | " ### END CODE HERE ###\n", 697 | " \n", 698 | " return e1, e2" 699 | ] 700 | }, 701 | { 702 | "cell_type": "code", 703 | "execution_count": 32, 704 | "metadata": { 705 | "scrolled": true 706 | }, 707 | "outputs": [ 708 | { 709 | "name": "stdout", 710 | "output_type": "stream", 711 | "text": [ 712 | "cosine similarities before equalizing:\n", 713 | "cosine_similarity(word_to_vec_map[\"man\"], gender) = -0.117110957653\n", 714 | "cosine_similarity(word_to_vec_map[\"woman\"], gender) = 0.356666188463\n", 715 | "\n", 716 | "cosine similarities after equalizing:\n", 717 | "cosine_similarity(e1, gender) = -0.716572752584\n", 718 | "cosine_similarity(e2, gender) = 0.739659647493\n" 719 | ] 720 | } 721 | ], 722 | "source": [ 723 | "print(\"cosine similarities before equalizing:\")\n", 724 | "print(\"cosine_similarity(word_to_vec_map[\\\"man\\\"], gender) = \", cosine_similarity(word_to_vec_map[\"man\"], g))\n", 725 | "print(\"cosine_similarity(word_to_vec_map[\\\"woman\\\"], gender) = \", cosine_similarity(word_to_vec_map[\"woman\"], g))\n", 726 | "print()\n", 727 | "e1, e2 = equalize((\"man\", \"woman\"), g, word_to_vec_map)\n", 728 | "print(\"cosine similarities after equalizing:\")\n", 729 | "print(\"cosine_similarity(e1, gender) = \", cosine_similarity(e1, g))\n", 730 | "print(\"cosine_similarity(e2, gender) = \", cosine_similarity(e2, g))" 731 | ] 732 | }, 733 | { 734 | "cell_type": "markdown", 735 | "metadata": {}, 736 | "source": [ 737 | "**Expected Output**:\n", 738 | "\n", 739 | "cosine similarities before equalizing:\n", 740 | "\n", 741 | " \n", 742 | " \n", 745 | " \n", 748 | " \n", 749 | " \n", 750 | " \n", 753 | " \n", 756 | " \n", 757 | "
\n", 743 | " **cosine_similarity(word_to_vec_map[\"man\"], gender)** =\n", 744 | " \n", 746 | " -0.117110957653\n", 747 | "
\n", 751 | " **cosine_similarity(word_to_vec_map[\"woman\"], gender)** =\n", 752 | " \n", 754 | " 0.356666188463\n", 755 | "
\n", 758 | "\n", 759 | "cosine similarities after equalizing:\n", 760 | "\n", 761 | " \n", 762 | " \n", 765 | " \n", 768 | " \n", 769 | " \n", 770 | " \n", 773 | " \n", 776 | " \n", 777 | "
\n", 763 | " **cosine_similarity(u1, gender)** =\n", 764 | " \n", 766 | " -0.700436428931\n", 767 | "
\n", 771 | " **cosine_similarity(u2, gender)** =\n", 772 | " \n", 774 | " 0.700436428931\n", 775 | "
" 778 | ] 779 | }, 780 | { 781 | "cell_type": "markdown", 782 | "metadata": { 783 | "collapsed": true 784 | }, 785 | "source": [ 786 | "Please feel free to play with the input words in the cell above, to apply equalization to other pairs of words. \n", 787 | "\n", 788 | "These debiasing algorithms are very helpful for reducing bias, but are not perfect and do not eliminate all traces of bias. For example, one weakness of this implementation was that the bias direction $g$ was defined using only the pair of words _woman_ and _man_. As discussed earlier, if $g$ were defined by computing $g_1 = e_{woman} - e_{man}$; $g_2 = e_{mother} - e_{father}$; $g_3 = e_{girl} - e_{boy}$; and so on and averaging over them, you would obtain a better estimate of the \"gender\" dimension in the 50 dimensional word embedding space. Feel free to play with such variants as well. \n", 789 | " " 790 | ] 791 | }, 792 | { 793 | "cell_type": "markdown", 794 | "metadata": {}, 795 | "source": [ 796 | "### Congratulations\n", 797 | "\n", 798 | "You have come to the end of this notebook, and have seen a lot of the ways that word vectors can be used as well as modified. \n", 799 | "\n", 800 | "Congratulations on finishing this notebook! \n" 801 | ] 802 | }, 803 | { 804 | "cell_type": "markdown", 805 | "metadata": {}, 806 | "source": [ 807 | "**References**:\n", 808 | "- The debiasing algorithm is from Bolukbasi et al., 2016, [Man is to Computer Programmer as Woman is to\n", 809 | "Homemaker? Debiasing Word Embeddings](https://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf)\n", 810 | "- The GloVe word embeddings were due to Jeffrey Pennington, Richard Socher, and Christopher D. Manning. (https://nlp.stanford.edu/projects/glove/)\n" 811 | ] 812 | } 813 | ], 814 | "metadata": { 815 | "coursera": { 816 | "course_slug": "nlp-sequence-models", 817 | "graded_item_id": "8hb5s", 818 | "launcher_item_id": "5NrJ6" 819 | }, 820 | "kernelspec": { 821 | "display_name": "Python 3", 822 | "language": "python", 823 | "name": "python3" 824 | }, 825 | "language_info": { 826 | "codemirror_mode": { 827 | "name": "ipython", 828 | "version": 3 829 | }, 830 | "file_extension": ".py", 831 | "mimetype": "text/x-python", 832 | "name": "python", 833 | "nbconvert_exporter": "python", 834 | "pygments_lexer": "ipython3", 835 | "version": "3.6.0" 836 | } 837 | }, 838 | "nbformat": 4, 839 | "nbformat_minor": 2 840 | } 841 | -------------------------------------------------------------------------------- /5. Sequence Models/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Objectives: 3 | - Learn about recurrent neural networks. This type of model has been proven to perform extremely well on temporal data. It has several variants including LSTMs, GRUs and Bidirectional RNNs. 4 | - Natural Language Processing & Word Embeddings. Examples of applications are sentiment analysis, named entity recognition and machine translation. 5 | - Sequence models & Attention mechanism. 6 | 7 | ## Week 1: 8 | - Building a Recurrent Neural Network - Step by Step - v3 9 | - Dinosaurus Island -- Character level language model final - v3 10 | - Improvise a Jazz Solo with an LSTM Network - v3 11 | 12 | ## Week 2: 13 | - Operations on word vectors - v2 14 | - Emojify - v2 15 | 16 | ## Week 3: 17 | - Neural machine translation with attention - v4.ipynb 18 | - Trigger word detection - v1 19 | 20 | -------------------------------------------------------------------------------- /5. Sequence Models/images/1-hot-vector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/1-hot-vector.png -------------------------------------------------------------------------------- /5. Sequence Models/images/LSTM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/LSTM.png -------------------------------------------------------------------------------- /5. Sequence Models/images/LSTM_rnn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/LSTM_rnn.png -------------------------------------------------------------------------------- /5. Sequence Models/images/RNN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/RNN.png -------------------------------------------------------------------------------- /5. Sequence Models/images/attn_mechanism.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/attn_mechanism.png -------------------------------------------------------------------------------- /5. Sequence Models/images/attn_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/attn_model.png -------------------------------------------------------------------------------- /5. Sequence Models/images/clip.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/clip.png -------------------------------------------------------------------------------- /5. Sequence Models/images/cosine_sim.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/cosine_sim.png -------------------------------------------------------------------------------- /5. Sequence Models/images/data_set.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/data_set.png -------------------------------------------------------------------------------- /5. Sequence Models/images/dataset_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/dataset_kiank.png -------------------------------------------------------------------------------- /5. Sequence Models/images/date_attention.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/date_attention.png -------------------------------------------------------------------------------- /5. Sequence Models/images/date_attention2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/date_attention2.png -------------------------------------------------------------------------------- /5. Sequence Models/images/dino.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/dino.jpg -------------------------------------------------------------------------------- /5. Sequence Models/images/dinos3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/dinos3.png -------------------------------------------------------------------------------- /5. Sequence Models/images/emb_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/emb_kiank.png -------------------------------------------------------------------------------- /5. Sequence Models/images/embedding1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/embedding1.png -------------------------------------------------------------------------------- /5. Sequence Models/images/emo_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/emo_model.png -------------------------------------------------------------------------------- /5. Sequence Models/images/emojifier-v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/emojifier-v2.png -------------------------------------------------------------------------------- /5. Sequence Models/images/emojifierv1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/emojifierv1.png -------------------------------------------------------------------------------- /5. Sequence Models/images/emojify_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/emojify_model.png -------------------------------------------------------------------------------- /5. Sequence Models/images/emojiss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/emojiss.png -------------------------------------------------------------------------------- /5. Sequence Models/images/equalize10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/equalize10.png -------------------------------------------------------------------------------- /5. Sequence Models/images/image_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/image_1.png -------------------------------------------------------------------------------- /5. Sequence Models/images/initial_state.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/initial_state.png -------------------------------------------------------------------------------- /5. Sequence Models/images/jazz.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/jazz.jpg -------------------------------------------------------------------------------- /5. Sequence Models/images/label_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/label_diagram.png -------------------------------------------------------------------------------- /5. Sequence Models/images/lookup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/lookup.png -------------------------------------------------------------------------------- /5. Sequence Models/images/mangosaurus.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/mangosaurus.jpeg -------------------------------------------------------------------------------- /5. Sequence Models/images/model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/model.png -------------------------------------------------------------------------------- /5. Sequence Models/images/music_gen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/music_gen.png -------------------------------------------------------------------------------- /5. Sequence Models/images/music_generation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/music_generation.png -------------------------------------------------------------------------------- /5. Sequence Models/images/neutral.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/neutral.png -------------------------------------------------------------------------------- /5. Sequence Models/images/neutralize.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/neutralize.png -------------------------------------------------------------------------------- /5. Sequence Models/images/neutralize_kiank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/neutralize_kiank.png -------------------------------------------------------------------------------- /5. Sequence Models/images/ones_reference.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/ones_reference.png -------------------------------------------------------------------------------- /5. Sequence Models/images/poorly_trained_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/poorly_trained_model.png -------------------------------------------------------------------------------- /5. Sequence Models/images/rnn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/rnn.png -------------------------------------------------------------------------------- /5. Sequence Models/images/rnn_cell_backprop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/rnn_cell_backprop.png -------------------------------------------------------------------------------- /5. Sequence Models/images/rnn_minusculas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/rnn_minusculas.png -------------------------------------------------------------------------------- /5. Sequence Models/images/rnn_step_forward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/rnn_step_forward.png -------------------------------------------------------------------------------- /5. Sequence Models/images/shakespeare.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/shakespeare.jpg -------------------------------------------------------------------------------- /5. Sequence Models/images/sound.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/sound.png -------------------------------------------------------------------------------- /5. Sequence Models/images/spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/spectrogram.png -------------------------------------------------------------------------------- /5. Sequence Models/images/table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/table.png -------------------------------------------------------------------------------- /5. Sequence Models/images/tmp: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /5. Sequence Models/images/train_label.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/train_label.png -------------------------------------------------------------------------------- /5. Sequence Models/images/train_reference.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/train_reference.png -------------------------------------------------------------------------------- /5. Sequence Models/images/woebot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/5. Sequence Models/images/woebot.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Gema Atienza 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deep Learning Specialization on Coursera 2 | 3 | **Master Deep Learning, and Break into AI** 4 | 5 | Instructor: [Andrew Ng](http://www.andrewng.org/) 6 | 7 | ## Introduction 8 | 9 | This repo contains all my work for this specialization. All the code base and images, are taken from [Deep Learning Specialization on Coursera](https://www.coursera.org/specializations/deep-learning). 10 | 11 | *In five courses, you will learn the foundations of Deep Learning, understand how to build neural networks, and learn how to lead successful machine learning projects. You will learn about Convolutional networks, RNNs, LSTM, Adam, Dropout, BatchNorm, Xavier/He initialization, and more. You will work on case studies from healthcare, autonomous driving, sign language reading, music generation, and natural language processing. You will master not only the theory, but also see how it is applied in industry. You will practice all these ideas in Python and in TensorFlow, which we will teach.* 12 | 13 | ## Programming Assignments 14 | 15 | - **Course 1: Neural Networks and Deep Learning:** 16 | 17 | Objectives: 18 | + Understand the major technology trends driving Deep Learning. 19 | + Be able to build, train and apply fully connected deep neural networks. 20 | + Know how to implement efficient (vectorized) neural networks. 21 | + Understand the key parameters in a neural network's architecture. 22 | 23 | Code: 24 | + [Week 2 - Phyton Basics with Numpy](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/1.%20Neural%20Networks%20and%20Deep%20Learning/Python%20Basics%20With%20Numpy%20v3.ipynb) 25 | + [Week 2 - Logistic Regression with a Neural Network mindset](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/1.%20Neural%20Networks%20and%20Deep%20Learning/Logistic%20Regression%20with%20a%20Neural%20Network%20mindset%20v3.ipynb) 26 | + [Week 3 - Planar data classification with a hidden layer](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/1.%20Neural%20Networks%20and%20Deep%20Learning/Planar%20data%20classification%20with%20one%20hidden%20layer%20v4.ipynb) 27 | + [Week 4 - Building your Deep Neural Network: Step by Step](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/1.%20Neural%20Networks%20and%20Deep%20Learning/Building%20your%20Deep%20Neural%20Network%20-%20Step%20by%20Step%20v5.ipynb) 28 | + [Week 4 - Deep Neural Network: Application](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/1.%20Neural%20Networks%20and%20Deep%20Learning/Deep%20Neural%20Network%20-%20Application%20v3.ipynb) 29 | 30 | - **Course 2: Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization** 31 | 32 | Objectives: 33 | + Understand industry best-practices for building deep learning applications. 34 | + Be able to effectively use the common neural network "tricks", including initialization, L2 and dropout regularization, Batch normalization, gradient checking, 35 | + Be able to implement and apply a variety of optimization algorithms, such as mini-batch gradient descent, Momentum, RMSprop and Adam, and check for their convergence. 36 | + Understand new best-practices for the deep learning era of how to set up train/dev/test sets and analyze bias/variance 37 | + Be able to implement a neural network in TensorFlow. 38 | 39 | Code: 40 | + [Week 1 - Initialization](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/2.%20Improving%20Deep%20Neural%20Networks-%20Hyperparameter%20tuning-%20Regularization%20and%20Optimization/Initialization.ipynb) 41 | + [Week 1 - Regularization](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/2.%20Improving%20Deep%20Neural%20Networks-%20Hyperparameter%20tuning-%20Regularization%20and%20Optimization/Regularization.ipynb) 42 | + [Week 1 - Gradient Checking](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/2.%20Improving%20Deep%20Neural%20Networks-%20Hyperparameter%20tuning-%20Regularization%20and%20Optimization/Gradient%20Checking%20v1.ipynb) 43 | + [Week 2 - Optimization](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/2.%20Improving%20Deep%20Neural%20Networks-%20Hyperparameter%20tuning-%20Regularization%20and%20Optimization/Optimization%20methods.ipynb) 44 | + [Week 3 - TensorFlow](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/2.%20Improving%20Deep%20Neural%20Networks-%20Hyperparameter%20tuning-%20Regularization%20and%20Optimization/Tensorflow%20Tutorial.ipynb) 45 | 46 | - **Course 3: Structuring Machine Learning Projects** 47 | 48 | Objectives: 49 | + Understand how to diagnose errors in a machine learning system, and 50 | + Be able to prioritize the most promising directions for reducing error 51 | + Understand complex ML settings, such as mismatched training/test sets, and comparing to and/or surpassing human-level performance 52 | + Know how to apply end-to-end learning, transfer learning, and multi-task learning 53 | 54 | Code: 55 | + There is no Program Assigments for this course. But this course comes with very interesting case study quizzes. 56 | 57 | - **Course 4: Convolutional Neural Networks** 58 | 59 | Objectives: 60 | + Understand how to build a convolutional neural network, including recent variations such as residual networks. 61 | + Know how to apply convolutional networks to visual detection and recognition tasks. 62 | + Know to use neural style transfer to generate art. 63 | + Be able to apply these algorithms to a variety of image, video, and other 2D or 3D data. 64 | 65 | Code: 66 | + [Week 1 - Convolutional Model: step by step](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/4.%20Convolutional%20Neural%20Networks/Convolution%20model%20-%20Step%20by%20Step%20-%20v2.ipynb) 67 | + [Week 1 - Convolutional Model: application](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/4.%20Convolutional%20Neural%20Networks/Convolution%20model%20-%20Application%20-%20v1.ipynb) 68 | + [Week 2 - Keras - Tutorial - Happy House](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/4.%20Convolutional%20Neural%20Networks/Keras%20-%20Tutorial%20-%20Happy%20House%20v2.ipynb) 69 | + [Week 2 - Residual Networks](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/4.%20Convolutional%20Neural%20Networks/Residual%20Networks%20-%20v2.ipynb) 70 | + [Week 3 - Autonomous driving application - Car detection](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/4.%20Convolutional%20Neural%20Networks/Autonomous%20driving%20application%20-%20Car%20detection%20-%20v1.ipynb) 71 | + [Week 4 - Face Recognition for the Happy House - v3](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/4.%20Convolutional%20Neural%20Networks/Face%20Recognition%20for%20the%20Happy%20House%20-%20v3.ipynb) 72 | + [Week 4 - Art Generation with Neural Style Transfer - v2.ipynb](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/4.%20Convolutional%20Neural%20Networks/Art%20Generation%20with%20Neural%20Style%20Transfer%20-%20v2.ipynb) 73 | 74 | - **Course 5: Sequence Models** 75 | Objectives: 76 | + Understand how to build and train Recurrent Neural Networks (RNNs), and commonly-used variants such as GRUs and LSTMs. 77 | + Be able to apply sequence models to natural language problems, including text synthesis. 78 | + Be able to apply sequence models to audio applications, including speech recognition and music synthesis. 79 | 80 | Code: 81 | + [Week 1 - Building a Recurrent Neural Network - Step by Step - v3](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/5.%20Sequence%20Models/Building%20a%20Recurrent%20Neural%20Network%20-%20Step%20by%20Step%20-%20v3.ipynb) 82 | + [Week 1 - Dinosaur Island - Character-Level Language Modeling](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/5.%20Sequence%20Models/Dinosaurus%20Island%20--%20Character%20level%20language%20model%20final%20-%20v3.ipynb) 83 | + [Week 1 - Improvise a Jazz Solo with an LSTM Network](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/5.%20Sequence%20Models/Improvise%20a%20Jazz%20Solo%20with%20an%20LSTM%20Network%20-%20v3.ipynb) 84 | + [Week 2 - Operations on word vectors - v2](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/5.%20Sequence%20Models/Operations%20on%20word%20vectors%20-%20v2.ipynb) 85 | + [Week 2 - Emojify - v2](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/5.%20Sequence%20Models/Emojify%2B-%2Bv2.ipynb) 86 | + [Week 3 - Neural machine translation with attention - v4](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/5.%20Sequence%20Models/Neural%20machine%20translation%20with%20attention%20-%20v4.ipynb) 87 | + [Week 3 - Trigger word detection - v1](https://github.com/gemaatienza/Deep-Learning-Coursera/blob/master/5.%20Sequence%20Models/Trigger%20word%20detection%20-%20v1.ipynb) 88 | -------------------------------------------------------------------------------- /prediction_video_compressed.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gemaatienza/Deep-Learning-Coursera/691374a31a373841bd3cad8cfa96fbef0b04a3dc/prediction_video_compressed.mp4 --------------------------------------------------------------------------------